Information processing apparatus and computer program

Abstract

To generate a video corresponding to attribute information by associating three dimensional model data to be generated with the attribute information of an object to be photographed when photographing the object and generating the video in which a three dimensional model is arranged on a virtual space by using the three dimensional model data of the photographed object.SOLUTION: An information processing apparatus includes a scan control unit configured to control a position and an angle of a scanner and scan an appearance of a subject to generate a scan image, a data generation unit configured to generate three dimensional model data from the scan image, a video generation unit configured to generate a video in which a three dimensional model based on the three dimensional model data is arranged in a virtual space, and a reception unit configured to receive an input of attribute information regarding the subject to be scanned and a representation of the subject, wherein the video generation unit generates the video so as to include the representation for each subject according to the attribute information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device and a computer program. [Background technology]

[0002] Patent document 1 describes a technology in which a texture generated from a group of captured images of an object taken from multiple angles by a photographing unit is mapped onto a primitive in a virtual space to generate a virtual space image as seen from a virtual camera in the virtual space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-107251 Summary of the Invention [Problem to be solved by the invention]

[0004] In generating virtual space images using the above technology, when an object is photographed and three-dimensional model data of the photographed object is used to generate an image in which a three-dimensional model is placed in a virtual space, it is expected that the generated three-dimensional model data will be associated with attribute information of the photographed object, so that an image corresponding to the attribute information can be generated.

[0005] Therefore, when photographing an object and using the three-dimensional model data of the photographed object to generate an image in which a three-dimensional model is placed in a virtual space, the object is to associate the generated three-dimensional model data with attribute information of the photographed object, thereby making it possible to generate an image according to the attribute information. [Means for solving the problem]

[0006] One aspect of the present invention to solve the above problem is an information processing device, a scan control means for controlling the position and angle of the scanner to scan the appearance of an object placed on a table and generate a scanned image; a data generation means for generating three-dimensional model data from the scanned image; an image generating means for generating an image in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; an output means for outputting the image generated by the image generation means to a display device; a receiving means for receiving input of attribute information relating to the subject to be scanned and the rendering of the subject; Equipped with The image generating means generates the image so as to include a rendition for each of the subjects according to the attribute information. [Effects of the Invention]

[0007] When photographing an object and using the three-dimensional model data of the photographed object to generate an image in which a three-dimensional model is placed in a virtual space, the three-dimensional model data can be generated in association with attribute information of the photographed object, and an image can be generated according to the attribute information. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of the configuration of an image processing system 10 according to an embodiment. [Figure 2] 1 is a diagram showing an example of a hardware configuration of an information processing device 100 according to an embodiment, and an example of a hardware configuration of a headset 150. [Figure 3] 1A and 1B are diagrams showing an example of the appearance of a doll-type model according to an embodiment. [Figure 4A] FIG. 1 is a diagram showing an example of implementation of an image processing system 10. [Figure 4B] FIG. 1 is a diagram showing an example of the configuration of an image processing system 10 according to an embodiment when performing scan photography. [Figure 5] 4A and 4B are diagrams for explaining the relationship between the rotational positions of a scanner device 413 and a turntable 406 during scanning and photographing according to an embodiment. [Figure 6]3 is a flowchart corresponding to an example of processing in the image processing system 10 according to the embodiment. [Figure 7] FIG. 10 is a diagram showing an example of an attribute specification screen according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] The following embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Any combination of two or more of the features described in the embodiments may be used. Identical or similar components are given the same reference numerals, and redundant descriptions will be omitted. In each figure, the up / down, left / right, front / back directions relative to the paper surface will be used in the description of the present embodiment as the up / down, left / right, front / back directions of components (or parts) in the present embodiment.

[0010] First, the configuration of an image processing system corresponding to this embodiment will be described. FIG. 1 is a diagram showing an example of the configuration of an image processing system 10 corresponding to this embodiment. The image processing system 10 is configured by connecting an information processing device 100 to a scanner 110, a support arm 120, a model support device 130, a display device 140, a headset 150, etc. Note that the system configuration is not limited to that shown in FIG. 1, and the information processing device 100 may be further connected to an external server, a cloud server, etc. via a network. The external server, etc., can execute at least a part of the processing of the embodiment described below.

[0011] The information processing device 100 controls the operation of at least one of the scanner 110, the support arm 120, and the model support device 130, photographs the object to be photographed from any angle to generate multiple images, and generates three-dimensional model data (the actual data) from the multiple images. If the object to be photographed is configured to be separable into multiple component parts, it is also possible to photograph each component part to generate three-dimensional model data, and then combine these to generate the three-dimensional model data of the target object.

[0012] The information processing device 100 can also function as an image generating device that generates images to be displayed in a virtual space by using the generated three-dimensional model data as virtual space data. In this embodiment, the subjects to be photographed are articles such as assemblable plastic models, action figures (figures with movable joints), toys, dolls, etc., which are hereinafter collectively referred to as "models."

[0013] Next, the scanner 110 is a three-dimensional scanner device that captures (scans) the three-dimensional shape of a model to be captured under the control of the information processing device 100 and outputs the three-dimensional shape and color information of the captured object. In this embodiment, the scan signals output from the scanner 110 are collectively referred to as "scanned images." For example, the scanner 110 may be a Space Spider manufactured by Artec. In this embodiment, for example, 3D scan data of the entire toy body can be obtained by capturing approximately 500 to 800 frames of scanned images. Alternatively, the scanner 110 may be a smartphone with a camera on which an application for capturing three-dimensional shapes is installed.

[0014] The support arm 120 is a position and orientation control device that moves the scanner 110 to a predetermined imaging position and orientation under the control of the information processing device 100. The support arm 120 may be configured to manually change the imaging position and orientation and to fix the changed position and orientation, or may be configured to be controllable by the information processing device 100. When the support arm 120 is configured to be controllable by the information processing device 100, for example, xArm 7 manufactured by UFACTORY Corporation can be used. The xArm 7 consists of seven joints and can move in the same manner as a human arm. Note that the scanner 110 may be positioned manually instead of using the support arm 120. For imaging positions and orientations that cannot be covered by the support arm 120, the scanner 110 may be manually operated to perform a scan.

[0015] The model support device 130 is a support stand (or a mounting stand on which the model is placed) that supports a model with a fixed pose. The model support device 130 may be configured to be rotatable, for example, with the model placed on the support stand or at the end of a support rod. In this embodiment, the scanner 110 is positioned at a desired imaging position and imaging angle using the support arm 120, and then the model support device 130 is rotated to perform imaging. The model support device 130 can rotate both clockwise and counterclockwise. The model support device 130 can also be stopped at any rotation angle between 0 and 360 degrees. It is also possible to stop the model support device 130 at a specific rotation angle and position it at any imaging position and imaging angle within the movable range of the scanner 110 to perform imaging. By performing this process at multiple rotation angles, imaging positions, and imaging angles, an image of the entire model can be acquired. Here, synchronous driving of the support arm 120 and the model support device 130 can facilitate imaging with high accuracy. Furthermore, instead of using the model support device 130, the scanner 110 may be manually moved around the model to scan from any desired imaging position and angle.

[0016] The display device 140 is a display device such as a liquid crystal display (LCD) and can display the processing results of the information processing device 100. Specifically, it can display images acquired by the scanner 110, display the original three-dimensional model data generated from the captured images, and display images (VR images) reconstructed using this data. It can also display a meter showing the progress of the scanner processing. The display device 140 can include an operation unit 140A that accepts operations from a user who is an observer of the displayed images, and the user can operate the operation unit 140A to input operations according to the content of the images displayed on the display device 140.

[0017] The headset 150 is composed of a head-mounted display 150A and a controller 150B, which will be described later. In particular, the headset 150 may be configured as a VR headset capable of providing moving images corresponding to the posture and tilt of a user who is an observer. A predetermined application is installed in the headset 150, and application data executed by the application can be downloaded from the information processing device 100 and executed. In this embodiment, the headset 150, like the information processing device 100, can function as an image generating device that uses three-dimensional model data as virtual space data to generate images to be displayed in a virtual space and provides the generated images as VR images. Note that the application data includes display data for displaying the VR images.

[0018] Next, the hardware configuration of the information processing device 100 corresponding to this embodiment will be described. FIG. 2A shows an example of the hardware configuration of the information processing device 100. The CPU 101 is a device that performs overall control of the information processing device 100 and performs calculations, processing, and management of data. For example, the CPU 101 controls the timing and number of images captured by the scanner 110, and can control the arm joint of the support arm 120 to position the scanner 110 at any desired imaging position and imaging angle. For example, after the imaging position and imaging angle of the scanner 110 are determined, the model support device 130 can be rotated to perform an imaging operation using the scanner 110. Furthermore, after the model support device 130 is stopped at a predetermined rotation angle, the scanner 110 can perform an imaging operation at any desired imaging position and imaging angle.

[0019] The CPU 101 can also function as an image processing unit that processes images output from the scanner 110. Specifically, the CPU 101 can perform processing to generate an image that displays the three-dimensional model in a virtual space, using three-dimensional model data generated based on the scan signal obtained by the scanner 110 as virtual space data.

[0020] The RAM 102 is a volatile memory and is used as a temporary storage area such as the main memory or work area of ​​the CPU 101. The ROM 103 is a non-volatile memory and stores image data and other data, various programs for the operation of the CPU 101, and the like, in predetermined areas. The CPU 101 controls each unit of the information processing device 100 using the RAM 102 as a work memory in accordance with, for example, a program stored in the ROM 103. Note that the program for the operation of the CPU 101 is not limited to being stored in the ROM 103, and may be stored in the storage device 104.

[0021] The storage device 104 is configured with a magnetic disk such as an HDD or flash memory. The storage device 104 stores application programs, an OS, control programs, related programs, game programs, etc. The storage device 104 can read and write data under the control of the CPU 101. The storage device 104 may be used in place of the RAM 102 or the ROM 103.

[0022] The communication device 105 is a communication interface for communicating with the scanner 110, the support arm 120, the model support device 130, the display device 140, and the headset 150 under the control of the CPU 101. The communication device 105 may further be configured to be able to communicate with an external server or the like. The communication device 105 may include a wireless communication module, which may include well-known circuitry including an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module card, a memory, etc. Here, communication between the information processing device 100 and the scanner 110, the support arm 120, the model support device 130, the display device 140, and the headset 150 may be performed via wireless communication.

[0023] The communication device 105 may also include a wired communication module for wired connection. The wired communication module enables communication with other devices, including the display device 140, via one or more external ports. It may also include various software components for processing data. The external ports are connected to other devices directly via Ethernet, USB, IEEE 1394, etc., or indirectly via a network. Note that software that realizes the same functions as each of the above devices may be configured as a substitute for the hardware devices.

[0024] The operation unit 106 is configured with, for example, buttons, a keyboard, a touch panel, a controller, etc., and accepts operation input from a user. The operation unit 106 may be shared with the operation unit 140A, or may be independent of the operation unit 140A. For example, if the operation unit 140A is intended to be an operation unit configured with a keyboard, a mouse, etc., the operation unit 106 can be shared with the operation unit 106. On the other hand, if the operation unit 140A is intended to be an operation unit configured with a touch panel, a controller, etc., the operation unit 106 can be a separate operation unit.

[0025] The display control unit 107 functions as a control unit for displaying information on the display device 140 connected to the information processing device 100, and controls the operation of the display device 140. Some of the functions of the operation unit 106 may be provided in the display device 140. For example, the display device 140 may be configured as a device equipped with a touch panel such as a tablet terminal.

[0026] Next, the hardware configuration of the headset 150 corresponding to this embodiment will be described. FIG. 2(B) shows an example of the hardware configuration of the headset 150. The headset 150 is composed of a head-mounted display (HMD) 150A and a controller 150B. The HMD 150A provides a right-eye image and a left-eye image to the user's left and right eyes, respectively, and enables a virtual reality (VR) experience by creating a sense of three-dimensionality due to the parallax between the images. The controller 150B is provided in a housing independent of the HMD 150A. The controller 150B is assumed to be configured as a pair of two controllers to be held and operated by the user's left and right hands, respectively, but may also be a single controller.

[0027] The CPU 151 is a device that performs overall control of the headset 150 and performs calculations, processing, and management of data. For example, it can execute application data downloaded from the information processing device 100 to display VR video on the display 157. Furthermore, based on operations received via the controller 156 or information detected by the detection unit 158, it can switch the VR video itself to be displayed, switch the viewpoint within the VR space, or change the position within the VR space.

[0028] RAM 152 is a volatile memory and is used as a temporary storage area such as the main memory or work area of ​​CPU 151. ROM 153 is a non-volatile memory in which image data and other data, various programs for operating CPU 151, and the like are stored in predetermined areas. CPU 151 controls each unit of headset 150 using RAM 152 as a work memory in accordance with, for example, a program stored in ROM 153. Note that the program for operating CPU 151 is not limited to being stored in ROM 153, and may also be stored in storage device 154.

[0029] The storage device 154 is configured by a magnetic disk such as an HDD or flash memory. The storage device 154 stores application programs, an OS, control programs, related programs, game programs, application data downloaded from the information processing device 100, display data, and the like. The storage device 154 can read and write data based on the control of the CPU 151. The storage device 154 may be used instead of the RAM 152 or the ROM 153.

[0030] The communication device 155 is a communication interface for communicating with the information processing device 100 and the controller 150B under the control of the CPU 151. The communication device 155 includes a wireless communication module that enables wireless communication based on Bluetooth or WiFi (IEEE802.11). The headset 150 can connect to the information processing device 100 via wireless communication and download display data for VR images. The headset 150 also communicates with the controller 150B and receives information on user operation instructions for the controller 150B.

[0031] The display 156 is configured to provide the right-eye image and left-eye image generated by the CPU 151 to the user's right eye and left eye, respectively. The detection unit 157 is a mechanism for detecting the gaze direction of the user's left and right eyes and the tilt of the head-mounted display. The detection unit 157 includes, for example, a sensor for detecting the gaze direction or gaze direction of the user wearing the HMD 150A. It also includes a gyroscope, a magnetometer, an accelerometer, a global positioning system (GPS), a compass, etc., and can identify the position, posture, tilt, etc. of the HMD 150A based on the detection information. The detection unit 157 detects information for determining the user's gaze direction and action in the HMD 150A and transmits it to the CPU 151. The CPU 151 determines the gaze direction and action based on the received detection information and adjusts the image presented on the display 156 of the HMD 150A to match the determined gaze direction and action.

[0032] The controller 150B is configured with, for example, a plurality of buttons, a cross key, etc., and receives operation inputs from the user such as selection operations, direction instruction operations, etc. The controller 150B is connected to the HMD 150A via a communication device 155 via wireless communication.

[0033] Next, referring to FIG. 3 , an example of a model and its accessories that are the subject of the image capture in this embodiment will be described. The model 300 is a model with a doll-like (robot or human) appearance. The model may be, for example, a plastic model that has been assembled and painted. Alternatively, it may be a completed model such as a figurine (action figure) with movable joints. The model in FIG. 2 is merely an example for illustrative purposes, and the shape of the model is not limited to a doll-like appearance, and may be a model of any shape, such as a general vehicle, racing vehicle, military vehicle, aircraft, ship, animal, or virtual creature. It should be noted that the object to be captured is not limited to a model, as long as the object's three-dimensional shape can be captured by the scanner 110.

[0034] The model 300 has a head 301, a chest 302, a right arm 303, a left arm 304, a right torso 305, a left torso 306, a right leg 307, a left leg 308, a right foot 309, and a left foot 310, which are joined together. At least a portion of each of the parts 301 to 310 is supported so as to be rotatable (or swingable) relative to the adjacent parts. For example, the head 301 is supported so as to be rotatable relative to the chest 302, and the right arm 303 and the left arm 304 are supported so as to be rotatable relative to the chest 302. In this way, each part of the model 300 is provided with a joint structure, so the model 300 can assume any posture.

[0035] The model 300 also includes an accessory 311. The accessory 311 is an accessory part that is expected to be used with the model 300, and in FIG. 3, it is a rifle-shaped weapon. That is, it is expected that the model 300 will use the accessory 311 by holding it with both hands. The shape of the weapon is not limited to a rifle, and can include any weapon shape, such as a sword, a shield, a bomb (e.g., a grenade), or a bazooka. The accessory is not limited to a weapon, and may be anything that can be used with the model 300, such as a decorative item. For example, if the model 300 is a doll-type model of a specific character, the accessory may be any item that the character carries (e.g., a carry-on bag, a hat, a bag, an accessory, a smartphone, or any other item that the character can carry).

[0036] Next, an overall implementation example of the image processing system 10 in this embodiment will be described with reference to Fig. 4A. Fig. 4A is a diagram showing a generalized implementation example capable of generating the main data generated by photographing a model.

[0037] 4A, case 401 contains information processing device 100, a drive system for driving support arm 402, a drive system for driving turntable 406, etc. Note that support arm 402 can also be manually adjusted for shooting direction and position. The surface of case 401 has a flat structure so that a promotional poster can be attached.

[0038] The support arm 402 corresponds to the support arm 120, and can support and fix the position of the terminal 403 functioning as the scanner 110 under the control of the information processing device 100 or manually. The support arm 402 can also operate to control the tilt of the terminal 403.

[0039] The terminal 403 is a touch-screen terminal with a built-in camera that can be used as the scanner 110. For example, a smartphone, tablet, or digital camera can be used. Instead of these terminals, Space Spider manufactured by Artec can also be used. FIG. 4A is a diagram showing a generalized example of the system configuration, and the configuration can be changed depending on the type of device used as the scanner 110. The terminal 403 can capture images of the model 300 and transmit them to the information processing device 100. The ring light 404 is a lighting device used when the terminal 403 captures an image of the model 300. It can evenly illuminate the model 300 and reduce the appearance of shadows. In addition to the ring light 404, additional light sources such as a top light or auxiliary lights on the left, right, and bottom sides can be installed. Instead of the terminal 403, a three-dimensional scanner device such as the one described above can also be used.

[0040] The background sheet 405 is a background sheet for photography, and a white sheet can be used, for example. The turntable 406 can mount the model 300 and rotate the model. The turntable 406 and its configuration, including its drive system, correspond to the model support device 130, and the turntable 406 rotates clockwise and counterclockwise and stops under the control of the information processing device 100. A plurality of predetermined markers 410 may be arranged on the turntable 406. The markers 410 can be used to adjust the orientation and position of the photographed model.

[0041] In FIG. 4A, the model is placed on a translucent (transparent) base, but other support devices such as an "Action Base" (registered trademark) may also be used. The Action Base has a support pole on top of the base that can be bent into a "L" shape, and the model can be attached to the tip of the pole. In this case, the above-mentioned marker 410 may be placed on the tip of the pole. The method of indicating the model can be changed depending on the pose. For example, in the case of an upright posture, the model can be placed on a transparent base and photographed. On the other hand, if it is desired to photograph the soles of the feet, such as in a flying posture, an Action Base may be used. Note that an Action Base may also be used to photograph an upright posture.

[0042] The display device 407 is a device corresponding to the display device 140 and may have a touch panel function. The user can perform a predetermined selection operation using the touch panel function. Alternatively, operations can be accepted using a controller or mouse (not shown) that can function as the operation unit 140A instead of the touch panel. The VR headset 408 is composed of an HMD 408A and a controller 408B, and corresponds to the HMDs 150A and 150B, respectively. The user can perform operations while viewing VR images by wearing the HMD 408A on their head and holding and operating the controllers 408B in their left and right hands.

[0043] Next, referring to FIG. 4B , a more specific configuration of the image processing system 10 corresponding to this embodiment when simultaneously scanning the model 300 and the accessory 311 will be described. In FIG. 4B , the model 300 and the accessory 311 are placed on a turntable 406. The model 300 is supported by an action base 411 and is located approximately in the center of the turntable 406. The accessory 311 is supported by a support member 412A in a frame 412, and is located adjacent to the model 300 and near the end of the turntable 406. The support member 412A can be, for example, a clip-shaped member. This allows the accessory 411 to be supported within the frame 412 by being clipped by the clip.

[0044] The scanner device 413 is supported by a support arm 402, and in this example, it is assumed that a Space Spider manufactured by Artec is used. The drive system for driving the support arm 402, the drive system for driving the turntable 406, etc. are the same as those described in Fig. 4A, and therefore description thereof will be omitted. In this embodiment, the turntable 406 can rotate in both clockwise and counterclockwise directions.

[0045] In this embodiment, model 300 and accessory 311 are placed together on turntable 406 in the configuration shown in Fig. 4B, and are then photographed together by scanner device 413 to generate a scanned image. The scanned image includes both model 300 and accessory 311, but the three-dimensional model data is generated by separating them.

[0046] The frame 412 that supports the accessory 311 is configured as a frame that surrounds the accessory 311 and is used as an indicator for identifying the area in which the accessory 311 is located. In other words, a three-dimensional object that exists inside the frame 412 is recognized as the accessory 311. It is desirable that the thickness of the frame 412 be as thin as possible within a range that maintains strength so as not to obstruct the line of sight of the scanner device 413 as much as possible. For example, the thickness can be about 3 mm.

[0047] Furthermore, model 300 and frame 412 are spaced apart to ensure a certain distance. Depending on the rotation angle of turntable 406, frame 412 may be positioned in front of model 300 relative to scanner device 413, blocking the capture of model 300. This blocked portion needs to be complemented at a different rotation angle, but if model 300 and frame 412 are too close, the complement accuracy will decrease, so a certain distance is maintained to ensure complement accuracy.

[0048] In this case, the sizes of model 300 and accessory 311 may be limited to a predetermined size or less so that a certain distance can be maintained between model 300 and frame 412 (or so that they are not too close). For example, the size of model 300 may be 15 cm in height, 15 cm in width, and 10 cm in depth or less, and the size of accessory 311 may be 15 cm in height, 5 cm in width, and 2 cm in thickness or less.

[0049] Furthermore, in this embodiment, the installation table 414 on which the action base 411 supporting the model 300 is installed is configured to be thicker than the installation table 415 on which the frame 412 supporting the accessory 311 is installed. That is, the model 300 is installed so as to be positioned relatively higher than the accessory 311. As an example, the model 300 can be installed so that at least the upper body or the upper half thereof is positioned higher than the frame 412. This reduces the proportion of the model 300 that is blocked by the accessory 311 when scanning the model 300, thereby increasing the success rate of scanning.

[0050] Next, referring to Fig. 5, the coordinated operation of the turntable 406 and support arm 120 during scanning according to this embodiment will be described. Fig. 5 is a diagram showing the relationship between the rotation direction and amount of the turntable 406 relative to the scanner device 413, and the installation stands 414 and 415. In Fig. 5, the model 300 installed on the installation stand 414 is positioned so that the direction marked "forward" is the front. The same can be said for the accessory 311 installed on the installation stand 415.

[0051] FIG. 5(A) shows the relationship between the scanner device 413 and the installation stands 414 and 415 at the start position of the scanning operation. The rotation angle of the turntable 406 at this point is set to 0 degrees, which serves as the reference. When the turntable 406 rotates 90 degrees to the right (clockwise) from this state, the state shown in FIG. 5(B) is reached. During the transition from FIG. 5(A) to FIG. 5(B), the scanner device 413 is positioned to the side of the turntable 406 and scans the model 300 and the accessory 311. Then, in the state shown in FIG. 5(B) where the fronts of the model 300 and the accessory 311 are positioned in front of the scanner device 413, the support arm 402 is moved to the limit of its range of motion to scan the front, left and right sides, top, and bottom of the model 300 and the accessory 311.

[0052] Next, from the state shown in Fig. 5(B), the turntable 406 is rotated rightward from the 90-degree position to the 270-degree position. During this time, the model 300 and the accessory 311 are scanned. Then, in the state shown in Fig. 5(C) where the backs of the model 300 and the accessory 311 are positioned in front of the scanning device 413, the support arm 402 is moved to the limit of its range of motion, and the back, left and right sides, top, and bottom of the model 300 and the accessory 311 are scanned.

[0053] Next, from the state shown in FIG. 5(C), the turntable 406 is further rotated to the right from the 270-degree position to the 330-degree position. During this time, the model 300 and the accessories 311 are scanned. Then, the state shown in FIG. 5(D) is reached, and the support arm 402 is moved to the limit of its range of motion from diagonally behind the right of the model 300 and the accessories 311, to scan the model 300 and the accessories 311. This scan allows for a complementary scan of the portions of the exterior of the model 300 that are obscured by the frame 412 and the accessories 311 (the surfaces of the model 300 on which the frame 412 and the accessories 311 are located). Note that although scanning from diagonally behind the right is described here, scanning may also be performed from diagonally in front of the right.

[0054] Next, from the state shown in FIG. 5(D), the turntable 406 is rotated leftward (counterclockwise) from the 330-degree position to the 30-degree position. During this time, the scanning device 413 is positioned so as to provide a bird's-eye view from diagonally above the model 300 and the accessory 311, and a scan is performed. This results in the state shown in FIG. 5(E). Then, from the state shown in FIG. 5(E), the turntable 406 is rotated rightward (clockwise) from the 30-degree position to the 360-degree position. During this time, the scanning device 413 is positioned so as to provide a sideways view from diagonally below the model 300 and the accessory 311, and a scan is performed. This results in the state shown in FIG. 5(F).

[0055] Note that if "fighting / punching" is selected in the setting of the accessory attribute 720 (described later), the accessory 311, including the frame 412, will not be placed. In this case, since the accessory 311 will not be placed, the operation of rotating the turntable from FIG. 5(E) back to FIG. 5(F) described above may be omitted. In this case, the scan may be terminated midway through the rotation from 330 degrees to the 30-degree position in FIG. 5(E). For example, the rotation may be terminated at 360 degrees or at an angle smaller than 30 degrees (e.g., 5 degrees, 10 degrees, etc.). In this way, the first scan range when the accessory 311 is placed is different from the second scan range when the accessory 311 is not placed, and the first scan range is larger (or wider) than the second scan range. Note that if the frame 412 remains, the operation of rotating the turntable from FIG. 5(E) back to FIG. 5(F) described above may not be omitted.

[0056] In the scanner device 413 assumed for use in this embodiment, scanning is performed at a position approximately midway (for example, when the distance between the scanner device 413 and the model 300 is approximately 130 mm) within the scannable range of the scanner device 413 (for example, a distance of 90 mm to 180 mm). In this case, if the accessory 311 passes between the scanner device 413 and the model 300, there is a risk that the accessory 311 will block the model 300 and cause the scan target to be lost. However, as described above, by rotating the turntable 406 clockwise and also counterclockwise to perform scanning, and by performing scanning to complement the blocked range, it is possible to reliably obtain three-dimensional model data for each of the model 300 and the accessory 311 without losing sight of the scan target.

[0057] Next, an example of processing executed by the image processing system 10 according to this embodiment will be described with reference to Fig. 6. At least a part of the processing corresponding to this flowchart is realized by the CPU 101 of the information processing device 100 executing a program stored in the ROM 103 or the storage device 104.

[0058] First, in S601, the CPU 101 accepts user registration. The CPU 101 accepts input of the user's name and contact information. Each user is assigned a user identifier to uniquely identify the individual user. The CPU 101 stores the input user information in the storage device 104 in association with the time the input was accepted and the user identifier.

[0059] Once user registration is complete, the user sets their own model on the model support device 130. The CPU 101 can determine whether the model 300 and accessories 311 have been set on the model support device 130 based on the scanned image by the scanner 110. Alternatively, a switch that turns on when the model 300 and accessories 311 are set on the model support device 130 may be provided, and the CPU 101 may detect a signal from the switch to make the determination. Alternatively, a button for accepting an operation when the setting of the model 300 and accessories 311 is complete may be displayed on the display device 140, and the CPU 101 may detect whether the operation in response to the button operation has been accepted. In S602, the CPU 101 detects, by one of the methods described above, that the model 300 and accessories 311 have been set on the model support device 130. In response to this detection, the processing proceeds to S603.

[0060] 7, which will be described later, there are cases where an attack is performed without using the accessory 311, so if a certain time has passed since the model 300 was set and the accessory 311 has not been set, the process may proceed to S603. Alternatively, it may be possible to detect whether the accessory 311 has been set only when the frame 412 has been set.

[0061] In S603, a process is executed to receive the specification of the attributes of the model 300 and the accessories 311. Here, for example, a screen 700 as shown in Fig. 7 is displayed, and it is possible to receive the selection of the attribute (type) of the model to be scanned, the attribute (type) of the accessories, and the color (effect color) to be used to effect the subject and accessories.

[0062] The screen 700 in FIG. 7 can be displayed on, for example, the display device 140 to accept operation input from the user. In the display of the screen 700, the user can select either realistic or deformed for the model attribute 710. Because the appearance characteristics of the model differ between realistic and deformed models, the scanning method can also be selected according to the selected model attribute. For example, in the case of deformed models, the operating range of the scanner device 413 may be set narrower than in the case of realistic models. Note that the model attribute 710 may further allow the user to specify a scale value of the model (1 / 144, 1 / 100, 1 / 60, etc.). This makes it possible to further set the operating range of the scanner device 413 according to the scale of the model.

[0063] Next, regarding the accessory attribute 720, assuming that the accessory is a weapon, the attribute can be selected from "fighting / punching," which uses a body part such as a fist as a weapon, or a "melee weapon" such as a "rifle," "machine gun," "bazooka," or sword as a ranged weapon. Note that the weapon attributes shown in FIG. 7 are merely examples, and more detailed attributes can be specified. In this embodiment, the accessory attribute 720 corresponds to specifying the type of effect using the accessory and whether or not the accessory is present. That is, as a performance method, it is possible to specify one of the following: an effect using a ranged weapon such as a "rifle," "machine gun," or "bazooka," an effect using a "melee weapon" such as a sword, or an effect using a punch without a weapon. Furthermore, specifying "fighting / punching" indicates that there is no accessory, while specifying any other attribute indicates that the corresponding accessory is present. In this embodiment, different effects are performed depending on the specified attribute.

[0064] When using a body part, in addition to punching, kicking, rotating an arm or leg, or other attacks using any body part may be performed, and these may be selectable as weapon attributes. Furthermore, when "fighting / punching" is specified as the accessory attribute 720, scanning of the accessory 311 (described below) may be omitted, or scanning of the accessory 311 may be performed in the same manner as when a ranged weapon or melee weapon is specified. Furthermore, while only offensive weapons are shown in FIG. 7, attributes such as defensive weapons and shields may also be specified. The effect color 730 can be selected from red, yellow, yellow-green, blue, and pink. The effect colors shown in FIG. 7 are merely examples, and colors other than those shown here can be specified.

[0065] Note that if it is not detected in S602 that the accessory 311 has been set, only "fighting / punching" may be selectable. Alternatively, if it is detected in S602 that the accessory 311 has been set, "fighting / punching" may be made unselectable or selectable.

[0066] A start scan button 740 is displayed on screen 700, and the user can instruct the start of scanning after completing the attribute specification from 710 to 730 described above. When the start of scanning is instructed, the process proceeds to S604. At this time, the attribute information entered via the screen of FIG. 7 is stored in the storage device 104 in association with user information. When the start scan button 740 is operated, a screen showing the progress of the scan is displayed on the display device 140. The user can understand the progress of the scan by referring to this screen. This screen includes messages (messages notifying the current status, such as "Scanning" or "Scan post-processing"), a progress meter (displaying the progress as a percentage using a pie chart or bar graph), and the like. A scan button for stopping the scan may also be included.

[0067] In S604, as described with reference to FIG. 5, the scanner 110, the support arm 120, and the model support device 130 cooperate to perform a scanning process (photographing process) to generate a scanned image. Specifically, the CPU 101 controls the support arm 120 to move the scanner 110 to one of the registered photographing positions, and the scanner 110 can perform a scan while rotating the model support device 130 at that photographing position. After stopping the model support device 130 at a predetermined rotation angle, the scanner 110 can perform a photographing operation at any photographing position and photographing angle. In S604, the scanning operation can be switched depending on the type of attribute accepted in the accessory attribute 720 in S603. That is, if an attribute other than "fighting / punching" is selected, the accessory 311 is installed, and therefore a first scanning operation corresponding to that case is performed. On the other hand, if "fighting / punching" is selected, the accessory 311 is not installed, and therefore a second scanning operation corresponding to that case is performed. The first scanning operation is performed in the first scanning range described above, and the second scanning operation is performed in the second scanning range described above. The captured scanned image is transmitted to the information processing device 100, and the CPU 101 stores the image in a table in the storage device 104 in association with the user identifier and the like.

[0068] In the next step S605, the CPU 101 performs post-scan processing on the scanned image acquired by the scanning process, generates three-dimensional model data for each of the model 300 and the accessory 311, and generates application data (display data) for video display. Here, if "fighting / punch" is specified as the accessory attribute 720, three-dimensional model data for the accessory 311 is not generated, but if other attributes such as "rifle," "machine gun," "bazooka," and "melee weapon" are specified, three-dimensional model data for the accessory 311 is generated. Furthermore, in generating the three-dimensional model data, if three-dimensional model data for the accessory 311 is generated, the three-dimensional model data for the accessory 311 is associated with the hand position in the three-dimensional model data for the model 300. On the other hand, if three-dimensional model data for the accessory 311 is not generated, such association is not performed.

[0069] Specifically, for rifles, machine guns, and bazookas, the grip portion is associated so that it comes to the position of the hand of the model 300. At this time, the angle at which the model 300 will grip the ranged weapon is also set. Furthermore, for close-range weapons such as sabers and swords, the handle portion is associated so that it comes to the position of the hand of the model 300, and the angle at which the model 300 will grip the close-range weapon is also set. In this way, it is possible to generate three-dimensional model data of the model 300 holding the accessory 311. If "fighting / punching" is specified as the attribute 720 of the accessory, the association process is omitted.

[0070] In post-scan processing, an area defining the range of unnecessary data is created in advance, the data is read, and the surfaces included in the unnecessary area are deleted. This makes it possible to remove almost all surfaces other than those of the model 300 and accessories 311. The unnecessary data range includes the areas where the action base 411, installation stand 414, frame 412, support member 412A, installation stand 415, etc. are located.

[0071] The generated application data can play back video when executed in a specific application on the information processing device 100. Also, when executed in the specific application on the head-mounted display 150A, it can play back VR video. The generated three-dimensional model data and display data are stored in the storage device 104 in association with the user information together with the attribute information received in S603.

[0072] In the following S606, video display processing is executed. When the video display processing is performed in the information processing device 100, the CPU 101 executes a specific application corresponding to the display data and displays the video on the display device 140 via the display control unit 107. Alternatively, in S606, the CPU 101 may control the communication device 105 to transmit the display data to the HMD 150A, and the CPU 151 of the HMD 150A may execute the received display data in the corresponding application to display the VR video, thereby performing the video display processing.

[0073] When displaying a video, the effects for the subject and accessories can be switched according to the attribute information. For example, if the attribute specified in attribute 720 in FIG. 7 is a machine gun, a rifle, or a bazooka, the main body is operated in a motion corresponding to the respective firearm, and an effect display of a bullet being fired from the firearm is performed. In this case, different effects may be performed for each attribute information, such as by varying the size of the bullet, the number of bullets fired per unit time, or the bullet speed in the effect display of the machine gun, rifle, or bazooka. In addition, the bullet is displayed in a color specified as the effect color.

[0074] Furthermore, if the attribute is "melee weapon," the main body is moved in a motion appropriate for a melee weapon, such as a saber or axe, and a slashing attack is displayed as the weapon is swung. The slashing attack is displayed in the color specified by the effect color. Furthermore, if the attribute is "fighting / punch," the main body is made to perform a punching motion, and an effect is displayed in which an energy bullet or attack wave equivalent to a slashing attack with a melee weapon is fired from the fist in response to the punching motion. Note that if a kick can also be specified, an effect can be displayed in which an energy bullet or attack wave is fired from the fist in response to the kicking motion. The attack is displayed in the color specified by the effect color.

[0075] In FIG. 6, a case where the specification of attribute information is accepted before the start of the scanning process is described, but the embodiment is not limited to this, and the specification of attribute information may be accepted after the scanning process of S604 or the post-scanning process of S605.

[0076] In this embodiment, a notification unit for notifying the status of the processing in FIG. 6 may be provided. For example, an LED strip may be attached to at least one of the turntable 406, the action base 411, the installation stand 414, the installation stand 415, etc., and the LED strip may emit light in different colors corresponding to the processing steps in the processing in FIG. 6 to notify the status. For example, the LED strip may emit light in a first color (e.g., red) during the processing in S601 and S602, and emit light in a second color (e.g., blue) while accepting input from the user in S603. Thereafter, when the user operates the scan start button 740, the LED strip may emit light in a third color (e.g., white). The LED strip may continue to emit light during each processing, or may emit light for a certain period of time (e.g., 5 seconds or 10 seconds) and then turn off.

[0077] As described above, in this embodiment, when three-dimensional model data generated from an image obtained by photographing a model with a scanner is placed in a virtual space and display data is generated for displaying an image, three-dimensional model data is generated by simultaneously photographing the model and its accessories, and this data can be used in the virtual space together with the three-dimensional model data of the model.

[0078] In addition, in this embodiment, attribute information can be specified regarding the type and presence / absence of accessories, so when using a three-dimensional model of the accessory in a virtual space, it is possible to perform a presentation according to the received attributes. This allows the user to reflect their own image in the presentation of the accessory, so that they do not feel uncomfortable. Furthermore, since the type of model can also be specified as attribute information, the scan range can be controlled according to the type of model, making it possible to obtain a scanned image with higher accuracy.

[0079] In the above-described embodiment, the accessories are described assuming weapons, but the type of accessories is not limited to weapons. For example, if the model is a specific humanoid character, the accessories may be clothing, accessories, etc. worn by the character. It is also possible to have the model appear in events, concerts, sports, online conferences, etc. held in a virtual space. Furthermore, the technology of this embodiment can also be applied to video technology such as cross reality (XR), which merges the real world with the virtual world and allows the perception of things that do not exist in real space.

[0080] In this way, in this embodiment, three-dimensional model data is generated from photographed images of the exterior of the model and its accessories, and it becomes possible to view a video in which the three-dimensional model is placed in a virtual space. Some models and accessories, such as prefabricated plastic models, are finished as unique works by the user through painting or other processes, and this individuality can be reflected in the video and character representation in the virtual space, greatly improving the user's preferences.

[0081] <Summary of the embodiment> The above-described embodiment discloses at least the following information processing device and computer program. (1) An information processing device, a scan control means for controlling the position and angle of the scanner to scan the appearance of an object placed on a table and generate a scanned image; a data generation means for generating three-dimensional model data from the scanned image; an image generating means for generating an image in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; a receiving means for receiving input of attribute information relating to the subject to be scanned and the rendering of the subject; Equipped with The image generating means generates the image so as to include a performance for each of the objects according to the attribute information. An information processing device. (2) The attribute information includes first attribute information representing a type of the subject and second attribute information relating to a rendering by the image generating means; The information processing device according to (1), wherein the image generating means generates the image so as to include a rendition for each of the subjects according to the first attribute information and the second attribute information. (3) The information processing device according to (2), wherein the second attribute information includes information indicating the type of the effect. (4) The information processing device according to (3), wherein the type of effect includes a plurality of different types of effect, and the image generating means includes a different effect image for each type. (5) The subject includes a first subject and a second subject, and the second subject is an accessory of the first subject; the first attribute information represents a type of the first subject; The information processing device according to any one of (2) to (4), wherein the second attribute information includes a type of the second subject used in the performance. (6) The second attribute information indicates the type of the accessory, thereby indicating the presence of the second subject; The information processing device according to (5), wherein the second attribute information indicates that a part of the first subject is used in the performance, thereby indicating that the second subject is not present. (7) The information processing device according to (5) or (6), wherein the data generation means performs a process of associating the second subject with a predetermined position of the first subject from a first scanned image of the first subject and a second scanned image of the second subject, thereby generating the three-dimensional model data as if the first subject is holding the second subject. (8) The second attribute information further indicates whether the second subject is present, If the second attribute information indicates that the second subject does not exist, the scanned image of the second subject is not generated; and The information processing device according to (7), wherein the data generating means generates the three-dimensional model data from the first scanned image without performing a process of associating the second subject with a predetermined position of the first subject. (9) The information processing device according to any one of (5) to (8), wherein the scanned image of the second subject is generated when the second attribute information indicates the presence of the second subject. (10) The information processing device according to any one of (5) to (9), wherein the scan control means differentiates a first scan range when the second subject is placed on the placement stand from a second scan range when the second subject is not placed on the placement stand, and the first scan range is larger than the second scan range. (11) The apparatus further includes a detection means for detecting whether the second subject is placed on the placement table, The information processing device according to (5) or (10), wherein, when the second subject is not detected, the accepting unit does not accept the second attribute information indicating the type of the accessory. (12) The information processing device according to any one of (5) to (11), wherein the first subject is a doll-type model, and the second subject is an accessory of the model. (13) The information processing device according to any one of (2) to (12), wherein the second attribute information further includes information about a color used in the effect. (14) A notification unit that notifies the operating state of the information processing device, The information processing device according to any one of (1) to (13), wherein the notification unit notifies that the information processing device is in at least one of a state where the information processing device is accepting input of the attribute information and a state where the scanning is being performed. (15) A computer program for causing a computer to function as the information processing device according to any one of (1) to (14).

[0082] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

Claims

1. An information processing device, a scan control means for controlling the position and angle of the scanner to scan the appearance of an object placed on a table and generate a scanned image; a data generation means for generating three-dimensional model data from the scanned image; an image generating means for generating an image in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; a receiving means for receiving input of attribute information relating to the subject to be scanned and the rendering of the subject; Equipped with The image generating means generates the image so as to include a rendition for each of the subjects according to the attribute information.

2. the attribute information includes first attribute information indicating a type of the subject and second attribute information related to a rendering by the image generating means, The information processing device according to claim 1 , wherein the image generating means generates the image so as to include a rendition for each of the subjects according to the first attribute information and the second attribute information.

3. The information processing device according to claim 2 , wherein the second attribute information includes information indicating a type of the effect.

4. The information processing device according to claim 3 , wherein the type of effect includes a plurality of different types of effect, and the image generating means generates different effect images for each type.

5. the subject includes a first subject and a second subject, the second subject being an accessory of the first subject; the first attribute information represents a type of the first subject; The information processing device according to claim 4 , wherein the second attribute information includes a type of the second subject used in the rendering.

6. the second attribute information indicates the type of the accessory, thereby indicating the presence of the second subject; The information processing device according to claim 5 , wherein the second attribute information indicates that a part of the first subject is used in the effect, thereby indicating that the second subject is not present.

7. 7. The information processing device according to claim 6, wherein the data generation means performs a process of associating the second subject with a predetermined position of the first subject from a first scanned image of the first subject and a second scanned image of the second subject, and generates the three-dimensional model data such that the first subject appears to be holding the second subject.

8. the second attribute information further indicates whether the second subject is present, If the second attribute information indicates that the second subject does not exist, the scanned image of the second subject is not generated; The information processing apparatus according to claim 7 , wherein the data generating means generates the three-dimensional model data from the first scanned image without performing a process of associating the second subject with a predetermined position of the first subject.

9. The information processing apparatus according to claim 8 , wherein the scanned image of the second subject is generated when the second attribute information indicates that the second subject is present.

10. 10. The information processing device according to claim 9, wherein the scan control means differentiates a first scan range when the second subject is placed on the placement stand from a second scan range when the second subject is not placed on the placement stand, and the first scan range is larger than the second scan range.

11. further comprising a detection means for detecting whether the second subject is placed on the placement table; The information processing apparatus according to claim 10 , wherein the accepting unit does not accept the second attribute information indicating the type of the accessory when the second subject is not detected.

12. The information processing device according to claim 11 , wherein the first subject is a doll-type model, and the second subject is an accessory of the model.

13. The information processing device according to claim 12 , wherein the second attribute information further includes information on a color used in the effect.

14. a notification unit that notifies an operating state of the information processing device; The information processing device according to claim 13 , wherein the notification unit notifies that the information processing device is in at least one of a state where the information processing device is accepting input of the attribute information and a state where the scanning is being performed.

15. A computer program for causing a computer to function as the information processing device according to any one of claims 1 to 14.

Images