Information processing device and computer program

Abstract

The invention relates to an information processing apparatus and a computer program. When an object is imaged and an image in which a three-dimensional model is disposed in a virtual space is generated using three-dimensional model data of the imaged object, the generated three-dimensional model data can be associated with attribute information of the imaged object to generate an image corresponding to the attribute information. This information processing device is provided with: a scanning control means for controlling the position and angle of a scanner, scanning the appearance of a subject, and generating a scanned image; a data generation mechanism which generates three-dimensional model data according to the scanning image; an image generation means for generating an image in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; and a receiving means for receiving input of attribute information relating to the object to be scanned and the performance of the object. The image generating means generates the image so as to include the performance of each of the subjects corresponding to the attribute information.

Description

Technical Field

[0001] The present invention relates to an information processing device and a computer program. Background Art

[0002] Patent document 1 describes a technology for mapping textures generated by a group of photographic images of an object captured from multiple photographic directions by a photographic unit onto graphic elements in a virtual space, thereby generating a virtual space image that can be viewed from a virtual camera in the virtual space.

[0003] [Background Art Literature]

[0004] [Patent Document]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2020-107251 Summary of the Invention

[0006] [Problems to be solved by the invention]

[0007] In the generation of virtual space images such as the above-mentioned technology, it is expected that when an object is photographed and the three-dimensional model data of the photographed object is used to generate an image in which a three-dimensional model is arranged in a virtual space, the generated three-dimensional model data can be associated with the attribute information of the photographed object to generate an image corresponding to the attribute information.

[0008] Therefore, its purpose is to be able to associate the generated three-dimensional model data with the attribute information of the photographed object when photographing the object and using the three-dimensional model data of the photographed object to generate an image corresponding to the attribute information.

[0009] [Technical means to solve the problem]

[0010] One aspect for solving the above-mentioned problem is an information processing device comprising:

[0011] A scanning control mechanism controls the position and angle of the scanner to scan the appearance of the object placed on the setting table and generate a scanned image;

[0012] A data generating mechanism for generating three-dimensional model data based on the scanned image;

[0013] an image generating unit for generating an image in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; an output unit for outputting the image generated by the image generating unit to a display device; and

[0014] an accepting unit that accepts input of attribute information related to a subject to be scanned and a performance of the subject; and

[0015] The image generation unit generates the image in a manner that includes a performance of each of the subjects corresponding to the attribute information.

[0016] [Effects of the Invention]

[0017] When photographing an object and using the three-dimensional model data of the photographed object to generate an image of the three-dimensional model arranged in a virtual space, it is possible to associate it with the attribute information of the photographed object, generate the three-dimensional model data, and generate an image corresponding to the attribute information. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a diagram showing a configuration example of an image processing system 10 according to the embodiment.

[0019] Figure 2 (A) and (B) in FIG. 1 are diagrams showing an example of the hardware configuration of the information processing device 100 and an example of the hardware configuration of the headphones 150 according to the embodiment.

[0020] Figure 3 This is a diagram showing an example of the appearance of a doll-type model according to the embodiment.

[0021] Figure 4A This is a diagram showing an example of implementation of the image processing system 10 .

[0022] Figure 4B 1 is a diagram showing a configuration example of the image processing system 10 according to the embodiment during scan imaging.

[0023] Figure 5 (A) to (F) are diagrams for explaining the relationship between the rotational positions of the scanner device 413 and the turntable 406 during scanning imaging according to the embodiment.

[0024] Figure 6 This is a flowchart corresponding to an example of processing in the image processing system 10 according to the embodiment.

[0025] Figure 7 This is a diagram showing an example of an attribute specification screen according to the embodiment. DETAILED DESCRIPTION

[0026] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention to which the claims relate, and all combinations of features described in the embodiments are not limited to those required by the invention. Two or more of the multiple features described in the embodiments may also be arbitrarily combined. In addition, the same reference numbers are given to the same or identical configurations, and repeated descriptions are omitted. In addition, in each figure, the up-down, down-down, left-right, front-to-back directions relative to the paper are used as the up-down, down-down, left-right, front-to-back directions of the parts (or components) in this embodiment, and are used in the description herein.

[0027] First, the configuration of an image processing system according to this embodiment will be described. Figure 1 1 is a diagram showing an example of the configuration of an image processing system 10 according to this embodiment. The image processing system 10 is configured by connecting a scanner 110, a support arm 120, a model support device 130, a display device 140, a headset 150, etc. to an information processing device 100. The system configuration is not limited to Figure 1 The information processing device 100 may be further connected to an external server, a cloud server, etc. via a network in the configuration shown. In the external server, etc., at least a part of the processing of the embodiment described below can be executed.

[0028] The information processing device 100 controls the operation of at least one of the scanner 110, the support arm 120, and the model support device 130 to capture images of the target object from any angle, generating multiple images and generating three-dimensional model data (true data) based on these images. If the target object is composed of multiple separable components, the 3D model data can be generated by capturing each component and then integrating these components to generate the 3D model data for the target object.

[0029] The information processing device 100 can also function as an image generation device that uses the generated three-dimensional model data as virtual space data to generate images for display in a virtual space. In this embodiment, the subject of the photographic object is an object such as an assembleable plastic model, an action figure (a figure with movable joints), a toy body, or a doll, which will be collectively referred to as a "model" below.

[0030] Next, the scanner 110 is a three-dimensional scanner device that captures (scans) the three-dimensional shape of the model of the photographed object under the control of the information processing device 100, and outputs the three-dimensional shape and color information of the photographed object. In this embodiment, the scanning signals output from the scanner 110 are collectively referred to as "scanned images." The scanner 110 can use, for example, Space Spider manufactured by Artec. In this embodiment, for example, by acquiring approximately 500 to 800 frames of scanned images, 3D scan data of the entire toy body can be obtained. In addition, as the scanner 110, for example, a smartphone with a camera installed with an application for capturing three-dimensional shapes can also be used.

[0031] The support arm 120 is a position and posture control device that moves the scanner 110 to a predetermined photographic position and posture according to the control of the information processing device 100. The support arm 120 can be configured to manually change the photographic position or posture and to fixedly maintain the changed position and posture, or it can be configured to be controlled by the information processing device 100. In the case where the support arm 120 is configured to be controlled by the information processing device 100, for example, the xArm7 manufactured by UFACTORY can be used. The xArm7 includes 7 joints and can perform the same movements as a human arm. In addition, the scanner 110 can also be positioned manually instead of the support arm 120. In addition, for photographic positions and postures that cannot be covered by the support arm 120, the scanner 110 can be manually operated to perform scanning.

[0032] The model support device 130 is a support table (or a setting table for setting the model) that supports a model in a fixed posture. The model support device 130 can also be configured to be rotatable, for example, while the model is set on the support table or at the front end of a support rod. In this embodiment, after the scanner 110 is positioned at any imaging position and imaging angle via the support arm 120, the model support device 130 is rotated to perform imaging. The model support device 130 can rotate in both clockwise and counterclockwise directions. In addition, the model support device 130 can be stopped at any rotation angle within the range of 0 degrees to 360 degrees, or it can be stopped at a specific rotation angle and positioned at any imaging position and imaging angle within the active range of the scanner 110 for imaging. By performing imaging at multiple rotation angles, imaging positions, and imaging angles, images of the entire model can be obtained. In addition, by driving the support arm 120 and the model support device 130 synchronously, imaging processing can be performed more simply and with higher precision. Alternatively, the scanner 110 may be manually moved around the model to perform scanning from any arbitrary imaging position and imaging angle, instead of the model support device 130 .

[0033] The display device 140 is a display device such as a liquid crystal display (LCD) and is capable of displaying the processing results of the information processing device 100. Specifically, it can display an image acquired by the scanner 110, the actual data of the three-dimensional model data generated based on the photographic image, or an image reconstructed using the actual data (VR (Virtual Reality) image). In addition, a meter can be displayed to indicate the progress of the scanning process. The display device 140 can include an operation unit 140A that accepts operations from a user who is viewing the displayed image. The user can operate the operation unit 140A to input operations corresponding to the content of the image displayed on the display device 140.

[0034] The headset 150 is composed of a head-mounted display 150A and a controller 150B, which will be described later. In particular, as a VR headset, it can also be configured to provide dynamic images corresponding to the posture or inclination of the user as an observer. The headset 150 is installed with a specified application and can download and execute application data executed in the application from the information processing device 100. In this embodiment, the headset 150, like the information processing device 100, can function as an image generating device, which uses three-dimensional model data as data for a virtual space, generates an image to be displayed in the virtual space, and provides the generated image as a VR image. In addition, the application data includes display data for displaying VR images.

[0035] Next, the hardware configuration of the information processing device 100 according to this embodiment will be described. Figure 2 (A) in the figure is an example of the hardware structure of the information processing device 100. The CPU (Central Processing Unit) 101 is a device that performs overall control of the information processing device 100 or calculation, processing, and management of data. For example, it is possible to control the imaging timing and number of images in the scanner 110, and control the joints of the arm of the support arm 120 to configure the scanner 110 at any imaging position and imaging angle. For example, after determining the imaging position and imaging angle of the scanner 110, the model support device 130 can be rotated, and the scanner 110 can perform the imaging action. In addition, after stopping the model support device 130 at a specified rotation angle, the scanner 110 can perform the imaging action at any imaging position and imaging angle.

[0036] The CPU 101 also functions as an image processing unit that processes images output from the scanner 110. Specifically, the CPU 101 uses three-dimensional model data generated based on scan signals obtained by the scanner 110 as virtual space data to generate an image that displays the three-dimensional model in the virtual space.

[0037] RAM (Random Access Memory) 102 is a volatile memory used as a temporary storage area, such as main memory and a work area, for CPU 101. ROM (Read Only Memory) 103 is a non-volatile memory, storing image data and other data, as well as various programs for CPU 101 operations, in designated areas. CPU 101 uses RAM 102 as work memory, for example, to control various components of information processing device 100 according to programs stored in ROM 103. Programs for CPU 101 operations are not limited to being stored in ROM 103; they may also be stored in storage device 104.

[0038] The storage device 104 is comprised of, for example, a hard disk drive (HDD) or flash memory. It stores application programs, an operating system (OS), control programs, related programs, game programs, and the like. The storage device 104 can read and write data under the control of the CPU 101. The storage device 104 can be used in place of the RAM 102 or ROM 103.

[0039] The communication device 105 is a communication interface for communicating with the scanner 110, the support arm 120, the model support device 130, the display device 140, and the headset 150, based on the control of the CPU 101. The communication device 105 can also be configured to communicate with an external server, etc. The communication device 105 can include a wireless communication module, which can include a well-known circuit structure, and the well-known circuit structure includes an antenna system (Antenna System), an RF (Radio Frequency) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identification module card, a memory, etc. Here, the information processing device 100 can also communicate with the scanner 110, the support arm 120, the model support device 130, the display device 140, and the headset 150 through wireless communication.

[0040] Furthermore, the communication device 105 may also include a wired communication module for wired connections. The wired communication module enables communication with other devices, including the display device 140, via one or more external ports. Furthermore, it may include various software components for processing data. The external port connects to other devices directly or indirectly via a network, such as Ethernet, USB (Universal Serial Bus), or IEEE 1394. Furthermore, it is also possible to implement software that implements equivalent functions to the above devices, thereby replacing the hardware devices.

[0041] Operation unit 106 is comprised of, for example, buttons, a keyboard, a touch panel, or a controller, and receives user input. Operation unit 106 may be shared with operation unit 140A or may be independent of operation unit 140A. For example, if operation unit 140A is configured as an operation unit such as a keyboard and a mouse, it may be shared with operation unit 106. On the other hand, if operation unit 140A is configured as an operation unit such as a touch panel or a controller, it may be a separate operation unit from operation unit 106.

[0042] The display control unit 107 functions as a control unit for displaying information on the display device 140 connected to the information processing device 100, and controls the operation of the display device 140. Some of the functions of the operation unit 106 may also be provided on the display device 140. For example, the display device 140 may be configured as a device having a touch panel such as a tablet terminal.

[0043] Next, the hardware configuration of the headphones 150 according to this embodiment will be described. Figure 2 (B) in the figure shows an example of the hardware configuration of the headset 150. The headset 150 is composed of a head-mounted display (HMD) 150A and a controller 150B. The HMD 150A provides right-eye images and left-eye images to the user's left and right eyes, respectively, and uses the parallax between them to allow the user to experience a three-dimensional sense, thereby enabling virtual reality (VR). The controller 150B is provided in a housing independent of the HMD 150A. Although the controller 150B is assumed to be a pair of two controllers so that they can be held and operated by the user's left and right hands respectively, it can also be a single controller.

[0044] The CPU 151 is a device that performs overall control of the headset 150 and performs data calculation, processing, and management. For example, it can execute application data downloaded from the information processing device 100 and display VR images on the display 157. Furthermore, based on operations received via the controller 156 or information detected by the detection unit 158, it can switch the displayed VR images, change the viewpoint within the VR space, or change the position within the VR space.

[0045] RAM 152 is a volatile memory used as a temporary storage area, such as main memory and a work area, for CPU 151. ROM 153 is a non-volatile memory, storing image data and other data, as well as various programs for CPU 151 operations, in designated areas. For example, CPU 151 uses RAM 152 as work memory to control various components of headset 150 according to the programs stored in ROM 153. Programs for CPU 151 operations are not limited to being stored in ROM 153; they may also be stored in storage device 154.

[0046] Storage device 154 is comprised of, for example, a magnetic disk such as an HDD or flash memory. Storage device 154 stores application programs, the operating system (OS), control programs, related programs, game programs, application data downloaded from information processing device 100, and display data. Storage device 154 can read and write data under the control of CPU 151. Storage device 154 can be used in place of RAM 152 or ROM 153.

[0047] The communication device 155 is a communication interface for communicating with the information processing device 100 or the controller 150B under the control of the CPU 151. The communication device 155 includes a wireless communication module capable of wireless communication based on Bluetooth or WiFi (IEEE802.11). The headset 150 is connected to the information processing device 100 via wireless communication and can download data for displaying VR images. Furthermore, the headset 155 communicates with the controller 150B to receive information regarding user instructions for operating the controller 150B.

[0048] Display 156 is configured to provide the right-eye image and left-eye image generated by CPU 151 to the user's right and left eyes, respectively. Detection unit 157 is a mechanism for detecting the gaze direction of the user's left and right eyes and the inclination of the head-mounted display. Detection unit 157 includes, for example, sensors for detecting the gaze direction or gaze direction of the user wearing HMD 150A. Furthermore, it includes a gyroscope, a magnetometer, an accelerometer, a Global Positioning System (GPS), a compass, and the like, and can specify the position, posture, inclination, and other aspects of HMD 150A based on this detection information. Detection unit 157 detects information used to determine the user's gaze direction or movement in HMD 150A and transmits it to CPU 151. Based on the received detection information, CPU 151 determines the gaze direction or movement and adjusts the image displayed on display 156 of HMD 150A to align with the determined gaze direction or movement.

[0049] The controller 150B is composed of, for example, a plurality of buttons and a cross key, and receives operation inputs such as selection operations and direction indication operations from the user. The controller 150B is wirelessly connected to the HMD 150A via the communication device 155 .

[0050] Next, refer to Figure 3 , an example of a model of a subject to be photographed in this embodiment and its accessories is described. Model 300 is a model having the appearance of a doll (robot or person). The model can be assembled and painted as a plastic model, for example. Alternatively, it can be a completed model such as a doll (action figure) with active joints. Figure 2 The model is merely an example for illustration. The model is not limited to a doll-like appearance and can also be any shape, such as a regular vehicle, a racing car, a military vehicle, an airplane, a ship, an animal, or a hypothetical life form. Furthermore, the object to be photographed is not limited to a model, as long as its three-dimensional shape can be captured by the scanner 110.

[0051] Model 300 comprises a head 301, chest 302, right wrist 303, left wrist 304, right torso 305, left torso 306, right leg 307, left leg 308, right foot 309, and left foot 310, all of which are joined together. At least a portion of each part 301-310 is supported so that it can rotate (or swing) relative to adjacent parts. For example, head 301 is supported so that it can rotate relative to chest 302, and right wrist 303 and left wrist 304 are supported so that they can rotate relative to chest 302. Because each part of model 300 is provided with a joint structure, model 300 can assume any desired pose.

[0052] In addition, the model 300 is provided with an accessory 311. The accessory 311 is an accessory part that is supposed to be used together with the model 300. Figure 3 In the figure, the weapon is in the shape of a rifle. That is, it is assumed that the model 300 uses the accessory 311 by holding it with both hands. The shape of the weapon is not limited to the shape of a rifle, and can include weapons in any shape, such as a sword (knife) shape, a shield (shield) shape, a bomb shape (such as a grenade), a bazooka shape, etc. In addition, the accessories are not limited to weapons and can also be decorative items as long as they can be used with the model 300. For example, in the case where the model 300 is a puppet-type model of a specific character, it can also be various props carried by the character (such as luggage, hats, bags, accessories, smartphones, and other items that the character can carry).

[0053] Next, refer to Figure 4A , an overall implementation example of the image processing system 10 in this embodiment is described. Figure 4AThis diagram generalizes an implementation example that can generate actual data generated by photographing a model.

[0054] exist Figure 4A In this example, housing 401 contains the information processing device 100, a drive system for driving a support arm 402, and a drive system for driving a turntable 406. Furthermore, support arm 402 allows for manual adjustment of the shooting direction and position. The surface of housing 401 is flat to allow for the attachment of promotional posters.

[0055] The support arm 402 corresponds to the support arm 120 and can support and fix the position of the terminal 403 functioning as the scanner 110 under the control of the information processing device 100 or manually. The support arm 402 can also operate to control the inclination of the terminal 403.

[0056] The terminal 403 is a touch-panel terminal with a built-in camera that can be used as the scanner 110 , and can be, for example, a smartphone, a tablet terminal, a digital camera, etc. Space Spider manufactured by Artec can be used instead of these terminals. Figure 4A The diagrams above generally illustrate system configurations, and can be configured to suit the type of device used as scanner 110. Terminal 403 can capture an image of model 300 and transmit it to information processing device 100. Ring light 404 is an illumination device used when terminal 403 captures image of model 300. It illuminates model 300 evenly, minimizing the appearance of shadows. In addition to ring light 404, auxiliary lights can be installed as overhead lights or on the left, right, or bottom sides as additional light sources. The aforementioned three-dimensional scanner device can be used in place of terminal 403.

[0057] Background plate 405 is used for photography and can be, for example, a white board. Turntable 406 can carry model 300 and rotate it. The structure comprising turntable 406 and its drive system corresponds to model support device 130. Under control of information processing device 100, turntable 406 rotates clockwise and counterclockwise, and stops. Turntable 406 may also be provided with a plurality of predetermined markings 410. Markings 410 can be used to adjust the orientation or position of the model being photographed.

[0058] exist Figure 4A In the present invention, the model is placed on a semi-transparent (transparent) table, but other supporting devices such as a "movable base" (registered trademark) can also be used. The movable base is provided on the table with a structure that is curved. The model can be mounted on the front end of a U-shaped support. In this case, the aforementioned marker 410 can also be placed on the front end of the support. The method of indicating the model can be changed depending on the posture. For example, in an upright position, the model can be placed on a transparent stand and filmed. On the other hand, if you want to film the soles of the feet, such as in a flying position, a movable base can be used. Furthermore, a movable base can also be used for filming an upright position.

[0059] The display device 407 is a device corresponding to the display device 140 and may also have a touch panel function. The user can use the touch panel function to perform a predetermined selection action. In addition, operations can also be accepted by a controller or mouse (not shown) that can function as the operation unit 140A instead of the touch panel. The VR headset 408 includes an HMD 408A and a controller 408B, which correspond to HMDs 150A and 150B, respectively. The user wears the HMD 408A on the head and holds the controller 408B in each hand to operate, thereby being able to operate while watching VR images.

[0060] Next, refer to Figure 4B , a more specific configuration of the image processing system 10 in the case of simultaneously scanning the model 300 and the accessory 311 according to this embodiment will be described. Figure 4B In the figure, the model 300 and the accessory 311 are placed on a turntable 406. The model 300 is supported by a movable base 411 and is positioned approximately in the center of the turntable 406. The accessory 311 is supported by a support member 412A within a frame 412, adjacent to the model 300 and positioned near the end of the turntable 406. The indicator member 412A can be, for example, a clamp-shaped member. This allows the accessory 411 to be clamped and supported within the frame 412.

[0061] The scanner device 413 is supported by the support arm 402. Here, it is assumed that the Space Spider manufactured by Artec Corporation is used. The drive system for driving the support arm 402 and the drive system for driving the turntable 406 are similar to those in FIG. Figure 4A In addition, in this embodiment, the turntable 406 can rotate in both clockwise and counterclockwise directions.

[0062] In this embodiment, Figure 4B In the illustrated embodiment, the model 300 and the accessory 311 are placed together on a turntable 406 and are imaged together by a scanner 413 to generate a scanned image. Although the scanned image includes both the model 300 and the accessory 311, the 3D model data is generated by separating them.

[0063] Furthermore, the frame 412 supporting the accessory 311 is configured as a frame surrounding the accessory 311 and serves as an indicator for identifying the area where the accessory 311 is located. In other words, a three-dimensional object within the frame 412 is identified as the accessory 311. To minimize obstruction of the scanner device 413's view, the frame 412 is desirably as thin as possible while maintaining strength. For example, a thickness of approximately 3 mm is possible.

[0064] Furthermore, model 300 and frame 412 are spaced apart to maintain a constant distance. Depending on the rotation angle of turntable 406, frame 412 may be positioned closer to model 300 than scanner device 413, potentially obstructing imaging of model 300. This obscured portion requires supplementation at a different rotation angle. However, if model 300 and frame 412 are too close, supplementation accuracy decreases. Therefore, a constant distance is maintained to ensure supplementation accuracy.

[0065] In this case, to ensure a constant distance between the model 300 and the frame 412 (or to prevent them from being too close), the dimensions of the model 300 and the accessory 311 can be limited to sizes smaller than predetermined dimensions. For example, the dimensions of the model 300 can be set to 15 cm in length, 15 cm in width, and 10 cm in depth, and the dimensions of the accessory 311 can be set to 15 cm in length, 5 cm in width, and 2 cm in thickness.

[0066] Furthermore, in this embodiment, the mounting platform 414, on which the movable base 411 supporting the model 300 is mounted, is configured to be thicker than the mounting platform 415, on which the frame 412 supporting the accessory 311 is mounted. Specifically, the model 300 is positioned above the accessory 311. For example, at least the upper body or upper half of the model 300 can be positioned above the frame 412. This reduces the amount of obstruction caused by the accessory 311 when scanning the model 300, improving the success rate of the scan.

[0067] Next, refer to Figure 5 , the cooperative action of the turntable 406 and the support arm 120 during scanning corresponding to this embodiment will be described. Figure 5 4 is a diagram showing the relationship between the rotation direction and amount of the turntable 406 relative to the scanner device 413 and the installation tables 414 and 415. Figure 5 In FIG. 4 , the model 300 placed on the installation table 414 is arranged so that the direction indicated as the front is the front. The accessories 311 placed on the installation table 415 can also be arranged in the same manner.

[0068] Figure 5(A) in FIG. 4 shows the relationship between the scanner device 413 and the installation platforms 414 and 415 at the start position of the scanning operation. The rotation angle of the turntable 406 at this point is set as 0 degrees as the reference. When the turntable 406 rotates 90 degrees to the right (clockwise) from the above state, it becomes Figure 5 The state shown in (B) in the figure. Figure 5 (A) in the transformation to Figure 5 During (B), the scanner device 413 is located on the side of the turntable 406 and scans the model 300 and the accessories 311. Figure 5 In the state shown in (B), when the front of the model 300 and the accessory 311 are located on the front side of the scanner device 413, the support arm 402 is moved within the limit of the active area to scan the front, left and right sides, flat surface and bottom surface of the model 300 and the accessory 311.

[0069] Then, from Figure 5 In the state (B), the turntable 406 is rotated to the right from the position of 90 degrees to the position of 270 degrees. During this period, the model 300 and the accessory 311 are also scanned. Then, it becomes Figure 5 In the state shown in (C), when the back of the model 300 and the accessory 311 are located on the front side of the scanner device 413, the support arm 402 is moved within the limit of the active area to scan the back, left and right sides, flat surface and bottom surface of the model 300 and the accessory 311.

[0070] Then, from Figure 5 In the state (C), the turntable 406 is further rotated to the right from the position of 270 degrees to the position of 330 degrees. During this period, the model 300 and the accessory 311 are also scanned. Then, it becomes Figure 5 In the state shown in (D) in the figure, support arm 402 is moved from the obliquely right rear side of model 300 and accessory 311 within the limits of its active range to scan model 300 and accessory 311. This scanning method complements the portion of model 300 obscured by frame 412 or accessory 311 (the surface of model 300 on which frame 412 or accessory 311 is located). While scanning from the obliquely right rear side is described here, scanning from the obliquely right front side is also possible.

[0071] Then, from Figure 5 In the state (D) in the middle, the turntable 406 is rotated to the left (counterclockwise) from the position of 330 degrees to the position of 30 degrees. During this period, the scanner device 413 is configured to scan from the upper side of the model 300 and the accessory 311 as a bird's-eye view. Figure 5 Then, from Figure 5 In the state (E) in the figure, the turntable 406 is rotated to the right (clockwise) from the position of 30 degrees to the position of 360 degrees. During this period, the scanner device 413 is configured to be viewed from the oblique lower side of the model 300 and the accessory 311 and scanned. Figure 5 The state shown in (F) in FIG.

[0072] In addition, when "Fighting / Boxing" is selected in the setting of the attribute 720 of the accessory described later, the accessory 311 is set without including the frame 412. In this case, since the accessory 311 is not set, it can also be omitted from the above description. Figure 5 (E) returns Figure 5 The turntable can be rotated by the method (F) in the figure. Figure 5 In (E), the scanning is not stopped from 330 degrees to 30 degrees. For example, the rotation may be stopped at 360 degrees or at an angle less than 30 degrees (e.g., 5 degrees, 10 degrees, etc.). In this way, the first scanning range when the accessory 311 is provided is different from the second scanning range when the accessory 311 is not provided, and the first scanning range is larger (or wider) than the second scanning range. In addition, when the frame 412 remains, it is not necessary to omit the scanning range from the above. Figure 5 (E) returns Figure 5 The turntable is rotated in the manner (F).

[0073] Assume that the scanner device 413 used in this embodiment performs scanning at a position approximately in the middle of the scanner device 413's scannable range (e.g., a distance of 90 mm to 180 mm) (e.g., the distance between the scanner device 413 and the model 300 is approximately 130 mm). In this case, if the accessory 311 passes between the scanner device 413 and the model 300, there is a concern that the accessory 311 may obscure the model 300, resulting in loss of the scanned object. However, as described above, by rotating the turntable 406 clockwise and then counter-rotating it counterclockwise while scanning, and by performing scanning in a manner that compensates for the obstructed range, it is possible to reliably acquire three-dimensional model data for each of the model 300 and the accessory 311 without losing the scanned object.

[0074] Next, refer to Figure 6 An example of processing executed by the image processing system 10 according to this embodiment is described below. At least a portion of the processing corresponding to the flowchart is implemented by the CPU 101 of the information processing device 100 executing a program stored in the ROM 103 or the storage device 104.

[0075] First, at S601, CPU 101 accepts user login. It accepts input of a user's name or contact information. Each user is assigned a user identifier to uniquely identify the user. CPU 101 associates the input user information with the time the input was accepted or the user identifier and stores them in storage device 104.

[0076] When the user login is complete, the user installs their model on the model support device 130. Based on the scanned image of the scanner 110, the CPU 101 can determine whether the model 300 and the accessories 311 are installed on the model support device 130. Alternatively, a switch that turns on when the model 300 and the accessories 311 are installed on the model support device 130 can be pre-configured, and the CPU 101 detects and determines the signal from the switch. Alternatively, a button for accepting an operation when the installation of the model 300 and the accessories 311 is completed can be pre-displayed on the display device 140, and the CPU 101 detects whether an operation for the button operation is accepted. In S602, the CPU 101 detects whether the model 300 and the accessories 311 are installed on the model support device 130 using any of the above methods. Based on the detection, the process enters S603.

[0077] In the following Figure 7 Since the attack performance may be performed without using the accessory 311 in the accessory attribute 720, the process may proceed to S603 if the attachment of the accessory 311 is not detected after a predetermined time has passed after at least the model 300 is attached. Alternatively, the attachment of the accessory 311 may be detected only when the frame 412 is attached.

[0078] In S603, a process of accepting the specification of the attributes of the model 300 and the accessory 311 is executed. Figure 7 The screen 700 shown receives selections of the attributes (type) of the model to be scanned next, the attributes (type) of the accessories, and the colors (rendering colors) used for rendering the subject and accessories.

[0079] Able to Figure 7 The screen 700 is displayed on the display device 140, for example, and accepts operation input from the user. The user can select either realistic or deformed for the model attribute 710 displayed on the screen 700. Because the appearance characteristics of the model are different in realism and deformation, the scanning method can be selected in accordance with the selected model attribute. For example, in the case of deformation, the range in which the scanner device 413 operates can be set narrower than in the case of realism. In addition, in the model attribute 710, the scale value of the model (1 / 144, 1 / 100, 1 / 60) can also be specified. In this way, the operating range of the scanner device 413 can be further set in accordance with the scale of the model.

[0080] Next, regarding the attribute 720 of the accessory, assuming that the accessory is a weapon, you can select from "Fighting / Boxing" which uses a body part such as a fist as a weapon, "Rifle" which is a shooting weapon, "Machine Gun", "Bazooka", and "Melee Weapon" such as a sword. In addition, as the attribute of the weapon, Figure 7 The above is just an example, and more detailed attributes can be specified. The attribute 720 of the accessory in this embodiment is equivalent to the specification of the type of performance using the accessory, and the specification related to the presence or absence of the accessory. That is, as a performance method, it is possible to specify any one of a performance using shooting weapons such as "rifles", "machine guns", and "bazookas", a performance using "melee weapons" such as swords, or a performance using boxing instead of weapons. In addition, if "fighting / boxing" is specified, it means that there are no accessories, and if an attribute other than this is specified, it means that there are corresponding accessories. In this embodiment, different performances corresponding to these specified attributes are performed.

[0081] When using the body parts, in addition to punching, kicking, rotating the wrist or foot, and attacks using any other body parts can also be performed, and these can also be selected as weapon attributes. In addition, when "fighting / boxing" is specified as the attribute 720 of the attachment, the scanning of the attachment 311 described later can be omitted, and the scanning of the attachment 311 can be performed in the same way as when a shooting weapon or a melee weapon is specified. In addition, although Figure 7 Only attacking weapons are shown in the figure, but you can also specify attributes such as defensive weapons and shields. For the performance color 730, you can select any one of red, yellow, yellow-green, blue, and pink. Figure 7 The color shown here is just an example, and colors other than those shown here can be specified.

[0082] Furthermore, if the attachment 311 is not detected in S602, only "Fighting / Boxing" may be selected. Alternatively, if the attachment 311 is detected in S602, "Fighting / Boxing" may or may not be selected.

[0083] The scan start button 740 is displayed on the screen 700. When the attributes of 710 to 730 are specified, the user can instruct the start of scanning. When the start of scanning is instructed, the process is transferred to S604. Figure 7The attribute information entered on the screen is associated with the user information and stored in the storage device 104. When the scan start button 740 is pressed, a screen showing the progress of the scan is displayed on the display device 140. The user can understand the progress of the scan by referring to the screen. The screen includes messages (such as scanning, post-scan processing in progress, and messages notifying the current status), a meter showing the progress (displaying the progress as a percentage using a pie chart or bar chart), etc. A scan button for stopping the scan may also be included.

[0084] In S604, as shown in reference Figure 5 As described, the scanner 110, support arm 120, and model support device 130 collaborate to perform a scanning process (photography process) to generate a scanned image. Specifically, the CPU 101 controls the support arm 120 to move the scanner 110 to any registered photography position. At that photography position, the model support device 130 is rotated while the scanner 110 scans. Furthermore, after the model support device 130 stops at a predetermined rotation angle, the scanner 110 can perform photography at any photography position and angle. In S604, the scanning operation can be switched based on the type of attribute received by the accessory attribute 720 in S603. Specifically, if an attribute other than "Fighting / Boxing" is selected, the first scanning operation corresponding to that condition is executed because the accessory 311 is installed. On the other hand, if "Fighting / Boxing" is selected, the second scanning operation corresponding to that condition is executed because the accessory 311 is not installed. The first scanning operation is performed within the first scanning range, and the second scanning operation is performed within the second scanning range. The captured scanned image is transmitted to the information processing apparatus 100 , and the CPU 101 associates the scanned image with a user identifier and the like and stores the image in a table in the storage device 104 .

[0085] In the following step S605, CPU 101 performs post-scan processing on the scanned image obtained through the scanning process, generating 3D model data for each of model 300 and accessory 311, and generating application data (display data) for image display. Here, if "Fighting / Boxing" is specified as accessory attribute 720, 3D model data for accessory 311 is not generated. However, if "Rifle," "Machine Gun," "Bazooka," or "Melee Weapon" is specified, 3D model data for accessory 311 is generated. Furthermore, when generating 3D model data for accessory 311, the 3D model data for accessory 311 is associated with the hand position of the 3D model data of model 300. On the other hand, if 3D model data for accessory 311 is not generated, this association is not established.

[0086] Specifically, for rifles, machine guns, and bazookas, the handle portion is associated so that it reaches the position of model 300's hand. This also sets the angle at which model 300 grips the weapon. Furthermore, for melee weapons like sideknives and swords, the handle portion is associated so that it reaches the position of model 300's hand, and the angle at which model 300 grips the weapon is also set. This generates three-dimensional model data for model 300 holding accessory 311. If "Fighting / Boxing" is specified as accessory attribute 720, this association process is omitted.

[0087] During post-scan processing, a region defining the area of ​​useless data is pre-created, and faces within this area where data reading is not necessary are deleted. This allows for the removal of faces other than those of the model 300 and the accessory 311. The useless data region includes the areas surrounding the movable base 411, the mounting platform 414, the frame 412, the support member 412A, and the mounting platform 415.

[0088] When the generated application data is executed in a specific application on the information processing device 100, images can be reproduced. Furthermore, when the specific application is executed on the head-mounted display 150A, VR images can be reproduced. The generated three-dimensional model data and display data are stored in the storage device 104 along with the attribute information received in S603, associated with the user information.

[0089] In the next step S606, image display processing is performed. When performing the image display processing in the information processing device 100, the CPU 101 executes the display data in the corresponding specific application and displays the image on the display device 140 via the display control unit 107. In addition, in S606, the CPU 101 controls the communication device 105 to transmit the display data to the HMD 150A. The CPU 151 of the HMD 150A executes the received display data in the corresponding application to perform processing to display the VR image, thereby performing image display processing.

[0090] When displaying an image, the performance of the subject or accessories can be switched according to the attribute information. Figure 7 If the attribute specified by attribute 720 is a machine gun, rifle, or bazooka, the main body is moved using behaviors corresponding to each weapon, creating a performance display of projectiles being fired from the weapon. In this case, different performances can be created based on attribute information such as the size of the projectiles, the number of projectiles fired per unit time, and the speed of the projectiles in the performance display for each machine gun, rifle, or bazooka. Furthermore, the projectiles are displayed in the color specified by the performance color.

[0091] Furthermore, when the attribute is "melee weapon," for example, the main body is moved by a behavior corresponding to a melee weapon such as a saber or an axe, and a display of a slashing strike from the melee weapon is performed in response to the weapon being swung. The display of the slashing strike is performed in the color specified by the performance color. Furthermore, when the attribute is "fighting / boxing," the main body is made to perform a boxing motion, and based on the boxing motion, a display of an energy bullet or attack wave equivalent to a slashing strike in a melee weapon is performed from the fist. Furthermore, the same is true when a kick can be specified; based on the kicking motion, a display of an energy bullet or attack wave being fired from the fist is performed. The display of the attack is performed in the color specified by the performance color.

[0092] exist Figure 6 In the description above, the case where the specification of attribute information is accepted before the start of the scanning process is described, but the embodiment is not limited thereto, and the specification of attribute information may be accepted after the scanning process of S604 or the post-scanning process of S605.

[0093] In this embodiment, it is possible to set a Figure 6 For example, a LED (Light Emitting Diode) strip may be installed in advance on at least one of the turntable 406, the movable base 411, the setting table 414, and the setting table 415, so that the LED strip can be used to Figure 6 The LED strip can be notified by emitting different colors corresponding to the processing steps in the processing. For example, during the processing in S601 and S602, the LED strip can be illuminated in a first luminous color (e.g., red), and while accepting input from the user in S603, the LED strip can be illuminated in a second luminous color (e.g., blue). Thereafter, if the user presses the scan start button 740, the LED strip can be illuminated in a third luminous color (e.g., white). The LED strip can continue to emit light during the processing, or it can emit light for a constant time (e.g., 5 seconds or 10 seconds) and then turn off.

[0094] As described above, in this embodiment, when the three-dimensional model data generated from the image obtained by photographing the model by the scanner is configured in a virtual space to generate display data for displaying the image, the three-dimensional model data can be generated by photographing its accessories simultaneously with the model, and used in the virtual space together with the three-dimensional model data of the model.

[0095] Furthermore, in this embodiment, since attribute information can be specified based on the type or presence of accessories, a performance corresponding to the accepted attributes can be created when using the three-dimensional model of the accessory in a virtual space. This allows the user to reflect their own image in the performance related to the accessory, thus eliminating any sense of incongruity. Furthermore, since attribute information can be specified based on the type of model, more accurate scanned images can be obtained by controlling the scanning range according to the type of model.

[0096] While weapons are described as accessories in the above embodiment, the types of accessories are not limited to weapons. For example, if the model is a specific humanoid character, the accessories may also be clothing or accessories worn by the character. Furthermore, the model can be made to appear in events, concerts, sports, online conferences, and the like taking place in a virtual space. Furthermore, the technology of this embodiment can also be applied to technologies that fuse the real world with the virtual world, such as Cross Reality (XR), to perceive images that do not exist in the real world.

[0097] Thus, in this embodiment, three-dimensional model data can be generated from images of the appearance of a model and its accessories, allowing the user to enjoy an image of the three-dimensional model arranged in a virtual space. Models and accessories, such as assembled plastic models, can be individually finished by the user through painting and other processes, and their individuality can be reflected in the dynamic image or character expression in the virtual space, significantly enhancing the enjoyment of the user experience.

[0098] <Summary of Implementation>

[0099] The above-described embodiments disclose at least the following information processing devices and computer programs.

[0100] (1) An information processing device comprising:

[0101] A scanning control mechanism controls the position and angle of the scanner to scan the appearance of the object placed on the setting table and generate a scanned image;

[0102] A data generating mechanism for generating three-dimensional model data based on the scanned image;

[0103] an image generating unit for generating an image in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; and

[0104] An accepting mechanism accepts input of attribute information related to the subject to be scanned and the performance of the subject; and

[0105] The image generation unit generates the image in a manner that includes a performance of each of the subjects corresponding to the attribute information.

[0106] (2) The information processing device according to (1), wherein the attribute information includes first attribute information indicating the type of the subject and second attribute information related to the performance in the image generation mechanism; and

[0107] The image generation unit generates the image in a manner that includes a performance of each of the subjects corresponding to the first attribute information and the second attribute information.

[0108] (3) The information processing device according to (2), wherein the second attribute information includes information indicating the type of the performance.

[0109] (4) The information processing device according to (3), wherein the type of performance includes a plurality of different types of performances, and the image generating means includes different performance images for each type.

[0110] (5) The information processing device according to any one of (2) to (4), wherein the subject includes a first subject and a second subject, and the second subject is an accessory of the first subject; and

[0111] The first attribute information indicates a type of the first subject;

[0112] The second attribute information includes the type of the second subject used in the performance.

[0113] (6) The information processing device according to (5), wherein the second attribute information indicates the presence of the second subject by indicating the type of the accessory; and

[0114] The second attribute information indicates that the second subject is absent by indicating that a part of the first subject is used for the performance.

[0115] (7) An information processing device according to (5) or (6), wherein the data generating mechanism performs processing to associate the second subject with a specified position of the first subject based on the first scanned image of the first subject and the second scanned image of the second subject, and generates the three-dimensional model data in a manner such that the first subject holds the second subject.

[0116] (8) The information processing device according to (7), wherein the second attribute information further indicates the presence or absence of the second subject;

[0117] If the second attribute information indicates that the second subject does not exist, not generating the scanned image of the second subject; and

[0118] The data generating unit generates the three-dimensional model data based on the first scanned image without performing a process of associating the second object with a predetermined position of the first object.

[0119] (9) The information processing device according to any one of (5) to (8), wherein when the second attribute information indicates the presence of the second subject, the scanned image of the second subject is generated.

[0120] (10) An information processing device according to any one of (5) to (9), wherein the scanning control mechanism causes a first scanning range when the second object is set on the setting table to be different from a second scanning range when the second object is not set on the setting table, and the first scanning range is larger than the second scanning range.

[0121] (11) The information processing device according to (5) or (10), further comprising: a detection mechanism for detecting whether the second subject is placed on the setting table; and

[0122] When the second subject is not detected, the acceptance unit does not accept the second attribute information indicating the type of the accessory.

[0123] (12) The information processing device according to any one of (5) to (11), wherein the first subject is a doll-type model, and the second subject is an accessory part of the model.

[0124] (13) The information processing device according to any one of (2) to (12), wherein the second attribute information further includes information on the color used for the performance.

[0125] (14) The information processing device according to any one of (1) to (13), further comprising: a notification unit for notifying the operating state of the information processing device; and

[0126] The notification unit notifies the information processing apparatus of at least one of a state of accepting input of the attribute information and a state of performing the scanning.

[0127] (15) A computer program for causing a computer to function as the information processing device according to any one of 1 to 14.

[0128] The present invention is not limited to the above-described embodiment, and various changes and modifications can be made within the scope of the gist of the invention.

Claims

1. An information processing device comprising: A scanning control mechanism controls the position and angle of the scanner to scan the appearance of the object placed on the setting table and generate a scanned image; A data generating mechanism for generating three-dimensional model data based on the scanned image; an image generating unit for generating an image in which a three-dimensional model based on the three-dimensional model data is arranged in a virtual space; and An accepting mechanism accepts input of attribute information related to the subject to be scanned and the performance of the subject; and The image generation unit generates the image in a manner that includes a performance of each of the subjects corresponding to the attribute information.

2. The information processing device according to claim 1 , wherein the attribute information includes first attribute information indicating the type of the subject and second attribute information related to a performance in the image generation mechanism; and The image generation unit generates the image in a manner that includes a performance of each of the subjects corresponding to the first attribute information and the second attribute information. 3 . The information processing device according to claim 2 , wherein the second attribute information includes information indicating a type of the performance.

4. The information processing device according to claim 3, wherein the types of performances include a plurality of different types of performances, and the image generating unit includes different performance images for each type.

5. The information processing device according to claim 4, wherein the subject comprises a first subject and a second subject, the second subject being an accessory of the first subject; and The first attribute information indicates a type of the first subject; The second attribute information includes the type of the second subject used in the performance.

6. The information processing device according to claim 5, wherein the second attribute information indicates the presence of the second subject by indicating the type of the accessory; and The second attribute information indicates that the second subject is absent by indicating that a part of the first subject is used for the performance.

7. The information processing device according to claim 6, wherein the data generating mechanism performs processing to associate the second subject with a specified position of the first subject based on the first scanned image of the first subject and the second scanned image of the second subject, and generates the three-dimensional model data in a manner in which the first subject holds the second subject.

8. The information processing device according to claim 7, wherein the second attribute information further indicates whether the second subject exists; If the second attribute information indicates that the second subject does not exist, not generating the scanned image of the second subject; and The data generating unit generates the three-dimensional model data based on the first scanned image without performing a process of associating the second object with a predetermined position of the first object. 9 . The information processing apparatus according to claim 8 , wherein when the second attribute information indicates the presence of the second subject, the scanned image of the second subject is generated.

10. The information processing device according to claim 9, wherein the scanning control mechanism makes a first scanning range when the second object is set on the setting table different from a second scanning range when the second object is not set on the setting table, and the first scanning range is larger than the second scanning range.

11. The information processing device according to claim 10, further comprising: a detection unit for detecting whether the second object is placed on the setting table; and When the second subject is not detected, the acceptance unit does not accept the second attribute information indicating the type of the accessory.

12. The information processing device according to claim 11, wherein the first subject is a doll-type model, and the second subject is an accessory part of the model. 13 . The information processing device according to claim 12 , wherein the second attribute information further includes information on a color used for the performance.

14. The information processing device according to claim 13, further comprising: a notification unit for notifying the operating status of the information processing device; and The notification unit notifies the information processing apparatus of at least one of a state of accepting input of the attribute information and a state of performing the scanning.

15. A computer program for causing a computer to function as the information processing apparatus according to any one of claims 1 to 14.

Images