Video generation device and computer program

Abstract

To generate a video in accordance with an operation received from a user in generating a video in which a three-dimensional model is arranged in a virtual space using three-dimensional model data of an imaged target.SOLUTION: In an image processing system 10, an information processing apparatus 100 functioning as a video generation device includes: generation means that generates a video in which a three-dimensional model is arranged in a virtual space, the three-dimensional model being based on three-dimensional model data generated from a scan image of an outer appearance of a subject scanned by a scanner; output means that outputs the video generated by the generation means to a display device; receiving means that receives an operation input from a first observer of the video displayed on the display device; and determination means that determines content of the operation input received by the receiving means. The generation means generates the video in accordance with a result determined by the determination means. When the determination means determines that the three-dimensional model is arranged at a predetermined position in the virtual space, the generation means includes an effect different in accordance with a determined arrangement position in the video.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image generating device and a computer program. [Background technology]

[0002] Patent document 1 describes a technology in which a texture generated from a group of captured images of an object taken from multiple angles by a photographing unit is mapped onto a primitive in a virtual space to generate a virtual space image as seen from a virtual camera in the virtual space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-107251 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the above technology, when an object is photographed and a video is generated in which a three-dimensional model is placed in a virtual space using three-dimensional model data of the photographed object, it is expected that the image will be generated in accordance with operations received from the user.

[0005] Therefore, the objective of the present invention is to enable the generation of an image according to operations received from a user when photographing an object and generating an image in which a three-dimensional model is placed in a virtual space using three-dimensional model data of the photographed object. [Means for solving the problem]

[0006] One aspect of the present invention to solve the above problem is an image generating device, a generating means for generating an image in which a three-dimensional model is arranged in a virtual space based on three-dimensional model data generated from a scanned image generated by scanning an image of an external appearance of a subject with a scanner; an output means for outputting the image generated by the generation means to a display device; a receiving means for receiving an operation input from a first observer of the image displayed on the display device; a determination means for determining the content of the operation input received by the reception means; Equipped with The generating means includes: generating the image according to the result of the determination by the determining means; When the determination means determines that the three-dimensional model has been placed at a predetermined position in the virtual space, a different effect corresponding to the determined placement position is included in the image. Effect of the Invention

[0007] When photographing an object and generating an image in which a three-dimensional model is placed in a virtual space using three-dimensional model data of the photographed object, it is possible to generate an image in accordance with operations received from the user. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the arrangement of an image processing system according to an embodiment. [Diagram 2] FIG. 1 is a diagram showing an example of the appearance of a doll-type model according to an embodiment. [Diagram 3] 1 is a diagram showing an example of a hardware configuration of an information processing device 100 and an example of a hardware configuration of a headset 150 according to an embodiment. [Figure 4] FIG. 1 is a diagram showing an example of implementation of an image processing system according to an embodiment. [Diagram 5] 4 is a flowchart corresponding to an example of processing in the image processing system 10 according to the embodiment. [Figure 6] 10 is a flowchart corresponding to an example of a video display process according to the embodiment. [Figure 7] 1A and 1B are diagrams showing an example of a menu screen corresponding to the embodiment, and a diagram showing an example of a node configuration of a virtual space corresponding to the embodiment. [Figure 8]10 is a flowchart showing an example of a video display process in response to a received operation according to the embodiment. [Figure 9A] FIG. 11 is a diagram showing an example of a screen display in a personal room according to an embodiment. [Figure 9B] FIG. 11 is a diagram showing an example of a screen display in a personal room according to an embodiment. [Figure 9C] FIG. 11 is a diagram showing an example of a screen display in a personal room according to an embodiment. [Figure 9D] FIG. 11 is a diagram showing an example of a screen display in a personal room according to an embodiment. [Figure 10A] FIG. 13 is a diagram showing an example of a screen display in a battle room according to an embodiment. [Figure 10B] FIG. 13 is a diagram showing an example of a screen display in a battle room according to an embodiment. [Figure 10C] FIG. 13 is a diagram showing an example of a screen display in a battle room according to an embodiment. [Figure 10D] FIG. 13 is a diagram showing an example of a screen display in a battle room according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more features among the multiple features described in the embodiments may be arbitrarily combined. Also, the same reference numbers are used for identical or similar configurations, and duplicated descriptions are omitted. Also, in each figure, the up, down, left, right, front and back directions relative to the paper surface are used in the description in the text as the up, down, left, right, front and back directions of the parts (or components) in this embodiment.

[0010] First, the configuration of an image processing system corresponding to this embodiment will be described. FIG. 1 is a diagram showing an example of the configuration of an image processing system 10 corresponding to this embodiment. The image processing system 10 is configured by connecting a scanner 110, a support arm 120, a model support device 130, a display device 140, a headset 150, and the like to an information processing device 100. Note that the system configuration is not limited to that shown in FIG. 1, and the information processing device 100 may be further connected to an external server, a cloud server, and the like via a network. The external server, etc., can execute at least a part of the processing of the embodiment described below.

[0011] The information processing device 100 controls the operation of at least one of the scanner 110, the support arm 120, and the model support device 130, photographs the object to be photographed from any angle to generate multiple images, and generates three-dimensional model data (main data) from the multiple images. If the object to be photographed is configured to be separable into multiple component objects, each component object may be photographed to generate three-dimensional model data, and the three-dimensional model data of the object may be generated by integrating the photographs.

[0012] The information processing device 100 can also function as an image generating device that generates images to be displayed in a virtual space by using the generated three-dimensional model data as virtual space data. In this embodiment, the subjects to be photographed are articles such as an assemblable plastic model, an action figure (a figure with movable joints), a toy, a doll, etc., which are collectively referred to as "models" below.

[0013] Next, the scanner 110 is a three-dimensional scanner device that captures (scans) the three-dimensional shape of a model to be captured under the control of the information processing device 100, and outputs the three-dimensional shape and color information of the captured object. In this embodiment, the scan signals output from the scanner 110 are collectively called "scanned images." For example, Space Spider manufactured by Artec can be used as the scanner 110. In this embodiment, for example, 500 to 800 frames of scanned images can be acquired to obtain 3D scan data of the entire toy body. In addition, for example, a smartphone with a camera on which an application for capturing three-dimensional shapes is installed may be used as the scanner 110.

[0014] The support arm 120 is a position and orientation control device that moves the scanner 110 to a predetermined photographing position and orientation under the control of the information processing device 100. The support arm 120 may be configured to be capable of manually changing the photographing position and orientation, and to be capable of fixing the changed position and orientation, or may be configured to be controllable by the information processing device 100. When the support arm 120 is configured to be controllable by the information processing device 100, for example, xArm 7 manufactured by UFACTORY can be used. The xArm 7 is made of seven joints and can move in the same manner as a human arm. Note that instead of using the support arm 120, the scanner 110 may be positioned manually. For photographing positions and orientations that cannot be covered by the support arm 120, the scanner 110 may be manually operated to perform scanning.

[0015] The model support device 130 is a support stand that supports a model with a fixed pose. The model support device 130 may be configured to be rotatable, for example, when the model is placed on the support stand or at the end of a support rod. In this embodiment, the scanner 110 is positioned at an arbitrary photographing position and photographing angle by the support arm 120, and then the model support device 130 is rotated once to photograph. By performing this at a plurality of photographing positions and photographing angles, an image of the entire model can be obtained. Here, the support arm 120 and the model support device 130 are driven synchronously to perform the photographing process more easily and with higher accuracy. Alternatively, instead of the model support device 130, the scanner 110 may be manually moved around the model to scan from an arbitrary photographing position and photographing angle.

[0016] The display device 140 is a display device such as a liquid crystal display (LCD), and can display the processing results of the information processing device 100. Specifically, it can display an image acquired by the scanner 110, display the main data of the three-dimensional model data generated from the captured image, and display an image (VR image) reconstructed using the main data. The display device 140 can include an operation unit 140A that accepts operations from a user who is an observer of the displayed image, and the user can operate the operation unit 140A to perform an operation input according to the content of the image displayed on the display device 140.

[0017] The headset 150 is composed of a head mounted display 150A and a controller 150B, which will be described later. In particular, the headset 150 may be configured as a VR headset capable of providing moving images corresponding to the posture and inclination of a user who is an observer. A specific application is installed in the headset 150, and application data executed in the application can be downloaded from the information processing device 100 and executed. In this embodiment, the headset 150, like the information processing device 100, can function as an image generating device that uses three-dimensional model data as virtual space data to generate an image to be displayed in a virtual space and provides the generated image as a VR image. The application data includes display data for displaying the VR image.

[0018] Next, the hardware configuration of the information processing device 100 corresponding to this embodiment will be described. FIG. 2(A) shows an example of the hardware configuration of the information processing device 100. The CPU 101 is a device that performs overall control of the information processing device 100 and performs calculation, processing, and management of data. For example, the CPU 101 controls the timing of image capture and the number of images captured by the scanner 110, and controls the joints of the support arm 120 to place the scanner 110 at an arbitrary image capture position and image capture angle. After the image capture position and image capture angle of the scanner 110 are determined, the model support device 130 is rotated to perform an image capture operation by the scanner 110. The CPU 101 can also function as an image processing unit that processes an image output from the scanner 110. The CPU 101 can perform a process of generating an image that displays a three-dimensional model in a virtual space by using three-dimensional model data generated based on a scan signal obtained by the scanner 110 as virtual space data. The display process will be described in detail later.

[0019] The RAM 102 is a volatile memory and is used as a temporary storage area such as the main memory and work area of ​​the CPU 101. The ROM 103 is a non-volatile memory, and image data and other data, various programs for the operation of the CPU 101, and the like are stored in respective predetermined areas. The CPU 101 controls each part of the information processing device 100 using the RAM 102 as a work memory according to, for example, a program stored in the ROM 103. Note that the program for the operation of the CPU 101 is not limited to being stored in the ROM 103, and may be stored in the storage device 104.

[0020] The storage device 104 is composed of a magnetic disk such as a HDD or a flash memory. Application programs, an OS, control programs, related programs, game programs, etc. are stored in the storage device 104. The storage device 104 can read and write data under the control of the CPU 101. The storage device 104 may be used in place of the RAM 102 or the ROM 103.

[0021] The communication device 105 is a communication interface for communicating with the scanner 110, the support arm 120, the model support device 130, the display device 140, and the headset 150 under the control of the CPU 101. The communication device 105 may further be configured to be able to communicate with an external server or the like. The communication device 105 may include a wireless communication module, which may include known circuitry including an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module card, a memory, and the like. Here, the communication between the information processing device 100 and the scanner 110, the support arm 120, the model support device 130, the display device 140, and the headset 150 may be performed by wireless communication.

[0022] The communication device 105 may also include a wired communication module for wired connection. The wired communication module enables communication with other devices including the display device 140 via one or more external ports. The communication device 105 may also include various software components for processing data. The external ports are connected to other devices directly via Ethernet, USB, IEEE1394, etc., or indirectly via a network. Note that software that realizes the same functions as the above devices may be used to replace the hardware devices.

[0023] The operation unit 106 is composed of, for example, a button, a keyboard, a touch panel, a controller, etc., and accepts operation input from a user. The operation unit 106 may be shared with the operation unit 140A, or may be independent of the operation unit 140A. For example, when the operation unit 140A is assumed to be an operation unit composed of a keyboard and a mouse, etc., it can be shared with the operation unit 106. On the other hand, when the operation unit 140A is assumed to be an operation unit composed of a touch panel, a controller, etc., it can be an operation unit separate from the operation unit 106.

[0024] The display control unit 107 functions as a control unit for displaying information on the display device 140 connected to the information processing device 100, and controls the operation of the display device 140. A part of the functions of the operation unit 106 may be provided in the display device 140. For example, the display device 140 may be configured as a device equipped with a touch panel such as a tablet terminal.

[0025] Next, a hardware configuration of the headset 150 corresponding to this embodiment will be described. FIG. 2B shows an example of the hardware configuration of the headset 150. The headset 150 is composed of a head mounted display (HMD) 150A and a controller 150B. The HMD 150A provides a right eye image and a left eye image to the left and right eyes of the user, respectively, and enables a virtual reality (VR) experience by creating a sense of stereoscopic effect due to the parallax between them. The controller 150B is provided in a housing independent of the HMD 150A. The controller 150B is assumed to be configured as a pair of two controllers to be held and operated by the user's left and right hands, respectively, but may be a single controller.

[0026] The CPU 151 is a device that performs overall control of the headset 150 and calculates, processes, and manages data. For example, it can execute application data downloaded from the information processing device 100 to display VR video on the display 157. In addition, based on an operation accepted via the controller 156 or information detected by the detection unit 158, it can switch the VR video itself to be displayed, switch the viewpoint within the VR space, or change the position within the VR space.

[0027] RAM 152 is a volatile memory and is used as a temporary storage area such as the main memory and work area of ​​CPU 151. ROM 153 is a non-volatile memory, and image data and other data, various programs for CPU 151 to operate, and the like are stored in respective predetermined areas. CPU 151 controls each part of headset 150 using RAM 152 as a work memory according to a program stored in ROM 153, for example. Note that the program for CPU 151 to operate is not limited to being stored in ROM 153, and may be stored in storage device 154.

[0028] The storage device 154 is composed of a magnetic disk such as a HDD or a flash memory. The storage device 154 stores application programs, an OS, control programs, related programs, game programs, application data downloaded from the information processing device 100, display data, and the like. The storage device 154 can read and write data based on the control of the CPU 151. The storage device 154 may be used instead of the RAM 152 or the ROM 153.

[0029] The communication device 155 is a communication interface for communicating with the information processing device 100 and the controller 150B under the control of the CPU 151. The communication device 155 includes a wireless communication module that enables wireless communication based on Bluetooth or WiFi (IEEE802.11). The headset 150 can connect to the information processing device 100 via wireless communication and download display data for VR images. In addition, the headset 150 communicates with the controller 150B and receives information on user operation instructions for the controller 150B.

[0030] The display 156 is configured to provide the right eye image and the left eye image generated by the CPU 151 to the right eye and the left eye of the user, respectively. The detection unit 157 is a mechanism for detecting the gaze direction of the left and right eyes of the user and the tilt of the head mounted display. The detection unit 157 includes, for example, a sensor for detecting the gaze direction or the gaze direction of the user wearing the HMD 150A. It also includes a gyroscope, a magnetometer, an accelerometer, a global positioning system (GPS), a compass, etc., and can specify the position, attitude, tilt, etc. of the HMD 150A according to the detection information. The detection unit 157 detects information for determining the gaze direction and action of the user in the HMD 150A and transmits it to the CPU 151. The CPU 151 determines the gaze direction and the action based on the received detection information, and adjusts the image presented on the display 156 of the HMD 150A so as to match the determined gaze direction and the action.

[0031] The controller 150B is configured with, for example, a plurality of buttons, a cross key, etc., and receives operation inputs from the user such as selection operations, direction instruction operations, etc. The controller 150B is connected to the HMD 150A via a communication device 155 via wireless communication.

[0032] Next, referring to FIG. 3, an example of a model that is a subject of the photographing target in this embodiment will be described. The model 300 is a model that has a doll-like (robot or human) appearance. The model can be, for example, a plastic model that has been assembled and painted. Alternatively, it may be a completed model such as a figure (action figure) with movable joints. The model in FIG. 2 is merely an example for the purpose of explanation, and the shape of the model is not limited to a doll-like appearance, but can be any shape such as a general vehicle, a racing vehicle, a military vehicle, an aircraft, a ship, an animal, a virtual life form, etc. It is to be noted that the object to be photographed is not limited to a model as long as the object can be photographed three-dimensionally by the scanner 110.

[0033] The model 300 has each of the parts of a head 301, a chest 302, a right arm 303, a left arm 304, a right torso 305, a left torso 306, a right leg 307, a left leg 308, a right foot 309, and a left foot 310, which are joined together to form the model. At least a part of each of the parts 301 to 310 is supported so as to be rotatable (or swingable) relative to the adjacent parts. For example, the head 301 is supported so as to be rotatable relative to the chest 302, and the right arm 303 and the left arm 304 are supported so as to be rotatable relative to the chest 302. In this way, each part of the model 300 is provided with a joint structure, so that the model 300 can take any posture.

[0034] Next, an implementation example of the image processing system 10 in this embodiment will be described with reference to Fig. 4. Fig. 4 shows an implementation example capable of generating this data by photographing a model.

[0035] 4, the case 401 contains the information processing device 100, a drive system for driving the support arm 402, a drive system for driving the turntable 406, and the like. The support arm 402 can also be manually adjusted in shooting direction and position. The surface of the case 401 has a flat structure so that a promotional poster can be attached.

[0036] The support arm 402 corresponds to the support arm 120, and can support and fix the position of a terminal 403 functioning as the scanner 110 under the control of the information processing device 100 or manually. The support arm 402 can also operate to control the inclination of the terminal 403.

[0037] The terminal 403 is a touch panel terminal with a built-in camera that can be used as the scanner 110, and can be, for example, a smartphone, a tablet terminal, or a digital camera. Instead of these terminals, Space Spider manufactured by Artec can be used. FIG. 4 is merely a diagram showing an example of the system configuration, and the configuration can be changed according to the type of device used as the scanner 110. The terminal 403 can take an image of the model 300 and transmit it to the information processing device 100. The ring light 404 is a lighting device used when the terminal 403 takes an image of the model 300, and can apply light evenly to the model 300 to make it difficult to cast a shadow. In addition to the ring light 404, a top light or auxiliary lights on the left, right, and bottom may be installed as an additional light source. Instead of the terminal 403, a three-dimensional scanner device as described above may be used.

[0038] The background sheet 405 is a background sheet for photography, and for example, a white sheet can be used. The turntable 406 can mount the model 300 and rotate the model. A plurality of predetermined markers 410 may be arranged on the turntable 406. The markers 410 can be used to adjust the orientation and position of the photographed model.

[0039] In FIG. 4, the model is placed on a semi-transparent (transparent) base, but other support devices such as an "Action Base" (registered trademark) may be used. The Action Base has a support pole configured to be bent into an "U" shape on a base, and the model can be attached to the tip of the pole. In this case, the above-mentioned marker 410 may be placed on the tip of the pole. The method of indicating the model can be changed depending on the pose. For example, in the case of an upright posture, the model can be placed on a transparent base and photographed. On the other hand, in the case of a flying posture, when it is desired to photograph the soles of the feet, an action base may be used. The action base may be used to photograph an upright posture.

[0040] The display device 407 is a device corresponding to the display device 140, and may have a touch panel function. The user can use the touch panel function to perform a predetermined selection operation. Also, operations can be accepted by a controller or mouse (not shown) that can function as the operation unit 140A instead of the touch panel. The VR headset 408 is composed of an HMD 408A and a controller 408B, and corresponds to the HMDs 150A and 150B, respectively. The user can perform operations while watching VR images by wearing the HMD 408A on the head and holding and operating the controller 408B in each hand.

[0041] Next, an example of processing executed by the image processing system 10 according to the present embodiment will be described with reference to Fig. 5. At least a part of the processing corresponding to the flowchart is realized by the CPU 101 of the information processing device 100 executing a program stored in the ROM 103 or the storage device 104, or by the CPU 151 of the headset 150 executing a program stored in the ROM 153 or the storage device 154.

[0042] First, in S501, the CPU 101 accepts user registration. The CPU 101 accepts input of the user's name and contact information. A user identifier is assigned to each user to uniquely identify each individual user. The CPU 101 stores the input user information in the storage device 104 in association with the time when the input was accepted and the user identifier.

[0043] When user registration is complete, the user sets his / her own model on the model support device 130. The CPU 101 can determine whether or not the model has been set on the model support device 130 based on the scanned image by the scanner 110. Alternatively, a switch that is turned on when the model is set on the model support device 130 may be provided, and the CPU 101 can detect a signal from the switch to make the determination. Alternatively, a button for accepting an operation when the setting of the model is completed may be displayed on the display device 140, and the CPU 101 can detect whether an operation in response to the button operation has been accepted. In S502, the CPU 101 detects that the model has been set on the model support device 130 by any of the methods described above. In response to this detection, the process proceeds to S503.

[0044] In S503, the scanner 110 executes a scan process (photographing process) to generate a scan image. Specifically, the CPU 101 controls the support arm 120 to move the scanner 110 to one of the registered photographing positions, and the scanner 110 performs scanning at that photographing position while rotating the model support device 130. The photographed scan image is transmitted to the information processing device 100, and the CPU 101 stores it in a table in the storage device 104 in association with a user identifier, etc.

[0045] In the next step S504, the CPU 101 executes post-scan processing on the scanned image acquired by the scan processing, generates three-dimensional model data, and generates application data (display data) for video display. The application data can play video when executed in a specific application on the information processing device 100. Also, the application data can play VR video when executed in the specific application on the head mounted display 150A. The generated three-dimensional model data and display data are stored in the storage device 104 in association with user information.

[0046] In the next step S505, image display processing is executed. When the image display processing is executed in the information processing device 100, the CPU 101 executes a specific application corresponding to the display data, and displays the image on the display device 140 via the display control unit 107. In addition, in step S505, the CPU 101 may control the communication device 105 to transmit the display data to the HMD 150A, and the CPU 151 of the HMD 150A may execute the received display data in the corresponding application to execute processing for displaying the VR image, thereby executing the image display processing.

[0047] Next, the display processing of the display data corresponding to S506 will be described with reference to Fig. 6. The processing of Fig. 6 is executed by the CPU 101 of the information processing device 100 using the display data, but may be executed by the CPU 151 of the HMD 150A.

[0048] An application for displaying VR images corresponding to the VR image development platform used to generate the VR data is installed in the information processing device 100, and the CPU 101 executes the application to display an image on the display device 140 using the display data. A similar application for displaying VR images is also installed in the HMD 150A, and the CPU 151 executes the application to display a VR image on the display 156 using the display data. The following mainly describes the processing in the information processing device 100, but unless otherwise specified, the same processing is also executed in the HMD 150A. Therefore, the processing performed by the CPU 101 can be interpreted as processing performed by the CPU 151.

[0049] First, in S601, the CPU 101 causes the display device 140 to display a menu screen via the display control unit 107. On the menu screen, the user can make a selection input via the operation unit 140A. Here, on the menu screen, a plurality of options are displayed. An example of the menu screen is shown in FIG. 7(A).

[0050] FIG. 7(A) is a diagram showing an example of a menu screen, and shows two options for the user's destination. One is "My Room", which is a space given to each user individually, where the user can store data of models that the user scanned and imported. The other is a battle room, where the user can play a battle game using three-dimensional model data of a model that the user scanned and imported. In the battle game, the user can play against other users or the CPU. In addition, any game other than the battle game can be played.

[0051] On the screen of FIG. 7(A), the user can operate the operation unit 140A to, for example, move the cursor displayed on the screen onto either "My Room" or "Battle Room" and perform a selection operation (for example, a click operation or a button operation) there to make a selection input. When the HMD 150A is used, the user can move the controller 150B to a position where it overlaps with "My Room" or "Battle Room" and perform a button operation on the controller 150B there to make a selection input. The correspondence between the operation unit 140A and the controller 150B is the same in the following explanation.

[0052] In the virtual space corresponding to this embodiment, as shown in FIG. 7(B), there are four main nodes, and players can travel between them. In this embodiment, the nodes are my room (first space), hangar (second space), friend room (third space), and battle room (fourth space). My room and hangar are connected, and players can move to the hangar from my room. Players can move to the friend room via the hangar. Players can move from my room to the battle room, but they can also move to the friend room and then move to the battle room. The nodes that make up the virtual space are not limited to those shown in FIG. 7(B), and players may be able to move between more nodes via any route.

[0053] Returning to the explanation of FIG. 6, when CPU 101 accepts a selection operation from the user via operation unit 140A in the following S602, in S603, CPU 101 causes display device 140 to display an image corresponding to the selected destination, either my room or the battle room. In the following S604, CPU 101 acquires operation information of the user on operation unit 140A. In the following S605, CPU 101 displays an image according to the operation information acquired in S604. The processing in S605 includes display of my room or the battle room, display according to the user operation accepted in the room, display of movement within the room, execution of the battle game, display of the hangar, etc.

[0054] More specifically, the details of the process in S605 will be described with reference to Fig. 8 to Fig. 10D. Fig. 8 is a flow chart showing an example of the details of the process in S605 in Fig. 6. Figs. 9A to 9D are diagrams showing an example of the display when "My Room" is selected in S602. Figs. 10A to 10D are diagrams showing an example of the display when "Battle Room" is selected in S602.

[0055] 8, CPU 101 receives an operation from a user via operation unit 140A. The operation includes a movement operation for changing the user's viewpoint position (including an operation for instructing a direction of travel and an operation of a cross key of a controller), an operation for selecting a displayed object, an operation for moving a selected displayed object, an operation for canceling the selection state of a selected object, etc. Note that the reception of the operation in S801 also includes acquisition by CPU 151 of detection information from detection unit 157 of HMD 150A.

[0056] For example, in the example of the screen display in my room shown in FIG. 9A, a message "Let's set the model in the hangar" is displayed in an area 901 of the display screen 900. In addition, a three-dimensional model 902 that is a selection candidate is displayed on the lower side of the display screen 900. The three-dimensional model 902 corresponds to the model captured by the scanning process of S503 described above. The user can select and hold the three-dimensional model 902 by operating the operation unit 140A according to the message "Hold the model with button X". Note that the size of the three-dimensional model 902 in this holding state is displayed so that it is the same as or close to the size in the real world of the actual model that was scanned and captured to become the three-dimensional model 902, or at least a size that does not feel strange when held in the hand. The user can move the three-dimensional model 902 that he / she is holding within his / her room and place it in a specified installation location, thereby appreciating the model in a state stored in the hangar. The size of the three-dimensional model 902 in the selection candidate state before being grasped may be made different from the size of the three-dimensional model 902 in the grasped state. Specifically, the three-dimensional model 902 in the selection candidate state may be displayed larger than the three-dimensional model 902 in the grasped state. In this case, the three-dimensional model 902 in the selection candidate state can be easily viewed in the user's room.

[0057] Also, for example, in the example of the screen display in the battle room shown in FIG. 10A, a message "Set the model on the battle table" is displayed in an area 1001 of the display screen 1000. Also, a three-dimensional model 1002 that is a selection candidate is displayed on the lower side of the display screen 1000. The three-dimensional model 1002 corresponds to the model captured by the scanning process of S503 described above. That is, the user can play a battle game using his / her own model. The user can select the three-dimensional model 1002 by operating the operation unit 140A according to the message "Select button X to hold the model, and release the button to set it", and move it onto the battle table 1003 to set it. The three-dimensional model 1002' corresponds to the three-dimensional model 1002 set on the battle table 1003.

[0058] Returning to the description of FIG. 8, in the following S802, CPU 101 judges whether the operation accepted in S801 is an operation to select a three-dimensional model displayed on the screen. The judgment of the operation can be performed based on whether a button operation corresponding to a selection button is accepted in a state in which the display of a cursor or a controller corresponding to controller 150B displayed on the screen is in a position superimposed on the three-dimensional model. When multiple three-dimensional models are displayed on the screen, it is judged that the three-dimensional model on which the cursor or controller is superimposed is selected. If the operation accepted in S801 is an operation to select a three-dimensional model, the process proceeds to S806. If the operation is not an operation to select a three-dimensional model, the process proceeds to S803.

[0059] In S806, CPU 101 performs display processing corresponding to the operation received in S801. For example, when a selection operation of a three-dimensional model is received, a state in which the three-dimensional model is being held is displayed. This allows the user to move the selected three-dimensional model to an arbitrary position by further inputting an operation. In addition, in HMD 150A, CPU 151 determines the gaze direction and action based on the detection information received from detection unit 157, and adjusts the image presented on display 156 of HMD 150A so as to match the determined gaze direction and action. Thereafter, the process returns to S801, and the above-mentioned process is repeated.

[0060] In S803, the CPU 101 determines whether or not the selected model has been placed in a predetermined location by the accepted operation. If the accepted operation is not the placement of the selected model in a predetermined location, the process proceeds to S806, where a display corresponding to the accepted operation is performed, and the process returns to S801.

[0061] FIG. 9B shows an example of a screen display when a three-dimensional model is placed in a predetermined location in the My Room display. In FIG. 9B, an indication of "Set Hangar Model" is displayed in an area 911 of a display screen 910 together with a message of "Release Button X". The user can follow this instruction to place the three-dimensional model 902 selected above on an installation stand 912 as shown in FIG. 9B. A door 913 leading to the Hangar is displayed on the back side of the area 911. Here, the My Room and the Hangar are shown to be installed adjacent to each other, but they may be connected via a passage. In the example shown in FIG. 9B, the CPU 101 can determine whether the three-dimensional model 902 has been installed in a predetermined location based on whether the button operation has been released when the three-dimensional model 902 is within the area of ​​the installation stand 912.

[0062] Similarly, in the example shown in FIG. 10A, CPU 101 can determine whether or not three-dimensional model 1002 has been placed in a specified location based on whether or not a button operation has been released while three-dimensional model 1002 is within the area of ​​battle table 1003.

[0063] 8, if it is determined in S803 that the operation received is to place the selected model in a predetermined location, the process proceeds to S804, where the CPU 101 performs a presentation display according to the location. Thereafter, it is determined whether or not to end this process, and if not, the process returns to S801 and the above-mentioned process is repeated, and if it is to end, the process ends.

[0064] Specifically, in the My Room display, when three-dimensional model 902 is placed on installation stand 912 as shown in Fig. 9C, hangar door 913 opens and the interior of the hangar is displayed. A three-dimensional model 921, which is an enlarged version of three-dimensional model 902 placed on installation stand 912, is displayed inside the hangar. Also, a passage 922 is displayed in front of three-dimensional model 921, and a sign 923 is displayed at the end of passage 922.

[0065] The size of the three-dimensional model 921 may be enlarged to a size that occupies a predetermined area within the space in the hangar in which the three-dimensional model 921 is placed. Also, the three-dimensional model 921 may be enlarged at a magnification corresponding to the actual size of the model that is the basis of the three-dimensional model 902. Alternatively, the magnification may be corresponding to the scale of the model (for example, 1 / 144, 1 / 100, 1 / 60, etc.).

[0066] In the display state of FIG. 9C, when an operation is further received and the received operation is a movement operation for changing the user's viewpoint position and an operation for proceeding in the direction of the passage 922, the display screen transitions to the content shown in FIG. 9D. Here, the three-dimensional model 921 is displayed in a larger scale in the display screen 930, and the sign 923 is also displayed in a similarly larger scale. A message 931 saying "To the friend room" is superimposed on the sign 923. Also, a message saying "Please follow the arrow" is displayed on the passage 922. This allows the user to perform a proceeding operation to proceed in the direction of the arrow, and when the user moves to a position within a certain distance of the sign 923, the user can move to the friend room. At this time, an image of the user boarding the cockpit of the three-dimensional model 921 may be displayed on the display screen as a dramatic display.

[0067] In this embodiment, if there are other users registered in this system, it may be possible to select one of them. Also, the user may pre-register a specific user from among the other users, and may be able to select one of the registered specific users. Since a friend room is assigned to each user, the above procedure allows the user to move to the friend room for each user and view the friend room.

[0068] In addition, in the battle room display, when the three-dimensional model 1002 is placed on the battle table 1003, the display content of the display screen 1010 is switched to that shown in Fig. 10B. The display screen 1010 includes display areas 1011, 1012, and 1013, and images of the battle game being executed are displayed in the display areas 1011 to 1013. In Fig. 10B, the battle game has not yet started, and the game is waiting for an instruction to start the game. An operation system 1014 is displayed below the display area 1011. An instruction receiving section 1015 is displayed in the operation system 1014.

[0069] When a cursor is moved to instruction receiving unit 115 and a click operation is performed, or when a controller is placed over 1015, it is determined that "game start" has been selected, and the display screen switches to the content shown in Fig. 10C. On display screen 1020 in Fig. 10C, in the display state of Fig. 10B, an image of a three-dimensional model 1021 is displayed in display area 1011. The three-dimensional model 1021 corresponds to three-dimensional model 1002 shown in Fig. 10A. After that, the display content of the display screen switches to that shown in Fig. 10D.

[0070] 10D shows an example of a screen displayed when a battle game is started. Battle game screens 1031, 1032, and 1033 are displayed in display areas 1011, 1012, and 1013, respectively. A three-dimensional model and the remaining time of the game are displayed on screen 1031. The user can proceed through the game by operating operation unit 140A or a controller.

[0071] As described above, in this embodiment, the information processing device 100 and the HMD 150A perform image display processing based on an operation received from a user. In the above embodiment, the display data for displaying an image by arranging three-dimensional model data generated from an image obtained by photographing a model in a virtual space is generated. In this case, in this embodiment, different performance images can be provided depending on the position where the user installs the three-dimensional model. In the above embodiment, the performance image shows a state in which the user's model is stored in a hangar at full size and a case in which a fighting game is performed using the model, but the performance method is not limited to these. For example, the model can be made to appear in an event, concert, sports, online conference, etc. held in a virtual space. Furthermore, the technology of this embodiment can be applied to a video technology such as cross reality (XR) that fuses the real world with the virtual world and makes it possible to perceive things that do not exist in the real space.

[0072] In this way, in this embodiment, three-dimensional model data is generated from images of the exterior of the model, and it becomes possible to view a video in which the three-dimensional model is placed in a virtual space. Some models, such as prefabricated plastic models, are finished as unique works by users through painting or other processes, and these individual characteristics can be reflected in the character representation in the video or virtual space, which can greatly improve the user's preference.

[0073] <Summary of the embodiment> The above embodiment discloses at least the following image generating device and computer program. (1) An image generating device, comprising: a generating means for generating an image in which a three-dimensional model is arranged in a virtual space based on three-dimensional model data generated from a scanned image generated by scanning an image of an external appearance of a subject with a scanner; an output means for outputting the image generated by the generation means to a display device; a receiving means for receiving an operation input from a first observer of the image displayed on the display device; a determination means for determining the content of the operation input received by the reception means; Equipped with The generating means includes: generating the image according to the result of the determination by the determining means; When the determination means determines that the three-dimensional model has been placed at a predetermined position within the virtual space, the image generation device includes in the image a different effect corresponding to the determined placement position. (2) the predetermined position includes a first position in a first space of the virtual space; The image generating device described in (1), wherein, in response to the three-dimensional model being placed at the first position, the generating means generates, as the performance, an image in which the three-dimensional model is enlarged and displayed in a second space connected to the first space. (3) the receiving means receives an operation input from the first observer to move a viewpoint position in the virtual space; The image generating device described in (2), wherein when the determination means determines that the first observer has moved near the enlarged and displayed three-dimensional model based on the operational input from the first observer, the generation means generates, as the performance, an image of movement from the second space to a third space different from the first space and the second space. (4) The image generating device described in (3), wherein the image of the movement into the third space includes an image of the first observer riding on the enlarged and displayed three-dimensional model. (5) the receiving means receives an operation input from the first observer to move a viewpoint position in the virtual space; The image generating device described in (2), wherein when the determination means determines that the viewpoint position has moved to the vicinity of the enlarged and displayed three-dimensional model based on the operational input from the first observer, the generation means generates, as the performance, an image of the first observer riding on the enlarged and displayed three-dimensional model. (6) the first space is associated with the first observer; The image generating device according to (3) or (4), wherein the third space is associated with a second observer different from the first observer. (7) The image generating device described in (6), wherein when it is determined that the object has moved to the vicinity of the enlarged and displayed three-dimensional model, the determination means further determines the second observer based on an operation of the first observer to select one of a plurality of observers. (8) The predetermined position includes a third position in a fourth space connected to the third space, The image generation device according to (6) or (7), wherein in response to the three-dimensional model being placed at the third position, the generation means generates an image of a game using the three-dimensional model data. (9) The image generation device according to (8), wherein the generation means generates an image of a start screen of the game in response to the three-dimensional model being placed at the third position. (10) The image generating device according to (8) or (9), wherein the game is a competitive game. (11) The image generating device according to (8) or (9), wherein the game is a competitive game with the second observer. (12) The predetermined position includes a third position in a fourth space of the virtual space, The image generating device according to (1), wherein in response to being placed at the third position, the generating means generates an image of a game using the three-dimensional model data. (13) The image generating device is a head mounted display, the display device is a display included in the head mounted display, The image generating device according to any one of (1) to (12), wherein the accepting means accepts the operation input via a controller associated with the head mounted display. (14) The display device is an external display device connected to the image generating device, The image generating device according to any one of (1) to (12), wherein the accepting means accepts the operation input via an operation unit for the image generating device. (15) A computer program for causing a computer to function as the image generating device according to any one of (1) to (14).

[0074] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

Claims

1. 1. An image generating device, comprising: a generating means for generating an image in which a three-dimensional model is arranged in a virtual space based on three-dimensional model data generated from a scanned image generated by scanning an image of an external appearance of a subject with a scanner; an output means for outputting the image generated by the generation means to a display device; a receiving means for receiving an operation input from a first observer of the image displayed on the display device; a determination means for determining the content of the operation input received by the reception means; Equipped with The generating means includes: generating the image according to the result of the determination by the determining means; When the determination means determines that the three-dimensional model has been placed at a predetermined position within the virtual space, the image generation device includes in the image a different effect corresponding to the determined placement position.

2. the predetermined position includes a first position in a first space of the virtual space, The image generating device according to claim 1, wherein in response to the three-dimensional model being placed at the first position, the generating means generates, as the performance, an image in which the three-dimensional model is enlarged and displayed in a second space connected to the first space.

3. the receiving means receives an operation input from the first observer to move a viewpoint position in the virtual space; The image generating device of claim 2, wherein when the determination means determines that the first observer has moved near the enlarged and displayed three-dimensional model based on the operational input from the first observer, the generation means generates, as the performance, an image of movement from the second space to a third space different from the first space and the second space.

4. The image generating device according to claim 3 , wherein the image of the movement into the third space includes an image of the first observer riding on the enlarged and displayed three-dimensional model.

5. the receiving means receives an operation input from the first observer to move a viewpoint position in the virtual space; The image generating device of claim 2, wherein when the determination means determines that the viewpoint position has moved to the vicinity of the enlarged and displayed three-dimensional model based on the operational input from the first observer, the generation means generates, as the performance, an image of the first observer riding on the enlarged and displayed three-dimensional model.

6. the first space is associated with the first observer; The image generating apparatus of claim 3 , wherein the third space is associated with a second observer different from the first observer.

7. 7. The image generating device according to claim 6, wherein when it is determined that the object has moved to the vicinity of the enlarged and displayed three-dimensional model, the determination means further determines the second observer based on an operation of the first observer to select one of a plurality of observers.

8. the predetermined position includes a third position in a fourth space connected to the third space, 7. The image generating device according to claim 6, wherein said generating means generates an image of a game using said three-dimensional model data in response to said three-dimensional model being placed at said third position.

9. 9. The image generating device according to claim 8, wherein said generating means generates an image of a start screen of said game in response to said three-dimensional model being placed at said third position.

10. The image generating device according to claim 8 , wherein the game is a competitive game.

11. The image generating device according to claim 8 , wherein the game is a competitive game with the second observer.

12. the predetermined position includes a third position in a fourth space of the virtual space, 2. The image generating device according to claim 1, wherein said generating means generates an image of a game using said three-dimensional model data in response to said placement at said third position.

13. the image generating device is a head mounted display, the display device is a display included in the head mounted display, The image generating device according to claim 1 , wherein the accepting unit accepts the operation input via a controller associated with the head mounted display.

14. the display device is an external display connected to the image generating device, The image generating device according to claim 1 , wherein the accepting means accepts the operation input via an operation unit for the image generating device.

15. A computer program for causing a computer to function as the image generating device according to any one of claims 1 to 14.

Images