Image generation device, computer program product, and computer-readable storage medium

By generating three-dimensional model images corresponding to user operations in the virtual space, the problem that image generation in the virtual space in the prior art is solved, and a more interactive and interesting virtual space experience is achieved.

CN119094660APending Publication Date: 2024-12-06BANDAI CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410995683.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-08-25
Filing Date
2024-07-24
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The prior art is difficult to realize image generation corresponding to user operations when generating three-dimensional model images in the virtual space.

Method used

An image generation device is designed, including a generation unit, an output unit, an acceptance unit and a determination unit. It generates three-dimensional model data through a scanner, and generates corresponding images based on user operation input, determines the position and configuration of the three-dimensional model, and generates and The performance image in the corresponding position.

Benefits of technology

It realizes the generation of images corresponding to user operations in the virtual space, enhancing user experience and interest, especially in model display and battle games in the virtual space.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119094660A_ABST
    Figure CN119094660A_ABST
Patent Text Reader

Abstract

The invention provides an image generation device, a computer program product, and a computer-readable storage medium. When generating an image in which a three-dimensional model is disposed in a virtual space using three-dimensional model data of an object obtained by imaging, it is possible to generate an image corresponding to an operation received from a user. An image generation device is provided with: a generation means for generating an image in which a three-dimensional model based on three-dimensional model data is arranged in a virtual space; an output unit that outputs the image generated by the generation unit to a display device; a receiving unit that receives an operation input from a first observer on the video displayed on the display device; and a determination unit that determines the content of the operation input received by the reception unit, in which the generation unit generates an image corresponding to the result of the determination by the determination unit, and when the determination unit determines that the three-dimensional model is disposed in the predetermined position in the virtual space, the generation unit generates the image corresponding to the result of the determination by the determination unit. The generation unit causes the image to include a different performance corresponding to the determined arrangement position.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an image generating device and a computer program. Background Art

[0002] Patent Document 1 describes a technique for mapping a texture generated from a group of photographic images obtained by photographing an object from a plurality of photographing directions by a photographing unit onto primitives in a virtual space to generate a virtual space image viewed from a virtual camera in the virtual space.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2020-107251 Summary of the invention

[0006] Problem that the invention aims to solve

[0007] In the above-mentioned technology, when an object is photographed and a video in which the three-dimensional model is arranged in a virtual space is generated using the three-dimensional model data of the photographed object, it is expected that an video corresponding to the operation received from the user is generated.

[0008] Therefore, an object is to capture an object and generate an image in which the three-dimensional model is arranged in a virtual space using the three-dimensional model data of the captured object, thereby enabling generation of an image corresponding to an operation received from a user.

[0009] Solutions for solving problems

[0010] One method for solving the above problem is an image generating device comprising:

[0011] a generating unit for generating an image in which a three-dimensional model based on three-dimensional model data is arranged in a virtual space, the three-dimensional model data being data generated based on a scanned image generated by scanning the appearance of a subject with a scanner;

[0012] an output unit configured to output the image generated by the generating unit to a display device;

[0013] a receiving unit that receives an operation input from a first observer on an image displayed on the display device; and

[0014] a determination unit configured to determine the content of the operation input received by the receiving unit,

[0015] The generating unit generates the image corresponding to the result of the determination in the determining unit.

[0016] When the determination unit determines that the three-dimensional model is arranged at a predetermined position in the virtual space, the generation unit causes the video to include a different performance corresponding to the determined arrangement position.

[0017] Effects of the Invention

[0018] When an object is photographed and an image in which the three-dimensional model is arranged in a virtual space is generated using three-dimensional model data of the photographed object, an image corresponding to an operation received from a user can be generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a diagram showing a configuration example of an image processing system according to an embodiment.

[0020] Figure 2 1 is a diagram showing an example of the hardware configuration of the information processing device 100 and an example of the hardware configuration of the head unit 150 according to the embodiment.

[0021] Figure 3 This is a diagram showing an example of the appearance of a doll-type model object according to the embodiment.

[0022] Figure 4 This is a diagram showing an example of implementation of an image processing system according to the embodiment.

[0023] Figure 5 This is a flowchart corresponding to an example of processing in the image processing system 10 according to the embodiment.

[0024] Figure 6 This is a flowchart corresponding to an example of image display processing according to the embodiment.

[0025] Figure 7 A diagram showing an example of a menu screen corresponding to the embodiment and a diagram showing an example of a node structure of a virtual space corresponding to the embodiment.

[0026] Figure 8 This is a flowchart showing an example of image display processing according to an accepted operation according to the embodiment.

[0027] Fig.9A It is a diagram showing an example of screen display in my room according to the embodiment.

[0028] Fig. 9B It is a diagram showing an example of screen display in my room according to the embodiment.

[0029] Fig. 9C It is a diagram showing an example of screen display in my room according to the embodiment.

[0030] Fig.9D It is a diagram showing an example of screen display in my room according to the embodiment.

[0031] Fig. 10A This is a diagram showing an example of screen display in a battle room according to the embodiment.

[0032] Fig. 10B This is a diagram showing an example of screen display in a battle room according to the embodiment.

[0033] Fig. 10C This is a diagram showing an example of screen display in a battle room according to the embodiment.

[0034] Fig. 10D This is a diagram showing an example of screen display in a battle room according to the embodiment. DETAILED DESCRIPTION

[0035] Below, the embodiments are described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention involved in the claims, and the combination of features described in the embodiments is not limited to all being necessary for the invention. Two or more of the multiple features described in the embodiments may also be arbitrarily combined. In addition, the same figure marks are marked on the same or similar structures, and repeated descriptions are omitted. In addition, in each figure, the up-down, down-down, left-right, front-back directions relative to the paper are set as the up-down, down-down, left-right, front-back directions of the components (or accessories) in this embodiment, and are used in the description of this article.

[0036] First, the configuration of an image processing system according to the present embodiment will be described. Figure 1 1 is a diagram showing an example of the structure of an image processing system 10 corresponding to the present embodiment. The image processing system 10 is configured by connecting a scanner 110, a support arm 120, a model object support device 130, a display device 140, a head unit 150, etc. to an information processing device 100. In addition, the system structure is not limited to Figure 1 The illustrated configuration may be such that the information processing device 100 is further connected to an external server, a cloud server, etc. via a network. In the external server, etc., at least a part of the processing in the embodiments described below can be executed.

[0037] The information processing device 100 controls the operation of at least one of the scanner 110, the support arm 120, and the model object support device 130, photographs the object to be photographed from any angle to generate multiple images, and generates three-dimensional model data (formal data) based on the multiple images. In the case where the object to be photographed is a structure that can be separated into multiple constituent objects, each constituent object can be photographed to generate three-dimensional model data, and the three-dimensional model data of the target object can be generated by integrating these three-dimensional model data.

[0038] The information processing device 100 can also function as an image generation device that generates images displayed in a virtual space by using the generated three-dimensional model data as virtual space data. In this embodiment, the subject to be photographed is an assembleable plastic model, an action figure (an action figure with movable joints), a toy, a doll, or other items, which are collectively referred to as "model objects" below.

[0039] Next, the scanner 110 is a three-dimensional scanner device that photographs (scans) the three-dimensional shape of the model object as the photographing object and outputs the three-dimensional shape and color information of the photographing object according to the control of the information processing device 100. In the present embodiment, the scanning signals output from the scanner 110 are collectively referred to as "scanned images". As the scanner 110, for example, the Space Spider manufactured by Artec (a registered trademark) can be used. In the present embodiment, for example, by acquiring the scanned image in a manner of about 500 to 800 frames, 3D scanning data of the entire toy can be obtained. In addition, as the scanner 110, for example, a smart phone with a camera installed with an application for photographing three-dimensional shapes can also be used.

[0040] The support arm 120 is a position and posture control device that moves the scanner 110 to a specified shooting position and posture according to the control of the information processing device 100. The support arm 120 can be configured to be able to change the shooting position and posture manually and to be able to fixedly maintain the changed position and posture, or it can be configured to be controlled by the information processing device 100. In the case where the support arm 120 is configured to be able to be controlled by the information processing device 100, for example, xArm 7 manufactured by UFACTORY (a registered trademark) can be used. xArm 7 is composed of seven joints and can perform the same movements as human arms. In addition, the scanner 110 can be positioned manually instead of the support arm 120. In addition, for shooting positions and postures that cannot be covered by the support arm 120, the scanner 110 can be manually operated to perform scanning.

[0041] The model object support device 130 is a support platform for supporting a model object with a fixed posture. The model object support device 130 can also be configured to allow the model object to rotate while being placed on the support platform or at the front end of a support rod. In the present embodiment, after the scanner 110 is positioned at an arbitrary shooting position and shooting angle by the support arm 120, the model object support device 130 is rotated once and shooting is performed. By performing this shooting at multiple shooting positions and shooting angles, an image of the entire model object can be acquired. In addition, here, by synchronously driving the support arm 120 and the model object support device 130, photographic processing can be performed more simply and with high precision. In addition, the scanner 110 can be manually moved around the model object instead of the model object support device 130 to perform scanning from an arbitrary shooting position and shooting angle.

[0042] The display device 140 is a display device such as a liquid crystal display (LCD), and can display the processing results in the information processing device 100. Specifically, it can display the image acquired by the scanner 110, or display the formal data of the three-dimensional model data generated based on the photographic image, or display the image (VR image) reconstructed using the formal data. The display device 140 can have an operation unit 140A for accepting operations from a user who is an observer of the displayed image, and the user can operate the operation unit 140A to perform operation input corresponding to the content of the image displayed on the display device 140.

[0043] The head assembly 150 is composed of a head-mounted display 150A and a controller 150B described later. In particular, it can also be configured to be able to provide dynamic images corresponding to the posture and inclination of the user as an observer as a VR head assembly. A prescribed application is installed in the head assembly 150, and application data executed in the application can be downloaded from the information processing device 100 and executed. In the present embodiment, the head assembly 150 can function as an image generating device in the same manner as the information processing device 100, and the image generating device uses three-dimensional model data as virtual space data, generates an image displayed in the virtual space, and provides the generated image as a VR image. In addition, the application data includes display data for displaying VR images.

[0044] Next, the hardware configuration of the information processing device 100 according to the present embodiment will be described. Figure 2(A) shows an example of the hardware structure of the information processing device 100. The CPU 101 is a device that performs overall control of the information processing device 100, data calculation, processing, and management. For example, it is possible to control the shooting timing and number of images in the scanner 110, and it is possible to control the joints of the arms of the support arm 120 to configure the scanner 110 to any shooting position and shooting angle. In addition, after determining the shooting position and shooting angle of the scanner 110, the model object support device 130 can be rotated and the shooting action can be performed by the scanner 110. In addition, the CPU 101 can also function as an image processing unit that processes the image output from the scanner 110. The CPU 101 can perform the following processing: using the three-dimensional model data generated based on the scanning signal obtained by the scanner 110 as virtual space data to generate an image that displays the three-dimensional model in the virtual space. The details of this display processing are described in detail in the following section.

[0045] The RAM 102 is a volatile memory used as a temporary storage area such as a main memory and a work area of ​​the CPU 101. The ROM 103 is a nonvolatile memory and stores image data, other data, various programs used by the CPU 101 to perform operations, etc. in respective predetermined areas. The CPU 101 uses the RAM 102 as a work memory, for example, to control various parts of the information processing device 100 according to the programs stored in the ROM 103. In addition, the programs used by the CPU 101 to perform operations are not limited to those stored in the ROM 103, but may also be stored in the storage device 104.

[0046] The storage device 104 is composed of a magnetic disk such as an HDD or a flash memory. The storage device 104 stores application programs, OS, control programs, related programs, game programs, etc. The storage device 104 can read or write data based on the control of the CPU 101. The storage device 104 can also be used as a substitute for the RAM 102 and the ROM 103.

[0047] The communication device 105 is a communication interface for communicating with the scanner 110, the support arm 120, the model object support device 130, the display device 140, and the head assembly 150 based on the control of the CPU 101. The communication device 105 can also be configured to communicate with an external server, etc. The communication device 105 can include a wireless communication module, which can include a well-known circuit structure, and the well-known circuit structure includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identification module card, a memory, etc. Here, the communication between the information processing device 100 and the scanner 110, the support arm 120, the model object support device 130, the display device 140, and the head assembly 150 can also be performed through wireless communication.

[0048] In addition, the communication device 105 can also include a wired communication module for wired connection. The wired communication module enables communication with other devices including the display device 140 via one or more external ports. In addition, various software components for processing data can be included. The external port is directly coupled to other devices via Ethernet, USB, IEEE 1394, etc. or indirectly via a network. In addition, it can also be constituted as a substitute for the hardware device by software for realizing the functions equivalent to the above devices.

[0049] The operation unit 106 is composed of, for example, a button, a keyboard, a touch panel, a controller, etc., and receives operation input from the user. The operation unit 106 may be shared with the operation unit 140A or may be independent of the operation unit 140A. For example, if the operation unit 140A is assumed to be an operation unit configured as a keyboard and a mouse, it can be shared with the operation unit 106. On the other hand, if the operation unit 140A is assumed to be an operation unit configured as a touch panel, a controller, etc., it can be an operation unit independent of the operation unit 106.

[0050] The display control unit 107 functions as a control unit for displaying information on the display device 140 connected to the information processing device 100, and controls the operation of the display device 140. Some functions of the operation unit 106 may be provided by the display device 140. For example, the display device 140 may be configured as a device having a touch panel such as a tablet terminal.

[0051] Next, the hardware configuration of the head unit 150 according to the present embodiment will be described. Figure 2(B) shows an example of the hardware structure of the head assembly 150. The head assembly 150 is composed of a head-mounted display (HMD) 150A and a controller 150B. The HMD 150A provides left-eye images and right-eye images to the left and right eyes of the user, respectively, and creates a three-dimensional sense through the parallax between the left-eye image and the right-eye image, thereby enabling a virtual reality (VR) experience. The controller 150B is provided in a housing independent of the HMD 150A. It is assumed that the controller 150B is a structure consisting of two controllers in a pair in order to be held and operated by the left and right hands of the user, respectively, but it can also be a single controller.

[0052] The CPU 151 is a device that performs overall control of the head unit 150, data calculation, processing, and management. For example, it can execute application data downloaded from the information processing device 100 to display a VR image on the display 157. In addition, based on the operation received via the controller 156 and the information detected by the detection unit 158, it is possible to switch the displayed VR image itself, switch the viewpoint in the VR space, or change the position in the VR space.

[0053] The RAM 152 is a volatile memory used as a temporary storage area such as a main memory and a work area of ​​the CPU 151. The ROM 153 is a nonvolatile memory and stores image data, other data, various programs used by the CPU 151 to perform operations, etc. in respective predetermined areas. The CPU 151 uses the RAM 152 as a work memory to control various parts of the head unit 150 according to the programs stored in the ROM 153, for example. In addition, the programs used by the CPU 151 to perform operations are not limited to being stored in the ROM 153, but may also be stored in the storage device 154.

[0054] The storage device 154 is composed of a magnetic disk such as an HDD or a flash memory. The storage device 154 stores application programs, OS, control programs, related programs, game programs, application data downloaded from the information processing device 100, display data, etc. The storage device 154 can read or write data based on the control of the CPU 151. The storage device 154 can also be used as a substitute for the RAM 152 and the ROM 153.

[0055] The communication device 155 is a communication interface for communicating with the information processing device 100 and the controller 150B based on the control of the CPU 151. The communication device 155 includes a wireless communication module capable of wireless communication based on Bluetooth (a registered trademark) and WiFi (IEEE802.11). The head component 150 can be connected to the information processing device 100 through wireless communication to download display data for VR images. In addition, it communicates with the controller 150B to receive information on the user's operation instructions to the controller 150B.

[0056] The display 156 is configured to provide the right eye image and the left eye image generated by the CPU 151 to the right eye and the left eye of the user, respectively. The detection unit 157 is a mechanism for detecting the sight lines of the left and right eyes of the user and detecting the tilt of the head mounted display. The detection unit 157 includes, for example, a sensor for detecting the sight line direction or the gaze direction of the user wearing the HMD 150A. In addition, it includes a gyroscope, a magnetometer, an accelerometer, a global positioning system (GPS), a compass, etc., and can determine the position, posture, tilt, etc. of the HMD 150A according to their detection information. The detection unit 157 detects information for determining the gaze direction and action of the user in the HMD 150A and sends the information to the CPU 151. The CPU 151 determines the gaze direction and action based on the received detection information, and adjusts the image presented in the display 156 of the HMD 150A so that it is consistent with the determined gaze direction and action.

[0057] The controller 150B is composed of, for example, a plurality of buttons, a cross key, etc., and receives operation inputs from the user such as selection operations and direction indication operations. The controller 150B and the HMD 150A are wirelessly connected via the communication device 155 .

[0058] Next, refer to Figure 3 An example of a model object that is a subject to be photographed in this embodiment is described below. Model object 300 is a model object having the appearance of a doll (robot or human). The model object can be, for example, a model object that is assembled and painted as a plastic model. Alternatively, it can be a completed model object such as a figure with movable joints (movable figure). Figure 3 The model object is only an example for explanation, and the shape of the model object is not limited to the appearance of a doll, and can be a model object of any shape such as a general vehicle, a racing car, a military vehicle, an airplane, a ship, an animal, a virtual life form, etc. In addition, the object to be photographed is of course not limited to a model object as long as the three-dimensional shape of the object can be photographed by the scanner 110.

[0059] The model object 300 includes a head 301, a chest 302, a right arm 303, a left arm 304, a right torso 305, a left torso 306, a right leg 307, a left leg 308, a right foot 309, and a left foot 310, and these parts are combined to form the model object 300. At least a part of each part 301 to 310 is supported in a manner that allows rotation (or swing) relative to an adjacent part. For example, the head 301 is supported in a manner that allows rotation relative to the chest 302, and the right arm 303 and the left arm 304 are supported in a manner that allows rotation relative to the chest 302. Since joint structures are provided at various parts of the model object 300 in this way, the model object 300 can take any posture.

[0060] Next, refer to Figure 4 An implementation example of the image processing system 10 in this embodiment will be described. Figure 4 An implementation example in which real data can be generated by photographing a model object is shown.

[0061] exist Figure 4 In the embodiment, the information processing device 100, a driving system for driving the support arm 402, a driving system for driving the rotating table 406, etc. are included in the box 401. In addition, the support arm 402 can also adjust the shooting direction and position manually. The surface of the box 401 is a flat structure so that a poster for promotion can be posted.

[0062] The support arm 402 corresponds to the support arm 120 and can support and fix the position of the terminal 403 functioning as the scanner 110 according to the control of the information processing device 100 or manually. The support arm 402 can also operate to control the inclination of the terminal 403.

[0063] The terminal 403 is a touch panel terminal with a built-in camera that can be used as the scanner 110, and can be, for example, a smartphone, a tablet terminal, a digital camera, etc. Instead of these terminals, Space Spider manufactured by Artec (registered trademark) can be used. Figure 4 However, this is a diagram showing an example of the structure of the system, and it can be set to a structure corresponding to the type of equipment used as the scanner 110. The terminal 403 can take an image of the model object 300 and send the image to the information processing device 100. The ring light 404 is a lighting device used when the model object 300 is photographed by the terminal 403, and can illuminate the model object 300 evenly, so that shadows are not easily generated. In addition, as an additional light source, in addition to the ring light 404, a ceiling light or auxiliary lights can be provided on the left, right, and bottom sides. Instead of the terminal 403, a three-dimensional scanner device as described above can also be used.

[0064] The background cloth 405 is a background cloth for photography, for example, a white cloth can be used. The rotating table 406 can carry the model object 300 and rotate the model object. A plurality of predetermined marks 410 can also be arranged on the rotating table 406. The marks 410 can be used to adjust the direction and position of the model object being photographed.

[0065] exist Figure 4 In the embodiment, the model object is placed on a translucent (transparent) table, but in addition to this, a supporting device such as a so-called "action base" (registered trademark) can also be used. The action base refers to a support arranged on a base and configured to be bent into a "く" shape, and the model object can be mounted on the front end of the support. In this case, the above-mentioned mark 410 can also be arranged on the front end portion of the support. The method of indicating the model object can be changed according to the posture. For example, in the case of an upright posture, it can be placed on a transparent table for shooting. On the other hand, in the case of a flying posture where you want to shoot the soles of the feet, you can also use a movable stand. In addition, the movable stand can also be used to shoot an upright posture.

[0066] The display device 407 is a device corresponding to the display device 140, and may also have a touch panel function. The user can use the touch panel function to perform a predetermined selection action. In addition, the operation can also be accepted by a controller or mouse not shown in the figure that can function as the operation unit 140A instead of the touch panel. The VR head assembly 408 is composed of an HMD 408A and a controller 408B, which correspond to HMDs 150A and 150B, respectively. The user wears the HMD 408A on the head, holds and operates the controller 408B with the left and right hands respectively, and can thus operate while watching the VR image.

[0067] Next, refer to Figure 5 An example of processing performed by the image processing system 10 according to the present embodiment is described below. At least a portion of the processing corresponding to the flowchart is implemented by the CPU 101 of the information processing device 100 executing a program stored in the ROM 103 and the storage device 104, or by the CPU 151 of the head unit 150 executing a program stored in the ROM 153 and the storage device 154.

[0068] First, in S501, the CPU 101 accepts user registration. The CPU 101 accepts input of the user's name and contact address. Each user is assigned a user identifier for uniquely identifying each user. The CPU 101 stores the input user information in the storage device 104 in association with the time when the input was accepted and the user identifier.

[0069] When the user registration is completed, the user sets his model object on the model object support device 130. The CPU 101 can determine whether the model object is set on the model object support device 130 based on the scanned image obtained by the scanner 110. Alternatively, a switch that becomes turned on when the model object is set on the model object support device 130 can be configured, and the CPU 101 detects the signal from the switch to make a judgment. Alternatively, a button for accepting an operation when the setting of the model object is completed is displayed on the display device 140, and the CPU 101 can detect whether an operation on the button is accepted. In S502, the CPU 101 detects that the model object is set on the model object support device 130 by any of the above methods. In response to this detection, the processing enters S503.

[0070] In S503, the scanner 110 performs scanning processing (photographic processing) to generate a scanned image. Specifically, the CPU 101 controls the support arm 120 to move the scanner 110 to an arbitrary registered shooting position, and at the shooting position, the model object support device 130 is rotated while the scanner 110 performs scanning. The scanned image obtained by shooting is sent to the information processing device 100, and the CPU 101 stores the scanned image in a table of the storage device 104 in association with the user identifier and the like.

[0071] In the next S504, the CPU 101 performs post-scanning processing on the scanned image acquired by the scanning process, generates three-dimensional model data, and generates application data (display data) for image display. In the information processing device 100, when the application data is executed in a specific application, the image can be played. In addition, in the head-mounted display 150A, when the application data is executed in the specific application, the VR image can be played. The generated three-dimensional model data and display data are stored in the storage device 104 in association with the user information.

[0072] In the next S505, the image display process is performed. When the image display process is performed in the information processing device 100, the CPU 101 executes the display data in the corresponding specific application, and displays the image on the display device 140 via the display control unit 107. In addition, in S505, the CPU 101 may control the communication device 105 to send the display data to the HMD 150A, and the CPU 151 of the HMD 150A may perform the process of displaying the VR image by executing the received display data in the corresponding application, thereby performing the image display process.

[0073] Next, refer to Figure 6 Next, the display processing of the display data corresponding to S505 is described. Figure 6The processing is executed by the CPU 101 of the information processing device 100 using the display data, but may be executed by the CPU 151 of the HMD 150A.

[0074] An application for displaying VR images corresponding to a VR image development platform for generating VR data is installed in the information processing device 100, and the CPU 101 can display images on the display device 140 using display data by executing the application. In addition, the same application for displaying VR images is also installed in the HMD 150A, and the CPU 151 can display VR images on the display 156 using display data by executing the application. In the following, the processing in the information processing device 100 is mainly described, but unless otherwise specified, it is assumed that the same processing is also performed in the HMD 150A. Therefore, the processing performed by the CPU 101 can be replaced by the processing performed by the CPU 151.

[0075] First, in S601, the CPU 101 causes the display device 140 to display a menu screen via the display control unit 107. In the menu screen, the user can make a selection input via the operation unit 140A. Here, a plurality of options are displayed in the menu screen. An example of the menu screen is shown in FIG. Figure 7 is shown in (A).

[0076] Figure 7 (A) is a diagram showing an example of a menu screen, showing two options as user destinations. One option is "My Room", which is a space given to each user, where the data of the model obtained by the user's own scanning can be stored. The other option is the Battle Room, where a battle game using the three-dimensional model data of the model object obtained by the user's scanning can be played. In this battle game, you can play against other users and the CPU. In addition, any game other than the battle game can also be played.

[0077] exist Figure 7 In the screen (A) of FIG. 1 , the user operates the operation unit 140A, for example, moves the cursor displayed on the screen to either "My Room" or "Battle Room", and performs a selection operation (for example, a click operation, a button operation) there, thereby enabling selection input. When using the HMD 150A, the user moves the controller 150B to a position overlapping with "My Room" or "Battle Room", and performs a button operation of the controller 150B there, thereby enabling selection input. The correspondence between the operation unit 140A and the controller 150B is the same in the following description.

[0078] In addition, in the virtual space corresponding to this embodiment, if Figure 7As shown in (B), there are generally four nodes, and it is possible to move back and forth between them. In this embodiment, the nodes include my room (first space), warehouse (second space), friend room (third space), and battle room (fourth space). My room is connected to the warehouse, and it is possible to move from my room to the warehouse. It is possible to move to the friend room via the warehouse. In addition, it is possible to move from my room to the battle room, but it is also possible to move to the friend room and from the friend room to the battle room. In addition, the nodes that constitute the virtual space are not limited to Figure 7 The nodes shown in (B) can also be set to be able to travel back and forth between more nodes through any path.

[0079] Back to Figure 6 As described above, when the CPU 101 receives a selection operation from the user via the operating unit 140A in the next S602, it causes the display device 140 to display an image corresponding to the selected arbitrary destination, that is, my room or battle room in S603. In the next S604, the CPU 101 obtains the user's operation information on the operating unit 140A. In the next S605, the CPU 101 displays an image corresponding to the operation information obtained in S604. The processing in S605 includes the display of my room or battle room, the display corresponding to the user operation received in the room, the movement display in the room, the execution of the battle game, the display of the warehouse, etc.

[0080] More specifically, refer to Figures 8 to 10D The details of the processing in S605 will be described. Figure 8 It is shown Figure 6 This is a flowchart of an example of details of the processing in S605. 9A to 9D This is a diagram showing an example of display when “My Room” is selected in S602 . FIG. 10A to FIG. 10D This is a diagram showing an example of display when “Battle Room” is selected in S602 .

[0081] exist Figure 8 In S801 of the flowchart of FIG. 1 , the CPU 101 receives an operation from the user via the operation unit 140A. The operation includes a movement operation for changing the user's viewpoint position (including an instruction operation for indicating the direction of travel, an operation on the cross key of the controller, etc.), an operation for selecting a display object, an operation for moving the selected display object, an operation for canceling the selection state of the selected object, etc. In addition, the received operation in S801 also includes the CPU 151 acquiring the detection information in the detection unit 157 of the HMD 150A.

[0082] For example, in Fig.9AIn the example of the screen display in my room shown in FIG. 1 , a display such as “Try to set the model in the warehouse” is displayed in area 901 of the display screen 900. In addition, a three-dimensional model 902 as a selection candidate is displayed at the bottom of the display screen 900. The three-dimensional model 902 corresponds to the model object obtained by the scanning process of S503 described above. The user can select and hold the three-dimensional model 902 by operating the operation unit 140A according to the message “Hold the model with button X”. In addition, the size of the three-dimensional model 902 in the holding state is displayed as a size that is the same as or close to the size of the actual model object in the real world that is obtained by scanning and becomes the three-dimensional model 902, or is displayed as a size that does not feel unnatural at least when held by hand. The user can move the held three-dimensional model 902 in my room and set it in a designated setting place, thereby appreciating the model object in the state of being stored in the warehouse. In addition, the size of the three-dimensional model 902 in the selection candidate state before being held can also be different from the size of the three-dimensional model 902 in the holding state. Specifically, the three-dimensional model 902 in the selection candidate state may be displayed larger than the three-dimensional model 902 in the gripping state. In this case, the three-dimensional model 902 in the selection candidate state can be easily appreciated in my room.

[0083] In addition, for example, Fig. 10A In the example of the screen display in the battle room shown, a display such as "Try to set the model to the battle stage" is performed in area 1001 of the display screen 1000. In addition, a three-dimensional model 1002 is displayed as a selection candidate at the bottom of the display screen 1000. The three-dimensional model 1002 corresponds to the model object obtained by the scanning process of S503 mentioned above. That is, the user can play a battle game using his own model object. The user can operate the operation unit 140A to select the three-dimensional model 1002 according to the message "Select button X to hold the model, release the button to set it", move it to the battle stage 1003, and set it on the battle stage 1003. The three-dimensional model 1002' is equivalent to the three-dimensional model 1002 set on the battle stage 1003.

[0084] Back to Figure 8As described above, in the next S802, the CPU 101 determines whether the operation received in S801 is an operation for selecting a three-dimensional model displayed on the screen. The determination of this operation can be based on whether a button operation equivalent to a selection button is received in a state where the cursor displayed on the screen or the display of a controller equivalent to the controller 150B is located at a position overlapping with the three-dimensional model. When multiple three-dimensional models are displayed on the screen, it is determined that the three-dimensional model overlapping with the cursor or the controller is selected. When the operation received in S801 is an operation for selecting a three-dimensional model, the process proceeds to S806. In addition, when it is not an operation for selecting a three-dimensional model, the process proceeds to S803.

[0085] In S806, the CPU 101 performs display processing corresponding to the operation received in S801. For example, when a selection operation for a three-dimensional model is received, the state of the three-dimensional model being held is displayed. Thus, the user can move the selected three-dimensional model to an arbitrary position through further operation input. In addition, in the HMD 150A, the CPU 151 determines the gaze direction and action based on the detection information received from the detection unit 157, and adjusts the image presented in the display 156 of the HMD 150A to make it consistent with the determined gaze direction and action. After that, the processing returns to S801 and the above processing is repeated.

[0086] In S803, the CPU 101 determines whether the selection model is set to a predetermined location based on the received operation. If the received operation is not to set the selection model to a predetermined location, the process moves to S806 to display the received operation and returns to S801.

[0087] Fig. 9B An example of a screen display when a three-dimensional model is set at a predetermined location in the My Room display is shown. Fig. 9B In the display screen 910, a message such as "warehouse set model" and "release button X to place" are displayed in area 911. The user can follow this instruction to place the selected three-dimensional model 902 as shown in FIG. Fig. 9B The room is set on a setting table 912 as shown. A door 913 leading to the warehouse is shown on the back side of the area 911. Here, the room and the warehouse are shown to be set adjacent to each other, but they can also be connected via a passage. Fig. 9B In the example shown, the CPU 101 can determine whether the three-dimensional model 902 is installed at a predetermined location based on whether the button operation is released while the three-dimensional model 902 is located within the area of ​​the installation table 912 .

[0088] Similarly, in Fig. 10AIn the example shown, the CPU 101 can determine whether the three-dimensional model 1002 is set to a predetermined location based on whether the button operation is released while the three-dimensional model 1002 is located in the area of ​​the fighting stage 1003.

[0089] Back to Figure 8 In the case where it is determined that the operation received in S803 is to set the selected model to a specified location, the process moves to S804, and the CPU 101 performs a performance display corresponding to the setting location. Afterwards, it is determined whether to end this process. If it is not ended, it returns to S801 and repeats the above process. If it is ended, the process ends.

[0090] Specifically, in my room display, if Fig. 9C As shown, in response to the three-dimensional model 902 being set on the setting table 912, the door 913 of the warehouse is opened to display the situation inside the warehouse. A three-dimensional model 921 obtained by enlarging the three-dimensional model 902 set on the setting table 912 is displayed in the warehouse. In addition, a passage 922 is displayed in front of the three-dimensional model 921, and a sign 923 is displayed at the end of the passage 922.

[0091] The size of the three-dimensional model 921 may be enlarged to a size that occupies a predetermined area in the space in the warehouse where the three-dimensional model 921 is arranged. Alternatively, the size may be enlarged to a factor corresponding to the actual size of the model object that is the prototype of the three-dimensional model 902. Alternatively, the size may be enlarged to a factor corresponding to the scale of the model object (e.g., 1 / 144, 1 / 100, 1 / 60, etc.).

[0092] exist Fig. 9C In the display state of further accepting an operation, if the accepted operation is a movement operation for changing the user's viewpoint position and is an operation for moving in the direction of passage 922, the display screen is transferred to Fig.9D Here, the three-dimensional model 921 is further enlarged and displayed in the display screen 930, and the logo 923 is also enlarged and displayed in the same manner. A message 931 such as "Go to the friend's room" is displayed superimposed on the logo 923. In addition, a message "Please move toward the arrow" is displayed at the passage 922. As a result, the user performs a moving operation for moving in the direction of the arrow, and when moving to a position within a certain distance from the logo 923, it is possible to move to the friend's room. At this time, an image of a person riding in the cockpit of the three-dimensional model 921 can also be displayed on the display screen as a performance display.

[0093] In this embodiment, if there are other users registered in the system, any one of them can be selected. In addition, the user can also pre-register a specific user among other users and select any one of the registered specific users. A friend room is allocated to each user, so through the above process, it is possible to move to each user's friend room and appreciate the friend room.

[0094] In addition, in the battle room display, in response to the three-dimensional model 1002 being set on the battle stage 1003, the display is switched to Fig. 10B The display content of the display screen 1010 shown in FIG. The display screen 1010 includes display areas 1011, 1012, and 1013, and the images of the battle game being executed are displayed in the display areas 1011 to 1013. Fig. 10B In the example, the battle game has not started yet, and the game is in a state of waiting for an instruction to start the game. An operating system 1014 is displayed below the display area 1011. An instruction receiving unit 1015 is displayed on the operating system 1014.

[0095] By moving the cursor to the instruction receiving unit 115 and clicking it, or overlapping the controller with the instruction receiving unit 1015, it is determined that the game start is selected, and the display screen is switched to Fig. 10C The content shown. Fig. 10C In the display screen 1020, Fig. 10B In the display state, the image of the three-dimensional model 1021 is displayed in the display area 1011. The three-dimensional model 1021 and Fig. 10A The three-dimensional model 1002 shown in FIG. 1 corresponds to the three-dimensional model 1002 shown in FIG. 1 . Then, the display content of the display screen switches to Fig. 10D The content shown.

[0096] exist Fig. 10D , an example of a screen displayed when a battle game starts is shown. Battle game screens 1031, 1032, and 1033 are displayed in display areas 1011, 1012, and 1013, respectively. A three-dimensional model and the remaining time of the game are displayed on screen 1031. The user can operate the operation unit 140A and the controller to progress the game.

[0097] As described above, in the present embodiment, a case where image display processing based on an operation received from a user is performed in the information processing device 100 and the HMD 150A is described. In the description of the above-mentioned embodiment, a case where display data is generated is described, and the display data is data for displaying an image by configuring three-dimensional model data generated based on an image obtained by shooting a model object in a virtual space. At this time, in the present embodiment, different performance images can be provided according to the position where the user sets the three-dimensional model. In the above-mentioned embodiment, as a performance image, a situation where the user's model object is kept in a warehouse in actual size is shown, and a situation where a battle game is implemented using the model object is shown, but the performance method is not limited to these. For example, the model object can also be made to appear in activities, concerts, sports, online meetings, etc. held in a virtual space. In addition, the technology of the present embodiment can also be applied to imaging technologies such as extended reality (XR) that merge the real world with the virtual world so that non-existent objects can be perceived in the real space.

[0098] Thus, in this embodiment, three-dimensional model data is generated based on an image obtained by photographing the appearance of the model object, and an image of the three-dimensional model arranged in a virtual space can be appreciated. Model objects, such as assembled plastic models, are model objects that users paint to create their own works, and since these personalities can be reflected in the performance of characters in moving images and virtual spaces, the interest can be significantly increased.

[0099] <Summary of Implementation Methods>

[0100] The above-mentioned embodiments disclose at least the following video generation device and computer program.

[0101] (1) An image generating device comprising:

[0102] a generating unit for generating an image in which a three-dimensional model based on three-dimensional model data is arranged in a virtual space, the three-dimensional model data being data generated based on a scanned image generated by scanning the appearance of a subject with a scanner;

[0103] an output unit configured to output the image generated by the generating unit to a display device;

[0104] a receiving unit that receives an operation input from a first observer on an image displayed on the display device; and

[0105] a determination unit configured to determine the content of the operation input received by the receiving unit,

[0106] The generating unit generates the image corresponding to the result of the determination in the determining unit.

[0107] When the determination unit determines that the three-dimensional model is arranged at a predetermined position in the virtual space, the generation unit causes the video to include a different performance corresponding to the determined arrangement position.

[0108] (2) The image generation device according to (1), wherein:

[0109] The specified position includes a first position in a first space in the virtual space,

[0110] In response to the three-dimensional model being arranged at the first position, the generation unit generates, as the performance, an image in which the three-dimensional model is enlarged and displayed in a second space connected to the first space.

[0111] (3) The image generation device according to (2), wherein:

[0112] The receiving unit receives an operation input for moving the viewpoint position in the virtual space from the first observer,

[0113] When the determination unit determines that the viewpoint position has moved to the vicinity of the enlarged displayed three-dimensional model based on the operation input from the first observer, the generation unit generates an image moving from the second space to a third space different from the first space and the second space as the performance.

[0114] (4) The image generation device according to (3), wherein:

[0115] The image moving to the third space includes an image of a performance of the first observer riding on the three-dimensional model displayed in the enlarged manner.

[0116] (5) The image generation device according to (2), wherein:

[0117] The receiving unit receives an operation input for moving the viewpoint position in the virtual space from the first observer,

[0118] When the determination unit determines that the viewpoint position has moved to the vicinity of the enlarged displayed three-dimensional model based on the operation input from the first observer, the generation unit generates, as the performance, an image of the first observer riding on the enlarged displayed three-dimensional model.

[0119] (6) The video generation device according to (3) or (4), wherein:

[0120] The first space is associated with the first observer,

[0121] The third space is associated with a second observer different from the first observer.

[0122] (7) The video generation device according to (6), wherein:

[0123] When it is determined that the viewpoint position has moved to a vicinity of the three-dimensional model displayed in the enlarged state, the determination unit further determines the second observer based on an operation from the first observer for selecting any one observer from a plurality of observers.

[0124] (8) The video generation device according to (6) or (7), wherein:

[0125] The specified position includes a third position in a fourth space connected to the third space,

[0126] The generation unit generates an image of a game using the three-dimensional model data in response to the three-dimensional model being arranged at the third position.

[0127] (9) The video generation device according to (8), wherein:

[0128] The generation unit generates an image of a start screen of the game in response to the three-dimensional model being arranged at the third position.

[0129] (10) The video generation device according to (8) or (9), wherein:

[0130] The game is a fighting game.

[0131] (11) The video generation device according to (8) or (9), wherein:

[0132] The game is a battle game with the second observer.

[0133] (12) The image generation device according to (1), wherein:

[0134] The specified position includes a third position in a fourth space in the virtual space,

[0135] The generation unit generates an image of a game using the three-dimensional model data in response to the three-dimensional model being arranged at the third position.

[0136] (13) The video generation device according to any one of (1) to (12), wherein:

[0137] The image generating device is a head mounted display,

[0138] The display device is a display included in the head mounted display.

[0139] The receiving unit receives the operation input via a controller associated with the head mounted display.

[0140] (14) The video generation device according to any one of (1) to (12), wherein:

[0141] The display device is an external display device connected to the image generating device.

[0142] The receiving unit receives the operation input via an operation unit for the video generating device.

[0143] (15) A computer program for causing a computer to function as the video generation device described in any one of (1) to (14).

[0144] The invention is not limited to the above-described embodiment, and various modifications and changes can be made within the scope of the gist of the invention.

Claims

1. An image generating device, comprising: a generating unit for generating an image in which a three-dimensional model based on three-dimensional model data is arranged in a virtual space, the three-dimensional model data being data generated based on a scanned image generated by scanning the appearance of a subject with a scanner; an output unit configured to output the image generated by the generating unit to a display device; a receiving unit that receives an operation input from a first observer on an image displayed on the display device; as well as a determination unit configured to determine the content of the operation input received by the receiving unit, The generating unit generates the image corresponding to the result of the determination in the determining unit. When the determination unit determines that the three-dimensional model is arranged at a predetermined position in the virtual space, the generation unit causes the video to include a different rendering corresponding to the determined arrangement position.

2. The image generation device according to claim 1, wherein: The specified position includes a first position in a first space in the virtual space, In response to the three-dimensional model being arranged at the first position, the generation unit generates, as the performance, an image in which the three-dimensional model is enlarged and displayed in a second space connected to the first space.

3. The image generation device according to claim 2, wherein: The receiving unit receives an operation input for moving the viewpoint position in the virtual space from the first observer, When the determination unit determines that the viewpoint position has moved to the vicinity of the enlarged displayed three-dimensional model based on the operation input from the first observer, the generation unit generates an image moving from the second space to a third space different from the first space and the second space as the performance.

4. The image generation device according to claim 3, wherein: The image moving to the third space includes an image of a performance of the first observer riding on the three-dimensional model displayed in the enlarged manner.

5. The image generation device according to claim 2, wherein: The receiving unit receives an operation input for moving the viewpoint position in the virtual space from the first observer, When the determination unit determines that the viewpoint position has moved to the vicinity of the enlarged displayed three-dimensional model based on the operation input from the first observer, the generation unit generates, as the performance, an image of the first observer riding on the enlarged displayed three-dimensional model.

6. The image generation device according to claim 3, wherein: The first space is associated with the first observer, The third space is associated with a second observer different from the first observer.

7. The image generation device according to claim 6, wherein: When it is determined that the viewpoint position has moved to a vicinity of the three-dimensional model displayed in the enlarged state, the determination unit further determines the second observer based on an operation from the first observer for selecting any one observer from a plurality of observers.

8. The image generation device according to claim 6, wherein: The specified position includes a third position in a fourth space connected to the third space, The generation unit generates an image of a game using the three-dimensional model data in response to the three-dimensional model being arranged at the third position.

9. The image generation device according to claim 8, wherein: The generation unit generates an image of a start screen of the game in response to the three-dimensional model being arranged at the third position.

10. The image generation device according to claim 8, wherein: The game is a fighting game.

11. The image generation device according to claim 8, wherein: The game is a battle game with the second observer.

12. The image generation device according to claim 1, wherein: The specified position includes a third position in a fourth space in the virtual space, The generation unit generates an image of a game using the three-dimensional model data in response to the three-dimensional model being arranged at the third position.

13. The image generation device according to claim 1, wherein: The image generating device is a head mounted display, The display device is a display included in the head mounted display. The receiving unit receives the operation input via a controller associated with the head mounted display.

14. The image generation device according to claim 1, wherein: The display device is an external display connected to the image generating device. The receiving unit receives the operation input via an operation unit for the video generating device. 15 . A computer program product, comprising a computer program for causing a computer to function as the video generation device according to claim 1 . 16 . A computer-readable storage medium storing a computer program for causing a computer to function as the video generation device according to claim 1 .