System and Sheet

Abstract

To provide a system that can easily generate 3D models of three-dimensional objects without requiring specialized equipment. [Solution] The system includes an image acquisition unit that acquires multiple images of a three-dimensional object placed on a predetermined sheet, taken from multiple directions, and a drawing target data setting unit that, based on the acquired multiple images, creates a three-dimensional model of the captured three-dimensional object, distinguishes between the three-dimensional model of the three-dimensional object on the defined three-dimensional object placement area on the sheet and the three-dimensional model of the three-dimensional object on an area other than the three-dimensional object placement area, and sets the three-dimensional model of the three-dimensional object on the three-dimensional object placement area as drawing target data.

Description

[Technical Field]

[0001] The present invention relates to a system for reproducing a three-dimensional object in a virtual space that has been photographed by a user using a terminal equipped with a camera, and to a sheet used in said system. [Background technology]

[0002] Conventionally, a technique has been known in which a three-dimensional object is scanned using specialized scanning equipment to generate a three-dimensional model of that object. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2005-346206 [Overview of the project] [Problems that the invention aims to solve]

[0004] Conventional technologies required specialized scanning equipment, making it difficult to easily generate 3D models of three-dimensional objects.

[0005] This invention has been made in view of the above-mentioned problems, and its purpose is to provide a system that can easily generate a three-dimensional model of a three-dimensional object without the need to prepare specialized equipment. [Means for solving the problem]

[0006] (1) The present invention relates to a system for reproducing a three-dimensional object in a virtual space that has been photographed by a user using a terminal equipped with a camera, and is characterized by including an image acquisition unit that acquires a plurality of images of a three-dimensional object placed on a predetermined sheet, taken from a plurality of directions, and a drawing target data setting unit that, based on the plurality of acquired images, creates a three-dimensional model of the photographed three-dimensional object, distinguishes between the three-dimensional model of the three-dimensional object on a defined three-dimensional object placement area on the sheet and the three-dimensional model of the three-dimensional object on an area other than the three-dimensional object placement area, and sets the three-dimensional model of the three-dimensional object on the three-dimensional object placement area as drawing target data. The present invention also relates to the sheet used in the above system. The present invention also relates to a program for causing a computer to function as each of the above units. The present invention also relates to a computer-readable information storage medium that stores the above program.

[0007] According to the present invention, a three-dimensional model of a three-dimensional object can be easily generated without the need to prepare specialized equipment.

[0008] (2) In addition, in the system according to the present invention, the sheet may be marked with an identification mark, and the drawing target data setting unit may recognize the three-dimensional object placement area based on the identification mark included in the plurality of acquired images.

[0009] (3) In addition, in the system according to the present invention, the sheet may be marked with an identification mark, and the drawing target data setting unit may create a three-dimensional model of the photographed three-dimensional object based on the feature points detected from the identification marks included in the plurality of acquired images.

[0010] (4) In addition, in the system according to the present invention, the sheet may be affixed with a plurality of identification marks, including at least two different identification marks.

[0011] (5) In addition, in the system according to the present invention, the drawing target data setting unit may create a three-dimensional model of a photographed three-dimensional object based on the image in which the number of recognized identification marks is equal to or greater than a predetermined number from among the plurality of acquired images.

[0012] (6) Also, in the system according to the present invention, an instruction label for indicating the placement location of the three-dimensional object is attached to the sheet, and the instruction label may be included in a polygon composed of a mark different from the instruction label.

[0013] (7) Also, in the system according to the present invention, the drawing target data setting unit may generate point cloud data from the plurality of acquired images to three-dimensionally model the photographed three-dimensional object.

[0014] (8) Also, in the system according to the present invention, the drawing target data setting unit acquires size information of the photographed three-dimensional object based on the acquired image, collates the three-dimensional model set as the drawing target data with a three-dimensional model stored in a database in advance, and specifies the type of the three-dimensional object based on the acquired size information and the collation result.

[0015] (9) Also, in the system according to the present invention, the drawing target data setting unit acquires sheet information of the sheet based on the acquired image, and sets additional information in the three-dimensional model set as the drawing target data based on the acquired sheet information.

Brief Description of the Drawings

[0016] [Figure 1] A diagram showing the system of the present embodiment. [Figure 2] A diagram showing an example of the functional blocks of the server of the present embodiment. [Figure 3] A diagram showing an example of a three-dimensional object placed on a predetermined sheet. [Figure 4] A diagram showing an example of a predetermined sheet [Figure 5] A diagram for explaining the photographing of a three-dimensional object. [Figure 6] A diagram for explaining the three-dimensional object placement area. [Figure 7] A flowchart showing the processing flow of the system of the present embodiment.

Mode for Carrying Out the Invention

[0017] Hereinafter, this embodiment will be described. Note that the embodiment described below does not unduly limit the content of the present invention described in the claims. Also, not all of the configurations described in this embodiment are essential constituent elements of the present invention.

[0018] 1. Configuration FIG. 1 is a diagram showing the system of this embodiment. In this embodiment, it is composed of a plurality of terminals 10 and a server 20. That is, as shown in FIG. 1, the system of this embodiment is configured such that the server 20 that provides services and the terminal 10 can be connected to a network.

[0019] The server 20 is an information processing device that provides an online service in response to a request from the terminal 10. The server 20 can be composed of a server group (authentication server, matching server, processing server, communication server, charging server, database server, etc.).

[0020] In this embodiment, the user uses the terminal 10 to photograph a three-dimensional object, and the terminal 10 transmits a plurality of images (still image group, moving image) generated by the photographing to the server 20. The server 20 three-dimensionally models the three-dimensional object based on the plurality of images received from the terminal 10 and reproduces the three-dimensional object in a three-dimensional virtual space. Also, in the server 20, information such as the user's account information, three-dimensional models and items available in the virtual space, and virtual currency is managed.

[0021] The terminal 10 is a portable terminal (smartphone, tablet PC, mobile phone, portable game machine, etc.) equipped with a camera (imaging unit), and is a device that can be connected to the server 20 via a network such as the Internet (WAN) or LAN.

[0022] <> FIG. 2 shows an example of the functional blocks of the server 20 of this embodiment. Note that the server of this embodiment may have a configuration in which some of the components (each part) in FIG. 2 are omitted.

[0023] The memory unit 270 stores programs and various data necessary for the computer to function as parts of the processing unit 200, and also functions as a work area for the processing unit 200, the functionality of which can be realized by a hard disk, RAM, etc. The memory unit 270 includes a storage unit 272 (for example, a database).

[0024] The storage unit 272 stores user information for each of the multiple users registered with the online service provided by the system of this embodiment. For example, the storage unit 272 stores user information such as username (user account), password, and destination information of terminal 10 (IP address, etc.), associated with the user identification information (user ID, terminal ID used by the user, etc.) of each of the multiple users. The storage unit 272 also stores information such as 3D models of three-dimensional objects photographed by the user, and information about items and virtual currency owned by the user, associated with the user identification information.

[0025] The communication unit 296 performs various controls for communication with the terminal 10 and other servers, and its functions can be realized by hardware such as various processors or communication ASICs, or by programs.

[0026] The processing unit 200 (processor) performs various processes such as user information management, login / logout processing, communication control processing, and 3D model generation processing based on data and programs transmitted from the terminal 10 and received via the communication unit 296. The processing unit 200 performs various processes using the storage unit 270 as the work area. The functions of the processing unit 200 can be realized by hardware such as various processors (CPU, DSP, etc.) and ASICs (gate arrays, etc.) or by programs. The processing unit 200 includes an image acquisition unit 210 and a drawing target data setting unit 212.

[0027] The image acquisition unit 210 acquires multiple images of a three-dimensional object transmitted from the terminal 10. The multiple images are a group of still images or moving images generated when a user uses the camera of the terminal 10 to photograph a three-dimensional object (e.g., a three-dimensional model) placed on a predetermined sheet from multiple directions. The predetermined sheet may be a sheet of paper on which identification marks and instruction marks described later are printed, the display surface of a display showing identification marks and instruction marks described later, or the mounting surface of a base or turntable with identification marks and instruction marks described later attached. The predetermined sheet is marked (printed or displayed) with identification marks (e.g., two-dimensional codes, barcodes, predetermined figures or marks) and instruction marks that indicate to the user the placement of the three-dimensional object on the predetermined sheet. The predetermined sheet may be marked with multiple identification marks, including at least two different identification marks. The instruction marks are arranged so that they are contained within a polygon composed of marks (e.g., identification marks) that are different from the instruction marks.

[0028] The drawing target data setting unit 212, based on multiple images acquired by the image acquisition unit 210, creates a 3D model of the captured (imaged) three-dimensional object, distinguishes between the 3D models of three-dimensional objects in the object placement area defined on a predetermined sheet and the 3D models of three-dimensional objects in areas other than the object placement area, and sets the 3D models of three-dimensional objects in the object placement area as the drawing target data. The 3D models set as the drawing target data are stored in the storage unit 272 in association with the user's (or terminal 10's) identification information. Drawing target data setting unit 212 The image acquisition unit 210 recognizes the area where the three-dimensional object is located based on the identification marks included in (imprinted) the image acquired by the image acquisition unit 210, and creates a three-dimensional model of the photographed three-dimensional object based on the feature points detected from the identification marks. If a predetermined sheet has multiple (N) identification marks, the drawing target data setting unit 212 may create a three-dimensional model of the photographed three-dimensional object based on the image among the multiple images acquired by the image acquisition unit 210 in which the number of recognized identification marks is equal to or greater than a predetermined number (n (n ≤ N)). Alternatively, the drawing target data setting unit 212 may create a three-dimensional model of the photographed three-dimensional object by generating point cloud data from the multiple images acquired by the image acquisition unit 210.

[0029] Furthermore, the drawing target data setting unit 212 may acquire size information (information regarding size) of the captured three-dimensional object based on the image acquired by the image acquisition unit 210, compare the three-dimensional model set as the drawing target data with three-dimensional models stored in the database in advance, identify the type of the three-dimensional object based on the acquired size information and the comparison result, and set the identified information (type information) in association with the three-dimensional model set as the drawing target data. Alternatively, the drawing target data setting unit 212 may acquire sheet information of a predetermined sheet based on the image acquired by the image acquisition unit 210, and set additional information in association with the three-dimensional model set as the drawing target data based on the acquired sheet information.

[0030] The processing unit 200 may, in response to a request from the terminal 10, send data of a 3D model set as data to be drawn to the terminal 10 and draw the 3D model on the terminal 10 (placing the 3D model in a 3D virtual space and generating an image as seen from a virtual camera in the 3D virtual space), or it may draw the 3D model set as data to be drawn and send the generated still image or video to the terminal 10 to display the still image or video.

[0031] 2. The method of this embodiment Next, the method of this embodiment will be explained with reference to the drawings. The system of this embodiment is configured to reproduce a three-dimensional object photographed by the user using terminal 10 in a three-dimensional virtual space. As shown in Figure 3, when taking a photograph, the user places a predetermined sheet ST on a stable flat surface such as a table top and places the three-dimensional object TO (in this case, a three-dimensional model) on the sheet ST. In Figure 3, the illustration of various signs attached to the sheet ST is omitted.

[0032] Figure 4 shows an example of sheet ST. Sheet ST has multiple identification marks IM attached (printed) to it. In this example, 12 identification marks IM1 to IM are attached to the outer perimeter of the rectangular sheet ST (along the outer edge). 12 Identification marks IM1~IM3,IM 11 ,IM 12 Identification marks IM5~IM9 are arranged along the long side of sheet ST, and identification marks IM3~IM5 and identification marks IM9~IM 11 These are arranged along the short side of sheet ST. Twelve identification markers IM1 to IM 12 Each of these is a uniquely identifiable (recognizable) two-dimensional code, for example, an identification code representing identification numbers "1" to "12". Note that parts of multiple identification marks (IM) may be identical.

[0033] Furthermore, sheet ST is equipped with an indicator mark IS that indicates the placement location of the three-dimensional object TO. In this example, the indicator mark IS is a crosshair (and a circle centered at the intersection of the crosshairs) placed in the center of sheet ST. The indicator mark IS is positioned so that the crosshair is contained within a rectangle (or a triangle formed by three of the four vertices of the rectangle) that encloses multiple identification marks IM. The user places the three-dimensional object TO according to the indicator mark IS so that its center aligns with the crosshair in a plan view. The user also places the three-dimensional object TO so that its front faces the side of identification mark IM1 (identification mark IM1 is in front of the three-dimensional object TO, identification mark IM4 is to the right of the three-dimensional object TO, identification mark IM7 is behind the three-dimensional object TO, and identification mark IM is to the left of the three-dimensional object TO). 10 Arrange the three-dimensional object TO such that it exists.

[0034] Terminal 10 has a dedicated application installed, which includes functions for printing Sheet ST, uploading video data (described later) to Server 20, and viewing the 3D model created on Server 20. When a user launches the application and instructs it to print (or display) Sheet ST, print data is sent from Terminal 10 to a printer connected to Terminal 10 via the network, and Sheet ST with identification markers IM and instruction markers IS printed on it is output from the printer. Alternatively, print data may be sent from Terminal 10 to the printer via Server 20. When printing Sheet ST, it is preferable to print on a specified paper size (e.g., A3 size) so that the size of the 3D object TO can be identified. In addition, user information set in the application (user name, experience points, information about evaluations from other users ("likes," etc.), information about effects to be added to the 3D model of the 3D object TO, information about user content (progress information, etc.), information entered by the user from the application, etc.) and a predetermined URL may be converted into a 2D code and embedded in one of the printed identification markers IM.

[0035] The user uses terminal 10 to film the three-dimensional object TO (and the sheet ST on which the three-dimensional object TO is placed) in video. Specifically, as shown in Figure 5, the camera 11 of terminal 10 is pointed towards the three-dimensional object TO, and filming begins from the front of the object TO. During filming, the entire sheet ST on which the three-dimensional object TO is placed is always visible. The user holds terminal 10 and moves around the three-dimensional object TO at a constant speed, one or more times, to film. It is preferable to move around the three-dimensional object TO multiple times at different angles of depression to film the object TO from multiple directions, including the height direction. If only two laps are performed, the height position and depression angle of the camera 11 are changed between the first and second laps. For example, the object TO is filmed from directly to the side in the first lap and from diagonally above in the second lap. Alternatively, the sheet ST on which the three-dimensional object TO is placed may be placed on a turntable or the like, and while fixing the position of the camera 11 on a plane using a tripod or the like, the sheet ST may be rotated multiple times while changing the height position and depression angle of the camera 11 to take pictures, thereby obtaining moving images of the three-dimensional object TO from multiple directions, including the height direction. In this case, an identification mark IM may be attached to the surface of the turntable so that the turntable itself functions as the sheet ST. The above application may also have a function to assist in video recording. In this case, the user starts recording by activating the camera 11 from the application, and during recording, the application displays on the terminal 10's display to instruct (guide) the user on the speed and direction of movement when moving around the sheet ST (or the rotation speed and direction of rotation when rotating the sheet ST), the height and orientation of the camera 11, etc.

[0036] The video data obtained from shooting using terminal 10 is stored in the storage unit of terminal 10. When a user launches the above application and performs an operation to instruct the upload of video data, the video data specified by the user is sent from terminal 10 to server 20. When the application is launched on terminal 10, the user is logged into server 20, and the video data uploaded from terminal 10 is stored in storage unit 272, associated with the user ID used during login. The video data may be sent to server 20 in multiple batches. For example, if there is a problem with the video data obtained from one shoot, the video data obtained from two shoots may be added and sent to server 20, or if there are two or more cameras, the video data captured by each camera may be sent to server 20 separately.

[0037] Server 20 creates a 3D model of the captured 3D objects (including 3D object TO on sheet ST, and all 3D objects captured in the image) based on the uploaded video data. Specifically, the 3D model is created using 3D Gaussian Splatting technology. First, using Structure from Motion (SfM) technology... First, feature points are detected from multiple images that make up the video (3D objects and identification markers IM reflected in the images), and point cloud data that becomes the initial position of a 3D Gaussian distribution is generated by matching the detected feature points across the multiple images. Next, the point cloud data generated by SfM is replaced with a 3D Gaussian distribution. Then, the 3D Gaussian distribution placed based on the point cloud data is rasterized (rendered), and the rasterized image is compared with the original image (images that make up the video). Parameters such as the position (world coordinates), orientation, shape (covariance), color, and transparency of the 3D Gaussian distribution are adjusted to minimize the difference between the two, or the 3D Gaussian distribution is duplicated (clone), split, or deleted. Note that each of the multiple identification markers IM on sheet ST is uniquely identifiable and is placed at a fixed position on sheet ST. Therefore, the relative position (horizontal position, height position) of camera 11 with respect to sheet ST (3D object TO) when the image was captured can be determined from the positional relationship of the multiple identification markers IM reflected in the images that make up the video. Using this, the drawing target data setting unit 212 may determine whether the video contains all the images necessary to create a 3D model of the three-dimensional object TO (images of the three-dimensional object TO taken from various directions, including the height direction and the front, back, left, and right directions), and if the necessary images are not included (the necessary images are missing), it may notify the user of the terminal 10 that uploaded the video. For example, if an image taken from the right side of the three-dimensional object TO (the direction where the identification mark IM4 is located) is missing, it may notify the user of that fact, and also if an image taken from the left side of the three-dimensional object TO (the direction where the identification mark IM4 is located) is missing, it may notify the user of that fact. 10 If there is an image taken from directly beside the object (in the direction in which it is located), but there is a lack of an image taken from diagonally above and to the left of the 3D object TO, a notification will be sent to that effect. If there are multiple images taken from multiple directions, the system may select the image with the highest accuracy suitable for 3D modeling from among the multiple images according to predetermined conditions. Also, if there is no image taken from a certain direction (or the accuracy of the image in question is insufficient), the system may generate an image taken from that direction by interpolation from images taken from other directions (or apply interpolation to the image with insufficient accuracy).

[0038] In this embodiment, among the generated three-dimensional models (point cloud data, three-dimensional Gaussian distribution arranged based on the point cloud data), the three-dimensional model of the solid object on the solid object placement area PA defined on the sheet ST and the three-dimensional model of the solid object on the area other than the solid object placement area PA are distinguished, and the three-dimensional model of the solid object on the solid object placement area PA is set as the drawing target data. The solid object placement area PA is recognized based on the identification label IM reflected in the image. For example, the four identification labels IM3, IM5, IM9, IM 11 are recognized, and the coordinates of these four identification labels IM3, IM5, IM9, IM 11 (in FIG. 4, the coordinates of the lower left corner of IM3, the upper left corner of IM5, the upper right corner of IM9, and IM 11 at the lower right corner) are used as vertices, and the area obtained by offsetting each side of the rectangle by a predetermined distance outward (in the direction away from the center of the rectangle) is the area corresponding to the entire surface of the sheet ST, which is set as the solid object placement area PA. Note that an area wider than the area corresponding to the entire surface of the sheet ST may be used as the solid object placement area PA, or an area narrower than the area corresponding to the entire surface of the sheet ST may be used as the solid object placement area PA. For example, the area corresponding to the rectangle surrounded by the twelve identification labels IM1 to IM 11 , where each side of the rectangle with the coordinates of the four identification labels IM3, IM5, IM9, IM 12 as vertices is offset by a predetermined distance inward (in the direction toward the center of the rectangle), may be used as the solid object placement area PA. As shown in FIG. 6, the three-dimensional model of the solid object on the solid object placement area PA is the three-dimensional model MD T (the three-dimensional model corresponding to the solid object TO on the sheet ST) existing inside a rectangular parallelepiped CB having the solid object placement area PA as the bottom surface and a predetermined height in the three-dimensional virtual space, and the three-dimensional model of the solid object on the area other than the solid object placement area PA is the three-dimensional model MD OThis is a 3D model corresponding to objects other than object TO on sheet ST. Alternatively, the point cloud data located inside the rectangular prism CB may be replaced and adjusted with a 3D Gaussian distribution, and the adjusted 3D Gaussian distribution may be set as the data to be plotted. It is also possible to do so. Furthermore, when the sheet ST itself (the surface of the sheet ST with identification marks IM etc. attached) is modeled in 3D along with the three-dimensional object TO, a given algorithm or program (for example, a program using given training information) may be used to delete point cloud data and 3D Gaussian distributions corresponding to the surface of the sheet ST. In addition, point cloud data and 3D Gaussian distributions that exist inside the rectangular parallelepiped CB but do not correspond to the three-dimensional object TO may be detected as noise and deleted using a given algorithm or program (for example, a program using given training information).

[0039] The 3D model set as the data to be drawn (3D model to be drawn) is stored in the storage unit 272, associated with the user ID of the user of terminal 10 that uploaded the video data. Furthermore, if user information is obtained (read) from an identification mark IM captured in the image, the obtained user information is stored in association with the 3D model to be drawn. The server 20 counts the number of times each user has created a 3D model of a 3D object TO (generated a 3D model to be drawn), sets an upper limit on this count, and prevents users from creating 3D models beyond this limit. Users can change (increase) this upper limit by using certain items (rights), such as tickets. The reason for limiting the number of times a 3D model can be created is that 3D modeling incurs significant processing costs. Alternatively, the count may be reset after a certain period (for example, allowing up to 10 3D models every two weeks). Furthermore, during the 3D modeling process, the server 20 transmits information regarding the progress of the process to the terminal 10, so that the progress of the process is displayed on the terminal 10's display.

[0040] When a user performs an operation from the above application to instruct the placement of a 3D model to be rendered in a 3D virtual space, the server 20 places the 3D model to be rendered associated with that user and the user's avatar in the 3D virtual space where other users also participate (where 3D models to be rendered associated with other users are also placed). At this time, the 3D model to be rendered is placed at a size determined based on the user information associated with the 3D model to be rendered and the size information described later. Specifically, if the object TO is a 1 / n scale 3D model, the 3D model to be rendered is placed in the 3D virtual space at the actual size (e.g., 18m), which is n times the size of the object TO, and the user's avatar is placed at human size (e.g., 1.8m). In addition, the user name and effect display may be added to the 3D model to be rendered based on the user information associated with the 3D model to be rendered. Server 20 transmits data necessary for rendering a 3D model or the like placed in a 3D virtual space to terminal 10. Terminal 10 generates an image visible from a given viewpoint in the 3D virtual space (a virtual camera set as the avatar's viewpoint, or a virtual camera that moves following the avatar) and displays the generated image on the display. Alternatively, Server 20 may generate an image visible from a given viewpoint in the 3D virtual space and sequentially transmit the generated images to terminal 10 (video streaming).

[0041] Users can operate terminal 10 to move their avatar in a 3D virtual space and experience what a real-world 3D model would look like from the avatar's perspective. Users can also view other users' 3D models placed in the 3D virtual space from their avatar's perspective and use terminal 10 to input comments and evaluations of other users' 3D models. Information regarding comments and evaluations of 3D models is sent to server 20 and stored in the user information associated with that 3D model. Based on this information, objects representing the comments and evaluations are placed alongside the 3D model in the 3D virtual space, or effects corresponding to the comments and evaluations are added to the 3D model. In this way, it is possible to simulate 3D models of objects in a 3D virtual space, something that cannot be achieved in real-world exhibitions where physical 3D models are displayed. By arranging the elements, you can create a unique visual effect.

[0042] According to this embodiment, a 3D model of a three-dimensional object TO can be easily generated without the need to prepare specialized equipment. Furthermore, by distinguishing between the 3D model of a three-dimensional object on the object placement area PA recognized based on the identification marker IM and the 3D models of other three-dimensional objects, and setting the 3D model of the three-dimensional object on the object placement area PA as the data to be drawn, it is possible to exclude three-dimensional objects other than the three-dimensional object TO placed on the sheet ST (three-dimensional objects that have unintentionally appeared in the image), and to use only the 3D model of the three-dimensional object TO on the sheet ST as the data to be drawn. In addition, the identification marker IM is used not only when recognizing the object placement area PA, but also when creating a 3D model of the three-dimensional object (feature points are extracted from the identification marker IM in the image when generating point cloud data). By extracting feature points from multiple identification markers IM that are different from each other (uniquely identifiable), the accuracy of matching feature points between images can be improved, and the accuracy of 3D modeling can be improved.

[0043] In this embodiment, a three-dimensional object may be modeled (point cloud data generated) based on images from the video footage obtained during capture in which the number of recognized identifiers IM is greater than or equal to a predetermined number. For example, images in which 10 or more identifiers IM can be recognized may be extracted from the group of images constituting the video, and point cloud data may be generated from the extracted images. In this way, the accuracy of the three-dimensional model can be further improved by generating point cloud data only from images containing a larger number of identifiers IM.

[0044] Furthermore, in this embodiment, a 3D model (3D Gaussian distribution) for each type of three-dimensional object (e.g., model number of a three-dimensional model) may be stored in a database (storage unit 272) in advance. The 3D model to be drawn may be compared with the 3D models of each type stored in the database, and the type of 3D model that most closely resembles the 3D model to be drawn may be identified as the type of three-dimensional object TO. The identified type may then be associated with the 3D model to be drawn. In addition, if there are three-dimensional objects of the same type but with different sizes (e.g., scale of a three-dimensional model), the size information of the three-dimensional object TO may be obtained from the images constituting the moving image obtained by shooting, and the type containing the size information of the three-dimensional object TO (e.g., whether it is model number "A" at 1 / 100 scale or model number "A" at 1 / 144 scale) may be identified. In this case, the size of the three-dimensional object TO may be estimated from the relationship between the size of the recognized identification mark IM or sheet ST and the size of the 3D model of the three-dimensional object TO. Furthermore, the user may input information about the type and size of the 3D object TO into the terminal 10 before or after taking a picture, and upload the input information along with the video image data to the server 20. In addition, when rendering a 3D model to be rendered (placing a 3D model to be rendered in a 3D virtual space), the 3D model to be rendered may be resized based on the size information set in association with the 3D model to be rendered (for example, increasing the size of a small 3D model to be rendered and decreasing the size of a large 3D model to be rendered).

[0045] Furthermore, in this embodiment, sheet information of sheet ST (for example, whether it is a sheet obtained for free (free sheet) or a sheet obtained by paying a fee (paid sheet)) may be obtained from the images constituting the moving image obtained by shooting, and additional information may be set on the 3D model to be drawn based on the obtained sheet information. In this case, the sheet information may be coded and embedded in one of the identification marks IM (2D code), and the sheet information may be obtained from the identification mark IM contained in the image. Also, if the sheet information indicates that it is a paid sheet, predetermined effects or marks may be added to the 3D model to be drawn and drawn, or predetermined items (weapons, etc.) may be equipped to the 3D model to be drawn and drawn, based on the additional information set according to the sheet information.

[0046] Alternatively, the system may allow multiple three-dimensional objects to be placed on Sheet ST, and each of these objects placed on Sheet ST may be converted into a 3D model and set as data to be drawn. In this case, the 3D models of the objects may be stored in a database beforehand, and the generated 3D model may be compared with the 3D model stored in the database to distinguish (identify) the 3D model corresponding to each of the multiple three-dimensional objects.

[0047] In addition to attaching the identification mark IM to the sheet ST (the surface on which the three-dimensional object TO is placed), the identification mark may also be placed on a vertical surface perpendicular to the surface on which the three-dimensional object TO is placed (for example, by fixing a flat plate with the identification mark attached vertically or suspending it from above). In this way, even when photographing the three-dimensional object TO placed on the sheet ST from an upward angle (making it difficult to capture the identification mark IM on the sheet ST), the identification mark on the vertical surface can still be captured.

[0048] Alternatively, a three-dimensional object TO may be placed on a designated pedestal marked with an identification mark IM, and the object TO placed on the pedestal may be photographed from multiple directions. The pedestal may be installed in stores selling three-dimensional models or at event venues, allowing users to bring their own models and take the photographs themselves. In addition to the three-dimensional object TO, the user themselves may also be photographed and a three-dimensional model created. Furthermore, if the three-dimensional object TO is large, a drone equipped with a camera may be flown around the object TO placed on the floor marked with an identification mark IM, and the camera may be used to photograph the object TO from multiple directions.

[0049] 3. Processing Next, an example of the processing of the system (server 20) of this embodiment will be explained using the flowchart in Figure 7.

[0050] First, the image acquisition unit 210 acquires video data transmitted from the terminal 10 (step S10). Next, the drawing target data setting unit 212 recognizes identification markers IM from each image constituting the video data and extracts images in which the number of recognized identification markers IM is equal to or greater than a predetermined number (step S11).

[0051] Next, the drawing target data setting unit 212 detects feature points from the three-dimensional objects and identification marks IM included in the multiple extracted images, and generates point cloud data by matching the detected feature points across the multiple images (step S12). Next, the drawing target data setting unit 212 replaces the generated point cloud data with a three-dimensional Gaussian distribution, compares the rasterized image of the three-dimensional Gaussian distribution with the original image, and adjusts the number, position, shape, color, transparency, etc. of the three-dimensional Gaussian distribution (step S13). Next, the drawing target data setting unit 212 identifies the world coordinates of the three-dimensional object placement area PA based on the identification marks IM included in the extracted images (step S14). Next, the drawing target data setting unit 212 sets the three-dimensional Gaussian distribution existing inside the rectangular parallelepiped CB with the three-dimensional object placement area PA as the drawing target data, and stores the three-dimensional Gaussian distribution (drawing target 3D model) set as the drawing target data in the storage unit 272, associating it with the user ID of the user of the terminal 10 that transmitted the moving image data (step S15).

[0052] Next, in response to a request from terminal 10, the processing unit 200 sends to terminal 10 a 3D model to be drawn that is associated with the user ID of the user of terminal 10 (step S16).

[0053] The present invention is not limited to the embodiments described above, and various modifications are possible. For example, terms cited as broad or synonymous in the specification or drawings may be replaced with broad or synonymous terms in other descriptions in the specification or drawings. [Explanation of symbols]

[0054] 10...Terminal, 11...Camera, 20...Server, 200...Processing unit, 210...Image acquisition unit, 21 2...Data setting unit for drawing targets, 270...Storage unit, 272...Storage unit, 296...Communication unit

Claims

1. A system that reproduces a three-dimensional object in a virtual space, which is photographed by a user using a device equipped with a camera. An image acquisition unit that acquires multiple images of a three-dimensional object placed on a predetermined sheet, taken from multiple directions, A system characterized by including a drawing target data setting unit that, based on a plurality of acquired images, creates a three-dimensional model of a photographed three-dimensional object, distinguishes between the three-dimensional model of a three-dimensional object in a defined three-dimensional object placement area on the sheet and the three-dimensional model of a three-dimensional object in an area other than the three-dimensional object placement area, and sets the three-dimensional model of the three-dimensional object in the three-dimensional object placement area as drawing target data.

2. In claim 1, The aforementioned sheet is marked with an identification sign. The drawing target data setting unit is: A system characterized by recognizing the area where a three-dimensional object is located based on the identification markers contained in a plurality of acquired images.

3. In claim 1, The aforementioned sheet is marked with an identification sign. The drawing target data setting unit is: A system characterized by creating a three-dimensional model of a photographed object based on feature points detected from the identification markers contained in a plurality of acquired images.

4. In claim 2 or 3, The system is characterized in that the sheet is affixed with a plurality of identification marks, including at least two different identification marks.

5. In claim 4, The drawing target data setting unit is: A system characterized by creating a three-dimensional model of a photographed object based on an image in which the number of recognized identification marks is equal to or greater than a predetermined number among a plurality of acquired images.

6. In claim 1, The aforementioned sheet is fitted with an indicator mark that shows where to place the three-dimensional object. The system is characterized in that the indicator sign is arranged such that it is contained within a polygon composed of marks different from the indicator sign.

7. In claim 1, The drawing target data setting unit is: A system characterized by generating point cloud data from multiple acquired images to create a three-dimensional model of a photographed object.

8. In claim 1, The drawing target data setting unit is: A system characterized by acquiring size information of a photographed three-dimensional object based on the acquired image, comparing the three-dimensional model set as the data to be drawn with a three-dimensional model stored in a database in advance, and identifying the type of the three-dimensional object based on the acquired size information and the comparison result.

9. In claim 1, The drawing target data setting unit is: A system characterized by acquiring sheet information of the sheet based on the acquired image, and setting additional information to the 3D model set as the data to be drawn based on the acquired sheet information.

10. A sheet used in the system described in claim 1.

Images