Information processing apparatus, information processing method, and program

Abstract

The present technology pertains to an information processing apparatus, an information processing method, and a program, which make it possible to easily and accurately edit a 3D map for VPS. The information processing apparatus edits an editing target region in a 3D model in a virtual space, and updates a 3D map on the basis of the 3D model in which the editing target region is edited. The present technology can be applied to an AR providing system or the like that provides an AR event by using, for example, a 3D map for VPS.

Description

Information processing device, method, and program

[0001] The present technology relates to an information processing device, method, and program, and in particular to an information processing device, method, and program that enable easy and accurate editing of a 3D map for a VPS (Visual Positioning Service).

[0002] VPS is a technology that can significantly upgrade the functionality of XR (Cross Reality) such as AR (Augmented Reality). VPS is a technology that measures position from images.

[0003] For example, suppose a 3D map for VPS (hereinafter referred to as a VPS map) was created in advance (e.g., one month in advance) at the request of the event maker for a recent AR event. However, if decorations for the AR event were added several days before the AR event, the content of the created VPS map would differ from reality, and there was no option but to create the VPS map again from scratch.

[0004] Regarding 2D images, partial replacement and generation techniques have already been proposed (see Non-Patent Document 1).

[0005] Johannes L. Schonberger, Jan-Michael Frahm, "Structure-from-Motion Revisited", [online], from June 26th to July 1st 2016, CVPR (Computer Vision and Pattern Recognition) 2016 in Las Vegas, [Retrieved October 25, 2023], Internet <https: / / openaccess.thecvf.com / content_cvpr_2016 / papers / Schonberger_Structure-From-Motion_Revisited_CVPR_2016_paper.pdf>

[0006] As described above, creating a VPS map from scratch takes time and effort, so there is a need for a technology that allows for easy and accurate editing of VPS maps.

[0007] The present technology has been made in view of such circumstances, and makes it possible to easily and accurately edit 3D maps for VPS.

[0008] An information processing device according to one aspect of the present technology includes a 3D model editing unit that edits an area to be edited in a 3D model in a virtual space, and a map update unit that updates a 3D map based on the 3D model in which the area to be edited has been edited.

[0009] In one aspect of the present technology, an area to be edited in a 3D model in a virtual space is edited, and a 3D map is updated based on the 3D model in which the area to be edited has been edited.

[0010] 10 is a diagram illustrating the timing of creating a conventional VPS map. FIG. 11 is a diagram illustrating an example of editing a 3D model according to the present technology. FIG. 12 is a diagram illustrating an example of merging a projection object image with a key frame. FIG. 13 is a diagram illustrating an example of updating a VPS map. FIG. 14 is a block diagram illustrating an example of the configuration of an AR provision system according to an embodiment of the present technology. FIG. 15 is a block diagram illustrating an example of the hardware configuration of the image acquisition device and VPS server of FIG. 5. FIG. 16 is a block diagram illustrating an example of the functional configuration of the VPS server. FIG. 17 is a block diagram illustrating an example of the functional configuration of the AR server. FIG. 18 is a flowchart illustrating pre-creation processing of the VPS server. FIG. 19 is a flowchart illustrating editing and updating processing of the VPS server. FIG. 19 is a flowchart illustrating an example of merging processing in step S33 of FIG. 10. FIG. 19 is a flowchart illustrating AR content provision processing of the AR server 23. FIG. 19 is a diagram illustrating an example of re-editing and re-projection of a 3D model. FIG. 20 is a flowchart illustrating updating processing of updated key frames in a real image. FIG. 21 is a block diagram illustrating an example of the configuration of a computer.

[0011] Hereinafter, embodiments of the present technology will be described. The description will be made in the following order: 1. Overview of the present technology 2. Configuration 3. Operation 4. Modifications 5. Other

[0012] 1. Overview of the Present Technology Prior Art FIG. 1 is a diagram showing the timing of creating a conventional VPS map.

[0013] For example, if a VPS map needs to be created on July 1st in preparation for an AR event on August 1st, the VPS map created on July 1st will include a 3D model created based on an image of the sign PS in its upright position.

[0014] After that, in the real space, on July 25th, decorations for the AR event on August 1st were added, and the signboard PS was changed to the signboard NS.

[0015] In this case, the 3D map created on July 1st will contain information about the past state in which the signboard PS was up, unlike the actual state in which the signboard NS was up.

[0016] In such a situation, with conventional technology, it would have been necessary to create a VPS map from scratch of the actual situation in which the sign NS would be displayed between July 25th and August 1st, the day of the AR event, which was time-consuming and labor-intensive.

[0017] <Editing a 3D Model According to the Present Technology> FIG. 2 is a diagram showing an example of editing a 3D model according to the present technology.

[0018] In Fig. 2A, a pre-created 3D model is shown in a virtual space, in Fig. 2B, a 3D model being edited in a virtual space, and in Fig. 2C, a 3D model being projected onto a keyframe in a virtual space.

[0019] 2, the base of the triangle represents an image captured at each of the vertex positions P1 to P3 of the triangle at the angle of view indicated by the vertex angle of the triangle. This is the same in the subsequent figures. This image is called a key frame, which is a frame required (i.e., a key) for recording the information of the 3D model as a VPS map, and the VPS map is created from the key frame information obtained from the 3D model based on the key frame.

[0020] First, 3D models of signs PS, buildings, etc. are created in advance in a virtual space based on images captured on July 1st from positions P1 to P3, as shown in A of Fig. 2. A VPS map is created using these images themselves (keyframes), the camera positions (e.g., positions P1 to P3) and postures when the images were captured, and feature quantities extracted from the images as keyframe information.

[0021] Next, as shown in B of FIG. 2 , editing is performed in a virtual space to replace the area of ​​the signboard PS, which is the area to be edited in the pre-created 3D model, with a 3D object (edit object) of the signboard NS that was created separately after July 25th, for example. Note that a part of the 3D model is referred to as a 3D object. Furthermore, the virtual space in which editing is performed is the virtual space in which the 3D model was created, but it may also be a virtual space different from the virtual space in which the 3D model was created.

[0022] Then, as shown in FIG. 2C, a 3D object of the signboard NS is projected from the edited 3D model onto the key frames from position P1 to position P3, thereby generating a projected object image.

[0023] The generated projected object image is merged with the keyframes, and the VPS map is updated based on the merged keyframes.

[0024] <Merging Projected Object Image and Key Frame> FIG. 3 is a diagram showing an example of merging a projected object image and a key frame.

[0025] The left side of Fig. 3 shows a key frame, which is a captured image, and the projected object image described with reference to Fig. 2. The projected object image is an image in which a 3D model including a 3D object edited in a virtual space is projected, and therefore differs from the actual captured image in that there is empty space between the 3D models, for example.

[0026] In FIG. 3, the cross in the middle represents merging the projected object image with the keyframe shown on the left.

[0027] For example, the portion of the key frame other than the signboard PS and the portion of the projected object image that is the signboard NS are merged, thereby generating an updated key frame in which the signboard PS has been updated to the signboard NS.

[0028] <Updating the VPS Map> FIG. 4 is a diagram showing an example of a VPS map.

[0029] In the center of FIG. 4, a VPS map DB1 is shown.

[0030] The VPS map DB1 stores a VPS map including a created 3D model and key frame information of the 3D model.

[0031] The keyframe information includes information about the entire image or a portion thereof encoded for matching between images, 3D measurement results related to the image, the device position and orientation, and the image (keyframe). Examples of the information about the entire image or a portion thereof encoded for matching between images include local features. Examples of the 3D measurement results related to the image include 3D landmarks.

[0032] A local feature is a set of characteristic corner points and hash values. A 3D landmark is information indicating where a corner point is located in three-dimensional space. A device position and orientation is information indicating the position and orientation of the device that captured the key frame. A key frame is composed of an actually captured image, etc. In addition, in positions where there is no captured image, a projected image in virtual space or a merged image of the surrounding captured image and the projected image becomes a key frame.

[0033] The right side of Figure 4 shows an updated key frame that is generated by editing the 3D model and merging the key frame with a projected object image onto which the edited 3D model (3D object) is projected.

[0034] This updated key frame replaces the key frame previously registered in the VPS map DB1, and the key frame information is updated based on the edited 3D model and the updated key frame, and the VPS map is updated.

[0035] As described above, in this technology, a 3D model created from multiple key frames with different viewpoints (i.e., positions and postures) is edited in a virtual space, and the projected object image is merged with the key frames, thereby updating the key frame information and updating the VPS map.

[0036] Therefore, according to the present technology, the VPS map DB1 can be easily edited with high accuracy.

[0037] 2. Configuration System Configuration FIG. 5 is a diagram illustrating an example of the configuration of an AR providing system according to an embodiment of the present technology.

[0038] In FIG. 5, the AR providing system 11 is a system that provides an AR event using AR content by providing AR content to a user terminal 24 owned by a user using a VPS map.

[0039] The AR providing system 11 is configured to include an image acquisition device 21 , a VPS server 22 , an AR server 23 , and a user terminal 24 .

[0040] The image acquisition device 21, the VPS server 22, the AR server 23, and the user terminal 24 are connected to a network 25. The image acquisition device 21, the VPS server 22, the AR server 23, and the user terminal 24 can communicate with each other via the network 25.

[0041] The image acquisition device 21 is configured by a smartphone, a digital camera, etc. The image acquisition device 21 acquires multiple pieces of actual image data of a building or the like in order to create a VPS model, and transmits the acquired multiple pieces of image data to the VPS server 22.

[0042] The VPS server 22 is a server that has a VPS map DB 1. The VPS server 22 creates a VPS model in a virtual space using an image of the image data transmitted from the image acquisition device 21, and registers a VPS map consisting of the VPS model and its key frame information in the VPS map DB 1.

[0043] In addition, the VPS server 22 edits the VPS model in the virtual space using the images transmitted from the image acquisition device 21 as necessary, and updates the VPS map (the VPS model and its key frame information) registered in the VPS map DB1.

[0044] The AR server 23 searches the VPS map DB1 of the VPS server 22 using an image of the image data transmitted from the user terminal 24 as a query image and associates the query image with the VPS map. The AR server 23 estimates (localizes) the position and orientation of the device (user terminal 24) based on the VPS map associated with the query image. Furthermore, the AR server 23 estimates the orientation of the registered AR content based on the estimated device position (orientation), that is, estimates how the AR content appears from the device, determines how the AR content should be displayed, and provides (transmits) the AR content to the user terminal 24.

[0045] The user terminal 24 is composed of a smartphone, a head-mounted display, or the like. The user terminal 24 acquires image data by capturing an image of a position that is the target of the AR event in real space, and transmits the acquired image data to the AR server 23. The user terminal 24 displays the AR content transmitted from the AR server 23 by superimposing it on an actual image captured by a camera or the like provided in the user terminal 24. This allows the user to enjoy the AR event.

[0046] Although FIG. 5 shows an example in which the VPS server 22 and the AR server 23 are configured separately, they may be configured as a single server.

[0047] <Hardware Configuration> FIG. 6 is a block diagram showing an example of the hardware configuration of the image acquisition device 21 and the VPS server 22 shown in FIG.

[0048] The image acquisition device 21 in FIG. 6 is configured to include an imaging unit 31 , a display unit 32 , and a communication unit 33 .

[0049] The imaging unit 31 captures an image of a subject for which a 3D model is to be created, generates an image signal, performs predetermined signal processing on the generated image signal, and outputs the processed image data to the display unit 32 and the communication unit 33.

[0050] The display unit 32 is configured by an LCD (Liquid Crystal Display) etc. The display unit 32 displays an image corresponding to the image data supplied from the imaging unit 31.

[0051] The communication unit 33 communicates with other devices via the network 25. For example, the communication unit 33 transmits image data supplied from the imaging unit 31 to the VPS server 22 via the network 25.

[0052] The VPS server 22 is configured to include a communication unit 41 , a CPU 42 , a data storage unit 43 , and a graphic display unit 44 .

[0053] The communication unit 41 communicates with other devices via the network 25. For example, the communication unit 41 exchanges data with the image acquisition device 21 and the AR server 23. For example, the communication unit 41 receives image data transmitted from the image acquisition device 21 and outputs the image data to the CPU 42 and the data storage unit 43.

[0054] The CPU 42 controls each part of the VPS server 22 so that the VPS server 22 functions as a VPS server that creates and edits VPS models, creates and manages VPS maps, and so on.

[0055] The data storage unit 43 stores programs, data, and image data. For example, the data storage unit 43 includes the VPS map DB1 shown in FIG.

[0056] The graphic display unit 44 displays an image corresponding to the image data stored in the data storage unit 43 .

[0057] Since the AR server 23 is configured in the same manner as the VPS server 22 in FIG. 6, the AR server 23 will be described below using the configuration of the VPS server 22 in FIG.

[0058] <Functional Configuration of VPS Server> FIG. 7 is a block diagram showing an example of the functional configuration of the VPS server 22. As shown in FIG.

[0059] Each function in FIG. 7 is realized by the CPU 42 executing a program stored in the data storage unit 43 or the like.

[0060] The VPS server 22 in FIG. 7 is configured to include a 3D model map creating / editing unit 51 and a 3D model creating unit 52 .

[0061] The 3D model map creating and editing unit 51 creates and stores 3D models and VPS maps, and also edits and updates the stored 3D models and VPS maps.

[0062] The 3D model map creating / editing unit 51 is configured to include the VPS map DB1, 3D model map creating unit 61, 3D model map registration unit 62, 3D model map display unit 63, 3D model editing unit 64, VPS map editing unit 65, and 3D model DB 66 shown in FIG.

[0063] The 3D model map creation unit 61 acquires multiple images of image data transmitted from the image acquisition device 21 and creates a 3D model using the acquired multiple images as key frames. The 3D model map creation unit 61 creates a 3D map based on the multiple key frames. 3D mapping refers to creating a VPS map consisting of a 3D model and key frame information. The 3D model map creation unit 61 outputs the VPS map to the 3D model map registration unit 62.

[0064] The 3D model map registration unit 62 registers the 3D model in the 3D model DB 66. The 3D model map registration unit 62 registers a VPS map made up of the 3D model and key frame information in the VPS map DB1.

[0065] The 3D model map display unit 63 displays the 3D models and VPS maps registered in the 3D model DB 66 and the VPS map DB 1. The 3D model map display unit 63 also displays, as necessary, the 3D models being created by the 3D model map creation unit 61 or edited by the 3D model editing unit 64, and the VPS maps being edited by the VPS map editing unit 65.

[0066] The 3D model editing unit 64 edits, in a virtual space, the 3D models of the VPS maps registered in the VPS map DB 1. That is, the 3D model editing unit 64 reads the 3D models of the VPS maps registered in the VPS map DB 1 into the virtual space and performs editing processing for the 3D models. The editing processing is, for example, processing to identify an area to be edited on the 3D model, identify a 3D object created by the 3D model creation unit 52, such as a signboard, as an editing object, and place the editing object in the editing area.

[0067] The 3D model editing unit 64 projects the 3D object from the edited 3D model onto the key frame to generate a projected object image. The 3D model editing unit 64 merges the key frame of the VPS map registered in the VPS map DB 1 with the projected object image to generate an updated key frame. The 3D model editing unit 64 outputs the edited 3D model and the updated key frame to the VPS map editing unit 65.

[0068] The VPS map editing unit 65 creates a 3D map using the updated keyframes and the 3D model, and updates the VPS map in the VPS map DB 1.

[0069] The 3D model DB 66 registers the 3D models created by the 3D model map creation unit 61 and the 3D models edited by the 3D model editing unit 64 .

[0070] The 3D model creation unit 52 is configured to include a 3D object creation unit 71 , a 3D object registration unit 72 , and a 3D object DB 73 .

[0071] The 3D object creation unit 71 acquires a plurality of images of the image data transmitted from the image acquisition device 21, and creates a 3D object using the acquired plurality of images. The 3D object creation unit 71 outputs the created 3D object to the 3D object registration unit 72.

[0072] The 3D object registration unit 72 registers the 3D object created by the 3D object creation unit 71 in the 3D object DB 73 .

[0073] The 3D object DB 73 registers the 3D objects created by the 3D object creation unit 71 .

[0074] <Functional Configuration of AR Server> FIG. 8 is a block diagram showing an example of the functional configuration of the AR server.

[0075] Each function in FIG. 8 is realized by the CPU 42 executing a program stored in the data storage unit 43 or the like.

[0076] The AR server 23 in FIG. 8 is configured to include an image receiving unit 81 , a correlation unit 82 , a device position and orientation estimation unit 83 , an AR content orientation estimation unit 84 , an AR content providing unit 85 , and an AR content DB 86 .

[0077] The image receiving unit 81 receives an image of the image data transmitted from the user terminal 24 as a query image, and outputs the received query image to the associating unit 82 .

[0078] The association unit 82 uses the query image to search the VPS map DB1 of the VPS server 22, associates the query image with the VPS map, and outputs information about the VPS map associated with the query image to the device position and orientation estimation unit 83.

[0079] The device position and orientation estimation unit 83 estimates the position and orientation of the device (user terminal 24) based on the VPS map associated with the query image, and outputs information indicating the estimated position and orientation of the user terminal 24 to the AR content orientation estimation unit 84.

[0080] The AR content attitude estimation unit 84 estimates the attitude of the AR content registered in the AR content DB 86 based on the device position (attitude) supplied from the device position and attitude estimation unit 83 .

[0081] The AR content providing unit 85 provides (transmits) the AR content in the posture estimated by the AR content posture estimating unit 84 to the user terminal 24 .

[0082] <3. Operation> <VPS Server Pre-Creation Process> FIG. 9 is a flowchart illustrating the VPS server 22 pre-creation process.

[0083] The image acquisition device 21 acquires a plurality of image data of an actual building or the like in order to generate a VPS model, and transmits the acquired image data to the VPS server 22 .

[0084] In step S11 , the 3D model map creation unit 61 receives a plurality of images of image data transmitted from the image acquisition device 21 .

[0085] In step S12, the 3D model map creation unit 61 creates a 3D model using the images of the plurality of image data received in step S11 as key frames.

[0086] In step S13, the 3D model map creation unit 61 creates a 3D map using the multiple key frames and the 3D model, and creates a VPS map consisting of the 3D model and key frame information. The 3D model map creation unit 61 outputs the VPS map to the 3D model map registration unit 62.

[0087] In step S14, the 3D model map registration unit 62 registers a VPS map consisting of the 3D model and key frame information in the VPS map DB1.

[0088] <Editing and Updating Process of VPS Server> FIG. 10 is a flowchart illustrating the editing and updating process of the VPS server 22. As shown in FIG.

[0089] In preparation for the upcoming AR event, 3D models of buildings and other objects that will be the subject of the AR event have been created as described above with reference to Figure 9, and have been converted into 3D maps in advance, with the VPS maps being registered in the VPS map DB1.

[0090] FIG. 10 illustrates an example in which, when a signboard on a building is subsequently actually replaced with a signboard for an AR event, a 3D object of the signboard for the AR event is created and replaced in the virtual space.

[0091] First, the 3D object creation unit 71 acquires a plurality of images of image data transmitted from the image acquisition device 21, and creates a 3D object of a signboard for the AR event using the acquired plurality of images. Note that the 3D object may be created in a virtual space without using image data. The 3D object registration unit 73 registers the 3D object created by the 3D object creation unit 71 in the 3D object DB 73.

[0092] Through the above processing, a 3D object of a signboard for an AR event is registered in the 3D object DB 73.

[0093] In step S31, the 3D model editing unit 64 reads out the 3D model of the VPS map registered in the VPS map DB1 into the virtual space, and performs editing processing to place the 3D object created by the 3D object creation unit 71 on the 3D model.

[0094] That is, the 3D model editing unit 64 reads out the 3D model of the VPS map registered in the VPS map DB1 into the virtual space, identifies the area to be edited on the 3D model, identifies a 3D object such as a signboard created by the 3D object creation unit 71 as the editing object, and places and edits the editing object in the editing area. Note that the method for identifying the editing area may be to identify an area designated by the creator using an operation unit (not shown) of the VPS server 22, or to identify an area designated by text information, etc.

[0095] In step S32, the 3D model editing unit 64 projects a 3D object from the edited 3D model onto the key frame to generate a projected object image.

[0096] In step S33, the 3D model editing unit 64 generates an updated key frame by merging the key frame of the VPS map registered in the VPS map DB 1 with the projected object image. Details of this merging process will be described later with reference to Fig. 11. The 3D model editing unit 64 outputs the edited 3D model and the updated key frame to the VPS map editing unit 65.

[0097] In step S34, the VPS map editing unit 65 reconstructs the VPS map by generating a 3D map based on the updated key frames.

[0098] In step S35, the VPS map editing unit 65 updates the VPS map in the VPS map DB1 with the reconfigured VPS map.

[0099] As described above, a 3D model is created in advance, and 3D objects are created for only the parts that have actually changed, and the VPS map is reconstructed and updated using these. This allows the VPS map to be easily updated without spending time and effort when there is little time for an AR event approaching.

[0100] <Merge Processing of VPS Server> FIG. 11 is a flowchart illustrating an example of the merge processing in step S33 of FIG.

[0101] In step S51, the 3D model editing unit 64 corrects the position of the projected object image using a key frame, which is an actual image, as a reference.

[0102] In step S52, the 3D model editing unit 64 uses the key frame as a mask to merge the key frame with the projected object image to generate an updated key frame. That is, in the updated key frame, the projected object image is used for the part of the edited object (e.g., a signboard), and the key frame is used for the part other than the edited object.

[0103] In step S53, the 3D model editing unit 64 corrects the shape and / or lighting of the edited object portion in the updated key frame. This completes an updated key frame in which only the edited object portion has been updated. By correcting the shape and / or lighting, it is possible to fine-tune the appearance of the edited object's shape and lighting in the projected object image so that it appears appropriate, for example, by matching it to the actual appearance.

[0104] The merging process of FIG. 11 may be performed automatically by a generative AI (generative AI) by inputting two images, a key frame and a projected object image, into the AI. At this time, editing content (text information) may be input and supplemented. In this case, the editing target area of ​​the key frame and the editing object are identified and editing is performed according to the text information.

[0105] <AR Content Providing Process by AR Server> FIG. 12 is a flowchart illustrating the AR content providing process by the AR server 23. As shown in FIG.

[0106] The user terminal 24 acquires image data of the position that is the target of the AR event in real space, and transmits the acquired image data to the AR server 23 .

[0107] In step S71, the image receiving unit 81 receives an image of the image data transmitted from the user terminal 24 as a query image, and outputs the received query image to the associating unit 82.

[0108] In step S72, the matching unit 82 uses the query image to search the VPS map DB1 of the VPS server 22, matches the query image with the VPS map, and outputs information about the VPS map associated with the query image to the device position and orientation estimation unit 83.

[0109] In step S73, the device position and orientation estimation unit 83 estimates the position and orientation of the device (user terminal 24) based on the VPS map associated with the query image, and outputs information indicating the estimated position and orientation of the user terminal 24 to the AR content orientation estimation unit 84.

[0110] In step S74, the AR content posture estimation unit 84 estimates the posture of the AR content registered in the AR content DB 86 based on the estimated device position (posture), and determines how the AR content should be displayed.

[0111] In step S75 , the AR content providing unit 85 provides (transmits) the AR content in the posture estimated by the AR content posture estimating unit 84 to the user terminal 24 .

[0112] 4. Modifications Note that, in the above explanation, an example of editing in which a sign is replaced with a new sign has been described, but editing may also be performed in which a building under construction is replaced with a completed building, or in which trees and flowers are placed around a building (in an area that was previously empty). Furthermore, editing may also be performed in which unilluminated trees are replaced with illuminated trees, or in which fresh green trees are replaced with trees with autumn leaves.

[0113] Furthermore, for example, by performing an editing process in which lighting is re-applied in a virtual space to a 3D model created based on an image captured during the day, it is possible to easily create key frames for, for example, night scenes.

[0114] Furthermore, when creating the original 3D model, acquiring the physical property values ​​(material values) of the object in the image that will become the 3D model at the same time allows for more precise editing. Physical property values ​​are values ​​that indicate, for example, whether the object is transparent glass, reflective like metal, or like paper or wood.

[0115] In the above description, an example of replacing a 3D object (e.g., a sign) has been described as an example of editing a 3D model, but it is also possible to add or delete a 3D object. For example, a mask area can be automatically set on a keyframe, and a portion of the 3D model can be deleted based on the mask area.

[0116] The above-described method for replacing a 3D object is merely an example and is not limiting. For example, a 3D object may be placed in a virtual space, and machine learning such as deep learning may be performed using a generative model that inputs the foreground and background, thereby replacing the 3D object and updating the VPS map.

[0117] Furthermore, in the above explanation, an example has been described in which the generation AI performs the merge process of Fig. 11, but for example, by inputting key frames, replacement 3D objects (if necessary), and editing content of the 3D model (text information) to the generation AI, the generation AI may automatically perform editing processes (replacement, addition, deletion) of the 3D model according to the text information, and even update the VPS map. In other words, the editing update process of Fig. 10 may be performed by the generation AI.

[0118] An example of text information used by the above-described generation AI when performing a merging process or editing process for a 3D model is text information such as "The signboard has been removed." A region to be deleted is identified and edited according to the text information. Furthermore, by inputting text information that combines physical property values ​​acquired from the above-described keyframe information, such as "Treat the placed 3D object as a tree," the physical property values ​​can be used in the editing process.

[0119] More ultimately, by inputting key frames and editing objects into the editing AI without entering any text information, the editing target area of ​​the key frame and the editing object can be identified, and the replacement process can be performed automatically by the editing AI.

[0120] <Re-editing and re-projection of 3D model> FIG. 13 is a diagram showing an example of re-editing and re-projection of a 3D model.

[0121] For example, from a 3D model in which object NS1 is positioned and reconstructed, object NS1 is projected from position P1 to a key frame at position P3, and the merged results are generated as an updated key frame 101 at position P1, an updated key frame 102 at position P2, and an updated key frame 103 at position P3.

[0122] When the projection results of the 3D model are merged with the key frames, the consistency of the image is lost, so for example, the color of the object NS1 may differ from each other between the updated key frames 101 and 103.

[0123] Therefore, the 3D model editing unit 64 compares the updated key frame 101 with the updated key frame 103, checks the consistency as a 3D object, re-edits the object NS1 in the 3D model to ensure consistency, and projects the re-edited object NS2 from the position P1 to the key frame at the position P3.

[0124] As described above, by repeating the process of recreating a 3D model based on the editing results and reprojecting it, it is possible to create a more consistent future 3D model and map information. The process in Figure 13 can also be automated using generation AI, etc.

[0125] <Updating of Update Key Frames in Real Images> FIG. 14 is a flowchart illustrating the update process of update key frames in real images.

[0126] For example, there is an updated keyframe that is created by editing a 3D model in a virtual space, projecting it onto a keyframe, and merging it with the keyframe, without using an image captured in real space.

[0127] The processing of Figure 14 begins when an image captured in real space is transmitted from the image acquisition device 21 at the time when the real-world situation catches up with the situation of the VPS map edited in virtual space as described above.

[0128] In step S101, the 3D model editing unit 64 aligns the real image transmitted from the image acquisition device 21 with the updated key frame registered in the VPS map DB1.

[0129] In step S102, the 3D model editing unit 64 extracts the difference between the real image and the updated key frame.

[0130] In step S103, the 3D model editing unit 64 updates the update key frame with the extracted difference. Thereafter, the VPS map editing unit 65 generates a 3D map based on the update key frame and updates the VPS map in the VPS map DB1.

[0131] As described above, when the situation in the real space catches up with the situation edited in the virtual space, it is possible to extract the difference and update it again. Also, the process in FIG. 14 can be automated using generation AI or the like.

[0132] In the above description, an example has been described in which a VPS map is constructed in advance on a VPS server and edited, but the present technology may also be used when a user constructs a VPS map in advance and edits it. In addition to providing an AR event using AR content, the present technology can also be applied to various other technologies, such as estimating the position of a camera in a studio in virtual production and autonomous movement of a robot.

[0133] <5. Other> <Effects of the Present Technology> As described above, in the present technology, an area to be edited in a 3D model in a virtual space is edited, and the 3D map is updated based on the 3D model in which the area to be edited has been edited.

[0134] This allows the 3D map to be edited easily and accurately.

[0135] <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware, or into a general-purpose personal computer, etc.

[0136] FIG. 15 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0137] The CPU 301 , ROM 302 , and RAM 303 are connected to one another via a bus 304 .

[0138] An input / output interface 305 is also connected to the bus 304. An input unit 306 including a keyboard, a mouse, etc., and an output unit 307 including a display, a speaker, etc. are connected to the input / output interface 305. In addition, a storage unit 308 including a hard disk, a nonvolatile memory, etc., a communication unit 309 including a network interface, etc., and a drive 310 that drives removable media 311 are also connected to the input / output interface 305.

[0139] In a computer configured as described above, the CPU 301 performs the above-described series of processes by, for example, loading a program stored in the storage unit 308 into the RAM 303 via the input / output interface 305 and the bus 304 and executing the program.

[0140] The program executed by the CPU 301 is provided, for example, by being recorded on a removable medium 311 or via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and is installed in the storage unit 308 .

[0141] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0142] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0143] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0144] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0145] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0146] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0147] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0148] <Examples of Combinations of Configurations> The present technology may also be configured as follows. (1) An information processing device comprising: a 3D model editing unit that edits an area to be edited in a 3D model in a virtual space; and a map update unit that updates a 3D map based on the 3D model in which the area to be edited has been edited. (2) The information processing device described in (1), wherein the map update unit updates the 3D map using the edited 3D model and an updated image that has been updated based on the 3D model. (3) The information processing device described in (2), wherein the 3D model editing unit generates a projected image by projecting the edited 3D model, and replaces the original image with the updated image generated by merging an original image and the projected image to create the 3D model, thereby updating the original image with the updated image. (4) The information processing device described in (3), wherein the 3D model editing unit edits the 3D model using an editing object created in the virtual space. (5) The information processing device according to (4), wherein the merging process includes at least merging an image of the editing object in the projected image with an image of the editing object in the original image other than the editing target area. (6) The information processing device according to (4) or (5), wherein the merging process includes correcting a shape and / or lighting of the editing object in the projected image based on the original image. (7) The information processing device according to any of (3) to (6), wherein the merging process includes aligning the projected image with the original image as a reference. (8) The information processing device according to any of (3) to (7), wherein the 3D map is configured to include the 3D model, the original image, and information acquired from the 3D model based on the original image. (9) The information processing device according to (8), wherein the information is configured to include information on the entire image or a part of the image encoded to establish correspondence between images, 3D measurement results related to the image, and the position and orientation of an imaging device when the original image was captured. (10) The information processing device according to any one of (3) to (9), wherein the 3D model editing unit is configured by a generation AI.(11) The information processing device according to any one of (4) to (10), wherein the 3D model editing unit specifies the editing target area and the editing object and edits the 3D model. (12) The information processing device according to any one of (4) to (11), further comprising an object creation unit that creates the editing object. (13) The information processing device according to any one of (2) to (12), wherein the 3D model editing unit re-edits the 3D model using a plurality of the update images. (14) The information processing device according to any one of (2) to (13), wherein the 3D model editing unit further updates the update image using a captured image captured in real space and the update image. (15) An information processing method, wherein an information processing device edits an edit target area in a 3D model in a virtual space, and updates a 3D map based on the 3D model in which the edit target area has been edited. (16) A program that causes a computer to function as a 3D model editing unit that edits an area to be edited in a 3D model in a virtual space, and a map updating unit that updates a 3D map based on the 3D model in which the area to be edited has been edited.

[0149] 1 VPS map DB, 11 AR provision system, 21 Data acquisition device, 22 VPS server, 23 AR server, 24 User terminal, 25 Network, 31 Liquid crystal display panel, 22 Backlight, 31 Imaging unit, 32 Display unit, 33 Communication unit, 41 Communication unit, 42 CPU, 43 Data storage unit, 44 Graphic display unit, 51 3D model map creation and editing unit, 52 3D model creation unit, 61 3D model map creation unit, 62 3D model map registration unit, 63 3D model map display unit, 64 3D model editing unit, 65 VPS map editing unit, 66 3D model DB, 71 3D object creation unit, 72 3D object registration unit, 73 3D object DB, 81 Image receiving unit, 82 Corresponding unit, 83: Device position and orientation estimation unit, 84: AR content orientation estimation unit, 85: AR content transmission unit, 86: AR content DB

Claims

1. An information processing device comprising: a 3D model editing unit that edits an area to be edited in a 3D model in a virtual space; and a map updating unit that updates a 3D map based on the 3D model in which the area to be edited has been edited.

2. The information processing device according to claim 1, wherein the map update unit updates the 3D map using the edited 3D model and an update image updated based on the 3D model.

3. The information processing device according to claim 2, wherein the 3D model editing unit generates a projection image by projecting the edited 3D model, and replaces the projection image with the updated image generated by merging the original image and the projected image to create the 3D model, thereby updating the original image to the updated image.

4. The information processing device according to claim 3, wherein the 3D model editing unit edits the 3D model using an editing object created in the virtual space.

5. The information processing device according to claim 4, wherein the merging process includes at least merging an image of the editing object in the projected image with an image of the original image other than the region to be edited.

6. The information processing device according to claim 4, wherein the merging process includes correcting a shape and / or lighting of the editing object in the projected image based on the original image.

7. The information processing device according to claim 3, wherein the merging process includes aligning the projected image with respect to the original image.

8. The information processing device according to claim 3, wherein the 3D map is configured to include the 3D model, the original image, and information obtained from the 3D model based on the original image.

9. The information processing device according to claim 8, wherein the information is configured to include information on the entire image or a portion of the image that has been encoded to establish correspondence between images, 3D surveying results related to the image, and the position and orientation of an imaging device when the original image was captured.

10. The information processing device according to claim 3, wherein the 3D model editing unit is configured by a generating AI.

11. The information processing device according to claim 4, wherein the 3D model editing unit specifies the region to be edited and the editing object, and edits the 3D model.

12. The information processing device according to claim 4, further comprising an object creation unit that creates the edit object.

13. The information processing device according to claim 2, wherein the 3D model editing unit re-edits the 3D model using a plurality of the updated images.

14. The information processing device according to claim 2, wherein the 3D model editing unit further updates the update image by using a captured image captured in real space and the update image.

15. An information processing method, in which an information processing device edits an area to be edited in a 3D model in a virtual space, and updates a 3D map based on the 3D model in which the area to be edited has been edited.

16. A program causing a computer to function as a 3D model editing unit that edits an area to be edited in a 3D model in a virtual space, and a map updating unit that updates a 3D map based on the 3D model in which the area to be edited has been edited.

Images