Scene modeling method, device and equipment, computer readable storage medium and product
By detecting and rendering feature points in videos and using loss functions to train the scene model, the problem of complex and poor performance of scene modeling algorithms in the prior art is solved, and simple, fast and high-quality three-dimensional model generation is achieved.
Patent Information
- Application Number
- CN202411099425.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-12-13
AI Technical Summary
The existing scene modeling methods have complex algorithms, and the generated three-dimensional model has poor effect, requires manual modification, and poor viewing from a free perspective.
By obtaining multiple original 2D pictures in the target video, detecting feature points, calculating camera poses, rendering 3D feature points, and using loss functions to train the scene model to achieve scene modeling.
The modeling process is simplified, the modeling speed and the quality of synthesis of new perspectives are improved, and the realism and reliability of the generated three-dimensional model is improved.
Smart Images
Figure CN119131236A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of communication technology, and in particular to a scene modeling method, device, equipment, computer-readable storage medium and product. Background Art
[0002] With current technology, if a scene needs to be viewed from a free perspective, the entire scene needs to be manually modeled or a wireless mesh needs to be reconstructed using a three-dimensional reconstruction algorithm, and the texture needs to be attached to the mesh surface to form a complete three-dimensional model. A virtual camera can then be placed at any position to view the scene, and finally the 3D range of the virtual camera's perspective is rendered to obtain a picture from that perspective.
[0003] The existing scene modeling method is based on geometry-based multi-view Figure 3 3D reconstruction: First, take multiple sets of pictures of the object or scene from multiple perspectives, then perform feature point detection and matching on the pictures to restore the camera pose, then perform dense matching to obtain a depth map and fuse multiple depth maps, then use Poisson reconstruction and other algorithms to obtain a 3D model, and finally use texture mapping to map the 3D model to obtain a complete and textured model. The algorithms involved in each node in the process of this solution are very complex and have a very high threshold. In real applications, the generated 3D models often have many bumps, lack of light and shadow, and lack of realism due to problems such as inaccurate camera pose and depth map estimation, which require manual subsequent modification and often have poor results when viewed from a free perspective.
[0004] When modeling is completed, free perspective generation can be achieved, but it has major defects, as follows:
[0005] The algorithms involved in each node in the process of this solution are very complex and have extremely high barriers to entry. At the same time, the final generated three-dimensional model is not necessarily optimal and often requires manual modification, resulting in the quality of the final synthesized image from a new perspective not necessarily being true and reliable. Summary of the invention
[0006] The purpose of the present application is to provide a scene modeling method, device, equipment, computer-readable storage medium and product to solve the problem that the existing scene modeling methods involve complex algorithms and poor results of the generated three-dimensional models.
[0007] In order to solve the above problems, the present application provides a scene modeling method, comprising:
[0008] Acquire multiple original 2D images in a target video; the target video is at least one video containing all angles of the target scene;
[0009] Detecting 2D feature points of each of the original 2D images, and determining a matching image of each of the original 2D images;
[0010] For each of the original 2D images and the matching images thereof, 2D feature point matching and posture calculation are performed to obtain the camera posture corresponding to each of the original 2D images and the 3D feature points corresponding to the matching 2D feature points on each of the original 2D images;
[0011] Rendering the 3D feature points corresponding to each of the original 2D images respectively to obtain a rendered 2D image corresponding to each of the original 2D images;
[0012] A similarity calculation is performed between the rendered 2D image and the corresponding original 2D image to obtain a loss function for training, thereby obtaining a scene model of the scene.
[0013] The detecting of the 2D feature points of each of the original 2D images and determining the matching images of each of the original 2D images includes:
[0014] Detecting 2D feature points of each of the original 2D images, and encoding the original 2D images into vectors;
[0015] The vector of each original 2D picture is respectively similar to the vectors of other original 2D pictures to determine a matching picture of each original 2D picture.
[0016] Among them, for each of the original 2D pictures and the matching pictures thereof, 2D feature point matching and posture calculation are performed to obtain the camera posture corresponding to each of the original 2D pictures and the 3D feature points corresponding to the matching 2D feature points on each of the original 2D pictures, including:
[0017] For each of the original 2D images and the matching images thereof, matching 2D feature points in the images is performed to obtain matching 2D feature points on each of the original 2D images;
[0018] Analyze the matching 2D feature points on each of the original 2D images by a triangulation method to obtain the original camera pose corresponding to each of the original 2D images;
[0019] The factor graph is used to restore the original camera pose corresponding to each of the original 2D images to obtain the camera pose corresponding to each of the original 2D images and the 3D feature points corresponding to the matching 2D feature points on each of the original 2D images.
[0020] For each of the original 2D images and the matching images thereof, matching of 2D feature points in the images is performed to obtain matching 2D feature points on each of the original 2D images, including:
[0021] For each of the original 2D images and its matching images, the LightGlue algorithm is used to match the 2D feature points in the images to obtain matching 2D feature points on each of the original 2D images.
[0022] The 3D feature points corresponding to each of the original 2D images are rendered respectively to obtain a rendered 2D image corresponding to each of the original 2D images, including:
[0023] Creating a set of three-dimensional Gaussian distributions according to the 3D feature points corresponding to each of the original 2D images;
[0024] Each group of the three-dimensional Gaussian distribution is rendered separately in the image space to obtain a rendered 2D picture corresponding to each of the original 2D pictures.
[0025] The similarity between the rendered 2D image and the corresponding original 2D image is calculated to obtain a loss function for training to obtain a scene model of the scene, including:
[0026] The parameters of the scene model are iteratively updated by minimizing the loss function to perform model training; when the similarity between the rendered 2D image and the original 2D image is higher than a preset value, the model training is completed to obtain the scene model of the scene.
[0027] Wherein, the method further comprises:
[0028] Get any pose parameters of the camera;
[0029] The arbitrary posture parameters are input into the scene model to obtain a scene perspective corresponding to the arbitrary posture parameters synthesized by the scene model.
[0030] An embodiment of the present application also provides a scene modeling device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor implements the scene modeling method as described above when executing the program.
[0031] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the scene modeling method as described above are implemented.
[0032] An embodiment of the present application also provides a computer program product, including computer instructions, which implement the steps of the scene modeling method described above when executed by a processor.
[0033] The above technical solution of the present application has at least the following beneficial effects:
[0034] In the scene modeling method, device, equipment, computer-readable storage medium and product of the embodiments of the present application, scene modeling can be completed by using a target video that can cover all angles of the scene, and images of any perspective can be freely output; the modeling method has a simple process, a fast modeling speed, and a good quality effect of synthesizing new perspectives. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 A flowchart showing the steps of the scene modeling method provided in an embodiment of the present application;
[0036] Figure 2 A schematic diagram showing the principle of camera pose estimation in the scene modeling method provided in an embodiment of the present application;
[0037] Figure 3 An example diagram showing camera poses and 3D feature points in the scene modeling method provided in an embodiment of the present application;
[0038] Figure 4 A schematic diagram showing the principle of three-dimensional Gaussian modeling of a scene modeling method provided in an embodiment of the present application;
[0039] Figure 5 A schematic diagram showing the structure of a scene modeling device provided in an embodiment of the present application;
[0040] Figure 6 A schematic diagram showing the structure of a scene modeling device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0041] In order to make the technical problems, technical solutions and advantages to be solved by the present application clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.
[0042] like Figure 1 As shown, the embodiment of the present application provides a scene modeling method, including:
[0043] Step 101, obtaining a plurality of original 2D images in a target video; the target video is at least one video containing all angles of the target scene;
[0044] Optionally, the target video is one or more videos obtained by a user using a mobile phone or other handheld camera device to take slow-motion shots of a scene in all directions. For example, the acquisition frame rate is generally 30fps or 60fps, and the resolution is 1080P or 4K. In which, all angles of the scene need to be captured and covered, and the obtained target video needs to cover all angles of the scene.
[0045] Step 102, detecting 2D feature points of each of the original 2D images, and determining a matching image of each of the original 2D images;
[0046] Step 103, performing 2D feature point matching and pose calculation for each of the original 2D images and the matching images thereof, to obtain a camera pose corresponding to each of the original 2D images and a 3D feature point corresponding to the matching 2D feature point on each of the original 2D images;
[0047] Step 104, rendering the 3D feature points corresponding to each of the original 2D pictures respectively to obtain a rendered 2D picture corresponding to each of the original 2D pictures;
[0048] Step 105 , performing similarity calculation between the rendered 2D image and the corresponding original 2D image to obtain a loss function for training, thereby obtaining a scene model of the scene.
[0049] In the embodiment of the present application, the scene modeling can be completed through the target video shot by the user's handheld camera or drone, and images of any perspective can be freely output; the modeling method has a simple process, fast modeling speed, and good quality of new perspective synthesis. After the scene modeling is completed, the model can be imported into an electronic device or other platform for editing, and digital humans or other 3D objects can also be placed in the scene to quickly build content.
[0050] In an optional embodiment, step 101 includes:
[0051] Performing frame cutting processing on the target video to obtain multiple 2D pictures;
[0052] An image quality evaluation algorithm is used to screen multiple 2D images to obtain multiple 2D original images.
[0053] Optionally, the frame rate of the frame capture processing is determined by the complexity of the scene; for example, if the scene is complex, 10 to 20 frames are taken, and if the scene is relatively simple, 1 to 10 frames are taken; this parameter will also affect the subsequent modeling speed. The higher the frame rate, the slower the modeling speed, and vice versa.
[0054] Optionally, the present application uses a neural radiation field-based algorithm to model the scene with high precision, and has extremely high requirements on the quality of the input image. Therefore, it is very important to eliminate images with poor quality problems such as blur and high noise. The image screening algorithm of the embodiment of the present application uses an image quality evaluation algorithm to calculate the distortion rate of the input image, and screens multiple 2D images according to the distortion rate to obtain multiple 2D original images. This type of algorithm can automatically filter out low-quality images such as damaged, blurred, and noisy images to ensure that all images obtained are undamaged, clear, and low-noise. For example, the image quality evaluation algorithm includes: one or more of the Blind / Referenceless Image Spatial QUalityEvaluator (BRISQUE) algorithm, the Naturalness Image QualityEvaluator (NIQE) algorithm, and the Blind Image Quality Index (BIQI) algorithm.
[0055] In at least one embodiment of the present application, step 102 includes:
[0056] Detecting 2D feature points of each of the original 2D images, and encoding the original 2D images into vectors;
[0057] The vector of each original 2D picture is respectively similar to the vectors of other original 2D pictures to determine a matching picture of each original 2D picture.
[0058] Accordingly, step 103 includes:
[0059] For each of the original 2D images and the matching images thereof, matching 2D feature points in the images is performed to obtain matching 2D feature points on each of the original 2D images;
[0060] Analyze the matching 2D feature points on each of the original 2D images by a triangulation method to obtain the original camera pose corresponding to each of the original 2D images;
[0061] The factor graph is used to restore the original camera pose corresponding to each of the original 2D images to obtain the camera pose corresponding to each of the original 2D images and the 3D feature points corresponding to the matching 2D feature points on each of the original 2D images.
[0062] Optionally, for each of the original 2D images and the matching images thereof, matching of 2D feature points within the images is performed to obtain matching 2D feature points on each of the original 2D images, including:
[0063] For each of the original 2D images and its matching images, the LightGlue algorithm is used to match the 2D feature points in the images to obtain matching 2D feature points on each of the original 2D images.
[0064] The embodiment of the present application uses the SuperPoint+scene image matching+LightGlue solution to achieve better image feature point detection performance. When faced with sparse poses, weak textures, and reflective and translucent scenes, the algorithm is still relatively robust and has higher matching efficiency. In addition, a factor graph algorithm is used to optimize the camera pose error, thereby more accurately restoring the camera pose and obtaining more accurate sparse 3D point positions.
[0065] For example, Figure 2 As shown, the process of camera pose recovery and 3D feature point determination in the embodiment of the present application includes: inputting multiple 2D images, and obtaining the camera pose and sparse 3D feature points through 2D feature point detection, scene image encoding and matching, 2D feature point matching and pose estimation. The details are as follows:
[0066] a. 2D feature point detection: that is, detecting the 2D feature points of each input 2D image. For example, using the Superpoint (deep learning feature point detection) feature point detection algorithm based on deep learning. This feature point detection algorithm is based on a deep learning model, and most of the training data used are marked as the endpoints of the graphics. Therefore, it is more robust when facing weak texture scenes and can detect more feature points. The implementation method is to input a 2D image, and the algorithm can detect more feature points of the image than other algorithms, and the output is the representation vector of each feature point.
[0067] b. Scene image encoding and matching: that is, according to step a, the representation vector of the 2D feature points of each image is detected, each image is encoded into a vector, and finally the vector is used to represent the image. After the image encoding is completed, the scene image matching can be quickly realized. For example, for image A, the vector (or representation vector) of image A is calculated with the representation vectors of all other original 2D images to quickly find other multiple original 2D images with similar scenes to the image A. For another example, the NxD-dimensional superpoint feature points extracted from each original 2D image are encoded as follows: first, a 1x1xK convolution is used to perform a convolution operation on it and a Softmax is performed to obtain an NxK feature vector, and then the NxD vector is multiplied by the NxK vector to obtain the KxD-dimensional representation vector of each image, thereby completing the encoding of each image, that is, each original 2D image can be represented by a KxD-dimensional vector. When matching between images, only the i-th image and other images are used to perform interpolation calculations between the representation vectors to quickly obtain the matching image of the image. Finally, the feature points in the image are matched using a feature point matching algorithm.
[0068] c. 2D feature point matching, this application uses the optimized version of the SuperGlue algorithm - the LightGlue algorithm. For the pictures matched in step b above, such as matching picture A to picture B / C / D, etc., 2D feature point matching is performed within the picture. LightGlue relies on a relative encoding that is more comparable between images and is added to each self-attention unit, which makes it easier to utilize positions and improves accuracy at a deeper level. In terms of the prediction head, SuperGlue predicts matches by using the Sinkhorn algorithm to solve the differentiable optimal transfer problem, while LightGlue decouples similarity and matchability. When the number of input 2D pictures is large, the LightGlue algorithm provided in the embodiment of the present application takes less time and has better performance.
[0069] d. Calculate the camera pose: Through steps a, b, and c above, we can obtain 2D feature points on each image that can be effectively matched with other images. Then we can use Figure 3 The triangulation method shown solves the position and posture of each camera, that is, the rotation matrix and translation matrix of the camera, and can also obtain the 3D position coordinates corresponding to the 2D feature points on the image. For example, in Figure 3, O1 and O2 represent the positions of the two cameras, the blue box represents the 2D image captured by the camera, and the points P1 and P2 represent the corresponding 2D feature point pairs detected and matched in steps a, b, and c, and the point P is the position of the 3D feature point corresponding to the 2D feature point.
[0070] However, the camera pose obtained by this method is not accurate. This proposal uses factor graphs to optimize pose recovery to minimize the error. The factor graph represents both pose and observation as variables.
[0071] After the above optimization, this step can obtain more accurate camera pose and sparse 3D point coordinates (i.e., 3D feature points).
[0072] In at least one embodiment of the present application, step 104 includes:
[0073] Creating a set of three-dimensional Gaussian distributions according to the 3D feature points corresponding to each of the original 2D images;
[0074] Each group of the three-dimensional Gaussian distribution is rendered separately in the image space to obtain a rendered 2D picture corresponding to each of the original 2D pictures.
[0075] For example, based on the 3D feature points (also called sparse 3D feature points) obtained by camera pose estimation, a set of three-dimensional Gaussian distributions is created, which is defined by position (mean), covariance matrix and opacity, as shown in the following formula:
[0076] Σ=RSS T R T
[0077] Where Σ is the covariance matrix, R is the rotation matrix, and S is the scaling matrix.
[0078] Optionally, the three-dimensional Gaussian distribution is rendered through the image space, and the rendered 2D picture is determined specifically as follows: a 2D picture is obtained by projecting through the image space (also referred to as a rendered 2D picture), or the three-dimensional Gaussian distribution is projected into the 2D space to obtain a 2D picture (also referred to as a rendered 2D picture).
[0079] The model training mentioned in the embodiment of the present application belongs to self-supervised training. Accordingly, step 105 includes:
[0080] The parameters of the scene model are iteratively updated by minimizing the loss function to perform model training;
[0081] When the similarity between the rendered 2D image and the original 2D image is higher than a preset value, the model training is completed and the scene model of the scene is obtained.
[0082] Optionally, the similarity between the rendered 2D image and the original 2D image is calculated to obtain a loss function, and then the parameters of each three-dimensional Gaussian distribution (also called the parameters of the scene model) are continuously and iteratively updated through reverse transmission. When the final loss function is small enough, that is, the rendered 2D image is very similar to the original 2D image, the entire training process is completed and the corresponding scene model is obtained.
[0083] For example, Figure 4 As shown, a three-dimensional Gaussian distribution is created based on the sparse 3D feature points, and the three-dimensional Gaussian distribution is projected into the 2D space to obtain a 2D image (i.e., a rendered 2D image). The loss function of the rendered 2D image and the original 2D image is calculated, and the three-dimensional Gaussian distribution parameters are iteratively updated. When the training is completed, the three-dimensional Gaussian distribution parameter model, i.e., the scene model, is output.
[0084] The above training strategy is to minimize the difference between the rendered image and the original input image, that is, to continuously iterate the model parameters by minimizing the loss function, so that the final rendered image is the best, thereby achieving the effect of self-supervision.
[0085] In the embodiment of the present application, the difference between the rendered 2D image and the original 2D image not only takes into account the difference in the images, but also takes into account the difference in the object categories in the scene and the difference in the edge quality of the rendering of each category.
[0086] Accordingly, the loss function provided in the embodiment of the present application includes the following parameters:
[0087] Color item;
[0088] Structural similarity index between rendered 2D image and original 2D image;
[0089] Loss of multiple categories of objects;
[0090] Edge detection loss within various object categories.
[0091] For example, the loss function in the embodiment of the present application is:
[0092]
[0093] in, is the color term; C k is the original 2D image; C′ k is the rendered 2D image; SSIM is the structural similarity index between the rendered 2D image and the original 2D image; L D-SSIM =1-L SSIM ; It is the category loss. Here, the semantic segmentation category loss is added to improve the model’s ability to generate main objects. It is the edge detection loss within the category, that is, here the canny edge detection algorithm (or other edge detection algorithms) is used to detect the edges of the original image and the rendered image that meet the 21 categories of object areas, and then the loss of the edge area (such as the 5x5 area of each edge point) is calculated. The effect of this loss is that the edges of the rendered image are often blurred. Using this loss can enhance the edge generation ability, that is, increase the high-frequency signal value. α, β, and γ are the weight values of the three major loss items, which are hyperparameters.
[0094] It should be noted that, in order to determine the various categories of objects in the loss function, the embodiment of the present application also provides a scene semantic segmentation step.
[0095] The embodiment of the present application combines the semantic segmentation of indoor scenes, and pre-trains the model to perform instance segmentation on the ScanNet dataset, which has 21 categories such as walls, tables, floors, beds, chairs, etc., which can largely meet the indoor scenes. At the same time, the BisenNet network trained with the ScanNet dataset has the ability to semantically segment these 21 categories of objects, and can segment these categories of objects in real scenes, and provide masks for these categories of objects.
[0096] The embodiment of the present application uses a 3D Gaussian distribution modeling solution, assisted by a semantic segmentation algorithm, and optimizes the loss function during training, so that the modeling of the main objects in the scene and the edges of the main objects (ie, high-frequency signals) are more prominent, and the final rendered image is also clearer.
[0097] Furthermore, in the embodiment of the present application, the method further includes:
[0098] Get any pose parameters of the camera;
[0099] The arbitrary posture parameters are input into the scene model to obtain a scene perspective corresponding to the arbitrary posture parameters synthesized by the scene model.
[0100] Optionally, after the model training is completed, the scene modeling is realized. At this time, only the arbitrary posture parameters of the camera need to be provided to realize the perspective synthesis under the camera posture of the scene, that is, the synthesis of completely free perspective. In addition, the obtained parameter model (i.e., the three-dimensional Gaussian distribution parameters of each 3D feature point) is imported into electronic equipment or platform software, so that effective editing of the scene can be realized, such as adding digital people to the scene, editing and modifying objects in the scene, etc., so as to quickly complete the scene construction and content generation, and can also be used in applications such as virtual reality and augmented reality.
[0101] In summary, in the embodiments of the present application, users do not need to use more expensive 3D acquisition equipment, and only need a mobile phone to complete data acquisition. By inputting the data into the entire process, scene modeling can be completed, and images of any perspective can be freely output.
[0102] like Figure 5 As shown, the embodiment of the present application also provides a scene modeling device, including:
[0103] The first acquisition module 501 is used to acquire multiple original 2D images in a target video; the target video is at least one video containing all angles of the target scene;
[0104] A first determining module 502, configured to detect 2D feature points of each of the original 2D images, and determine a matching image of each of the original 2D images;
[0105] The second acquisition module 503 is used to perform 2D feature point matching and pose calculation for each of the original 2D pictures and the matching pictures thereof, to obtain the camera pose corresponding to each of the original 2D pictures and the 3D feature points corresponding to the matching 2D feature points on each of the original 2D pictures;
[0106] The second determining module 504 is used to render the 3D feature points corresponding to each of the original 2D pictures respectively to obtain a rendered 2D picture corresponding to each of the original 2D pictures;
[0107] The model training module 505 is used to calculate the similarity between the rendered 2D image and the corresponding original 2D image to obtain a loss function for training, so as to obtain a scene model of the scene.
[0108] As an optional embodiment, the first determining module includes:
[0109] A first detection submodule, configured to detect 2D feature points of each of the original 2D images, and encode the original 2D images into vectors;
[0110] The first determination submodule is used to calculate the similarity between the vector of each original 2D picture and the vectors of other original 2D pictures respectively, so as to determine the matching picture of each original 2D picture.
[0111] As an optional embodiment, the second acquisition module includes:
[0112] The first submodule is used to match the 2D feature points in each original 2D picture and its matching picture, so as to obtain the matching 2D feature points on each original 2D picture;
[0113] The second submodule is used to analyze the matching 2D feature points on each of the original 2D pictures by a triangulation method to obtain the original camera pose corresponding to each of the original 2D pictures;
[0114] The third submodule is used to restore the original camera pose corresponding to each of the original 2D images using the factor graph to obtain the camera pose corresponding to each of the original 2D images and the 3D feature points corresponding to the matching 2D feature points on each of the original 2D images.
[0115] As an optional embodiment, the first submodule includes:
[0116] The first determination unit is used to match the 2D feature points in each of the original 2D images and the matching images thereof by using the LightGlue algorithm to obtain the matching 2D feature points on each of the original 2D images.
[0117] As an optional embodiment, the second determining module includes:
[0118] A fourth submodule is used to create a set of three-dimensional Gaussian distributions according to the 3D feature points corresponding to each of the original 2D images;
[0119] The fifth submodule is used to render each group of the three-dimensional Gaussian distributions respectively through the image space to obtain a rendered 2D picture corresponding to each of the original 2D pictures.
[0120] As an optional embodiment, the model training module includes:
[0121] The model training submodule is used to iteratively update the parameters of the scene model by minimizing the loss function to perform model training; when the similarity between the rendered 2D image and the original 2D image is higher than a preset value, the model training is completed to obtain the scene model of the scene.
[0122] As an optional embodiment, the device further includes:
[0123] Parameter acquisition module, used to obtain arbitrary posture parameters of the camera;
[0124] A synthesis module is used to input the arbitrary posture parameters into the scene model to obtain a scene perspective corresponding to the arbitrary posture parameters synthesized by the scene model.
[0125] In the embodiment of the present application, scene modeling can be completed by using a target video that can cover all angles of the scene, and images of any perspective can be freely output; the modeling method has a simple process, a fast modeling speed, and a good quality effect of synthesizing new perspectives.
[0126] It should be noted that the scene modeling device provided in the embodiment of the present application is a device capable of executing the above-mentioned scene modeling method. All embodiments of the above-mentioned scene modeling method are applicable to the device and can achieve the same or similar beneficial effects, which will not be repeated here.
[0127] like Figure 6 As shown, an embodiment of the present application also provides a scene modeling device, including a memory 610, a processor 600, and a computer program stored in the memory 610 and executable on the processor 600. When the processor 600 executes the program, each process in the scene modeling method embodiment as described above is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0128] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, each process in the above-mentioned scene modeling method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0129] The embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the various processes of the scene modeling method embodiment shown above and can achieve the same technical effect. To avoid repetition, they are not repeated here.
[0130] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.
[0131] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A device that specifies functions in one or more processes and / or one or more blocks.
[0132] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable storage medium produce a paper product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0133] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that the computer or other programmable device executes a series of operating steps to produce a computer-implemented process, thereby providing instructions executed on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0134] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A scene modeling method, characterized in that: include: Acquire multiple original 2D images in a target video; the target video is at least one video containing all angles of the target scene; Detecting 2D feature points of each of the original 2D images, and determining a matching image of each of the original 2D images; For each of the original 2D images and the matching images thereof, 2D feature point matching and posture calculation are performed to obtain the camera posture corresponding to each of the original 2D images and the 3D feature points corresponding to the matching 2D feature points on each of the original 2D images; Rendering the 3D feature points corresponding to each of the original 2D images respectively to obtain a rendered 2D image corresponding to each of the original 2D images; A similarity calculation is performed between the rendered 2D image and the corresponding original 2D image to obtain a loss function for training, thereby obtaining a scene model of the scene.
2. The method according to claim 1, characterized in that The detecting the 2D feature points of each of the original 2D pictures and determining the matching pictures of each of the original 2D pictures comprises: Detecting 2D feature points of each of the original 2D images, and encoding the original 2D images into vectors; The vector of each original 2D picture is respectively similar to the vectors of other original 2D pictures to determine a matching picture of each original 2D picture.
3. The method according to claim 1, characterized in that: For each of the original 2D images and its matching images, 2D feature point matching and posture calculation are performed to obtain the camera posture corresponding to each of the original 2D images and the 3D feature points corresponding to the matching 2D feature points on each of the original 2D images, including: For each of the original 2D images and the matching images thereof, matching 2D feature points in the images is performed to obtain matching 2D feature points on each of the original 2D images; Analyze the matching 2D feature points on each of the original 2D images by a triangulation method to obtain the original camera pose corresponding to each of the original 2D images; The factor graph is used to restore the original camera pose corresponding to each of the original 2D images to obtain the camera pose corresponding to each of the original 2D images and the 3D feature points corresponding to the matching 2D feature points on each of the original 2D images.
4. The method according to claim 3, characterized in that For each of the original 2D images and the matching images thereof, 2D feature points in the images are matched to obtain matching 2D feature points on each of the original 2D images, including: For each of the original 2D images and its matching images, the LightGlue algorithm is used to match the 2D feature points in the images to obtain matching 2D feature points on each of the original 2D images.
5. The method according to claim 1, characterized in that Rendering the 3D feature points corresponding to each of the original 2D images respectively to obtain a rendered 2D image corresponding to each of the original 2D images includes: Creating a set of three-dimensional Gaussian distributions according to the 3D feature points corresponding to each of the original 2D images; Each group of the three-dimensional Gaussian distribution is rendered separately in the image space to obtain a rendered 2D picture corresponding to each of the original 2D pictures.
6. The method according to claim 1, characterized in that The similarity between the rendered 2D image and the corresponding original 2D image is calculated to obtain a loss function for training to obtain a scene model of the scene, including: The parameters of the scene model are iteratively updated by minimizing the loss function to perform model training; when the similarity between the rendered 2D image and the original 2D image is higher than a preset value, the model training is completed to obtain the scene model of the scene.
7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Get any pose parameters of the camera; The arbitrary posture parameters are input into the scene model to obtain a scene perspective corresponding to the arbitrary posture parameters synthesized by the scene model.
8. A scene modeling device, comprising a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that: When the processor executes the program, the scene modeling method according to any one of claims 1 to 7 is implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the scene modeling method according to any one of claims 1 to 7 are implemented.
10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the scene modeling method according to any one of claims 1 to 7.