Method, device and equipment for generating new view images based on multi-spherical scene representation

Abstract

The application discloses a new view image generation method, device and equipment based on a multi-spherical surface scene expression, and the method comprises the following steps: firstly, collecting a scene image set; then, performing image pose estimation on all images in the scene image set to obtain pose information of all images; then, modeling a multi-spherical surface scene model according to the pose information of each image in the scene image set; and finally, generating a target image under an arbitrary new view according to the multi-spherical surface scene model. The application improves the image generation efficiency and image generation quality, and can be widely applied to the technical field of image processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a new view image generation method, device and equipment based on multi-spherical scene expression. BACKGROUND

[0002] New view generation is a long-standing problem in computer vision and computer graphics, which takes several images of a scene as input and outputs an image at an arbitrary view. In practical applications, efficient scene modeling and real-time photorealistic image rendering are two crucial parts.

[0003] Based on motion recovery structure or unstructured multi-view stereo geometry is a classic new view generation method, which first models the scene as a point cloud or a triangular patch, and then uses a physics-based rendering algorithm to generate a new view image. However, these methods separate the scene modeling and new view generation parts, which makes the final result suboptimal. The development of differentiable rendering makes the scene modeling and new view generation parts can be trained end-to-end. By unifying the scene modeling and new view generation parts into a differentiable framework, these methods can obtain more realistic images than classic methods.

[0004] Currently, the neural radiance field algorithm proposes to represent a three-dimensional scene as an implicit function parameterized by a multi-layer perceptron, and then uses differentiable voxel rendering technology to generate a new view. Although the neural radiance field algorithm and its subsequent work can generate high-quality images, these works consume a large amount of computing resources. For example, rendering an image with a resolution of 800x800 requires querying the multi-layer perceptron millions of times, which is very time-consuming. In order to solve this problem, many methods study how to reduce the number of queries or reduce the time of each query. In order to reduce the number of multi-layer perceptron queries, these methods propose to use a sphere tracing algorithm, a closed-form solution to the voxel rendering equation, and to use depth information. In order to reduce the time of each query, special data structures (such as three-dimensional grids, octrees, etc.) are used to store local features, and shallow multi-layer perceptrons are used to improve query speed. On the other hand, training a neural radiance field to fit a three-dimensional scene requires dozens of hours of convergence time. Although some scholars propose to use meta-learning to initialize in order to improve the convergence speed, this method has poor generalization in different scenes. In summary, inefficient rendering and slow convergence speed are two major shortcomings of the neural radiance field algorithm.

[0005] Unlike the neural radiation field method, the new perspective generation algorithm based on multi-plane images uses a set of mutually parallel planes at different depths to represent the three-dimensional scene, where each plane is an RGBA image (that is, each pixel of the image includes RGB colors and a transparency A). When generating a new perspective, this method performs a homography transformation on each plane to the target perspective, and then uses the transparency combination method to superimpose all parallel planes to obtain the final perspective. Although proxy networks (such as convolutional neural networks, multi-layer perceptrons, etc.) are required during training, these methods can achieve real-time effects during the rendering process. Although new perspective generation algorithms based on multi-plane images can achieve real-time rendering, they can only generate new perspective images at a smaller perspective relative to the reference image.

[0006] In summary, the following problems exist in the application field of new perspective generation:

[0007] 1. Long model training time. Most existing methods require training a model for each new scenario, and training a model takes dozens of hours to converge.

[0008] 2. Slow generation of new perspectives. After model training, generating new perspective images from the model requires multiple queries of the multi-layer perceptron, which consumes a large amount of computing resources. This results in slow generation of new perspective images. Generating an 800×800 resolution image takes about 30 seconds. Summary of the Invention

[0009] In view of this, embodiments of the present invention provide a method, apparatus, and device for generating new perspective images based on multi-spherical scene expression with high efficiency and high image generation quality.

[0010] An aspect of an embodiment of the present invention provides a method for generating a new perspective image based on multi-spherical scene expression, comprising:

[0011] Collecting a set of scene images;

[0012] Performing image pose estimation on all images in the scene image set to obtain pose information of all images;

[0013] Modeling a multi-spherical scene model based on the pose information of each image in the scene image set;

[0014] According to the multi-spherical scene model, a target image under any new viewing angle is generated.

[0015] Optionally, the collecting scene image set includes:

[0016] Shooting a scene video of the target scene with a camera according to a preset motion trajectory; or shooting a scene video of the target scene around the target object with a camera according to a motion trajectory moving around the target object;

[0017] According to a preset time interval, images are extracted frame by frame from the scene video to obtain a scene image set.

[0018] Optionally, performing image pose estimation on all images in the scene image set to obtain pose information of all images includes:

[0019] Extracting feature points of each image in the scene image set and a descriptor of each feature point;

[0020] According to the descriptors of each feature point, calculate the matching relationship between any two feature points;

[0021] Calculating the position relationship between the images based on the matching relationship between any two feature points;

[0022] The pose information of all images is globally optimized through bundle adjustment to obtain the pose information of each image, and then a set of pose images is obtained.

[0023] Optionally, modeling a multi-spherical scene model according to the pose information of each image in the scene image set includes:

[0024] Based on the imaging principle of the pinhole camera model, the position of the shooting camera is used as the starting point of the ray, and the direction pointing to the corresponding pixel position on the imaging plane is used as the ray direction to generate the ray corresponding to each pixel point in each image;

[0025] According to the ray corresponding to each pixel point, obtaining point attribute information of any sampling point on each ray; wherein the point attribute information includes the density and color of the sampling point;

[0026] Calculate the ray color value of each ray according to the point attribute information of each ray;

[0027] According to the ray color value of each ray and the image pixel value corresponding to each ray, the objective function is optimized, and a multi-sphere scene model is constructed according to the optimized objective function.

[0028] Optionally, obtaining point attribute information of any sampling point on each ray according to the ray corresponding to each pixel point includes:

[0029] Representing a three-dimensional scene as multiple spheres with the same center but different radii, and configuring the maximum range of the three-dimensional scene and the radius of each sphere; wherein each sphere is represented by a two-dimensional image; wherein each pixel in the two-dimensional image stores density and color spherical harmonic function coefficients; wherein the color spherical harmonic function is used to calculate the color value of each pixel;

[0030] uniformly sampling multiple sampling points from each ray, and determining the nearest neighbor point of each sampling point on the corresponding two spherical surfaces;

[0031] Obtaining point attribute information of the nearest neighbor point by linear interpolation;

[0032] Calculating the point attribute information of the sampling point according to the point attribute information of the nearest neighbor point;

[0033] The two-dimensional images are superimposed to form a three-dimensional voxel, and then the point attribute information of any sampling point on each ray is determined.

[0034] Optionally, in the step of calculating the ray color value of each ray according to the point attribute information of each ray, the calculation formula of the ray color value is:

[0035]

[0036] in, Represents the color value of ray r; T i Indicates the cumulative transmittance; σ i represents the density of sampling point i; δ i Represents the distance between adjacent sampling points; c i Indicates the color of sampling point i; N s Indicates the total number of sampling points.

[0037] Optionally, generating a target image at any new perspective based on the multi-spherical scene model includes:

[0038] Given the camera pose at the new perspective, calculate the ray corresponding to each pixel in the new perspective image;

[0039] Sampling is performed on the ray, obtaining the attributes of each sampling point on the ray, and then the color value of the ray is calculated;

[0040] Assign the color value of the ray to the corresponding pixel, and finally form an image under a new perspective.

[0041] Another aspect of the present invention provides a device for generating a new perspective image based on multi-spherical scene expression, comprising:

[0042] The first module is used to collect a set of scene images;

[0043] The second module is used to estimate the image pose of all images in the scene image set to obtain pose information of all images;

[0044] The third module is used to model a multi-spherical scene model based on the pose information of each image in the scene image set;

[0045] The fourth module is used to generate a target image at any new viewing angle based on the multi-spherical scene model.

[0046] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0047] The memory is used to store programs;

[0048] The processor executes the program to implement the method described above.

[0049] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0050] The present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the above method.

[0051] The embodiments of the present invention first collect a set of scene images; then perform image pose estimation on all images in the set to obtain pose information for all images; then, based on the pose information of each image in the set, a multi-spherical scene model is built; and finally, based on the multi-spherical scene model, a target image is generated from any new perspective. This invention improves image generation efficiency and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0053] Figure 1 The following is a flowchart of the overall steps provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0055] In order to solve the problems existing in the prior art, an embodiment of the present invention provides a method for generating a new perspective image based on multi-spherical scene representation, comprising:

[0056] Collecting a set of scene images;

[0057] Performing image pose estimation on all images in the scene image set to obtain pose information of all images;

[0058] Modeling a multi-spherical scene model based on the pose information of each image in the scene image set;

[0059] According to the multi-spherical scene model, a target image under any new viewing angle is generated.

[0060] Optionally, the collecting scene image set includes:

[0061] Shooting a scene video of the target scene with a camera according to a preset motion trajectory; or shooting a scene video of the target scene around the target object with a camera according to a motion trajectory moving around the target object;

[0062] According to a preset time interval, images are extracted frame by frame from the scene video to obtain a scene image set.

[0063] Optionally, performing image pose estimation on all images in the scene image set to obtain pose information of all images includes:

[0064] Extracting feature points of each image in the scene image set and a descriptor of each feature point;

[0065] According to the descriptors of each feature point, calculate the matching relationship between any two feature points;

[0066] Calculating the position relationship between the images based on the matching relationship between any two feature points;

[0067] The pose information of all images is globally optimized through bundle adjustment to obtain the pose information of each image, and then a set of pose images is obtained.

[0068] Optionally, modeling a multi-spherical scene model according to the pose information of each image in the scene image set includes:

[0069] Based on the imaging principle of the pinhole camera model, the position of the shooting camera is used as the starting point of the ray, and the direction pointing to the corresponding pixel position on the imaging plane is used as the ray direction to generate the ray corresponding to each pixel point in each image;

[0070] According to the ray corresponding to each pixel point, obtaining point attribute information of any sampling point on each ray; wherein the point attribute information includes the density and color of the sampling point;

[0071] Calculate the ray color value of each ray according to the point attribute information of each ray;

[0072] According to the ray color value of each ray and the image pixel value corresponding to each ray, the objective function is optimized, and a multi-sphere scene model is constructed according to the optimized objective function.

[0073] Optionally, obtaining point attribute information of any sampling point on each ray according to the ray corresponding to each pixel point includes:

[0074] Representing a three-dimensional scene as multiple spheres with the same center but different radii, and configuring the maximum range of the three-dimensional scene and the radius of each sphere; wherein each sphere is represented by a two-dimensional image; wherein each pixel in the two-dimensional image stores density and color spherical harmonic function coefficients; wherein the color spherical harmonic function is used to calculate the color value of each pixel;

[0075] uniformly sampling multiple sampling points from each ray, and determining the nearest neighbor point of each sampling point on the corresponding two spherical surfaces;

[0076] Obtaining point attribute information of the nearest neighbor point by linear interpolation;

[0077] Calculating the point attribute information of the sampling point according to the point attribute information of the nearest neighbor point;

[0078] The two-dimensional images are superimposed to form a three-dimensional voxel, and then the point attribute information of any sampling point on each ray is determined.

[0079] Optionally, in the step of calculating the ray color value of each ray according to the point attribute information of each ray, the calculation formula of the ray color value is:

[0080]

[0081] in, Represents the color value of ray r; T i Indicates the cumulative transmittance; σ i represents the density of sampling point i; δ i Represents the distance between adjacent sampling points; c iIndicates the color of sampling point i; N s Indicates the total number of sampling points.

[0082] Optionally, generating a target image at any new perspective based on the multi-spherical scene model includes:

[0083] Given the camera pose at the new perspective, calculate the ray corresponding to each pixel in the new perspective image;

[0084] Sampling is performed on the ray, obtaining the attributes of each sampling point on the ray, and then the color value of the ray is calculated;

[0085] Assign the color value of the ray to the corresponding pixel, and finally form an image under a new perspective.

[0086] Another aspect of the present invention provides a device for generating a new perspective image based on multi-spherical scene expression, comprising:

[0087] The first module is used to collect a set of scene images;

[0088] The second module is used to estimate the image pose of all images in the scene image set to obtain pose information of all images;

[0089] The third module is used to model a multi-spherical scene model based on the pose information of each image in the scene image set;

[0090] The fourth module is used to generate a target image at any new viewing angle based on the multi-spherical scene model.

[0091] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0092] The memory is used to store programs;

[0093] The processor executes the program to implement the method described above.

[0094] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0095] The present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the above method.

[0096] The specific implementation process of the present invention is described in detail below with reference to the accompanying drawings:

[0097] The present invention provides an efficient new perspective generation technology that uses multiple concentric spheres of different radii to represent three-dimensional scenes. This technology can complete model training convergence in about 20 minutes, which is more than 100 times faster than the existing neural radiation field method that takes dozens of hours to converge. This technology can achieve real-time (>30 frames per second) in new perspective generation, which is more than 1,000 times faster than the existing neural radiation field method that takes about 30 seconds to generate. This technology is comparable to existing methods in terms of the quality of new perspective generated images, and even surpasses existing methods.

[0098] Specifically, if Figure 1 As shown, the method of the present invention comprises the following steps:

[0099] 1. Collect scene image collection;

[0100] 2. Performing image pose estimation on all images in the scene image set to obtain pose information of all images;

[0101] 3. Modeling a multi-spherical scene model based on the pose information of each image in the scene image set;

[0102] 4. Generate a target image at any new perspective based on the multi-spherical scene model.

[0103] Specifically, step 1: scene image acquisition. The detailed steps are as follows:

[0104] Step 1: Use the camera to shoot the scene video. The camera's motion trajectory in the scene can be around the object, or you can set any motion trajectory.

[0105] Step 2: Extract each frame from the captured video. Since the perspective of consecutive frames varies little and the image content is similar, images are extracted at regular intervals. The resulting image set is used in subsequent steps.

[0106] Step 2: Image pose estimation. The specific steps are as follows:

[0107] Step 1: Use feature extraction algorithm (for example, SIFT algorithm) to extract feature points and their descriptors of all images in the image set.

[0108] Step 2: Using the descriptor, calculate the matching relationship between all feature points. Specifically, assuming that image I i and I j The feature point descriptors in are x i and x j , calculate the Euclidean distance between them:

[0109] D(x i ,x j )=||i x -x j ||

[0110] If the Euclidean distance D(x i ,x j ) is less than a given threshold T, then the two feature points match; otherwise, they do not match.

[0111] Step 3: Use the matching relationship to calculate the pose relationship between images. First, select two images from the image set as the initial image pair, and use them to match feature points to obtain the image pose and the three-dimensional space point position as the initialization model; then, gradually select an image from the remaining image set, and use the matching relationship between the image feature points and the three-dimensional space points to calculate the image pose. This process is called image registration; at the same time, use the registration relationship between the feature points in the image and the feature points of the registered image to calculate the position of the feature points in the three-dimensional space and add them to the three-dimensional model. This process is called triangulation; finally, since image registration and triangulation are performed independently, but their calculation results are highly correlated, in order to better optimize the camera pose, the bundle adjustment method is used for global optimization. The optimization objective function is as follows:

[0112]

[0113] Among them, X k is a point in three-dimensional space, x j is the corresponding projection point on the camera, π(·) is the camera projection function, M is the camera pose parameter, ρ j Used to reduce the weight of outliers.

[0114] Finally, we get the pose of each image, and we can get a pose image set by combining each image. Indicates that I i represents the i-th image, M i represents the pose matrix of the i-th image, and N represents the number of images.

[0115] Step 3: Multi-sphere scene modeling. This step is used to model a multi-sphere scene representation from the existing set of pose-based images:

[0116] Step 1: Ray generation. According to the imaging principle of the pinhole camera model, each pixel in the image corresponds to a ray. The starting point of the ray is the camera position, and the direction of the ray points to the corresponding pixel position on the imaging plane. Based on the image and its position, calculate the ray r(t) = o + td corresponding to all pixels, where o∈R 3 Indicates the starting point of the ray, d∈R 3Expressed as ray direction, t>0.

[0117] Step 2: Point attribute acquisition. The present invention represents the three-dimensional scene as N spheres with the same center and different radii, denoted as S = {S1, ..., S N}. Assume that the maximum range of the scene is r max , then the sphere S n The radius is set to Each sphere is represented by a two-dimensional image with a resolution of H×W. The pixel point (i, j) of the two-dimensional image corresponds to the coordinates (x, y, z) on the three-dimensional sphere. The relationship between them is as follows:

[0118]

[0119] in,

[0120] Each pixel in a two-dimensional image stores two attributes: density σ and color spherical harmonic function coefficient H. Using spherical harmonics to represent color is to deal with objects with high-reflectivity surfaces, because these surfaces will show different colors when viewed from different angles. The color spherical harmonic function coefficient contains the coefficient of each RGB channel, that is, H = [H R , H G , H B ],in l max is the maximum degree of spherical harmonics. Therefore, the spherical harmonic function coefficients H are known. * and viewing direction d∈R 2 , the present invention can calculate the color value by the following spherical harmonic function:

[0121]

[0122] Among them, Y l m :R 2 →R is the spherical harmonics basis function, Used to normalize color values.

[0123] Note that all the density σ and color spherical harmonic function coefficients H in the two-dimensional image representing the sphere are trainable. After training, they are directly used to render new perspective images.

[0124] In the case of obtaining the ray in the previous step, the present invention uniformly samples N on the ray r. s points Assume that the sampling points p are located at radii r o and r i The two spheres between which the nearest neighboring points to the sampling point are and The present invention obtains spherical points by linear interpolation Attributes (i.e., density and color spherical harmonics coefficients):

[0125]

[0126] in, and are the attributes of the four nearest neighbors of pixel (i, j). (i0, j0) and (i1, j1) are the coordinates of the upper left and lower right corner pixels, respectively. Then, the attributes of sampling point p can be obtained as follows:

[0127]

[0128] in,

[0129] A three-dimensional voxel with a resolution of N×H×W is formed by superimposing the above two-dimensional images. The above process of obtaining the attributes of the sampling point is equivalent to the process of linear interpolation on the three-dimensional voxel.

[0130] Through this step, the present invention can obtain the properties of any sampling point p on the ray r(t)=o+td: density σ and color spherical harmonic function coefficient H, wherein the color c of the point can be obtained through the spherical harmonic function coefficient and the ray direction.

[0131] Step 3: Get a ray r(t) = o + td on N s The density of sampling points σ i and color c i Finally, the present invention calculates the ray color value through the following rendering equation:

[0132]

[0133] in, is the cumulative transmittance, which is understood as the probability that the ray can reach the point, δ i =t i+1 -t i is the distance between adjacent sampling points.

[0134] Step 4: Through the above steps, the present invention can obtain the color value of each ray from the multi-sphere scene representation. At the same time, the ray corresponds to the pixel value of the collected image C(r). The present invention minimizes the following objective function by optimizing the density and color spherical harmonic function coefficients in the multi-sphere scene representation:

[0135]

[0136] During the optimization process, the present invention uses a gradient descent optimization algorithm.

[0137] Step four: After the multi-spherical scene modeling is completed, the application can realize the generation of any new view image. The specific refinement steps are as follows:

[0138] Step1: Given the camera pose under the new view, the application calculates the corresponding ray of each pixel in the new view image.

[0139] Step2: Sampling is performed on the ray, and the attributes of each point, i.e. the density and color spherical harmonic function coefficients, are obtained through the process of Step three Step2, and finally the color value of the ray is calculated through the method in Step three Step3.

[0140] Step3: Assign the ray color value to the corresponding pixel, and finally form the image under the new view.

[0141] In summary, in the new view generation technology realized by the application, the three-dimensional reconstruction is modeled as a plurality of concentric spheres with different radii, and the point attributes on the sphere are explicitly represented by three-dimensional voxels, thereby completing efficient modeling and rendering.

[0142] Compared with the prior art, the application has the following outstanding substantive features:

[0143] 1. The three-dimensional scene is represented as a plurality of spheres.

[0144] 2. The method for calculating the attributes of any point in space from the plurality of spherical representations.

[0145] 3. The method for using three-dimensional voxels to represent and display the sphere. Here, the three-dimensional voxels can be octree structure, multi-resolution voxels, spatial hash table, etc.

[0146] In addition, compared with the prior art, the application has the following significant advantages:

[0147] 1. Short model training time: the existing method needs to train a model for each new scene, and it takes tens of hours of convergence time to train a model, while the application method can complete the training in minutes.

[0148] 2. Fast image generation speed: the existing method takes about 30 seconds to generate an 800x800 resolution image, while the application method can achieve real-time speed (more than 30 frames per second).

[0149] 3. High image generation quality: the image generation quality is better than that of the existing method.

[0150] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0151] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0152] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0153] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0154] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0155] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0156] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0157] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0158] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A new perspective image generation method based on multi-spherical scene expression, characterized in that: include: Collecting a set of scene images; Performing image pose estimation on all images in the scene image set to obtain pose information of all images; Modeling a multi-spherical scene model based on the pose information of each image in the scene image set; generating a target image at any new perspective based on the multi-spherical scene model; The method of modeling a multi-spherical scene model based on the pose information of each image in the scene image set includes: Based on the imaging principle of the pinhole camera model, the position of the shooting camera is used as the starting point of the ray, and the direction pointing to the corresponding pixel position on the imaging plane is used as the ray direction to generate the ray corresponding to each pixel point in each image; According to the ray corresponding to each pixel point, obtaining point attribute information of any sampling point on each ray; wherein the point attribute information includes the density and color of the sampling point; Calculate the ray color value of each ray according to the point attribute information of each ray; According to the ray color value of each ray and the image pixel value corresponding to each ray, the objective function is optimized, and a multi-sphere scene model is constructed according to the optimized objective function; The step of obtaining point attribute information of any sampling point on each ray according to the ray corresponding to each pixel point includes: Representing a three-dimensional scene as multiple spheres with the same center but different radii, and configuring the maximum range of the three-dimensional scene and the radius of each sphere; wherein each sphere is represented by a two-dimensional image; wherein each pixel in the two-dimensional image stores density and color spherical harmonic function coefficients; wherein the color spherical harmonic function is used to calculate the color value of each pixel; uniformly sampling a plurality of sampling points from each of the rays, and determining the nearest neighbor point of each sampling point on the corresponding two spherical surfaces; Obtaining point attribute information of the nearest neighbor point by linear interpolation; Calculating the point attribute information of the sampling point according to the point attribute information of the nearest neighbor point; The two-dimensional images are superimposed to form a three-dimensional voxel, and then the point attribute information of any sampling point on each ray is determined.

2. The method for generating new perspective images based on multi-spherical scene expression according to claim 1, characterized in that: The acquisition scene image set includes: Shooting a scene video of the target scene with a camera according to a preset motion trajectory; or shooting a scene video of the target scene around the target object with a camera according to a motion trajectory moving around the target object; According to a preset time interval, images are extracted frame by frame from the scene video to obtain a scene image set.

3. The method for generating new perspective images based on multi-spherical scene expression according to claim 1, characterized in that: The performing image pose estimation on all images in the scene image set to obtain pose information of all images includes: Extracting feature points of each image in the scene image set and a descriptor of each feature point; According to the descriptors of each feature point, calculate the matching relationship between any two feature points; Calculating the position relationship between the images based on the matching relationship between any two feature points; The pose information of all images is globally optimized through bundle adjustment to obtain the pose information of each image, and then a set of pose images is obtained.

4. The method for generating new perspective images based on multi-spherical scene expression according to claim 1, characterized in that: In the step of calculating the ray color value of each ray according to the point attribute information of each ray, the calculation formula of the ray color value is: in, Represents the color value of ray r; T i Indicates the cumulative transmittance; σ i represents the density of sampling point i; δ i Represents the distance between adjacent sampling points; c i Indicates the color of sampling point i; N s Indicates the total number of sampling points.

5. The method for generating new perspective images based on multi-spherical scene expression according to claim 1, characterized in that: Generating a target image at any new perspective based on the multi-spherical scene model includes: Given the camera pose at the new perspective, calculate the ray corresponding to each pixel in the new perspective image; Sampling is performed on the ray, obtaining the attributes of each sampling point on the ray, and then the color value of the ray is calculated; Assign the color value of the ray to the corresponding pixel, and finally form an image under a new perspective.

6. A device for implementing the new perspective image generation method based on multi-spherical scene expression according to any one of claims 1 to 5, characterized in that: include: The first module is used to collect a set of scene images; The second module is used to estimate the image pose of all images in the scene image set to obtain pose information of all images; A third module is used to model a multi-spherical scene model based on the pose information of each image in the scene image set; The fourth module is used to generate a target image at any new viewing angle based on the multi-spherical scene model.

7. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 5.

Images