A pre-bidirectional compensation based panoramic image registration method for virtual reality metaverse
The pre-bidirectional compensation-based panoramic image registration method improves image alignment efficiency and accuracy in virtual reality metaverse scenes by using adaptive distortion correction and bidirectional compensation with DSCNN and VGG16 networks for unsynchronized cameras with moving foregrounds.
Patent Information
- Application Number
- JP2023517808
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2022-03-28
- Filing Date
- 2022-07-22
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2042-07-22
AI Technical Summary
In conventional virtual reality metaverse scenes, camera clocks are not synchronized, leading to low efficiency and poor effectiveness of image alignment when there is a moving foreground in the image.
A pre-bidirectional compensation-based panoramic image registration method involving motion object extraction, adaptive distortion correction, and bidirectional compensation using dense convolutional neural networks (DSCNN) and VGG16 network for accurate image alignment.
The method enhances image alignment accuracy and reduces calculation time by employing different alignment strategies based on the presence of moving objects, effectively addressing asynchronization issues.
Smart Images

Figure 0007578327000056 
Figure 0007578327000057 
Figure 0007578327000058
Abstract
Description
[Technical field]
[0001] The present invention relates to the technical fields of deep learning and image processing, and in particular to a pre-bidirectional compensation based panoramic image registration method for virtual reality metaverse. [Background technology]
[0002] Image registration refers to mapping one or more images to a target image through a certain optimal transformation. Image registration method based on grayscale information is a process of maximizing the similarity between the registration image and the image to be registered based on grayscale information. The similarity measurement method generally adopts methods such as mutual information, squared difference, Hausdorff distance, cross-correlation coefficient, squared difference and equidistance. Among them, mutual information is frequently used in image fusion. The process of maximizing the similarity between the registration image and the image to be registered is actually an optimization process, and by continuing to adjust the parameters of the transformation model, only when the similarity is maximized, the parameters of the transformation model can reach the optimum, and the registration target image is transformed according to the optimal model to complete the registration with the registration image. Image registration based on feature information uses a feature extraction algorithm to extract features, and then constructs a mapping relationship of feature points between the registration image and the image to be registered to obtain a transformation model to complete the image registration.
[0003] When different types of images are aligned, the algorithms and evaluation standards adopted are also different. With the daily advancement of various instruments in the medical field, image alignment is becoming more and more important in medical image processing. By aligning images of the same patient taken at different times, we can better analyze the dynamic changes of the patient's organs and pathology, make more accurate medical diagnoses, and formulate more appropriate treatment plans. Image alignment is also one of the methods to study mechanical properties in the field of material mechanics. Information such as temperature and shape collected by different sensors is fused and compared to obtain various values such as temperature field and strain field. Furthermore, various values are substituted into theoretical models to optimize parameters. Image alignment can also be used for automatic tracking of moving objects, pattern recognition, video analysis, etc.
[0004] However, in traditional virtual reality metaverse scenes, there are technical challenges such as low efficiency and poor effectiveness of image registration when there is a moving foreground in the image due to the camera clocks not being synchronized. Summary of the Invention [Problem to be solved by the invention]
[0005] In traditional virtual reality metaverse scenes, the camera clocks are not synchronized, which leads to low efficiency and poor effectiveness of image alignment when there is a moving foreground in the image. [Means for solving the problem]
[0006] 1. A pre-bidirectional compensation based panoramic image registration method for a virtual reality metaverse, comprising: S1, performing moving object extraction on the two images to be aligned using a background subtraction method based on Vibe, performing adaptive distortion pixel correction on the foreground set of images from which the moving objects are extracted, extracting image features using a SURF feature extraction algorithm, and completing moving object matching according to a feature threshold; S2, performing motion direction and speed detection for the successfully matched moving object using the sparse optical flow method, calculating logical time according to the physical time of the camera, calculating the speed of the moving object according to the difference between the physical times of the two cameras, and compensating for each moving object; S3, constructing a first image alignment model based on a dense convolutional neural network, constructing a second image alignment model based on a VGG16 network, performing image alignment on images with failed object extraction, images with failed object matching, and images with bidirectional compensation, and classifying the images into two types according to whether images are bidirectionally compensated or not, in which the first image alignment model is used to perform image alignment on images after bidirectional compensation based on position prediction, and the second image alignment model is used to perform image alignment on images without bidirectional compensation, including images with failed object extraction and images with failed object matching.
[0007] Further, in step S1, the process of performing moving object extraction on two images to be aligned using a Vibe-based background subtraction method and performing adaptive distortion pixel correction on a foreground set of images from which the moving object is extracted is as follows: constructing a background model M(x) containing N sample values for each pixel value in the image sequence to be detected;
number
number
number
number
[0008] In addition, we use the sparse optical flow method to calculate the speed and direction of the moving object in the foreground, and obtain the speed in two directions, x and y, and u and v, respectively.
number
number
number
number
number
number
number
number
number
number
number
number
number
number
[0009] Furthermore, in step S3, the first image alignment model comprises three dense network blocks, one convolution layer and one first pooling layer connected in sequence, each dense network block comprises one convolution layer, one second pooling layer and one dense network connected in sequence, the output of the first pooling layer is flattened and then connected to a regression network, which comprises five fully connected layers connected in sequence, and is used to output parameters of geometric transformation for image alignment.
[0010] Furthermore, in step S3, the second image alignment model includes a feature extraction network, a feature matching network and a regression network connected in sequence, where the feature extraction network includes two convolution layers and one pooling layer connected in sequence, and is used to extract related features of the images to be aligned and generate a feature map, and the feature extraction network includes two matching networks, and the two matching networks share weights, and each matching network is used to perform correlation calculation and match feature descriptors using the related maps, and outputs the parameters of geometric transformation for image alignment output by the regression network.
[0011] Further, the virtual reality metaverse scene includes a virtual reality scene. Effect of the Invention
[0012] The present invention proposes a compensation scheme to address the problem of image registration efficiency caused by camera asynchrony and foreground moving objects, firstly, object extraction is performed on the image using background subtraction, then adaptive distortion correction is performed taking into account the distortion of the two cameras when performing object matching, and finally, compensation operations are performed on the foreground in the two images respectively. The present invention further proposes an image registration algorithm based on a dense convolutional neural network, provides a method for peer-to-peer acquisition of transformation parameters, and finally adopts different registration schemes according to whether bidirectional compensation is performed on the images to improve the algorithm efficiency.
[0013] 1. For images taken by unsynchronized cameras, the present invention adopts different alignment methods according to whether there is a moving object in the foreground. When there is a moving object in the foreground, the adaptive distortion pixel correction and bidirectional compensation based on position prediction of the present invention can effectively avoid the deviation caused by the moving object in the unsynchronized time difference, and make the alignment result more accurate.
[0014] 2. The image alignment method based on DSCNN of the present invention can realize peer-to-peer output of the transformation parameters required for alignment. Compared with traditional image alignment algorithms, the calculation time is significantly reduced, and the method has great future potential. [Brief description of the drawings]
[0015] [Figure 1] 1 is an overall flowchart of the pre-bidirectional compensation-based panoramic image registration method for the virtual reality metaverse of the present invention. [Diagram 2] 13 is a flowchart of object extraction. [Diagram 3] 1 is a flowchart of object matching taking distortion into account. [Figure 4] 1 is a flowchart of image registration based on bidirectional compensation. [Diagram 5] 1 is a flowchart of DSCNN image alignment. [Figure 6] DSCNN image alignment network structure diagram. [Figure 7] 1 is a structural diagram of an image registration network based on VGG-16. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0016] The following examples will enable those skilled in the art to more fully understand the present invention, but are not intended to limit the present invention in any way.
[0017] This invention proposes a pre-bidirectional compensation-based panoramic image registration method for the virtual reality metaverse, which basically includes moving object extraction, distortion-aware moving object matching, and bidirectional compensation based on position prediction, and adopts different registration methods for different foreground sets, and the scenes in the virtual reality metaverse include, for example, virtual reality scenes, VR surveillance scenes, etc. As shown in Figure 1, the method specifically includes the following steps 1 to 4.
[0018] Step 1: Use the ViBe algorithm to construct a background model M(x) containing N sample values for each pixel in the target image sequence, and v i represents the background sample with index i,
number
[0019] ViBe assumes that the spatial distribution relationship of adjacent pixel points is also similar. The initialization of the ViBe background model is done by a single frame image, and the technique can respond quickly when the light source suddenly changes, i.e., discard all the original sample points and reinitialize.
number
[0020] The adaptive distortion pixel correction is mainly carried out on the basis that different focal lengths will lead to different types of distortion. For example, if the focal length is the middle value of the device, it is likely to produce barrel distortion. If the focal length is too long, it is likely to produce a mixture of barrel distortion and pincushion distortion. Any distortion will have a negative effect on the matching result. Therefore, the formula
number
number
[0021] Step 2: Calculate the displacement of the matched foreground moving object in each direction per unit time using the LK optical flow method.
number
[0022] Perform a Taylor series expansion on the right hand side and divide both sides by dt to get the equation
number
number
number
[0023] LK optical flow finds the optical flow equations of nine pixels in a square with a side length of three based on the fact that they have the same motion, constructs a system of nine equations and two unknowns, and solves them using least squares fitting. The solution is:
number
[0024] Due to various reasons such as the internal crystal oscillator of the camera, there is a small time interval Δt between the right and left images, and the actual physical time t′ of the right image and the time t of the left image satisfy t′=t+Δt. At time t, the moving object extracted from the left image to be aligned is
number
number
number
[0025] The foreground images of the left and right alignment target images after position compensation are
number
number
number
number
number
number
number
[0026] Step 3: Feature extraction is performed on the target image A and the target image B for alignment using the DSCNN network, and the two feature extraction networks share weight parameters. Then, processing is performed using a regression network consisting of five fully connected layers, and finally the transformation parameters for image alignment are output.
[0027] The feature extraction network used for image registration is based on a dense convolutional neural network, and the network structure consists of three dense network blocks, one convolution layer, and one pooling layer. The dense network block consists of one convolution layer, one pooling layer, and one dense network. The output is flattened and then connected to the regression network. With the increase in the number of CNN convolution layers, the input information and gradient information of the network gradually disappear with the increase in the depth of the network. ResNets and Highway Netwoks use different methods to prove that the shorter the path from the start value to the end of the neural network, the better the problem of gradient vanishing can be alleviated. ResNets uses identifier connection to split the signal and obtain better gradient flow and information. FractalNets ensures a short path in the network, and the effect of gradient vanishing is reduced. Therefore, the present invention introduces a dense network block into the feature extraction network and proposes a feature extraction network based on a dense network, which not only ensures the extraction of more complete feature information, but also prevents the problem of gradient vanishing, as shown in Figure 6. Each layer of the dense network block in the figure is directly connected to ensure the maximization of information flow between the networks of each layer. The input of each layer in the dense network block is the output of all previous layers, ensuring the feedforward property of the network, and the feature mapping of this layer is also transmitted to the next layer as input. In this invention, the recurrent network of DSCNN consists of five fully connected layers. The fully connected layers are used to find the nonlinear relationship between features and high-level inference. Finally, it outputs the transformation parameters that can align the two images.
[0028] Step 4: Perform image registration based on VGG16 for images without extracted moving objects and images with failed moving object matching. The reason is that the core idea of the image registration algorithm based on bidirectional compensation according to the present invention is to adopt different registration algorithms for images with different foregrounds to achieve high registration accuracy, as shown in Fig. 1. Perform image registration based on DSCNN for images with successful detection and matching of moving foregrounds, and if no object is detected or matching fails, perform image registration based on VGG16 directly without performing bidirectional compensation operation, thereby reducing the time complexity of the entire algorithm.
[0029] The image alignment algorithm based on VGG16 mainly includes three parts: feature extraction network, feature matching network and regression network. The feature extraction network part adopts the improved VGG-16, the two matching networks share weights, the matching network is used to match feature descriptors, and outputs the parameters of the geometric transformation output by the regression network. The feature extraction network adopts a standard convolutional neural network architecture, and the image to be aligned is input into a convolutional neural network without a fully connected layer to generate a feature map. The convolutional neural network is a cut-out of a part of the VGG-16 network, and its network structure is shown in Figure 7.
[0030] Classical image registration geometric estimation discards the original descriptors and focuses on the similarity between the descriptor pairs, because the similarity and spatial location of the descriptor pairs contain the necessary information required for geometric estimation, and by thresholding the similarity value, only the most similar neighboring matches are kept to construct the descriptor pairs. The matching network of the present invention also uses this idea, adopting a correlation layer, not considering the original descriptors, but only considering the spatial location and similarity between the descriptor pairs. The matching network of the present invention adopts the structure of a correlation layer and a normalization layer. The correlation layer is used to calculate the similarity of all feature descriptors, and the normalization layer is used to process and normalize the similarity scores to eliminate fuzzy matching.
[0031] It should be noted that the above description of the embodiments is merely for the purpose of understanding the method and core idea of the present application. Those skilled in the art may make various improvements and modifications to the present application without departing from the principles of the present application, and these improvements and modifications are also within the scope of protection of the claims of the present application.
Claims
1. A pre-bidirectional compensation based panoramic image registration method for a virtual reality metaverse, comprising: S1: performing moving object extraction on two images to be aligned using a background subtraction method based on Vibe, performing adaptive distortion pixel correction on the foreground set of images from which the moving objects are extracted, and extracting image features using a SURF feature extraction algorithm, and completing moving object matching according to a feature threshold; S2: performing motion direction and speed detection for the successfully matched moving objects using a sparse optical flow method, calculating logical time according to the physical time of the cameras, calculating the speed of the moving objects according to the difference between the physical times of the two cameras, and compensating the positions of the moving objects based on the difference between the physical times; S3. A method for pre-bidirectional compensation based panoramic image registration for a virtual reality metaverse, comprising the steps of: constructing a first image registration model based on a convolutional neural network; constructing a second image registration model based on a VGG16 network; performing motion direction and speed detection for an image in which a moving object extraction fails, an image in which a moving object matching fails, and a moving object in which matching succeeds; calculating a logical time according to the physical time of the camera; calculating the speed of the moving object according to the difference between the physical times of the two cameras; performing image registration for images that have been bidirectionally compensated to compensate for each moving object; and classifying the images into two types according to whether or not bidirectional compensation is performed for the images, wherein the first image registration model is used to perform image registration for images that have been bidirectionally compensated, and the second image registration model is used to perform image registration for images that have not been bidirectionally compensated, including images in which a moving object extraction fails and images in which a moving object matching fails.
2. In step S1, the process of performing motion object extraction for two images to be aligned using a Vibe-based background subtraction method and performing adaptive distortion pixel correction for a foreground set of images from which motion objects have been extracted is as follows: constructing a background model M(x) containing N sample values for each pixel value in the image sequence to be detected; [Equation 38] (v i represents the background sample with index i, i=1, 2, ..., N); The background model M(x) [0039] (In the formula, N G (x) represents the spatial neighborhood of the pixel at x location, and v y represents the color values of pixel points in the spatial neighborhood, and sample points of the background model are generated by randomly selecting N pixel points from the neighborhood; Correcting the pixel according to the following formula: [0040] In the formula, (x d , y d ) represents the pixel coordinate where distortion occurs, and (x c , y c ) represents the distorted center pixel coordinate, and (x u , y u ) represents the pixel coordinates after correction, and K 1 and K. 2 are the linear and quadratic radial distortion coefficients, respectively, r is the distance from the distorted coordinate to the center pixel coordinate, α and 1-α are the relative positions of the camera focal lengths, [0041] and In the formula, f max , f min represent the maximum and minimum focal lengths of the camera, respectively, and f c and wherein x represents the actual focal length of the camera.
3. In step S2, the process of compensating for each motion object includes: Using the sparse optical flow method, we calculate the speed and direction of the moving object in the foreground, and obtain the speed in two directions, x and y, and u and v, respectively. [0042] During the ceremony, [0043] represents the image gradient in the x and y directions for the i-th pixel, [0044] represents the gradient in time of the i-th pixel, Of the two images to be aligned, A and B, the foreground set of the left image is [0045] It is expressed as The foreground set in the right image is [0046] It is expressed as In the formula, p Ai , p Bi represents the pixels that make up the foreground set in image A and image B, The actual physical time t' of the right image and the time t of the left image satisfy t'=t+Δt, where Δt is the difference in physical time between the two images, and the time it takes to perform position compensation for image A is t A and the time t required to perform position compensation on image B. B are respectively [0047] and The left foreground set after compensation is [0048] and During the ceremony, [0049] represents the pixels constituting the compensated foreground set of image A, and p Aix represents the pixel component in the x direction before compensation, [Number 50] represents the compensated pixel component in the x direction, and u A represents the velocity of the pixel in the x direction, and p Aiy represents the pixel component in the y direction before compensation, [0051] represents the compensated pixel component in the y direction, and V A represents the velocity of the pixel in the y direction; The foreground set in the right image after compensation is [0052] and During the ceremony, [0,53] represents the pixels constituting the compensated foreground set of image B, and p Bix represents the pixel component in the x direction before compensation, [0,54] represents the compensated pixel component in the x direction, and u B represents the velocity of the pixel in the x direction, and p Biy represents the pixel component in the y direction before compensation, [0.55] represents the compensated pixel component in the y direction, and v B 2. The method of claim 1, further comprising: representing the velocity of the pixel in the y-direction.
4. The pre-bidirectional compensation based panoramic image registration method for the virtual reality metaverse of claim 1, characterized in that in step S3, the first image registration model consists of three network blocks, one convolutional layer and one first pooling layer connected in sequence, each network block consisting of one convolutional layer, one second pooling layer and one network connected in sequence, the output of the first pooling layer is flattened and then connected to a regression network, which consists of five fully connected layers connected in sequence, and is used to output parameters of geometric transformation for image registration.
5. The method of claim 1 , wherein the virtual reality metaverse scene comprises a virtual reality scene.
Citation Information
Patent Citations
Method for registering infrared light image to visible light image
CN114066955A
