A panoramic image registration method based on prior-driven bidirectional compensation for virtual reality

Abstract

The present invention discloses a panoramic image registration method with prior-driven bidirectional compensation for virtual reality, including: extracting moving targets from two images to be registered, performing adaptive deformation pixel correction on the image foreground set of the extracted moving targets, extracting image features, and completing moving target matching; detecting the moving direction and speed of the successfully matched moving targets, calculating the speed of the moving targets according to the difference in physical time between two cameras, and compensating for each moving target; and performing image registration on images with failed target extraction, images with failed target matching, and images with bidirectional compensation. The present invention can solve the technical problems of low efficiency and poor effect of image registration when the moving foreground is caused by images with asynchronous camera clocks in existing VR monitoring scenarios.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning and image processing, and in particular relates to a panoramic image registration method of a priori driven bidirectional compensation for virtual reality. Background Art

[0002] Image registration is to map a single or multiple images to the target image according to a certain optimal transformation. The image registration method based on grayscale information is a process of maximizing the similarity between the registered image and the image to be registered based on grayscale information. Similarity measurement methods generally use mutual information, square difference, Hausdorff distance, mutual correlation coefficient, square difference sum and other measurement methods. Among them, mutual information is more frequently used in image fusion. The process of maximizing the similarity between the registered image and the image to be registered is actually an optimization process. By continuously adjusting the parameters of the transformation model, only when the similarity is maximized can the parameters of the transformation model reach the optimal state. The image to be registered is transformed according to the optimal model to complete the registration with the registered image. Image registration based on feature information uses feature extraction algorithms to extract features, and then obtains the transformation model by establishing a mapping relationship between the feature points of the registered image and the image to be registered, thereby completing image registration.

[0003] Different types of images use different algorithms and evaluation criteria for registration. With the advancement of various medical equipment, image registration is becoming more and more important in medical image processing. By registering images of the same patient taken at different times, we can analyze the dynamic changes of the patient's organs and pathologies, so as to make more accurate medical diagnoses and develop more targeted treatment plans. Image registration is also a method for studying mechanical properties in the field of material mechanics. The temperature, shape and other information collected by different sensors are fused and compared to obtain various values ​​such as temperature field and deformation field. Then, various values ​​are brought into the theoretical model for parameter optimization. Image registration can also be used for automatic tracking of moving targets, pattern recognition and video analysis.

[0004] However, in existing VR monitoring scenarios, technical problems such as low image registration efficiency and poor effect occur when there is a moving foreground in the image due to camera clock asynchrony. Summary of the invention

[0005] Technical problems solved: In existing VR monitoring scenarios, due to the asynchrony of camera clocks, there are technical problems such as low image registration efficiency and poor effect when there is a moving foreground in the image.

[0006] Technical solution:

[0007] A panoramic image registration method for virtual reality-oriented priori driven bidirectional compensation, the panoramic image registration method comprising the following steps:

[0008] S1. Use the Vibe-based background subtraction method to extract the moving target from the two images to be registered, perform adaptive deformation pixel correction on the image foreground set of the extracted moving target, use the SURF feature extraction algorithm to extract image features, and complete the moving target matching according to the feature threshold;

[0009] S2. Use the sparse optical flow method to detect the moving direction and speed of the successfully matched moving target, calculate the logical time according to the physical time of the camera, calculate the speed of the moving target according to the difference between the physical times of the two cameras, and compensate for each moving target;

[0010] S3. Construct a first image registration model based on a dense convolutional neural network, and construct a second image registration model based on a VGG16 network; perform image registration on images in which target extraction fails, images in which target matching fails, and images in which two-way compensation is performed, and divide the images into two categories based on whether or not two-way compensation is performed, wherein the first image registration model is used to perform image registration on images after two-way compensation based on position prediction, and the second image registration model is used to perform image registration on images that have not undergone two-way compensation, including images in which target extraction fails and target registration fails.

[0011] Furthermore, in step S1, the process of extracting moving targets from the two images to be registered by using a background subtraction method based on Vibe and performing adaptive deformation pixel correction on the image foreground set of the extracted moving targets includes the following steps:

[0012] For each pixel value in the image sequence to be detected, a background model M(x) containing N sample values ​​is established, where M(x) = {v1, v2…v N},v i represents the background sample with index i, i = 1, 2, ..., N;

[0013] Update the background model M(x) as follows:

[0014] M 0 (x) = {v y (y|y∈N G (x))}

[0015] Where N G (x) represents the spatial neighborhood of the pixel at position x, v y Represents the color value of the pixel in the neighborhood of the space; the sample points of the background model are randomly selected from the neighborhood pixels.

[0016] The pixels are corrected according to the following formula:

[0017]

[0018]

[0019]

[0020] In the formula, (x d ,y d ) represents the pixel coordinates where deformation occurs, (x c ,y c ) represents the coordinates of the deformed center pixel, (x u ,y u ) represents the pixel coordinates after correction, K1 and K2 represent the first-order and second-order radial distortion coefficients respectively; r represents the distance from the deformed coordinates to the center pixel coordinates; α and 1-α represent the relative position of the camera focal length:

[0021]

[0022]

[0023] In the formula, f max 、f min Respectively represent the maximum and minimum focal length of the camera, f c Indicates the actual focal length of the camera.

[0024] Furthermore, the sparse optical flow method is used to calculate the speed and direction of the moving target in the foreground, and the speeds in the x and y directions are u and v respectively:

[0025]

[0026] in and represents the image gradient of the ith pixel in the x and y directions, Represents the gradient of the i-th pixel in time;

[0027] The foreground set of the left image in the two images A and B to be registered is expressed as:

[0028] A N ={p A1 , p A2 …p AN};

[0029] The foreground set of the right image is represented as:

[0030] B N ={p B1 , p B2 …p BN};

[0031] where pAi 、p Bi represents the pixels constituting the foreground set of image A and image B;

[0032] The real physical time t′ of the right image and the time t of the left image satisfy t′=t+Δt, where Δt is the difference in physical time between the two images; the time t for position compensation of image A A and the time t for position compensation of image B B They are:

[0033] t A =t+Δt / 2

[0034] t B =t′-Δt / 2=t+Δt / 2;

[0035] The left foreground set after compensation is:

[0036] A N ′={p A1 ′,p A2 ′…p AN ′}

[0037]

[0038]

[0039] where p Ai ′ represents the pixels constituting the foreground set of image A after compensation, p Aix represents the pixel component in the x direction before compensation, p Aix ′ represents the pixel component after compensation in the x direction, u A Indicates the speed of the pixel in the x direction, p Aiy Represents the pixel component in the y direction before compensation, p Aiy ′ represents the pixel component after compensation in the y direction, u B Indicates the speed of the pixel in the y direction;

[0040] After compensation, the foreground set of the right image is:

[0041] B N ′={p B1 ′,p B2 ′…p BN ′}

[0042]

[0043]

[0044] where p Bi ′ represents the pixels constituting the foreground set of image B after compensation, p Bixrepresents the pixel component in the x direction before compensation, p Bix ′ represents the pixel component after compensation in the x direction, u B Indicates the speed of the pixel in the x direction, p Biy Represents the pixel component in the y direction before compensation, p Biy ′ represents the pixel component after compensation in the y direction, v B Indicates the speed of the pixel in the y direction.

[0045] Furthermore, in step S3, the first image registration model is composed of three dense network blocks, a convolutional layer and a first pooling layer connected in sequence, wherein each dense network block is composed of a convolutional layer, a second pooling layer and a dense network connected in sequence; the output of the first pooling layer is connected to the regression network after flattening, and the regression network is composed of five fully connected layers connected in sequence to output the parameters of the geometric transformation of the image registration.

[0046] Furthermore, in step S3, the second image registration model includes a feature extraction network, a feature matching network and a regression network connected in sequence, wherein the feature extraction network includes two convolutional layers and a pooling layer connected in sequence, which are used to extract relevant features of the image to be registered and generate a feature map; the feature extraction network includes two matching networks, which share weights, and each matching network is used to use the correlation map to perform correlation calculation and match feature descriptors, and its output is output through the regression network as the parameters of the geometric transformation of the image registration.

[0047] The present invention proposes a compensation scheme for the image registration efficiency problem caused by camera asynchrony and foreground moving targets. First, the background subtraction method is used to extract the target from the image. Then, the deformation of the two cameras is considered during target matching, and adaptive deformation correction is performed. Finally, compensation operations are performed on the foregrounds of the two images respectively. The present invention also proposes an image registration algorithm based on a dense convolutional neural network, provides an end-to-end method for obtaining transformation parameters, and finally adopts different registration schemes for images that require or do not require bidirectional compensation, thereby improving the efficiency of the algorithm.

[0048] Beneficial effects:

[0049] First, the present invention adopts different registration methods for images taken by asynchronous cameras according to whether there are moving objects in the foreground; when there are moving objects in the foreground, the adaptive deformation pixel correction and bidirectional compensation based on position prediction of the present invention can effectively avoid the offset caused by moving objects within the asynchronous time difference, and the registration result is more accurate.

[0050] Second, the DSCNN-based image registration method proposed in the present invention can achieve the conversion parameters required for end-to-end output registration, and the calculation time is greatly reduced compared with the traditional image registration algorithm, which has a broader application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is an overall flow chart of the panoramic image registration method of the prior-driven bidirectional compensation for virtual reality of the present invention;

[0052] Figure 2 Extract a flow chart for the target;

[0053] Figure 3 Flowchart for target matching considering deformation;

[0054] Figure 4 It is a flowchart of image registration based on bidirectional compensation;

[0055] Figure 5 This is the DSCNN image registration flow chart;

[0056] Figure 6 This is the DSCNN image registration network structure diagram;

[0057] Figure 7 This is the structure diagram of the image registration network based on VGG-16. DETAILED DESCRIPTION

[0058] The following examples will enable those skilled in the art to more fully understand the present invention, but are not intended to limit the present invention in any way.

[0059] The present invention proposes a panoramic image registration method for virtual reality-oriented prior-driven bidirectional compensation, which basically includes moving target extraction, moving target matching considering deformation, bidirectional compensation based on position prediction, and using different registration methods for different foreground sets. Figure 1 As shown, the specific steps include:

[0060] Step 1: The ViBe algorithm establishes a background model M(x) containing N sample values ​​for each pixel value in the image sequence to be detected. i Represents the background sample with index i:

[0061] M(x)={v1,v2…v N}

[0062] v(x) represents the color value of the pixel at position x in a given color space, and the background model of the pixel is M(x).

[0063] ViBe assumes that the spatial distribution relationship of adjacent pixels is also similar. ViBe background model initialization can be completed using a single frame image. This technology can respond quickly when facing sudden changes in light sources, that is, all original sample points are discarded and reinitialized:

[0064] M 0 (x) = {v y (y|y∈N G (x))}

[0065] N G (x) represents the spatial neighborhood of the pixel at position x, v y Indicates the color value of the pixel in the neighborhood of the space. In specific implementation, the sample points of the background model are randomly selected from the neighborhood pixels.

[0066] Adaptive deformation pixel correction is mainly based on the different types of deformation caused by different focal lengths. For example, when the focal length is in the middle value of the device, barrel deformation is likely to occur. When the focal length is too long, mixed deformation of barrel and pincushion deformation is likely to occur. No matter which deformation is present, it will have a negative impact on the matching results. Therefore, the following formula is proposed to correct the pixels:

[0067]

[0068]

[0069]

[0070] Where (x d ,y d ) represents the pixel coordinates where deformation occurs, (x c ,y c ) represents the coordinates of the deformed center pixel, (x u ,y u ) represents the pixel coordinates after correction, K1 and K2 represent the first-order and second-order radial distortion coefficients respectively. r represents the distance from the deformed coordinates to the center pixel coordinates. α and 1-α represent the relative position of the camera focal length:

[0071]

[0072]

[0073] where f max 、f min Respectively represent the maximum and minimum focal length of the camera, f c Indicates the actual focal length of the camera.

[0074] Step 2: Use the LK optical flow method to calculate the displacement of the matched foreground moving target in each direction per unit time:

[0075] I(x,y,t)=I(x+dx,y+dy,t+dt)

[0076] Perform Taylor series expansion on the right side and divide both sides by dt to obtain the formula:

[0077] f x u+f y v+f t =0

[0078] in f x , f y represents the image gradient, f t Represents the gradient in time. LK optical flow finds their optical flow equations based on the same motion of 9 pixels in a square with a side length of 3, forming a system of 9 equations and 2 unknowns, and uses least squares fitting to solve. The solution is as follows:

[0079]

[0080] Due to various reasons such as the internal crystal oscillator of the camera, there is a small time interval Δt between the right image and the left image. The real physical time t′ of the right image and the time t of the left image satisfy t′=t+Δt. The moving target extracted from the left image to be registered at time t is represented as A N ={p A1 , p A2 …p AN}, at the same time, the moving target extracted from the right image to be registered is represented by B N ={p B1 , p B2 …p BN}, the time for position compensation of the two images is:

[0081] t A =t+Δt / 2

[0082] t B =t′-Δt / 2=t+Δt / 2.

[0083] After position compensation, the foreground images of the left and right images to be registered are:

[0084] A N ′={p A1 ′,p A2 ′…p AN ′}

[0085]

[0086]

[0087] B N ′={p B1 ′,p B2 ′…p BN ′}

[0088]

[0089]

[0090] Among them, p Ai ′ represents the pixels that constitute the foreground set after A compensation, p Aix represents the pixel component in the x direction before compensation, p Aix ′ represents the pixel component after compensation in the x direction, u A Indicates the speed of the pixel in the x direction, p Aiy Represents the pixel component in the y direction before compensation, p Aiy ′ represents the pixel component after compensation in the y direction, u B represents the speed of the pixel in the y direction, Δt is the difference in physical time between the two images; p Bi ′ represents the pixels that constitute the foreground set after B compensation, p Bix represents the pixel component in the x direction before compensation, p Bix ′ represents the pixel component after compensation in the x direction, uB represents the speed of the pixel in the x direction, and p Biy Represents the pixel component in the y direction before compensation, p Biy ′ represents the pixel component after compensation in the y direction, v B Indicates the speed of the pixel in the y direction.

[0091] Step 3: The images A and B to be registered are respectively subjected to the DSCNN network to extract features. The two feature extraction networks share weight parameters and then pass through a regression network consisting of 5 fully connected layers to finally output the transformation parameters for image registration.

[0092] The feature extraction network used for image registration is based on a dense convolutional neural network. The network structure consists of three dense network blocks, a convolutional layer, and a pooling layer. The dense network block consists of a convolutional layer, a pooling layer, and a dense network. The output is flattened and connected to the regression network. As the number of CNN convolutional layers increases, the input information and gradient information of the network will gradually disappear as the depth of the network increases. ResNets and Highway Netwoks use different methods to confirm that the shorter the path from the starting value to the end of the neural network, the more it can alleviate the problem of gradient disappearance. ResNets diverts signals through identification connections to obtain better gradient flow and information. FractalNets ensures a short path in the network and reduces the impact of gradient disappearance. Therefore, the present invention introduces a dense network block into a feature extraction network and proposes a feature extraction network based on a dense network, which not only ensures the extraction of more comprehensive feature information, but also avoids the problem of gradient disappearance, such as Figure 6 As shown. The layers of the dense network block in the figure are directly connected to each other, ensuring that the information flow between each layer of the network is maximized. The input of each layer in the dense network block is the output of all previous layers, ensuring the feedforward characteristics of the network, and the feature map of this layer will also be passed to the subsequent layer as input. The regression network of DSCNN in the present invention is composed of 5 fully connected layers. The fully connected layer is used to find nonlinear relationships and high-level reasoning between features. Finally, the output can align the conversion parameters of the two images.

[0093] Step 4: Perform image registration based on VGG16 for images that do not extract moving targets or fail to match moving targets. This is because the core idea of ​​the image registration algorithm based on bidirectional compensation proposed in the present invention is to use different registration algorithms for images with different foregrounds in order to achieve a higher registration accuracy, such as Figure 1 As shown in the figure, for images that are successfully detected and matched to the moving foreground, image registration based on DSCNN is performed. When the target cannot be detected or the match is unsuccessful, image registration based on VGG16 is directly performed without the need for bidirectional compensation operations, so as to reduce the time complexity of the overall algorithm.

[0094] The image registration algorithm based on VGG16 mainly includes three parts: feature extraction network, feature matching network and regression network. The feature extraction network uses the improved VGG-16. The two matching networks share weights. The matching network is used to match feature descriptors, and its output is output as the parameters of geometric transformation through the regression network. The feature extraction network uses the standard convolutional neural network architecture. The image to be registered is input into the convolutional neural network without a fully connected layer to generate a feature map. The convolutional neural network intercepts part of the VGG-16 network. Its network structure is as follows: Figure 7 shown.

[0095] Classical image registration geometric estimation discards the original descriptors and focuses on the similarity between descriptor pairs. This is because the similarity and spatial position of descriptor pairs already contain the necessary information required for geometric estimation. By thresholding the similarity value, only the matches of the most similar neighbors are retained to construct the descriptor pairs. The matching network of the present invention also draws on this idea, using an association layer, which only considers the spatial position and similarity between descriptor pairs without considering the original descriptors. The matching network of the present invention adopts a structure of a correlation layer and a normalization layer. Among them, the correlation layer calculates the similarity of all feature descriptors, and the normalization layer is used to process and normalize the similarity scores to remove fuzzy matches.

[0096] It should be pointed out that the description of the above embodiments is only used to help understand the method of the present application and its core idea. For ordinary technicians in this technical field, several improvements and modifications can be made to the present application without departing from the principles of the present application. These improvements and modifications are also within the scope of protection of the claims of the present application.

Claims

1. A panoramic image registration method with prior-driven bidirectional compensation for virtual reality, characterized in that: The panoramic image registration method comprises the following steps: S1. Use the Vibe-based background subtraction method to extract the moving target from the two images to be registered, perform adaptive deformation pixel correction on the image foreground set of the extracted moving target, use the SURF feature extraction algorithm to extract image features, and complete the moving target matching according to the feature threshold; S2. Use the sparse optical flow method to detect the moving direction and speed of the successfully matched moving target, calculate the logical time according to the physical time of the camera, calculate the speed of the moving target according to the difference between the physical times of the two cameras, and compensate for each moving target; S3. Constructing a first image registration model based on a dense convolutional neural network, and constructing a second image registration model based on a VGG16 network; performing image registration on images where target extraction fails, images where target matching fails, and images where bidirectional compensation is performed, and dividing the images into two categories based on whether bidirectional compensation is performed, wherein the first image registration model is used to perform image registration on images after bidirectional compensation based on position prediction, and the second image registration model is used to perform image registration on images where bidirectional compensation is not performed, including images where target extraction fails and target registration fails; In step S2, the process of compensating the respective moving targets includes the following steps: The sparse optical flow method is used to calculate the speed and direction of the moving target in the foreground, and the speeds in the x and y directions are u and v respectively: in and represents the image gradient of the ith pixel in the x and y directions, Represents the gradient of the i-th pixel in time; The foreground set of the left image in the two images A and B to be registered is expressed as: A N ={p A1 ,p A2 …p AN }; The foreground set of the right image is represented as: B N ={p B1 ,p B2 …p BN }; where p Ai 、p Bi represents the pixels constituting the foreground set of image A and image B; The real physical time t′ of the right image and the time t of the left image satisfy t′=t+Δt, where Δt is the difference in physical time between the two images; the time t for position compensation of image A A and the time t for position compensation of image B B They are: t A =t+Δt / 2 t B =t′-Δt / 2=t+Δt / 2; The left foreground set after compensation is: A N ′={p A1 ′,p A2 ′…p AN ′} where p Ai ′ represents the pixels constituting the foreground set of image A after compensation, p Aix represents the pixel component in the x direction before compensation, p Aix ′ represents the pixel component after compensation in the x direction, u A Indicates the speed of the pixel in the x direction, p Aiy Represents the pixel component in the y direction before compensation, p Aiy ′ represents the pixel component after compensation in the y direction, u B Indicates the speed of the pixel in the y direction; After compensation, the foreground set of the right image is: B N ′={p B1 ′,p B2 ′…p BN ′} where p Bi ′ represents the pixels constituting the foreground set of image B after compensation, p Bix represents the pixel component in the x direction before compensation, p Bix ′ represents the pixel component after compensation in the x direction, u B Indicates the speed of the pixel in the x direction, p Biy Represents the pixel component in the y direction before compensation, p Biy ′ represents the pixel component after compensation in the y direction, v B Indicates the speed of the pixel in the y direction.

2. The virtual reality-oriented prior-driven bidirectional compensation panoramic image registration method according to claim 1, characterized in that: In step S1, the process of extracting moving targets from the two images to be registered by using the background subtraction method based on Vibe and performing adaptive deformation pixel correction on the image foreground set of the extracted moving targets includes the following steps: For each pixel value in the image sequence to be detected, a background model M(x) containing N sample values ​​is established, M(x) = {v1, v2…v N },v i represents the background sample with index i, i = 1, 2, ..., N; Update the background model M(x) as follows: M 0 (x)={v y (y|y∈N G (x))} Where N G (x) represents the spatial neighborhood of the pixel at position x, v y Represents the color value of the pixel in the neighborhood of the space; the sample points of the background model are randomly selected from the neighborhood pixels. The pixels are corrected according to the following formula: In the formula, (x d ,y d ) represents the pixel coordinates where deformation occurs, (x c ,y c ) represents the center pixel coordinate of the deformed pixel, (x u ,y u ) represents the pixel coordinates after correction, K1 and K2 represent the first-order and second-order radial distortion coefficients respectively; r represents the distance from the deformed coordinates to the center pixel coordinates; α and 1-α represent the relative position of the camera focal length: In the formula, f max 、f min Respectively represent the maximum and minimum focal length of the camera, f c Indicates the actual focal length of the camera.

3. The panoramic image registration method with prior-driven bidirectional compensation for virtual reality according to claim 1, characterized in that: In step S3, the first image registration model is composed of three dense network blocks, a convolutional layer and a first pooling layer connected in sequence, wherein each dense network block is composed of a convolutional layer, a second pooling layer and a dense network connected in sequence; the output of the first pooling layer is connected to the regression network after flattening, and the regression network is composed of five fully connected layers connected in sequence to output the parameters of the geometric transformation of the image registration.

4. The virtual reality-oriented prior-driven bidirectional compensation panoramic image registration method according to claim 1, characterized in that: In step S3, the second image registration model includes a feature extraction network, a feature matching network and a regression network connected in sequence, wherein the feature extraction network includes two convolutional layers and a pooling layer connected in sequence, which are used to extract relevant features of the image to be registered and generate a feature map; the feature extraction network includes two matching networks, which share weights, and each matching network is used to use the correlation map to perform correlation calculation and match feature descriptors, and its output is output through the regression network as the parameters of the geometric transformation of the image registration.

Images