Optimal seam image fusion method considering spatiotemporal relationship for virtual reality

Abstract

The present invention mentions an optimal stitching line image fusion method for virtual reality that considers the time-space relationship, including: extracting a foreground set of moving targets, and constructing a stitching line search model that does not include the foreground set according to the extracted foreground moving target set; according to the spatial relationship between the two cameras, the midline of the overlapping field of view of the target image is obtained from the midline of the overlapping field of view of the image to be fused, and the final mapping is obtained from the known transformation matrix; the stitching lines are calculated on both sides of the midline, and the respective optimal solutions are calculated according to the preset energy function to calculate the final optimal stitching line; a linear fusion model is constructed based on a learnable sparse convolutional neural network, and the two images to be fused are fused using the linear fusion model. The present invention can achieve a better fusion effect, and is not prone to ghosting and splitting. In particular, when there is a moving object in the foreground, the stitching line of the present invention can avoid the moving object and thus achieve a better fusion effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and deep learning, and in particular relates to an optimal seam line image fusion method considering time and space relationship for virtual reality. Background Art

[0002] The research focus of image stitching varies according to different application scenarios, such as multi-focal length image fusion, multi-exposure image fusion, visible light image and infrared image fusion, etc. The effects of various image fusion methods are also different. Ghosting, splitting, brightness changes, etc. of the image fusion part will affect the user experience, especially in scenarios such as VR monitoring that require high fusion effects.

[0003] Traditional image fusion algorithms work well when fusing static images, but they are prone to ghosting and double images when fusing images containing moving objects, making the image fusion results unclear. The weighted fusion method is simple to calculate but has poor results; pyramid fusion first decomposes the image into different frequency band spaces, and then fuses them separately, achieving the fusion of details and features of different data, but the calculation is complex and is not suitable for systems with high real-time requirements. The traditional best seam algorithm can effectively prevent the objects on both sides of the seam from being split, and the fusion effect is good. However, when the clock of the shooting device is not synchronized, ghosting will still occur. For example, when the optimal solution searched by the traditional best seam algorithm passes through a moving object, the moving object will still be split, and when the color brightness difference of the fused image is large, there will also be obvious light and dark differences. Summary of the invention

[0004] Technical problems solved: In response to the above technical problems, the present invention proposes an optimal stitching line image fusion method for virtual reality that considers the time-space relationship. The optimal stitching line method is improved based on the time-space relationship. Compared with the traditional image fusion method, the method proposed by the present invention has a better fusion effect, is less likely to have ghosting and splitting, and especially when there are moving objects in the foreground, the stitching line of the present invention can avoid the moving objects and thus achieve a better fusion effect. When the brightness difference of the images to be fused is too large, the method can also fuse the brightness well and reduce the brightness difference. It is better than the traditional algorithm in monitoring scenarios, has higher practicality and broad application prospects.

[0005] Technical solution:.

[0006] An optimal seam line image fusion method considering time and space relationship for virtual reality, the image fusion method comprising the following steps:

[0007] S1. A background subtraction method based on Vibe is used to extract a moving target foreground set, and a seam line search model that does not include the foreground set is constructed according to the extracted foreground moving target set;

[0008] S2. According to the spatial relationship between the two cameras, the midline of the overlapping visual field of the target image is obtained from the midline of the overlapping visual field of the image to be fused, and the final mapping is obtained from the known transformation matrix;

[0009] S3, calculating the suture lines on both sides of the midline respectively, calculating the respective optimal solutions according to the preset energy function, and calculating the final optimal suture line;

[0010] S4. Use a linear fusion algorithm to stitch the images on both sides of the optimal stitching line; build a brightness correction model based on a learnable sparse convolutional neural network, and the brightness correction model combines the part A of the source image to be stitched, the part B of the target image to be stitched, and the stitched part C of the final image stitched through the optimal stitching line to perform brightness correction on the stitched image.

[0011] Furthermore, in step S1, a background subtraction method based on Vibe is used to extract a moving target foreground set, and a process of constructing a seam line search model that does not include the foreground set according to the extracted foreground moving target set includes the following steps:

[0012] After the foreground moving target is found and matched successfully by the background subtraction method, two mask images I1 and I2 containing the moving target are obtained respectively. The pixel point set constituting the moving object in the two images is called the foreground set. The pixels in the foreground set are used to constrain the search model of the stitching line. The extracted foreground sets are expressed as follows:

[0013] G1={p1,p2,p3…p N}

[0014] G2={p′1, p′2, p′3…p′ M}

[0015] Where G1 represents the foreground set of the left image, G2 represents the foreground set of the right image, and p 1～N represents the pixels that make up the foreground set G1, p′ 1～M Represents the pixels constituting the foreground set G2. The foreground set of I1 contains N pixels, and the foreground set of I2 contains M pixels.

[0016] In the constructed seam line search model, the pixels in the two foreground sets are separated and do not participate in the energy intensity calculation. Then, when the corresponding position of the next row of a pixel and the three positions to its left and right contain foreground set pixels that do not participate in the calculation, the energy intensity of this point is calculated only with the two nearest non-foreground pixels and accumulated, and finally multiple seam lines are obtained.

[0017] Furthermore, in step S2, according to the spatial relationship between the two cameras, the midline of the overlapping visual field of the target image is obtained from the midline of the overlapping visual field of the image to be fused, and the process of obtaining the final mapping from the known transformation matrix includes the following steps:

[0018] The center line of the overlap is calculated using the following formula:

[0019]

[0020]

[0021]

[0022]

[0023] Where [u0, v0] represents the pixel value on the midline of the overlapping area of ​​the target image, [u1, v1] represents the pixel value on the midline of the overlapping area of ​​the image to be stitched, F0 represents the parameter matrix of the camera that shoots the target image, F1 represents the parameter matrix of the camera that shoots the image to be stitched, and Q represents the spatial transformation matrix between the two cameras. d 0，x Indicates the unit pixel size of the target image in the horizontal direction x, d 1，x Indicates the unit pixel size of the image to be stitched in the horizontal direction, d 0，y Indicates the unit pixel size of the target image in the vertical direction y; y and d y Respectively represent the size of the unit pixel in the x and y directions, and s, z, e and w represent the values ​​of the quaternion of the spatial transformation relationship between the two cameras;

[0024]

[0025] The center line [u1, v1] in the image to be fused is converted by the image registration transformation parameter T to obtain the center line [u2, v2] in the final fused image.

[0026] Furthermore, in step S3, the preset energy function is:

[0027] E=E c +E g +E f

[0028] Where E c 、E g and E f They are:

[0029] E c =|I1(i, j)-I2(i, j)|

[0030]

[0031]

[0032] Where I1 and I2 are the target image and the image to be stitched, respectively. c represents the intensity difference of the same pixel position in these two images, E g Represents the gradient difference of the same pixel position in two images, E f represents the gradient value at the seam line, where represents the pixel gradient of I1 at the i-th row and j-th column position, represents the pixel gradient of I2 at the i-th row and j-th column position, and k represents a constant between -1 and 1.

[0033] Furthermore, the suture lines are calculated on both sides of the midline respectively, and the respective optimal solutions are calculated according to the preset energy function. The process of calculating the final optimal suture line includes the following steps:

[0034] The local optimal solution is calculated using the following formula:

[0035]

[0036] G=G1∪G2

[0037] In the formula, S p (i, j) represents the sum of the energies of the pixel p located at the i-th row and j-th column and the previous pixel, E(i, j) represents the energy of the pixel located at (i, j), G1 represents the foreground set of the left image, G2 represents the foreground set of the right image, and G represents the set of two foreground sets; the three adjacent pixels in the next row of the current pixel, that is, any pixel p located in (i-1, j-1)(i-1, j)(i-1, j+1) * If it does not belong to the foreground set G, the calculation is normal. Otherwise, only the two pixels (i-1, ja) (i-1, jb) closest to the pixel (i, j) in the i-1th row are selected for calculation, where a and b represent the minimum and second minimum values ​​of the distance j in the i-1th row respectively; when the foreground set is included in the adjacent three pixels, the two pixels closest to the point in the non-foreground set and the pixel (i-1, j) are selected;

[0038] The global optimal solution is calculated using the following formula:

[0039] S o =min(S el (i, j), S er (i, j))

[0040] In the formula, S el(i, j) represents the optimal solution on the left, S er (i, j) represents the optimal solution on the right side, S o represents the global optimal solution.

[0041] Further, in step S3, the midline left suture line set L l It is expressed as:

[0042]

[0043] in Indicates the left suture line of the midline, From 1 to n;

[0044] Best suture line L on the left side of the midline lo It is expressed as:

[0045]

[0046] in Indicates that it is located The total energy of the pixels on ;

[0047] Midline right suture set L r It is expressed as:

[0048]

[0049] in Indicates the right suture line of the midline, From 1 to m;

[0050] Best suture line L on the right side of the midline ro It is expressed as:

[0051]

[0052] in Indicates that it is located The total energy of the pixels on .

[0053] Further, in step S4, a brightness correction model is constructed based on a learnable sparse convolutional neural network, and the brightness correction model includes a first color space conversion module, a common feature extraction network, a detail feature extraction network, a fusion correction module and a second color space conversion module;

[0054] The first color space conversion module is used to convert the part A of the source image to be spliced, the part B of the target image to be spliced, and the spliced ​​part C of the final image spliced ​​by the best stitching line from the RGB space into the YCbCr space; then the image Y1 of the part A of the source image to be spliced ​​after the color space conversion and the image Y2 of the part B of the target image to be spliced ​​after the color space conversion are input into the detail feature extraction network to extract the detail features, and the image Y3 of the spliced ​​image after the color space conversion is input into the common feature extraction network to extract the common features; the fusion correction module is used to fuse the extracted brightness features and color features respectively, and then fuse the fused brightness components and color components and convert them into the RGB space to complete the brightness correction of the spliced ​​part of the fused image.

[0055] Furthermore, the common feature extraction network is composed of four convolutional layers connected in series; the network structure of the independent feature extraction network is LSCNN, including a first convolutional layer, a learnable sparse convolutional network block, a residual network and a second convolutional layer connected in sequence; the learnable sparse convolutional network block includes two activation layers and two convolutional layers

[0056] Furthermore, in step S4, the brightness correction model is trained using a weighted structural similarity loss function, and the loss function is expressed as follows:

[0057] SSIM(i)=g1(i)×SSIM1(i)+g2(i)×SSIM2(i)+g3(i)×SSIM3(i)

[0058] in:

[0059]

[0060]

[0061]

[0062]

[0063] In the formula, is the mean of image sample x1, is the mean of image sample x2, μ y is the mean of image y, is the variance of image x1, is the variance of image x2, is the variance of image x3, σ y is the variance of image sample y, is the covariance of x3 and y, x1 and x2 represent the window areas of the two images respectively, x3 represents the window area of ​​the fused image, k′ is a constant ranging from 1 to 3, y is the brightness fusion output result, c1, c2, c3 are constants, and n′ ranges from 1 to 3.

[0064] Beneficial effects:

[0065] First, the present invention proposes an optimal seam line image fusion method based on the improvement of space-time relationship for virtual reality. The improvement of the optimal seam line method based on space-time relationship is mainly reflected in three aspects: first, the moving foreground set is obtained according to the time relationship when constructing the search model, and the seam line is successfully avoided from passing through the foreground moving target; second, the spatial relationship between the two cameras is introduced when solving the model, and the mapping of the midline of the overlapping part in the final image is obtained, and then the local optimal solution is obtained from both ends, which greatly improves the time complexity of the algorithm; third, the energy function is improved by introducing the pixels on both sides of the seam line pixel to predict the neighborhood pixel points, thereby improving the algorithm performance.

[0066] Second, the present invention also proposes a brightness correction method based on a learnable sparse convolutional neural network. The method takes the overlapping parts of two images to be spliced, the sum of the two images, and the spliced ​​parts of the spliced ​​images as input, converts the three images from RGB space to YCbCr space, and then fuses the brightness and chromaticity separately. The brightness fusion adopts an architecture for extracting common information and unique information separately, uses the proposed learnable sparse convolutional neural network to extract the unique information of the two images to be spliced, and uses an ordinary convolutional neural network to extract the common features in the spliced ​​images. Finally, the brightness features are fused and fused with the fused chromaticity components. After conversion to RGB space, the brightness correction of the image is completed.

[0067] Third, the present invention also proposes a weighted structural similarity loss function to train a learnable sparse convolutional neural network, so that the network can better learn features. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 Schematic diagram of the traditional optimal suture line search model;

[0069] Figure 2 A schematic diagram of the optimal suture line search model proposed by the present invention;

[0070] Figure 3 is an overall flow chart of the brightness correction method;

[0071] Figure 4 This is a diagram of the structure of a learnable sparse convolutional neural network;

[0072] Figure 5 This is a diagram of the structure of a learnable sparse convolution module;

[0073] Figure 6 This is the structure diagram of the residual learning module;

[0074] Figure 7 This is a flow chart of an optimal seam line image fusion method considering time and space relationship for virtual reality according to an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The following examples will enable those skilled in the art to more fully understand the present invention, but are not intended to limit the present invention in any way.

[0076] Figure 7 FIG. 1 is a flowchart of an optimal stitching line image fusion method considering time and space relationship for virtual reality according to an embodiment of the present invention. Figure 7 The present invention proposes an optimal stitching line image fusion method considering the time-space relationship for virtual reality, including the construction of a basic stitching line search model, the solution of the center line of the overlapping part of the fused image, the solution of the local optimal stitching line, the solution of the global optimal stitching line, and brightness correction. Specifically, the following steps are included:

[0077] Step 1: Use the background subtraction method based on Vibe to extract the foreground set of the moving target. After finding the foreground moving target and matching it successfully using the background subtraction method, two mask images I1 and I2 containing the moving target are obtained respectively. The pixel point set that constitutes the moving object in the two images is called the foreground set. The two foreground sets are expressed as:

[0078] G1(x)={p1,p2,p3 . . . p N}

[0079] G2(x)={p′1, p′2, p′3…p′ M}

[0080] Where G1 represents the foreground set of the left image, G2 represents the foreground set of the right image, p represents the pixels constituting the foreground set, the foreground set of I1 contains N pixels, and the foreground set of I2 contains M pixels.

[0081] In the newly constructed search model, the pixels in the two foreground sets are separated and do not participate in the energy intensity accumulation calculation. In this way, when the corresponding position of the next row of a pixel and the three positions to the left and right contain foreground set pixels that do not participate in the calculation, the energy intensity of this point is calculated only with the two nearest non-foreground pixels and accumulated, and finally multiple seam lines are obtained. The traditional optimal seam line search model is as shown in the attached manual. Figure 1 As shown in the figure, the search model diagram is as shown in the appendix of the manual. Figure 2As shown. The present invention constrains the pixels in the search process, shortens the seam line search time, and ensures that each seam line generated in the end will not pass through the foreground moving object. This ensures that all parts of the moving object come from the same image source when the final image is fused, avoids the splitting of the moving object caused by the moving objects on the left and right sides of the optimal seam line coming from two images, and can produce a better image fusion effect.

[0082] Step 2: The midline of the overlapping field of view between the two cameras is solved using the following formula:

[0083]

[0084]

[0085]

[0086]

[0087] Where [u0, v0] represents the pixel value on the midline of the overlapping area of ​​the target image, and [u1, v1] represents the pixel value on the midline of the overlapping area of ​​the image to be registered. F0 represents the parameter matrix of the camera that shoots the target image, and F1 represents the parameter matrix of the camera that shoots the image to be registered. M represents the spatial transformation matrix between the two cameras. d x and d y Respectively represent the size of the unit pixel in the x and y directions. x, z, y and w represent the values ​​of the quaternion of the spatial transformation relationship between the two cameras.

[0088] The center line [u1, v1] in the image to be fused is converted by the image registration transformation parameter T to obtain the center line in the final fused image:

[0089]

[0090] where [u2, v2] represents the centerline of the spliced ​​part of the final fused image.

[0091] Step 3: Use a new energy function to solve the local optimal solution, which takes into account the influence of the most similar pixels as well as the pixels on the left and right sides, so as to predict the fused image.

[0092] E=E c +E g +E f

[0093] E c =|I1(i, j)-I2(i, j)|

[0094]

[0095]

[0096] Where I1 and I2 are the target image and the image to be stitched, respectively. c represents the intensity difference of the same pixel position in these two images, E g Represents the gradient difference of the same pixel position in two images, E f It can also be regarded as the gradient value at the seam line. Gradient minimization can make the seam line avoid the foreground object better to a certain extent, thus preventing the seam line from becoming the boundary of the image.

[0097] In step 3, the set of sutures on the left side of the midline can be expressed as:

[0098]

[0099] The optimal suture line to the left of the midline can be expressed as:

[0100]

[0101] The set of sutures on the right side of the midline can be expressed as:

[0102]

[0103] The optimal suture line to the right of the midline can be expressed as:

[0104]

[0105] Traversing the energy map, the total energy S accumulated by the pixels on the seam line e Minimum:

[0106]

[0107] G=G1∪G2

[0108] In the formula, S p (i, j) represents the sum of the energy of the pixel p at (i, j) and the previous pixel, E(i, j) represents the energy of the pixel at (i, j), G represents the set of two foreground sets, and the formula shows that any one of the three adjacent pixels in the next row of the current pixel p * If it does not belong to the foreground set G, the calculation is normal. Otherwise, only the two pixels closest to j are calculated. a and b represent the two pixels closest to j when the three adjacent pixels in the next row contain the foreground set.

[0109] The global optimal solution is calculated using the following formula:

[0110] S o =min(S el(i, j), S er (i, j))

[0111] Where S e l(i, j) represents the optimal solution on the left, S er (i, j) represents the optimal solution on the right side, S o represents the global optimal solution.

[0112] Step 4: Perform brightness correction on the spliced ​​part of the fused image, such as Figure 3 As shown. The input of the method consists of three parts, namely the part A of the source image to be spliced, the part B of the target image to be spliced, and the spliced ​​part C of the final image spliced ​​by the best stitching line. First, the three images are converted from the RGB space to the YCbCr space. Y represents the brightness space. Cb represents the blue component of the color space. Cr represents the red component of the color space. Since the human eye is more aware of the brightness channel and many information and features in the image are contained in this channel, the brightness and chromaticity are fused separately. First, for the fusion of the brightness space, the present invention proposes a learnable sparse convolutional network LSCNN (Learnable Sparse Convolutional Neural Network), which extracts the detail information of the image to be spliced ​​and the common information of the two images to be spliced ​​contained in the spliced ​​image, and fuses them.

[0113] The schematic diagram of the brightness channel neural network is as follows Figure 4 As shown in FIG. 1 , the image Y1 of the source image to be spliced ​​after color space conversion and the image Y2 of the target image to be spliced ​​after color space conversion are input into the detail feature extraction network, and the image Y3 of the spliced ​​image after color space conversion is input into the common feature extraction network. Figure 4 As shown in the left branch, it first passes through a convolutional layer, then enters a learnable sparse network block, followed by a residual network, and finally passes through a convolutional layer. The learnable sparse convolutional network block is as follows Figure 5 As shown in Figure 1, it includes two activation layers and two convolutional layers. The sparse network blocks that can be learned are as follows: Figure 6 As shown in the figure, it includes two convolutional layers. The common feature extraction network is relatively simple, consisting of four convolutional layers. The first three layers have the same structural parameters, all of which are 3×3 convolution kernels, and the number of feature maps output is 32. The number of feature maps output by the last convolutional layer is 1. Finally, the common information and independent information are synthesized, and the output result is the fused brightness. The brightness channel is combined with the chrominance channel to complete the brightness correction of the spliced ​​part of the fused image.

[0114] For the learnable sparse convolutional neural network in step S4, the present invention proposes a new loss function. The depth of the detail extraction network branch is deeper than the common feature extraction network branch. Although such a network structure can achieve the predetermined effect, training with an appropriate loss function can make the network fusion performance even better. The present invention selects the weighted structural similarity WSSIM (Weighted Structural SIMilarity) between the input image and the output final brightness fusion image as the loss function to measure the richness of the fused image information. The general SSIM consists of three parts, l(x, y) represents the brightness between image x and image y, c(x, y) represents the contrast, and s(x, y) represents the structure:

[0115] SSIM(x,y)=l(x,y)c(x,y)s(x,y)

[0116]

[0117]

[0118]

[0119] where μ x is the mean of image sample x, μ y is the mean of image y, σ x is the variance of image x, σ y is the variance of image sample y, σ xy is the covariance of x and y. The weighted structural similarity function proposed in the present invention is shown in the following formula:

[0120] SSIM(i)=g1(i)×SSIM1(i)+g2(i)×SSIM2(i)+g3(i)×SSIM3(i)

[0121]

[0122]

[0123]

[0124]

[0125] x1 and x2 represent the window areas of the two images to be fused that are input into the LSCNN detail processing network, x3 represents the window area of ​​the spliced ​​image that is input into the LSCNN common information processing network, and less represents the output result of brightness fusion. The window is a 5×5 area centered on pixel i, and |||| represents the search for the 2-normal form.

[0126] It should be pointed out that the description of the above embodiments is only used to help understand the method of the present application and its core idea. For ordinary technicians in this technical field, several improvements and modifications can be made to the present application without departing from the principles of the present application. These improvements and modifications are also within the scope of protection of the claims of the present application.

Claims

1. An optimal seam line image fusion method considering spatiotemporal relationship for virtual reality, characterized in that: The image fusion method comprises the following steps: S1. A background subtraction method based on Vibe is used to extract a moving target foreground set, and a seam line search model that does not include the foreground set is constructed according to the extracted foreground moving target set; S2. According to the spatial relationship between the two cameras, the midline of the overlapping visual field of the target image is obtained from the midline of the overlapping visual field of the image to be fused, and the final mapping is obtained from the known transformation matrix; S3, calculating the suture lines on both sides of the midline respectively, calculating the respective optimal solutions according to the preset energy function, and calculating the final optimal suture line; S4, using a linear fusion algorithm to stitch the images on both sides of the best stitching line; constructing a brightness correction model based on a learnable sparse convolutional neural network, the brightness correction model combines the source image to be stitched part A, the target image to be stitched part B and the stitched part C of the final image stitched through the best stitching line, and performs brightness correction on the stitched image; The suture lines are calculated on both sides of the midline, and the respective optimal solutions are calculated according to the preset energy function. The process of calculating the final optimal suture line includes the following steps: The local optimal solution is calculated using the following formula: G=G1∪G2 In the formula, S p (i,j) represents the sum of the energies of the pixel p located at the i-th row and j-th column and the previous pixel, E(i,j) represents the energy of the pixel located at (i,j), G1 represents the foreground set of the left image, G2 represents the foreground set of the right image, and G represents the set of two foreground sets; the three adjacent pixels in the next row of the current pixel, that is, any pixel p located in (i-1,j-1)(i-1,j)(i-1,j+1) * If it does not belong to the foreground set G, the calculation is normal. Otherwise, only the two pixels (i-1, ja) (i-1, jb) closest to the pixel (i, j) in the i-1th row are selected for calculation, where a and b represent the minimum and second minimum values ​​of the distance j in the i-1th row respectively; when the foreground set is included in the adjacent three pixels, the two pixels closest to the point in the non-foreground set and the pixel (i-1, j) are selected; The global optimal solution is calculated using the following formula: S o =min(S el (i,j),S er (i,j)) In the formula, S el (i,j) represents the optimal solution on the left, S er (i,j) represents the optimal solution on the right side, S o represents the global optimal solution.

2. The optimal seam line image fusion method considering time and space relationship for virtual reality according to claim 1 is characterized in that: In step S1, a background subtraction method based on Vibe is used to extract a moving target foreground set, and a process of constructing a seam line search model that does not include the foreground set according to the extracted foreground moving target set includes the following steps: After the foreground moving target is found and matched successfully by the background subtraction method, two mask images I1 and I2 containing the moving target are obtained respectively. The pixel point set constituting the moving object in the two images is called the foreground set. The pixels in the foreground set are used to constrain the search model of the stitching line. The extracted foreground sets are expressed as follows: G1={p1,p2,p3…p N } G2={p'1,p'2,p'3…p' M } Where G1 represents the foreground set of the left image, G2 represents the foreground set of the right image, and p 1～N represents the pixels that make up the foreground set G1, p' 1～M Represents the pixels constituting the foreground set G2. The foreground set of I1 contains N pixels, and the foreground set of I2 contains M pixels. In the constructed seam line search model, the pixels in the two foreground sets are separated and do not participate in the energy intensity calculation. Then, when the corresponding position of the next row of a pixel and the three positions to its left and right contain foreground set pixels that do not participate in the calculation, the energy intensity of this point is calculated only with the two nearest non-foreground pixels and accumulated, and finally multiple seam lines are obtained.

3. The optimal stitching line image fusion method considering time and space relationship for virtual reality according to claim 1 is characterized in that: In step S2, according to the spatial relationship between the two cameras, the midline of the overlapping visual field of the target image is obtained from the midline of the overlapping visual field of the image to be fused, and the process of obtaining the final mapping from the known transformation matrix includes the following steps: The center line of the overlap is calculated using the following formula: Where [u0, v0] represents the pixel value on the midline of the overlapping area of ​​the target image, [u1, v1] represents the pixel value on the midline of the overlapping area of ​​the image to be stitched, F0 represents the parameter matrix of the camera that shoots the target image, F1 represents the parameter matrix of the camera that shoots the image to be stitched, and Q represents the spatial transformation matrix between the two cameras. d 0,x Indicates the unit pixel size of the target image in the horizontal direction x, d 1,x Indicates the unit pixel size of the image to be stitched in the horizontal direction, d 0,y Indicates the unit pixel size of the target image in the vertical direction y; y and d y Respectively represent the size of the unit pixel in the x and y directions, and s, z, e and w represent the values ​​of the quaternion of the spatial transformation relationship between the two cameras; The center line [u1, v1] in the image to be fused is converted by the image registration transformation parameter T to obtain the center line [u2, v2] in the final fused image.

4. The optimal seam line image fusion method considering time and space relationship for virtual reality according to claim 1, characterized in that: In step S3, the preset energy function is: E=E c +E g +E f Where E c 、E g and E f They are: E c =|I1(i,j)-I2(i,j) Where I1 and I2 are the target image and the image to be stitched, respectively. c represents the intensity difference of the same pixel position in these two images, E g Represents the gradient difference of the same pixel position in two images, E f represents the gradient value at the seam line, where represents the pixel gradient of I1 at the i-th row and j-th column position, represents the pixel gradient of I2 at the i-th row and j-th column position, and k represents a constant between -1 and 1.

5. The optimal seam line image fusion method considering time and space relationship for virtual reality according to claim 1, characterized in that: In step S3, the midline left suture line set L l It is expressed as: in Indicates the left suture line of the midline, From 1 to n; Best suture line L on the left side of the midline lo It is expressed as: in Indicates that it is located The total energy of the pixels on ; Midline right suture set L r It is expressed as: in Indicates the midline right suture line, From 1 to m; Best suture line L on the right side of the midline ro It is expressed as: in Indicates that it is located The total energy of the pixels on .

6. The optimal stitching line image fusion method considering time and space relationship for virtual reality according to claim 1, characterized in that: In step S4, a brightness correction model is constructed based on a learnable sparse convolutional neural network, and the brightness correction model includes a first color space conversion module, a common feature extraction network, a detail feature extraction network, a fusion correction module, and a second color space conversion module; The first color space conversion module is used to convert the to-be-joined part A of the source image, the to-be-joined part B of the target image, and the joined part C of the final image joined by the best stitching line from the RGB space into the YCbCr space; Then, the image Y1 of the source image to be spliced ​​part A after color space conversion and the image Y2 of the target image to be spliced ​​part B after color space conversion are input into the detail feature extraction network to extract detail features, and the image Y3 of the spliced ​​image after color space conversion is input into the common feature extraction network to extract common features; The fusion correction module is used to fuse the extracted brightness features and color features respectively, and then fuse the fused brightness components and color components and convert them into RGB space to complete the brightness correction of the spliced ​​part of the fused image.

7. The optimal stitching line image fusion method considering time and space relationship for virtual reality according to claim 6, characterized in that: The common feature extraction network is composed of four convolutional layers connected in series; the network structure of the detail feature extraction network is LSCNN, including a first convolutional layer, a learnable sparse convolutional network block, a residual network and a second convolutional layer connected in sequence; the learnable sparse convolutional network block includes two activation layers and two convolutional layers.

8. The optimal stitching line image fusion method considering time and space relationship for virtual reality according to claim 1, characterized in that: In step S4, the brightness correction model is trained using a weighted structural similarity loss function, and the loss function is expressed as follows: SSIM(i)=g1(i)×SSIM1(i)+g2(i)×SSIM2(i)+g3(i)×SSIM3(i) in: In the formula, is the mean of image sample x1, is the mean of image sample x2, μ y is the mean of image y, is the variance of image x1, is the variance of image x2, is the variance of image x3, σ y is the variance of image sample y, is the covariance of x3 and y, x1 and x2 represent the window areas of the two images respectively, x3 represents the window area of ​​the fused image, k′ is a constant ranging from 1 to 3, y is the brightness fusion output result, c1, c2, c3 are constants, and n′ ranges from 1 to 3.

Images