Brightness-robust panoramic image stitching generation method and device

Abstract

According to the brightness-robust panoramic image splicing generation method and device provided by the invention, the feature point matching of the image is assisted by referring to the feature point matching result of the motion field which is slightly influenced by the brightness change, and the influence of the visual angle brightness change on the feature point matching is reduced. Moreover, in order to achieve natural color transition at the crack, pixel values from the two view images are weighted based on the distance. Based on the provided motion guide feature point matching and suture line brightness transition method, panoramic image splicing generation with relatively high brightness robustness can be realized.

Description

Technical Field

[0001] The present invention relates to the field of computational imaging, and in particular to a method and device for generating panoramic image stitching with robust brightness. Background Art

[0002] With the continuous advancement of image sensors and digital image processing technology, the image quality of digital imaging devices has been significantly improved and has been widely used in many fields. For example, the spatial resolution of images has gradually increased from the initial low resolution of hundreds of thousands of pixels to the current million pixels or even tens of millions of pixels, achieving significant development. However, for emerging application fields such as virtual reality, map making, and tourism navigation, the field of view of images captured by traditional digital devices is relatively limited, which has become one of the bottlenecks in their development.

[0003] As a key image acquisition and processing technology, panoramic imaging technology has been widely used in scenes that require wide viewing angle coverage. Panoramic imaging technology uses special imaging equipment to achieve ultra-wide viewing angle image acquisition. The core of the technology is to form a seamless, panoramic image by stitching multiple images together. Panoramic images are particularly important in virtual reality. Through virtual video devices, users can have an immersive experience in a virtual environment. Whether it is virtual tourism or remote conferencing, panoramic images can provide realistic visual experience. In the field of map making and tourism navigation, panoramic images can improve navigation accuracy and user experience by displaying real street view information.

[0004] Panoramic imaging devices usually use multi-sensor array technology to capture images in different directions simultaneously through multiple cameras. These cameras cover the entire scene according to a certain arrangement, such as a ring arrangement or a matrix arrangement. The advantage of a multi-sensor array is that it can capture images from multiple perspectives at the same time, thereby reducing the shooting time and the timing errors that occur during the image stitching process. Another way to achieve panoramic imaging is to use rotating lens technology. A single camera gradually captures all directions of the scene by rotating or swinging the lens. Although this method has a slower shooting speed, it can reduce equipment costs and is suitable for shooting needs in some fixed scenes. The rotating lens technology ensures that the overlapping parts of each image are consistent by precisely controlling the rotation angle of the lens, providing high-quality raw data for subsequent image stitching.

[0005] In the actual shooting process, the quality of panoramic images will be affected by changes in lighting conditions. Specifically, the angle and intensity of sunlight, changes in shadows, etc. will lead to inconsistent image brightness in different directions. In addition, the exposure parameters of each camera of a panoramic camera may be different. Even cameras of the same model may have different exposure effects due to individual differences. These problems will bring challenges to the stitching and fusion of panoramic images, and will appear as inconsistent brightness when they are finally stitched. Summary of the invention

[0006] The present invention provides a brightness-robust panoramic image stitching generation method and device, which are used to provide a brightness-robust panoramic image stitching generation method to reduce the brightness influence of adjacent viewing angle video frame images.

[0007] The present invention provides a brightness robust panoramic image stitching generation method, comprising: obtaining video frame sequences corresponding to each viewing angle of a panoramic camera; based on the optical flow field registration results of the video frame sequences at each two adjacent viewing angles in the respective viewing angles, calibrating the image registration results of the video frame sequences at each two adjacent viewing angles, and determining an image feature point matching set; the image feature point matching set is used to reflect the relative positions and overlapping areas between video frame images at the same time in the video frame sequences at adjacent viewing angles; based on the distance between the pixel points in the overlapping area and the optimal stitching line, determining the brightness signal weights corresponding to the corresponding pixel points in the video frame images at the target time at the adjacent viewing angles; wherein the optimal stitching line is the optimal stitching line of the video frame images at the target time at the adjacent viewing angles determined based on the image feature point matching set; after fusing the brightness signals of the video frame images at the target time at the adjacent viewing angles based on the brightness signal weights, the brightness signals are fused with the texture signals of the video frame images at the target time at the adjacent viewing angles to determine a panoramic image.

[0008] In one of the embodiments, based on the optical flow field registration results of the video frame sequence at every two adjacent viewing angles in the respective viewing angles, the image registration results of the video frame sequence at every two adjacent viewing angles are calibrated to determine a matching set of image feature points, including: determining a changing speed of the scene in the video frame image at the corresponding viewing angle based on the sum of the absolute values ​​of the pixel values ​​in the comprehensive optical flow field of each viewing angle; for each matching feature pair in the video frame sequence at every two adjacent viewing angles, performing the following steps respectively: determining the changing speed of the scene as the weight of the confidence of the optical flow field registration result, and determining the weighted confidence of each matching feature pair based on the weight of the confidence of the optical flow field registration result; and determining the matching set of image feature points based on the weighted confidence of each matching feature pair.

[0009] In one of the embodiments, the determining the weighted confidence of each matching feature pair based on the weight of the confidence of the optical flow field registration result includes: determining the weight of the confidence of the corresponding video frame image registration result based on the weight of the confidence of the optical flow field registration result; adding the product of the confidence of the optical flow field registration result and the weight of the confidence of the optical flow field registration result, and the product of the confidence of the corresponding video frame image registration result and the weight of the confidence of the corresponding video frame image registration result, to determine the weighted confidence of each matching feature pair; the determining the image feature point matching set based on the weighted confidence of each matching feature pair includes: when the weighted confidence of the matching feature pair is greater than a preset threshold, determining the matching feature pair as a matching feature pair in the image feature point matching set.

[0010] In one of the embodiments, before calibrating the image registration results of the video frame sequence at every two adjacent viewing angles based on the optical flow field registration results of the video frame sequence at every two adjacent viewing angles, the method further includes: performing optical flow field feature point detection and registration on the video frame sequence at every two adjacent viewing angles to determine the corresponding optical flow field registration results at every two adjacent viewing angles.

[0011] In one of the embodiments, the optical flow field feature point detection and alignment of the video frame sequence under every two adjacent viewing angles to determine the optical flow field alignment results corresponding to every two adjacent viewing angles include: for the video frame sequence under each viewing angle, determining the average value of the optical flow field from at least one video frame image at a time point to be reconstructed in the video frame sequence to the video frame images at other non-reconstructed time points; determining the average value as the comprehensive optical flow field of the corresponding viewing angle; and performing feature point detection and alignment on the comprehensive optical flow fields corresponding to each two adjacent viewing angles to determine the optical flow field alignment results for each two adjacent viewing angles.

[0012] In one of the embodiments, the method of determining the brightness signal weights corresponding to the corresponding pixels in the video frame image at the target moment under adjacent viewing angles based on the distance between the pixels in the overlapping area and the optimal stitching line comprises: for each pixel in the overlapping area, based on the distance between the pixel and the optimal stitching line, and the length between the pixel and the overlapping area, determining the brightness signal weights corresponding to the pixels on the same side and the pixels on the opposite side of the pixel; fusing the brightness signals of the video frame images at the target moment under adjacent viewing angles based on the brightness signal weights, and fusing them with the texture signals of the video frame images at the target moment under adjacent viewing angles to determine the panoramic image comprises: determining the fused brightness signal based on the brightness signal weights corresponding to the pixels on the same side and the pixels on the opposite side; fusing the fused brightness signal with the pre-processed texture signal to obtain the panoramic image.

[0013] In one of the embodiments, the method of determining the brightness signal weights corresponding to the pixel points on the same side and the pixel points on the opposite side of the pixel point based on the distance between the pixel point and the optimal stitching line and the length between the pixel point and the overlapping area, includes: adding the ratio of the distance and the length to a preset constant to obtain a first weight of the pixel points on the same side; subtracting the result of the first weight from 1 to determine the second weight of the pixel points on the opposite side; and determining the fused brightness signal based on the brightness signal weights corresponding to the pixel points on the same side and the pixel points on the opposite side, includes: summing the product of the first weight and the low-frequency signal corresponding to the pixel points on the same side and the product of the second weight and the low-frequency signal corresponding to the pixel points on the opposite side to obtain the fused brightness signal.

[0014] The present invention also provides a brightness robust panoramic image stitching generation device, comprising: an acquisition module, used to acquire the video frame sequences corresponding to each viewing angle of the panoramic camera; a first determination module, used to calibrate the image registration results of the video frame sequences under each two adjacent viewing angles based on the optical flow field registration results of the video frame sequences under each two adjacent viewing angles, and determine an image feature point matching set; the image feature point matching set is used to reflect the relative position and overlapping area between the video frame images at the same time in the video frame sequences under adjacent viewing angles; a second determination module, used to determine the brightness signal weights corresponding to the corresponding pixel points in the video frame images at the target time under adjacent viewing angles based on the distance between the pixel points in the overlapping area and the optimal stitching line; wherein the optimal stitching line is the optimal stitching line of the video frame images at the target time under the adjacent viewing angle determined based on the image feature point matching set; a third determination module, used to fuse the brightness signals of the video frame images at the target time under the adjacent viewing angles based on the brightness signal weights, and then fuse them with the texture signals of the video frame images at the target time under the adjacent viewing angles to determine the panoramic image.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned brightness-robust panoramic image stitching generation methods are implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned brightness-robust panoramic image stitching generation methods are implemented.

[0017] The brightness-robust panoramic image stitching generation method and device provided by the present invention assists the image feature point matching by referring to the feature point matching results of the motion field (optical flow field) that is less affected by brightness changes, thereby reducing the influence of the brightness change of the viewing angle on the feature point matching. In addition, in order to achieve a natural color transition at the gap, the pixel values ​​from the two viewing angle images are weighted based on the distance. Based on the proposed motion-guided feature point matching and the brightness transition method of the stitching line, the present invention can achieve relatively brightness-robust panoramic image stitching generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a schematic diagram of an application scenario provided by the present invention;

[0020] Figure 2 This is one of the flow charts of the brightness robust panoramic image stitching generation method provided by the present invention;

[0021] Figure 3 This is the second flow chart of the brightness robust panoramic image stitching generation method provided by the present invention;

[0022] Figure 4 This is a third flow chart of the brightness robust panoramic image stitching generation method provided by the present invention;

[0023] Figure 5 This is a fourth flow chart of the brightness robust panoramic image stitching generation method provided by the present invention;

[0024] Figure 6 is a schematic diagram of the overlapping area and the optimal suture line provided by the present invention;

[0025] Figure 7 It is a schematic diagram of feature point registration for auxiliary calibration of image registration results based on optical flow field registration results provided by the present invention;

[0026] Figure 8 It is a structural schematic diagram of a brightness robust panoramic image stitching generation device provided by the present invention;

[0027] Fig. 9 It is a schematic diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] To facilitate understanding, the inventive concept of the present invention is first explained.

[0030] The core of panoramic image generation lies in the image stitching algorithm. The image stitching algorithm mainly includes the steps of feature point detection, image registration and image fusion. First, the key points with significant features are extracted from the image through the feature point detection algorithm. Then, the feature points in adjacent images are matched through the image registration algorithm to determine the relative position and overlapping area between the images. Finally, the image fusion algorithm is used to seamlessly combine multiple images into a panoramic image.

[0031] However, in the actual shooting process, the quality of panoramic images will be affected by changes in lighting conditions. Specifically, the angle and intensity of sunlight, changes in shadows, etc. will lead to inconsistent image brightness in different directions. In addition, the exposure parameters of each camera of a panoramic camera may be different. Even cameras of the same model may have different exposure effects due to individual differences. These problems will bring challenges to the stitching and fusion of panoramic images, and will appear as inconsistent brightness when they are finally stitched.

[0032] The present invention aims to design a brightness-robust panoramic image stitching generation method, which assists the image feature point matching by referring to the feature point matching results of the motion field that is less affected by brightness changes, thereby reducing the influence of the brightness change of the viewing angle on the feature point matching. In addition, in order to achieve a natural color transition at the gap, the pixel values ​​from the two viewing angles are weighted based on the distance. Based on the proposed motion-guided feature point matching and the brightness transition method of the seam line, the present invention can achieve relatively brightness-robust panoramic image stitching generation.

[0033] The brightness robust panoramic image stitching generation method and device of the present invention are described below with reference to the accompanying drawings.

[0034] First, combine Figure 1 The application scenarios of the present invention are described as follows: Figure 1 As shown, based on a panoramic camera with n viewing angles, the scene is continuously photographed to obtain a sequence of m frames corresponding to each viewing angle at synchronized time points. Where j represents the time index of the video frame image, i represents the index of the panoramic camera viewing angle, It represents the jth video frame image shot by the i-th viewing angle. Generally speaking, m should not be too large, for example, equal to m=3, otherwise it is easy to cause complex motion and affect the accuracy of subsequent feature point matching.

[0035] Figure 2 1 is a flow chart of a brightness robust panoramic image stitching generation method provided by the present invention. It can be understood that the brightness robust panoramic image stitching generation method can be executed by a brightness robust panoramic image stitching generation device, such as Figure 2 As shown, the method comprises the following steps:

[0036] Step 210: Obtain video frame sequences corresponding to each viewing angle of the panoramic camera.

[0037] The video frame sequence is a video frame image corresponding to a period of time. Specifically, the following can be obtained: Figure 1 The video frame sequences corresponding to the same scene at different viewing angles of the panoramic camera are shown.

[0038] Step 220, based on the optical flow field registration results of the video frame sequence at each two adjacent viewing angles in the said viewing angles, calibrate the image registration results of the video frame sequence at each two adjacent viewing angles to determine the image feature point matching set; wherein the image feature point matching set is used to reflect the relative position and overlapping area between the video frame images at the same time in the video frame sequence at adjacent viewing angles.

[0039] In combination with the foregoing, the video frame sequence includes video frame images at at least one moment within a period of time.

[0040] Among them, the image registration result can be understood as directly based on the feature point detection, image registration and other processes in the existing image stitching algorithm, the video frame images at each moment in the video frame sequence of adjacent perspectives are registered respectively, and the initial relative position and overlapping area between the video frame images at the same moment in the video frame sequence of two adjacent perspectives are obtained. The optical flow field registration result can be understood as the result obtained by performing optical flow estimation on the video frame sequence at each perspective, and then performing feature point detection and registration of the optical flow field, which is used to calibrate the image registration result.

[0041] It can be understood that the optical flow field in the same scene is less affected by the brightness changes of each viewing angle. Therefore, it is proposed to calibrate the image registration structure through the optical flow field registration result to obtain a more accurate registration result.

[0042] Specifically, it can be combined Figure 7 Understand the calibration process described above.

[0043] Step 230 , for each pixel point in the overlapping area, based on the distance between the pixel point and the optimal stitching line, determine the brightness signal weights corresponding to the corresponding pixel points in the video frame image at the target moment under adjacent viewing angles.

[0044] The optimal stitching line is the optimal stitching line of the video frame image at the target moment under adjacent viewing angles determined based on the image feature point matching set. Specifically, an optimal stitching line search algorithm can be used to determine the optimal stitching line of the video frame image at the target moment under adjacent viewing angles from the image feature point matching set. The optimal stitching line search algorithm can be, for example, a graph cut algorithm, a dynamic programming algorithm, or an energy minimization algorithm, etc., which is not limited in the present invention. For example, a dynamic programming algorithm is used to find a stitching line that minimizes the stitching error in images of adjacent viewing angles.

[0045] The video frame images at the target moment under the adjacent viewing angles can be understood as the video frame images corresponding to the target moment in the video frame sequences under the two adjacent viewing angles. The corresponding pixel points correspond to the pixel points in the overlapping area, which can be understood as the pixels at the same position in the overlapping area in the video frame images corresponding to the target moment under the two adjacent viewing angles.

[0046] It can be understood that in order to achieve the gradual fusion of the brightness of the stitching line, the brightness signal weight of the pixel points in the overlapping area is made relevant to the distance of the optimal stitching line.

[0047] Step 240 , after fusing the brightness signals of the video frame images at the target moment under the adjacent viewing angles based on the brightness signal weights, the brightness signals are fused with the texture signals of the video frame images at the target moment under the adjacent viewing angles to determine a panoramic image.

[0048] The brightness-robust panoramic image stitching generation method provided by the present invention assists the image feature point matching by referring to the feature point matching results of the motion field (optical flow field) that is less affected by brightness changes, thereby reducing the influence of the brightness change of the viewing angle on the feature point matching. In addition, in order to achieve a natural color transition at the gap, the pixel values ​​from the two viewing angle images are weighted based on the distance. Based on the proposed motion-guided feature point matching and the brightness transition method of the stitching line, the present invention can achieve relatively brightness-robust panoramic image stitching generation.

[0049] In one of the embodiments, before the above step 220, the method further includes: performing optical flow field feature point detection and registration on the video frame sequences under every two adjacent viewing angles, and determining the optical flow field registration results corresponding to the video frame sequences under every two adjacent viewing angles.

[0050] Specifically, Figure 3As shown, the optical flow field feature point detection and registration are performed on the video frame sequences under each two adjacent viewing angles to determine the optical flow field registration results corresponding to each two adjacent viewing angles, including the following steps:

[0051] Step 310: for each video frame sequence at each viewing angle, determine the average value of the optical flow field from at least one video frame image at a time point to be reconstructed in the video frame sequence to other video frame images at non-reconstructed time points.

[0052] It can be understood that in order to comprehensively utilize more temporal motion information, the average value of all optical flow fields of a perspective can be taken to obtain the comprehensive optical flow field under this perspective.

[0053] Step 320: determine the average value as the comprehensive optical flow field of the corresponding viewing angle.

[0054] The average value may be an arithmetic average value or a weighted average value, which is not limited in the present invention.

[0055] Here, we take the arithmetic mean as an example to illustrate the calculation process of the comprehensive optical flow field. For example, the comprehensive optical flow field of the i-th perspective The corresponding mathematical expression is:

[0056]

[0057] in, Represents the video frame image at each viewing angle at the time point to be reconstructed To the optical flow field of other frames. The corresponding mathematical expression is:

[0058]

[0059] Among them, t is fixed and represents the index of the time point to be reconstructed, and k represents any non-reconstruction time. It represents an optical flow estimation algorithm. Specifically, the optical flow field can be determined using any optical flow estimation algorithm based on deep learning, classical optical flow method, dense optical flow method and sparse optical flow method, and the present invention is not limited to this.

[0060] Step 330 , performing feature point detection and registration for the integrated optical flow fields corresponding to each two adjacent perspectives, so as to determine the optical flow field registration result for each two adjacent perspectives.

[0061] Specifically, for each perspective, the comprehensive optical flow field Feature point detection algorithms can be used to detect feature points. Feature point detection algorithms include but are not limited to (Scale-invariant feature transform, SIFT) algorithm, (Speeded up robust features, SURF) algorithm and (Oriented FAST and rotated BRIEF, ORB) algorithm. Based on the result of feature point detection, feature point matching algorithms can be used to match feature points in the optical flow field. For example, the RANSAC algorithm is used to align the image and the optical flow field respectively, eliminate wrongly matched feature point pairs, and obtain a more accurate matching relationship. Similarly, Figure 3 As shown, for each video frame image at time t The feature point detection algorithm is also first used to check the feature points, and then the aligned feature points are obtained based on the feature point matching algorithm.

[0062] In one embodiment, if Figure 4 As shown, based on the optical flow field registration results of the video frame sequences under each two adjacent viewing angles in the viewing angles, the image registration results of the video frame sequences under each two adjacent viewing angles are calibrated to determine the image feature point matching set, including steps 410 to 430.

[0063] Step 410 : determining the change speed of the scene in the video frame image at the corresponding viewing angle based on the sum of the absolute values ​​of the pixel values ​​in the integrated optical flow field at each viewing angle.

[0064] Among them, the speed of scene change can be understood as the speed of scene movement.

[0065] In one embodiment, the comprehensive optical flow field of the i-th viewing angle can be calculated The sum of the absolute values ​​of the pixel values ​​in the image is used to roughly estimate the speed of the motion.

[0066] Step 420, for each matching feature pair in the video frame sequence under every two adjacent perspectives, perform the following steps respectively: determine the changing speed of the scene as the weight of the confidence of the optical flow field registration result, and determine the weighted confidence of each matching feature pair based on the weight of the confidence of the optical flow field registration result.

[0067] It can be understood that the present application uses both optical flow field registration and image registration. Therefore, the weighted confidence of each matching feature pair can be understood as including the confidence of the optical flow field registration result and the confidence of the image registration result.

[0068] It can also be understood that in order to fuse the optical flow field registration results and the image registration results, the speed of scene movement needs to be considered. Specifically, when the scene moves faster, a greater weight is given to the feature point matching of the optical flow field, that is, when the scene changes faster, a greater weight is given to the confidence of the optical flow field registration result; when the sum of the absolute values ​​is almost 0, it means that the scene changes slowly, so the weight w of the confidence of the optical flow field registration result can be made larger. optical flow is 0.

[0069] Step 430: Determine a matching set of image feature points based on the weighted confidence of each matching feature pair.

[0070] In one embodiment, step 430 may specifically include: when a weighted confidence of the matching feature pair is greater than a preset threshold, determining the matching feature pair as a matching feature pair in the image feature point matching set.

[0071] In one embodiment, if Figure 5 As shown, the step 420 described above of determining the weighted confidence of each matching feature pair based on the weight of the confidence of the optical flow field registration result includes the following steps 510 to 520.

[0072] Step 510: Determine the confidence weight of the corresponding video frame image registration result based on the confidence weight of the optical flow field registration result.

[0073] Specifically, the sum of the weight of the confidence of the optical flow field registration result and the weight of the confidence of the image registration result can be set to 1, that is, w image +w optical flow =1, where w image Represents the confidence weight of the image registration result, w optical flow The weight representing the confidence of the optical flow field registration result.

[0074] Step 520, sum the product of the confidence of the optical flow field registration result and the weight of the confidence of the optical flow field registration result, and the product of the confidence of the corresponding video frame image registration result and the weight of the confidence of the corresponding video frame image registration result, to determine the weighted confidence of each matching feature pair.

[0075] Specifically, each matching feature pair (p i ,p j The calculation formula of the weighted confidence of ) is: weighted (p i ,p j )=w image ·c image (p i ,p j (+woptical flow ·c optical flow (p i ,p j ), where c image (p i ,p j ) represents the confidence of the image registration result, c optical flow (p i ,p j ) represents the confidence of the optical flow field registration result, w image Represents the confidence weight of the image registration result, w optical flow The weight representing the confidence of the optical flow field registration result.

[0076] In one of the embodiments, based on the distance between the pixel point in the overlapping area and the optimal stitching line, the brightness signal weights corresponding to the corresponding pixel points in the video frame image at the target moment under adjacent viewing angles are determined, including: for each pixel point in the overlapping area, based on the distance between the pixel point and the optimal stitching line, and the length between the pixel point and the overlapping area, determining the brightness signal weights corresponding to the pixel point on the same side and the pixel point on the opposite side of the pixel point.

[0077] Specifically, the overlapping area and the optimal stitching line can refer to Figure 6 .

[0078] It can be understood that for any pixel P(x,y) in the overlapping area, its prediction value comes from the pixels P1(x,y) and P2(x,y) corresponding to the matching feature pairs in the adjacent view registration results. Here, P1(x,y) is the pixel on the same side of the best stitching line as P(x,y), referred to as the pixel on the same side of the pixel point P(x,y). Similarly, P2(x,y) is the pixel on the opposite side of the best stitching line as P(x,y), referred to as the pixel on the opposite side of the pixel point P(x,y).

[0079] In one of the embodiments, the brightness signal weights corresponding to the pixel points on the same side and the pixel points on the opposite side of the pixel point are determined based on the distance between the pixel point and the optimal stitching line, and the length between the pixel point and the overlapping area, including: adding the ratio of the distance and the length to a preset constant to obtain the first weight of the pixel point on the same side; and subtracting the result of the first weight from 1 to determine the second weight of the pixel point on the opposite side.

[0080] Specifically, for two adjacent video frame images, the high-frequency and low-frequency signals can be separated by Fourier transform and wavelet transform to obtain the high-frequency signal corresponding to P1(x, y). High-frequency signal corresponding to P2(x,y) And the low-frequency signal corresponding to P1(x,y) and P2(x,y) corresponds to the low frequency signal Among them, for the high-frequency signal related to texture and The high-frequency signal of the viewing angle on the same side of the optimal stitching line can be retained; for the low-frequency signal related to brightness, the low-frequency signals of two pixels can be fused based on the designed weights.

[0081] For example, in combination Figure 6 , let the position of the optimal suture line be x seam , for a pixel P(x,y), its distance a from the optimal stitching line can be expressed as a=|xx seam |. Let the length of the left side of the overlapping area be b, and the left and right boundaries be x l and x r , then b=x seam -x l , then for the same-side pixel P1(x,y), the expression of the corresponding first weight w1 can be: For the opposite side pixel P2 (x, y), its second weight is w2 = 1-w1.

[0082] In one of the embodiments, the fused brightness signal is determined based on the brightness signal weights corresponding to the same-side pixel points and the opposite-side pixel points, including: summing the product of the first weight and the low-frequency signal corresponding to the same-side pixel point, and the product of the second weight and the low-frequency signal corresponding to the opposite-side pixel point, to obtain the fused brightness signal.

[0083] Combined with the above example, the expression corresponding to the fused brightness signal is:

[0084] In one of the embodiments, the step of fusing the brightness signals of the video frame images of the target moment under the adjacent viewing angles based on the brightness signal weights and fusing them with the texture signals of the video frame images of the target moment under the adjacent viewing angles to determine the panoramic image includes:

[0085] Based on the brightness signal weights corresponding to the same-side pixel points and the opposite-side pixel points, a fused brightness signal is determined; and the fused brightness signal and the pre-processed texture signal are fused to obtain a panoramic image.

[0086] The preprocessed texture signal may be a high-frequency signal of the same-side viewing angle as the best suture line.

[0087] Figure 7 FIG. 1 is a schematic diagram of feature point registration for assisting calibration of image registration results based on optical flow field registration results of the present invention. Figure 7As shown, two adjacent viewpoints a and b are respectively subjected to feature point registration of the optical flow field and feature point registration of the image, and the feature point registration of the optical flow field is used to assist the feature point registration of the image to achieve image fusion of viewpoint a and viewpoint b.

[0088] The brightness robust panoramic image stitching generation device provided by the present invention is described below. The brightness robust panoramic image stitching generation device described below and the brightness robust panoramic image stitching generation method described above can be referred to each other.

[0089] Figure 8 is a schematic diagram of a brightness robust panoramic image stitching generation device provided by the present invention, such as Figure 8 As shown, the brightness robust panoramic image stitching generation device provided by the embodiment of the present invention includes:

[0090] An acquisition module 810 is used to acquire a video frame sequence corresponding to each viewing angle of the panoramic camera; the video frame sequence includes a video frame image;

[0091] A first determination module 820 is configured to calibrate the image registration results of the video frame sequences under each two adjacent viewing angles based on the optical flow field registration results of the video frame sequences under each two adjacent viewing angles, and determine an image feature point matching set; the image feature point matching set is used to reflect the relative positions and overlapping areas between the video frame images at the same time in the video frame sequences under adjacent viewing angles;

[0092] A second determination module 830 is used to determine the brightness signal weights corresponding to the corresponding pixel points in the video frame image at the target moment under adjacent viewing angles based on the distance between the pixel points in the overlapping area and the optimal stitching line; wherein the optimal stitching line is the optimal stitching line of the video frame image at the target moment under adjacent viewing angles determined based on the image feature point matching set;

[0093] The third determination module 840 is used to fuse the brightness signal of the video frame image at the target moment under the adjacent viewing angles based on the brightness signal weight, and then fuse it with the texture signal of the video frame image at the target moment under the adjacent viewing angles to determine the panoramic image.

[0094] The brightness-robust panoramic image stitching generation device provided by the present invention assists the image feature point matching by referring to the feature point matching results of the motion field (optical flow field) that is less affected by brightness changes, thereby reducing the influence of the brightness change of the viewing angle on the feature point matching. In addition, in order to achieve a natural color transition at the gap, the pixel values ​​from the two viewing angle images are weighted based on the distance. Based on the proposed motion-guided feature point matching and the brightness transition method of the stitching line, the present invention can achieve relatively brightness-robust panoramic image stitching generation.

[0095] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the brightness robust panoramic image stitching generation method described in any embodiment of the present invention.

[0096] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program codes.

[0097] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the brightness-robust panoramic image stitching generation method described in any embodiment of the present invention.

[0098] On the other hand, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the brightness-robust panoramic image stitching generation method described in any embodiment of the present invention.

[0099] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0101] It can be understood that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A brightness robust panoramic image stitching generation method, characterized in that: include: Obtain the video frame sequences corresponding to each viewing angle of the panoramic camera; Based on the optical flow field registration results of the video frame sequences at each two adjacent viewing angles in each viewing angle, the image registration results of the video frame sequences at each two adjacent viewing angles are calibrated to determine an image feature point matching set; the image feature point matching set is used to reflect the relative positions and overlapping areas between the video frame images at the same time in the video frame sequences at adjacent viewing angles; Determining, based on the distance between the pixel points in the overlapping area and the optimal stitching line, the luminance signal weights corresponding to the corresponding pixel points in the video frame images at the target moment under adjacent viewing angles; wherein the optimal stitching line is the optimal stitching line of the video frame images at the target moment under adjacent viewing angles determined based on the image feature point matching set; After fusion of the brightness signals of the video frame images at the target moment under adjacent viewing angles based on the brightness signal weights, the fusion is performed with the texture signals of the video frame images at the target moment under adjacent viewing angles to determine a panoramic image.

2. The brightness robust panoramic image stitching generation method according to claim 1, characterized in that: The step of calibrating the image registration results of the video frame sequences under each two adjacent viewing angles based on the optical flow field registration results of the video frame sequences under each two adjacent viewing angles to determine the image feature point matching set includes: Based on the sum of the absolute values ​​of the pixel values ​​in the integrated optical flow field of each perspective, the changing speed of the scene in the video frame image at the corresponding perspective is determined; For each matching feature pair in a video frame sequence under each two adjacent viewing angles, the following steps are performed respectively: determining the change speed of the scene as the weight of the confidence of the optical flow field registration result, and determining the weighted confidence of each matching feature pair based on the weight of the confidence of the optical flow field registration result; An image feature point matching set is determined based on the weighted confidence of each matching feature pair.

3. The brightness robust panoramic image stitching generation method according to claim 2, characterized in that: The determining the weighted confidence of each matching feature pair based on the weight of the confidence of the optical flow field registration result includes: Determining the confidence weight of the corresponding video frame image registration result based on the confidence weight of the optical flow field registration result; Determine a weighted confidence of each matching feature pair by adding the product of the confidence of the optical flow field registration result and the weight of the confidence of the optical flow field registration result to the product of the confidence of the corresponding video frame image registration result and the weight of the confidence of the corresponding video frame image registration result; The determining of the image feature point matching set based on the weighted confidence of each matching feature pair includes: When the weighted confidence of the matching feature pair is greater than a preset threshold, the matching feature pair is determined as a matching feature pair in the image feature point matching set.

4. The brightness robust panoramic image stitching generation method according to claim 1, characterized in that: Before calibrating the image registration results of the video frame sequences at every two adjacent viewing angles based on the optical flow field registration results of the video frame sequences at every two adjacent viewing angles, the method further includes: Optical flow field feature point detection and registration are performed on the video frame sequence under every two adjacent viewing angles to determine the corresponding optical flow field registration results under every two adjacent viewing angles.

5. The brightness robust panoramic image stitching generation method according to claim 4, characterized in that: The optical flow field feature point detection and registration are performed on the video frame sequences under each two adjacent viewing angles to determine the optical flow field registration results corresponding to each two adjacent viewing angles, including: For each video frame sequence at each viewing angle, determining an average value of the optical flow field from at least one video frame image at a time point to be reconstructed in the video frame sequence to other video frame images at non-reconstructed time points; Determine the average value as the comprehensive optical flow field corresponding to the viewing angle; For the comprehensive optical flow fields corresponding to each two adjacent perspectives, feature point detection and registration are performed to determine the optical flow field registration results of each two adjacent perspectives.

6. The brightness robust panoramic image stitching generation method according to any one of claims 1 to 5, characterized in that: The determining, based on the distance between the pixel points in the overlapping area and the optimal stitching line, the brightness signal weights corresponding to the corresponding pixel points in the video frame images at the target moment under adjacent viewing angles, includes: For each pixel in the overlapping area, determining the brightness signal weights corresponding to the pixel on the same side and the pixel on the opposite side of the pixel based on the distance between the pixel and the optimal stitching line and the length between the pixel and the overlapping area; The step of fusing the brightness signals of the video frame images at the target moment under the adjacent viewing angles based on the brightness signal weights and fusing them with the texture signals of the video frame images at the target moment under the adjacent viewing angles to determine the panoramic image includes: Determining a fused brightness signal based on the brightness signal weights corresponding to the same-side pixel points and the opposite-side pixel points respectively; The fused brightness signal and the pre-processed texture signal are fused to obtain a panoramic image.

7. The brightness robust panoramic image stitching generation method according to claim 6, characterized in that: The determining, based on the distance between the pixel point and the optimal stitching line and the length between the pixel point and the overlapping area, the brightness signal weights corresponding to the pixel point on the same side and the pixel point on the opposite side, respectively, includes: Adding a ratio of the distance to the length to a preset constant to obtain a first weight of the pixel point on the same side; Subtracting the first weight from 1 to determine the result as the second weight of the opposite-side pixel; The determining of the fused brightness signal based on the brightness signal weights corresponding to the same-side pixel points and the opposite-side pixel points respectively includes: The product of the first weight and the low-frequency signal corresponding to the pixel point on the same side is summed with the product of the second weight and the low-frequency signal corresponding to the pixel point on the opposite side to obtain a fused brightness signal.

8. A brightness robust panoramic image stitching generation device, characterized in that: include: An acquisition module is used to obtain the video frame sequences corresponding to each viewing angle of the panoramic camera; a first determining module configured to calibrate the image registration results of the video frame sequences at each of the two adjacent viewing angles based on the optical flow field registration results of the video frame sequences at each of the two adjacent viewing angles, and determine an image feature point matching set; the image feature point matching set is configured to reflect the relative positions and overlapping areas between the video frame images at the same moment in the video frame sequences at the adjacent viewing angles; a second determination module, configured to determine, based on the distance between the pixel points in the overlapping area and the optimal stitching line, the luminance signal weights corresponding to the corresponding pixel points in the video frame images at the target moment under adjacent viewing angles; wherein the optimal stitching line is the optimal stitching line of the video frame images at the target moment under adjacent viewing angles determined based on the image feature point matching set; The third determination module is used to fuse the brightness signals of the target moment video frame images under adjacent viewing angles based on the brightness signal weights, and then fuse them with the texture signals of the target moment video frame images under adjacent viewing angles to determine a panoramic image.

9. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, wherein: When the computer-readable instructions are executed by the processor, the processor is caused to perform the steps of the brightness-robust panoramic image stitching generation method according to any one of claims 1 to 7.

10. A storage medium storing computer-readable instructions, characterized in that: When the computer-readable instructions are executed by one or more processors, the one or more processors are caused to perform the steps of the brightness-robust panoramic image stitching generation method according to any one of claims 1 to 7.

Images