Video stitching method and device based on embedded platform

Abstract

The application discloses a video splicing method and device based on an embedded platform, which comprises the following steps: collecting video streams with overlapping areas by two cameras; processing first frame data on the embedded platform to obtain a homography matrix and calculate the overlapping area; determining a seam by using an optimal seam algorithm; performing color correction and fusion on the first frame by using an optimized histogram matching method and a fade-in and fade-out method; detecting a moving object by using a frame difference method, setting a current frame as an updated frame when the moving object passes near the seam, updating the seam and a color correction function, avoiding ghosting and blurring, and regularly updating the homography matrix to eliminate errors; generating pixel mapping of data of the first frame and the updated frame, saving the color correction function, directly performing coordinate mapping and color correction on other frames, using hardware acceleration to fuse the frames, and finally displaying the spliced video on a display device in real time. The application can improve splicing quality while ensuring real-time performance, quickly obtain a spliced image, and display the spliced image on the display device in real time.

Description

Technical Field

[0001] This invention relates to the field of video information processing technology, and specifically to a video stitching method and apparatus based on an embedded platform. Background Technology

[0002] The most efficient way for humans to acquire information is through vision; approximately 80% of information is perceived visually. Today, multimedia information such as video and images plays a crucial role in various fields, including security surveillance, autonomous driving, medical diagnosis, and cultural heritage preservation. With continuous technological advancements, the demand for wide-field-of-view and high-resolution video is increasing. However, most video shooting equipment on the market has a relatively small field of view, less than the wide field of view of the human eye, making it difficult to meet this demand. To address this challenge, video stitching technology has emerged, providing a solution.

[0003] Video stitching is a technique that combines multiple video frames in chronological order into a continuous video. This technology is widely used in virtual reality, video surveillance, autonomous driving, and digital entertainment; however, most current video stitching methods cannot guarantee real-time performance and quality requirements.

[0004] Traditional stitching algorithms can cause obvious ghosting and stitching seams due to parallax, color, and brightness differences caused by different camera shooting angles. At the same time, the biggest difference between video stitching and image stitching is that there are moving objects in the video. If the stitching line is not updated when the moving object passes through it, ghosting and blurring problems will occur. Summary of the Invention

[0005] The present invention proposes a video splicing method, device and storage medium based on an embedded platform, which can at least solve one of the technical problems in the background art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A video stitching method based on an embedded platform includes the following steps:

[0008] Step 1: Extract feature points from the first frame image data of the two acquired video streams, and match and purify the feature points to obtain the homography transformation matrix;

[0009] Step 2: In the first frame of the two video streams, select the left image to be stitched as the reference plane, perform perspective transformation on the right image to be stitched based on the homography matrix to align it with the reference plane, and obtain the overlapping area of ​​the first frame of the two video streams.

[0010] Step 3: Calculate the optimal stitching line for the overlapping region using the optimal stitching line algorithm based on dynamic programming for the first frame image of the two video streams;

[0011] Step 4: Based on the found best stitching line, the optimized histogram matching method and fade-in / fade-out method are used for the overlapping area of ​​the first frame. First, the fade-in / fade-out fusion area is determined. Then, the color of the video frame image to be stitched is corrected based on the color change of the video frame image within the range. The acquired video frame images are segmented and fused to obtain the fused panoramic image.

[0012] Step 5: Use the three-frame difference method to separate the moving region from the original image. When the moving region passes near the seam line, set the current frame as the update frame, update the seam line and color correction function, and update the homography matrix periodically to eliminate accumulated errors.

[0013] Step 6: Generate a pixel map from the data of the first frame and the updated frame, save the color correction function, and use the hardware acceleration method of the embedded platform to process the stitching and fusion of other frames.

[0014] Furthermore, the specific method for image registration described in step 1 is as follows:

[0015] Step 1-1: Use the ORB algorithm to extract feature points from the first frame of the video stream and generate corresponding feature descriptors;

[0016] Step 1-2: Apply Hamming distance-based nearest neighbor matching to the generated feature points, and perform coarse purification of the matched feature point pairs using the neighbor and second nearest neighbor method, taking the point pairs with values ​​less than the threshold as the matching point pairs;

[0017] Steps 1-3: Use the RANSAC algorithm to perform secondary purification on the matching point pairs, and finally obtain the homography matrix of the image.

[0018] Furthermore, the optimal suture detection method described in step 3 is as follows:

[0019] Step 3-1: The optimal stitching line algorithm based on dynamic programming aims to find an optimal stitching line that maximizes the visual effect of the stitched area. This is achieved by performing pixel-level analysis of the overlapping areas of the image, establishing an energy function to measure the matching degree between pixels. Then, using dynamic programming, starting from one end of the image, the pixel with the smallest energy function value is selected step-by-step to form a continuous stitching line.

[0020] First, we define an energy function E(x,y) for the overlapping region to measure the color and geometric differences in the overlapping areas of the image, where x and y are the image coordinates:

[0021] E(x,y)=rE c (x,y)+(1-r)E g (x,y)

[0022] Among them, E c (x,y) measures color difference, which is obtained by calculating the intensity difference between the coordinate pixels, E g (x,y) measures the difference in geometric gradient changes. It is obtained by acquiring the gradient maps in the horizontal and vertical directions in the overlapping areas of the image frames and then calculating the gradient difference. r is the weight between color and geometric differences. Generally, gradient changes are more sensitive, so r is usually taken as 0.3.

[0023] The suture line is searched using dynamic programming, specifically as follows:

[0024] First, initialize the path weights and path indices for the first row. Then, perform dynamic programming calculations for each row of pixels, considering boundary conditions, and calculate the cumulative intensity value for each pixel. For the selection of the next pixel, if it is within the range of the moving object, skip it; otherwise, determine the next pixel by taking the minimum value, progressively updating the path weights and path indices. Finally, obtain the optimal stitching line by backtracking to the path with the minimum cumulative weight.

[0025] Furthermore, the method for fusing the acquired first frame image of the video in step 4 is as follows:

[0026] Step 4-1: Fade-in and fade-out blending is a technique commonly used in video splicing and image fusion. Its purpose is to achieve a smooth transition in the splicing area, thereby reducing visual discontinuity. By weighted averaging the overlapping areas, the fade-in and fade-out blending method gradually adjusts the transparency of the image, so that the spliced ​​image transitions naturally in the visual sense.

[0027] First, determine the extent of the fade-in / fade-out blending region, which is determined by the average brightness difference of the overlapping regions:

[0028]

[0029] Where B represents the range of the fade-in / fade-out fusion region, B min To minimize the range, pre-set according to the actual situation, B max The maximum range is the shortest distance from the suture line to the edge of the overlapping area.

[0030] B max =min(C(x,y)-O(x,y)) L O(x,y) R -C(x,y))

[0031] Where C(x,y) represents the coordinates of the suture line, and O(x,y) represents the coordinates of the suture line. L And O(x,y) R These are the coordinates of the left and right boundaries of the overlapping region, respectively.

[0032] L diff L represents the average brightness difference in the overlapping regions of image frames. avg The average brightness is L. diff The calculation method is as follows:

[0033]

[0034] L avg The calculation method is as follows:

[0035]

[0036] Where overlap represents the overlapping region, w and h are the width and height of the overlapping region, respectively, and L... point1 (x,y) represents the grayscale intensity of pixels within the overlapping area of ​​the left image to be stitched, and L point2 (x,y) represents the pixel grayscale intensity within the overlapping area of ​​the right image to be stitched together;

[0037] Step 4-2: Histogram matching, also known as histogram specification, is an image processing technique used to transform the histogram of an image into a histogram with a specific shape. This technique can be used to enhance image contrast and can selectively enhance contrast within a certain range of gray values.

[0038] After determining the fade-in / fade-out blending area, an optimized histogram matching method is used to color-correct the video frame images to be stitched based on the color changes of the video frame images to be stitched within the fade-in / fade-out blending area.

[0039] The fade-in / fade-out blending region is divided into several parts according to a set threshold. For each region, the color channels of the video frame images to be stitched are converted to Lab color. The histograms and normalized cumulative histograms of the two video frame images to be stitched are calculated for each color channel. The histogram calculation method is as follows:

[0040]

[0041] Where h s (i) is the histogram of the reference video frame image in each color channel, h t (i) is the histogram of the target video frame image in each color channel, where W and H are the width and height of the divided region, and I is the histogram of the target video frame image in each color channel. s (x,y) and I t (x,y) represent the color channel values ​​of the corresponding coordinates of the reference video frame image and the target video frame image, respectively. The δ function is 1 when the color channel value is equal to i, and 0 otherwise.

[0042] The normalized cumulative histogram is calculated as follows:

[0043]

[0044] Where C s (i) is the cumulative histogram of the reference video frame image in each color channel, C t (i) is the cumulative histogram of the target video frame image in each color channel, where W and H are the width and height of the divided region. After calculation, a preliminary color mapping function M is established for each color channel:

[0045] M(j)={i|C s (i)≤C t (j)≤C s (i+1)}

[0046] Where C s (i)≤C t (j)≤C s (i+1) represents the mapping condition, j is the color channel value corresponding to the target video frame image, and i is the color channel value corresponding to the reference video frame image.

[0047] After the initial color mapping function is established, considering the parallax effect of the two images to be stitched in the fade-in and fade-out fusion area, the calculated histograms are sorted in ascending order and the normalized cumulative histogram is recalculated. The numerical indices of the normalized cumulative histograms that are less than the set threshold are selected, and their numerical positions in the initial color mapping function are inferred and removed as noise areas. The non-continuous parts in the processed initial color mapping function are smoothed to obtain the corrected color mapping function. The corrected color mapping function is used to complete the color correction of each color channel of the target video frame image.

[0048] Repeat the above process until color correction of all divided areas is complete;

[0049] Step 4-3: The first frame of the video image is fused based on the optimal stitching algorithm. The fade-in and fade-out fusion areas are fused according to the existing fade-in and fade-out method, and then the other areas are added to complete the final fusion.

[0050]

[0051] Where I1 is the range from the reference image to the left boundary of the fade-in / fade-out blending region, B is the range of the fade-in / fade-out blending region, I2 is the range from the right boundary of the fade-in / fade-out blending region to the image to be stitched after projection transformation, and r is the weighting coefficient, calculated as follows:

[0052]

[0053] Where, x r x represents the x-coordinate of the right boundary of the fade-in / fade-out merging region. lx represents the x-coordinate of the left boundary of the fade-in / fade-out merging region. i This represents the x-coordinate of the current pixel.

[0054] Furthermore, the specific method for separating the moving image and detecting the updated frame in step 5 is as follows:

[0055] Step 5-1: Extract three consecutive frames F from the video stream n-1 F n and F n+1 The two difference images D1 and D2 are calculated using the following formula:

[0056] D1=|F n+1 -F n |

[0057] D2=|F n -F n-1 |

[0058] Where D1 is frame F n and F n+1 The difference image, D2 is frame F n-1 and F n The difference image is calculated, and the final motion detection result M is obtained by using the following formula:

[0059] M = D1∩D2

[0060] The detection result M is separated from the original image as the moving region.

[0061] Determine if any pixels in the current motion region coincide with the position of the stitching line pixels. If so, use the current frame as the update frame and update the optimal stitching line and color mapping function. Otherwise, continue using the optimal stitching line and color mapping function from the previous frame.

[0062] Furthermore, the specific implementation method of using other frame processing in step 6 is as follows:

[0063] Step 6-1: Convert the homography matrix obtained by registration of the first frame image, the optimal stitching line obtained by dynamic programming, and the weight coefficients obtained by optimizing the fade-in / fade-out method into the mapping coordinates corresponding to each pixel of the original image. Save the color mapping function obtained by optimizing the histogram matching method. After updating the frame and calculating the new homography matrix, optimal stitching line, weight coefficients, and color mapping function, update the mapping coordinates and the old color mapping function in real time. Through hardware interface functions, realize the fast coordinate mapping and color correction through hardware acceleration to achieve fast stitching and fusion of video frames.

[0064] On the other hand, the present invention also provides a multi-screen splicing device for display, comprising:

[0065] The main control module, using RK3588 as the device's main controller, is configured to drive the corresponding display to show specific modules according to user settings; drive multiple video walls based on multiple display data streams; ensure that each video wall presents content based on its corresponding display data stream; and process the video stream data to be spliced ​​as a display data stream and output it to multiple video walls to display a complete picture corresponding to the image to be displayed.

[0066] The splicing logic processing module is configured to establish a two-dimensional coordinate system according to user settings, convert the display order of each splicing display into two-dimensional coordinates, determine the range of each display area by traversing the two-dimensional coordinates of each splicing display, determine the range of each display area by the first division by the system after the first division, determine the range of each display area by the current division area and the display range of each splicing display after the second division by the division chip, and output the data stream of each division area to multiple splicing displays.

[0067] The display module is configured to receive the segmented data stream after secondary segmentation by the processing module and execute the multi-screen splicing display.

[0068] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0069] As described above, the video stitching method and apparatus based on an embedded platform of the present invention first acquires video streams with overlapping areas using two cameras. Then, the first frame is processed on the embedded platform to obtain a homography matrix. Next, an optimal stitching line algorithm is used to search for stitching lines in the overlapping areas. An optimized histogram matching method and fade-in / fade-out method are used to color-correct and stitch the first frame data. A frame difference method is used to detect moving objects. If a moving object passes near the stitching line, the current frame is used as the update frame, updating the stitching line and color correction function to ensure that moving objects do not exhibit ghosting or blurring near the stitching line. Simultaneously, the homography matrix is ​​updated periodically to eliminate errors. Pixel mapping is generated from the data of the first and update frames, and the color correction function is saved. Other frames are directly mapped and color-corrected. Hardware acceleration is used to speed up the fusion process. Finally, the stitched video is displayed in real-time on a display device. This method improves stitching quality while ensuring real-time performance, quickly obtaining high-quality stitched images and displaying them in real-time on a display device.

[0070] The advantages and positive effects of this invention are as follows: After processing the first frame image to obtain the homography matrix and the optimal stitching line, subsequent frames are directly stitched using hardware acceleration and the first frame data, ensuring the real-time performance of video stitching. During image stitching, the optimal stitching line is updated using a method based on frame difference to calculate the foreground region of moving objects in the video. If the optimal stitching line calculated from a previous frame image crosses the foreground region of the current frame, the optimal stitching line needs to be recalculated for the current frame; otherwise, the stitching line from the previous frame is used. This method significantly reduces the processing time of the stitching system. During image fusion, an improved fade-in / fade-out algorithm is used to fuse video frame images based on the optimal stitching line algorithm. The brightness difference between overlapping images is used to determine the range of the fade-in / fade-out fusion region, accelerating the processing speed. Based on the range of the fade-in / fade-out fusion region, the video frame images to be stitched are divided into blocks and histogram matching correction is used to balance colors while ensuring more detailed correction of color differences, adapting to color changes in the vertical direction of the image. The fade-in / fade-out fusion region is fused using the fade-in / fade-out method, and with the addition of other regions, the overlapping areas of the stitched images can be smoothly stitched. This method effectively avoids problems such as ghosting, blurring, seams, and uneven transitions caused by the presence of moving objects and differences in lighting, resulting in seamlessly fused and stitched images with good visual effects. Attached Figure Description

[0071] Figure 1 A flowchart of a video stitching method based on an embedded platform provided in an embodiment of the present invention;

[0072] Figure 2 A flowchart of the first frame image registration stage provided in an embodiment of the present invention;

[0073] Figure 3 This is a flowchart of searching for stitching lines in the first frame of an image to be stitched, provided in an embodiment of the present invention.

[0074] Figure 4 This is a flowchart of fusing the first frame image provided in an embodiment of the present invention;

[0075] Figure 5 This is a flowchart of motion region separation and frame detection provided in an embodiment of the present invention;

[0076] Figure 6 A flowchart for processing other frames provided in an embodiment of the present invention;

[0077] Figure 7 A logic diagram of a display device provided in an embodiment of the present invention;

[0078] Figure 8The image shows the frame stitching effect provided in the embodiment of the present invention, where (a) is the left image, (b) is the right image, and (c) is the stitching result image. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0080] like Figure 1 As shown, the video stitching method based on an embedded platform described in this embodiment includes the following steps:

[0081] First, the first frame image is registered using feature points. Then, the first frame image is color-corrected and blended using the best stitching line algorithm, optimized histogram matching algorithm, and optimized fade-in / fade-out method. The motion region is separated using the three-frame difference method. The need to update the stitching line and fade-in / fade-out region is determined by real-time monitoring of whether the motion region passes through the stitching line. The hardware acceleration function of the embedded platform is used to complete the stitching and fusion of other frames. Finally, the fused video frame is sent to the display device for output.

[0082] The specific steps of video stitching based on an embedded platform described in this invention are as follows:

[0083] Step 1: Extract feature points from the first frame image data of the two acquired video streams, and match and purify the feature points to obtain the homography transformation matrix;

[0084] Step 2: Select the left image to be stitched as the reference plane, perform perspective transformation on the right image to be stitched according to the homography matrix to align it with the reference plane, and obtain the overlapping area of ​​the first frame images of the two video streams;

[0085] Step 3: Calculate the optimal stitching line for the overlapping region using the optimal stitching line algorithm based on dynamic programming for the first frame image of the two video streams;

[0086] Step 4: Based on the found best stitching line, the optimized histogram matching method and fade-in / fade-out method are used for the overlapping area of ​​the first frame. First, the fade-in / fade-out fusion area is determined. Then, the color of the video frame image to be stitched is corrected based on the color change of the video frame image within the range. The acquired video frame images are segmented and fused to obtain the fused panoramic image.

[0087] Step 5: Use the three-frame difference method to separate the moving region from the original image. When the moving region passes near the seam line, set the current frame as the update frame, update the seam line and color correction function, and update the homography matrix periodically to eliminate accumulated errors.

[0088] Step 6: Generate a pixel map from the data of the first frame and the updated frame, save the color correction function, and use the hardware acceleration method of the embedded platform to process the stitching and fusion of other frames.

[0089] like Figure 2 As shown, the specific method for registering the first frame image according to the present invention is as follows:

[0090] In step 1, feature points are extracted from the first frame image data of the two acquired video streams using the ORB algorithm, and the Hamming distance and RANSAC algorithms are used to match and purify the feature points to obtain the homography transformation matrix.

[0091] The specific method for step 1 is as follows:

[0092] Step 1-1: Use the ORB algorithm to extract feature points from the first frame of the video stream and generate corresponding feature descriptors;

[0093] Step 1-2: For the generated feature points, the nearest neighbor matching method based on Hamming distance is used, and the matching feature point pairs are coarsely matched by the neighbor and second nearest neighbor methods. Point pairs with less than the threshold are used as matching point pairs. According to a large number of experimental results, the threshold is generally set to 0.6.

[0094] Steps 1-3: Use the RANSAC algorithm to refine the matching point pairs, and finally obtain the homography matrix of the image.

[0095] In step 2, the images to be stitched are subjected to perspective transformation to keep them in the same plane as the reference image. The specific method is as follows:

[0096] Step 2-1: Align the coordinates of the image to be stitched with the coordinates of the reference image according to the homography transformation formula:

[0097]

[0098] Among them, (x r ,y r ,z r Let (x, y) be the transformed world coordinates. After calculation, convert them to transformed two-dimensional coordinates (x, y). H is the homography matrix. a ,y a ,1) Convert the coordinates of the images to be stitched into world coordinates;

[0099] Step 2-2: Adjust the size of the canvas of the converted images to be stitched, expanding the canvas width to the sum of the widths of the two images, to facilitate subsequent stitching with the reference image;

[0100] Steps 2-3: Use the homography matrix to determine the overlapping area of ​​corresponding frames in the two video streams.

[0101] like Figure 3 As shown, the specific method for searching for the stitching line in the first frame of the image to be stitched, as described in step 3, is as follows:

[0102] Step 3-1: The optimal stitching line algorithm based on dynamic programming finds an optimal stitching line to optimize the visual effect of the stitched area. This method performs detailed analysis of the pixels in the overlapping area and constructs an energy function to evaluate pixel matching. Then, starting from one end of the image, dynamic programming is used to progressively select the pixels with the lowest energy function values ​​to form a continuous stitching line, ensuring the best stitching effect.

[0103] First, we define an energy function E(x,y) for the overlapping region to measure the color and geometric differences in the overlapping areas of the image, where x and y are the image coordinates:

[0104] E(x,y)=rE c (x,y)+(1-r)E g (x,y)

[0105] Among them, E c (x,y) measures color difference, which is obtained by calculating the intensity difference between the coordinate pixels, E g (x,y) measures the difference in geometric gradient changes. It is obtained by acquiring the gradient maps in the horizontal and vertical directions in the overlapping areas of the image frames and then calculating the gradient difference. r is the weight between color and geometric differences. Generally, gradient changes are more sensitive, so r is usually taken as 0.3.

[0106] The suture line is searched using dynamic programming, specifically as follows:

[0107] First, initialize the path weights and path indices for the first row. Then, perform dynamic programming calculations for each row of pixels, considering boundary conditions, and calculate the cumulative intensity value for each pixel. For the selection of the next pixel, if it is within the range of the moving object, skip it; otherwise, determine the next pixel by taking the minimum value, progressively updating the path weights and path indices. Finally, obtain the optimal stitching line by backtracking to the path with the minimum cumulative weight.

[0108] like Figure 4 As shown, the specific method for fusing the first frame image of the acquired video in step 4 is as follows:

[0109] Step 4-1: Fade-in and fade-out blending is a technique commonly used in video splicing and image fusion. Its purpose is to achieve a smooth transition in the splicing area, thereby reducing visual discontinuity. By weighted averaging the overlapping areas, the fade-in and fade-out blending method gradually adjusts the transparency of the image, so that the spliced ​​image transitions naturally in the visual sense.

[0110] This invention uses an optimized fade-in / fade-out method, first determining the range of the fade-in / fade-out blending region, which is determined by the average brightness difference of the overlapping regions:

[0111]

[0112] Where B is the calculated fade-in / fade-out blending region. min For the minimum range, B max The maximum range is the shortest distance from the suture line to the edge of the overlapping area.

[0113] B max =min(C(x,y)-O(x,y)) L O(x,y) R -C(x,y))

[0114] Where C(x,y) represents the coordinates of the suture line, and O(x,y) represents the coordinates of the suture line. L And O(x,y) R These are the coordinates of the left and right boundaries of the overlapping region, respectively.

[0115] L diff L represents the average brightness difference in the overlapping regions of image frames. avg For the maximum brightness difference, where L diff The calculation method is as follows:

[0116]

[0117] L avg The calculation method is as follows:

[0118]

[0119] Where overlap represents the overlapping region, w and h are the width and height of the overlapping region, respectively, and L... point1 (x,y) represents the grayscale intensity of pixels within the overlapping area of ​​the left image to be stitched, and L point2 (x,y) represents the pixel grayscale intensity within the overlapping area of ​​the right image to be stitched together;

[0120] Step 4-2: Histogram matching, also known as histogram specification, is an image processing technique used to transform the histogram of an image into a histogram with a specific shape. This technique can be used to enhance image contrast and can selectively enhance contrast within a certain range of gray values.

[0121] After determining the fade-in / fade-out blending area, an optimized histogram matching method is used to color-correct the video frame images to be stitched based on the color changes of the video frame images to be stitched within the fade-in / fade-out blending area.

[0122] The fade-in / fade-out blending region is divided into several parts according to a set threshold. For each region, the color channels of the video frame images to be stitched are converted to Lab color. The histograms and normalized cumulative histograms of the two video frame images to be stitched are calculated for each color channel. The histogram calculation method is as follows:

[0123]

[0124] Where h s (i) is the histogram of the reference video frame image in each color channel, h t (i) is the histogram of the target video frame image in each color channel, where W and H are the width and height of the divided region, and I is the histogram of the target video frame image in each color channel. s (x,y) and I t (x,y) represent the color channel values ​​of the corresponding coordinates of the reference video frame image and the target video frame image, respectively. The δ function is 1 when the color channel value is equal to i, and 0 otherwise.

[0125] The normalized cumulative histogram is calculated as follows:

[0126]

[0127] Where C s (i) is the cumulative histogram of the reference video frame image in each color channel, C t (i) is the cumulative histogram of the target video frame image in each color channel, where W and H are the width and height of the divided region. After calculation, a preliminary color mapping function M is established for each color channel:

[0128] M(j)={i|C s (i)≤C t (j)≤C s (i+1)}

[0129] Where C s (i)≤C t (j)≤c s (i+1) represents the mapping condition, j is the color channel value corresponding to the target video frame image, and i is the color channel value corresponding to the reference video frame image.

[0130] After the initial color mapping function is established, considering the parallax effect of the two images to be stitched in the fade-in and fade-out fusion area, the calculated histograms are sorted in ascending order and the normalized cumulative histogram is recalculated. The numerical indices of the normalized cumulative histograms that are less than the set threshold are selected, and their numerical positions in the initial color mapping function are inferred and removed as noise areas. The non-continuous parts in the processed initial color mapping function are smoothed to obtain the corrected color mapping function. The corrected color mapping function is used to complete the color correction of each color channel of the target video frame image.

[0131] Repeat the above process until color correction of all divided areas is complete;

[0132] Step 4-3: The first frame of the video is fused using the optimal stitching algorithm. The fade-in / fade-out fusion region is fused using the existing fade-in / fade-out method, and then combined with other regions to complete the final fusion.

[0133]

[0134] Where I1 is the range from the reference image to the left boundary of the fade-in / fade-out blending region, B is the range of the fade-in / fade-out blending region, I2 is the range from the right boundary of the fade-in / fade-out blending region to the image to be stitched after projection transformation, and r is the weighting coefficient, calculated as follows:

[0135]

[0136] Where, x r x represents the x-coordinate of the right boundary of the fade-in / fade-out merging region. k x represents the x-coordinate of the left boundary of the fade-in / fade-out merging region. i This represents the x-coordinate of the current pixel.

[0137] like Figure 5 As shown, the specific method for separating the moving image and detecting the updated frame in step 5 is as follows:

[0138] Step 5-1: Extract three consecutive frames F from the video stream n-1 F n and F n+1 The two difference images D1 and D2 are calculated using the following formula:

[0139] D1=|F n+1 -F n |

[0140] D2=|F n -F n-1 |

[0141] Where D1 is frame F n and F n+1The difference image, D2 is frame F n-1 and F n The difference image is calculated, and the final motion detection result M is obtained by using the following formula:

[0142] M = D1∩D2

[0143] The detection result M is separated from the original image as the moving region;

[0144] Determine if any pixels in the current motion region coincide with the position of the stitching line pixels. If so, use the current frame as the update frame and update the optimal stitching line and color mapping function. Otherwise, continue using the optimal stitching line and color mapping function from the previous frame.

[0145] like Figure 6 As shown, the specific implementation methods for other frame processing in step 6 are as follows:

[0146] Step 6-1: Convert the homography matrix obtained by registration of the first frame image, the optimal stitching line obtained by dynamic programming, and the weight coefficients obtained by optimizing the fade-in / fade-out method into the mapping coordinates corresponding to each pixel of the original image. Save the color mapping function obtained by optimizing the histogram matching method. After updating the frame and calculating the new homography matrix, optimal stitching line, weight coefficients, and color mapping function, update the mapping coordinates and the old color mapping function in real time. Through hardware interface functions, realize the fast coordinate mapping and color correction through hardware acceleration to achieve fast stitching and fusion of video frames.

[0147] like Figure 7 As shown, the present invention also provides a multi-screen display device for verifying splicing effects, comprising:

[0148] The main control module, using RK3588 as the device's main controller, is configured to drive the corresponding display to show specific modules according to user settings; drive multiple video walls based on multiple display data streams; ensure that each video wall presents content based on its corresponding display data stream; and process the video stream data to be spliced ​​as a display data stream and output it to multiple video walls to display a complete picture corresponding to the image to be displayed.

[0149] The splicing logic processing module is configured to establish a two-dimensional coordinate system according to user settings, convert the display order of each splicing display into two-dimensional coordinates, determine the range of each display area by traversing the two-dimensional coordinates of each splicing display, determine the range of each display area by the first division by the system after the first division, determine the range of each display area by the current division area and the display range of each splicing display after the second division by the division chip, and output the data stream of each division area to multiple splicing displays.

[0150] The display module is configured to receive the segmented data stream after secondary segmentation by the processing module and execute the multi-screen splicing display.

[0151] Figure 8 The input consists of test video frames and the stitched result, where... Figure 8 (a) is the left image to be stitched together. Figure 8 (b) is the right image to be pieced together. Figure 8 (c) shows the stitching result. The stitching result demonstrates that this invention, through color correction and dynamic adjustment of the fade-in / fade-out blending area range B, expands the blending range when brightness differences are large and maintains a minimum width when differences are small, avoiding excessive smoothing or residual seams. Simultaneously, the blending width range is limited by B. min ≤B≤B max Ensure the transition area maintains a fade-in / fade-out effect without exceeding the overlap boundary (as shown by the dashed box E in the figure), avoiding image content misalignment or distortion due to overexpansion. The stitching results show that in areas with abrupt changes in indoor and outdoor lighting and complex textures, this method performs well in areas B. min ≤B≤B max Under the constraints, it can stably handle various brightness and color differences, and output splicing results without visual incongruity.

[0152] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0153] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0154] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the video stitching methods based on an embedded platform described above.

[0155] It is understood that the systems, devices, and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0156] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0157] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0158] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0159] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video stitching method based on an embedded platform, characterized in that, The specific steps for video stitching based on an embedded platform are as follows: Step 1: Extract feature points from the first frame of the two acquired video streams, and match and refine the feature points to obtain the homography transformation matrix; Step 2: In the first frame of the two video streams, select the left image to be stitched as the reference plane, perform perspective transformation on the right image to be stitched based on the homography matrix to align it with the reference plane, and obtain the overlapping area of ​​the first frame of the two video streams. Step 3: Calculate the optimal stitching line for the overlapping region using the optimal stitching line algorithm based on dynamic programming for the first frame image of the two video streams; Step 4: Based on the found best stitching line, the optimized histogram matching method and fade-in / fade-out method are used for the overlapping area of ​​the first frame. First, the fade-in / fade-out fusion area is determined. Then, the color of the video frame image to be spliced ​​is corrected based on the color change of the video frame image within the range. The acquired video frame images are segmented and fused to obtain the fused first frame image. Step 5: Use the three-frame difference method to separate the moving region from the original image. When the moving region passes near the seam line, set the current frame as the update frame, update the seam line and color correction function, and update the homography matrix periodically to eliminate accumulated errors. Step 6: Generate pixel mapping from the data of the first frame and the updated frame, save the color correction function, and use the hardware acceleration method of the embedded platform to process the stitching and fusion of other frames; The method for calculating the fade-in / fade-out blending region in step 4 is as follows: The extent of the fade-in / fade-out blending region is determined by the average brightness difference in the overlapping regions: in To gradually enter and exit the integration area, To minimize the range, preset according to the actual situation. The maximum range is the shortest distance from the suture line to the edge of the overlapping area. in Let's say the coordinates of the suture line are... and These are the coordinates of the left and right boundaries of the overlapping region, respectively. The average brightness difference in the overlapping areas of image frames. The average brightness is denoted as , where The calculation method is as follows: The calculation method is as follows: in The overlapping area and These represent the width and height of the overlapping region, respectively. The grayscale intensity of the pixels within the overlapping area of ​​the left image to be stitched together. The grayscale intensity of the pixels in the overlapping area of ​​the right image to be stitched together.

2. The video stitching method based on an embedded platform according to claim 1, characterized in that: The method for color correction of the first frame image in step 4 is as follows: After determining the fade-in / fade-out blending area, an optimized histogram matching method is used to color-correct the video frame images to be stitched based on the color changes of the video frame images to be stitched within the fade-in / fade-out blending area. The fade-in / fade-out blending region is divided into several parts according to a set threshold. For each region, the color channels of the video frame images to be stitched are converted to Lab color. The histograms and normalized cumulative histograms of the two video frame images to be stitched are calculated for each color channel. The histogram calculation method is as follows: in For reference, the histograms of the video frame images in each color channel are used. This is the histogram of the target video frame image in each color channel. and To divide the width and height of the area, and These represent the color channel values ​​at corresponding coordinates in the reference and target video frame images, respectively. The function has color channel values ​​equal to The value is 1 if it is true, and 0 otherwise. The normalized cumulative histogram is calculated as follows: in For reference, the cumulative histogram of the video frame image in each color channel, This is the cumulative histogram of the target video frame image across each color channel. and To define the width and height of the regions, a preliminary color mapping function M for each color channel is established after calculation: in The mapping condition is represented by j, which is the color channel value corresponding to the target video frame image, and i is the color channel value corresponding to the reference video frame image. After the initial color mapping function is established, considering the parallax effect of the two images to be stitched in the fade-in and fade-out fusion area, the calculated histograms are sorted in ascending order and the normalized cumulative histogram is recalculated. The numerical indices of the normalized cumulative histograms that are less than the set threshold are selected, and their numerical positions in the initial color mapping function are inferred and removed as noise areas. The non-continuous parts in the processed initial color mapping function are smoothed to obtain the corrected color mapping function. The corrected color mapping function is used to complete the color correction of each color channel of the target video frame image. Repeat the above process until the color correction of all divided areas is complete.

3. The video stitching method based on an embedded platform according to claim 1, characterized in that: The method for fusing the first frame of the video in step 4 is as follows: The first frame of the video is fused based on the optimal stitching algorithm. The fade-in and fade-out fusion regions are fused using the existing fade-in and fade-out method, and then other regions are added to complete the final fusion. in, The range from the reference image to the left boundary of the fade-in / fade-out blend region. The scope of the gradual entry and exit fusion zone. The range from the right boundary of the fade-in / fade-out blending region to the area of ​​the image to be stitched after projection transformation. The weighting coefficient is calculated using the following formula: in, The x-coordinate of the right boundary of the gradual entry and exit fusion region. The x-coordinate of the left boundary of the gradual entry and exit fusion region. This represents the x-coordinate of the current pixel.

4. The video stitching method based on an embedded platform according to claim 1, characterized in that: The specific method for registering the first frame image in step 1 is as follows: Step 1-1: Use the ORB algorithm to extract feature points from the first frame of the video stream and generate corresponding feature descriptors; Step 1-2: The generated feature points are matched using the nearest neighbor matching method based on Hamming distance, and the matched feature point pairs are coarsely purified by the neighbor and second nearest neighbor method, and the point pairs with less than the threshold are taken as the matching point pairs. Steps 1-3: Use the RANSAC algorithm to perform secondary purification on the matching point pairs, and finally obtain the homography matrix of the image.

5. The video stitching method based on an embedded platform according to claim 1, characterized in that: The optimal method for suture detection in step 3 is as follows: Define the energy function of the overlapping region This is used to measure the color and geometric differences in overlapping areas of an image, where x and y are image coordinates. in, Color differences are measured by calculating the intensity difference between coordinate pixels. To measure the difference in geometric gradient changes, the gradient maps in the horizontal and vertical directions are obtained from the overlapping areas of the image frames, and then the gradient difference is calculated. r is the weight between color and geometric differences.

6. The video stitching method based on an embedded platform according to claim 1, characterized in that: Step 3 uses dynamic programming to search for suture lines. The specific method is as follows: First, initialize the path weights and path indices of the first row. Then, perform dynamic programming calculations for each row of pixels, and calculate the cumulative intensity value of each pixel after considering boundary processing. For the selection of the next pixel, if the next pixel is within the range of the moving object, skip the point. If it is not within the range of the moving object, determine the next pixel by taking the minimum value. Update the path weights and path indices step by step. Finally, obtain the optimal stitching line by backtracking the path with the minimum cumulative weight.

7. The video stitching method based on an embedded platform according to claim 1, characterized in that: The specific method for separating the moving image and detecting the updated frame in step 5 is as follows: Step 5-1: Extract three consecutive frames from the video stream , and The two difference images are obtained by calculating using the following formula. and : in, For frames and The difference image, For frames and The difference image is calculated, and the final motion detection result is obtained by using the following formula. : Test results Separate it from the original image as a region of motion; Determine if any pixels in the current motion region coincide with the position of the stitching line pixels. If so, use the current frame as the update frame and update the optimal stitching line and color mapping function. Otherwise, continue using the optimal stitching line and color mapping function from the previous frame.

8. The video stitching method based on an embedded platform according to claim 1, characterized in that: The specific implementation method for processing other frames using hardware acceleration in step 6 is as follows: The homography matrix obtained by registration of the first frame image, the optimal stitching line obtained by dynamic programming, and the weight coefficients obtained by optimizing the fade-in / fade-out method are converted into the mapping coordinates corresponding to each pixel of the original image. The color mapping function obtained by optimizing the histogram matching method is saved. After the new frame is updated, the new homography matrix, optimal stitching line, weight coefficients, and color mapping function are calculated. The mapping coordinates and the old color mapping function are also updated in real time. Through hardware interface functions, the coordinate mapping and color correction are quickly performed through hardware acceleration to achieve fast stitching and fusion of video frames.

9. A multi-screen splicing device for display, used to implement the video splicing method based on an embedded platform as described in any one of claims 1-8, characterized in that: include, The main control module, using RK3588 as the device's main controller, is configured to drive the corresponding display to show specific modules according to user settings; Based on multiple display data streams, multiple video walls are driven respectively; each video wall presents content based on its corresponding display data stream; the video stream data to be spliced ​​is processed and output as a display data stream to multiple video walls to display a complete picture corresponding to the image to be displayed; The splicing logic processing module is configured to establish a two-dimensional coordinate system according to user settings, convert the display order of each splicing display into two-dimensional coordinates, determine the range of each display area by traversing the two-dimensional coordinates of each splicing display, determine the range of each display area by the first division by the system after the first division, determine the range of each display area by the current division area and the display range of each splicing display after the second division by the division chip, and output the data stream of each division area to multiple splicing displays. The display module is configured to receive the segmented data stream after secondary segmentation by the processing module and execute the multi-screen splicing display.

Images