Panoramic image stitching method, stitching device, stitching equipment and storage medium

Abstract

The present application provides a panoramic image stitching method, stitching device, stitching equipment, and storage medium. The panoramic image stitching method includes: capturing a video stream, obtaining stitching parameters, wherein the stitching parameters include stitching lines and edge lines of the panoramic image; obtaining a brightness offset mapping table, wherein the brightness offset mapping table is calculated by selecting multiple key points from the intersections between stitching lines and the intersections between stitching lines and edge lines; and stitching and brightness adjusting multiple images of the video stream according to the stitching parameters and the brightness offset mapping table to obtain a panoramic image. The stitching method of the present application can make the brightness of the stitched panoramic image more balanced, and the panoramic image after brightness adjustment has less difference from the original image quality.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a panoramic image stitching method, stitching device, stitching equipment and storage medium. Background Art

[0002] The existing method for generating panoramic videos generally uses multiple lenses to simultaneously shoot to obtain multiple sub-videos in different directions, and finally combines the multiple sub-videos into a panoramic video image.

[0003] Since the video sources used in panoramic videos come from multiple cameras, in most scenes, the light sources of each camera will be different. Especially in scenes like stage, the differences are more obvious. Therefore, brightness balancing is required in the production of panoramic videos.

[0004] In traditional panoramic videos, the brightness equalization method of panoramic images will cause the image quality of the video to deteriorate. Summary of the Invention

[0005] The present application provides a panoramic image stitching method, a stitching device, a stitching equipment and a storage medium to solve the technical problem in the prior art that the brightness equalization method of panoramic images may cause the quality of video images to deteriorate.

[0006] To solve the above technical problems, the present application proposes a panoramic image stitching method, which includes: capturing a video stream and obtaining stitching parameters, where the stitching parameters include stitching lines and edge lines of the panoramic image; obtaining a brightness offset mapping table, wherein the brightness offset mapping table is calculated by selecting multiple key points from the intersections between stitching lines and the intersections between stitching lines and edge lines; and stitching and brightness adjusting multiple images of the video stream according to the stitching parameters and the brightness offset mapping table to obtain a panoramic image.

[0007] Furthermore, a brightness offset mapping table is calculated by selecting multiple key points from the intersections between stitching lines and the intersections between stitching lines and edge lines, including: calculating the brightness offset values of multiple key points corresponding to each image; and calculating the brightness offset mapping table based on the brightness offset values of the multiple key points.

[0008] Furthermore, the brightness offset values of multiple key points corresponding to each image are calculated, including: selecting four key points from the intersection points between stitching lines and the intersection points between stitching lines and edge lines, wherein the four key points are respectively distributed in the four quadrants of the panoramic image; based on the four key points, four sampling areas are divided around the four key points, and each sampling area does not exceed the overlapping area of multiple images; the brightness average of each sampling area of each image is calculated respectively; based on the brightness average of the same sampling area of multiple images, the brightness offset value of the same sampling area of each image is calculated to obtain the brightness offset values of the four sampling areas corresponding to each image, and the brightness offset value of the sampling area is used as the brightness offset value of the key point corresponding to the sampling area to obtain the brightness offset values of the four key points in each image.

[0009] Furthermore, based on the brightness offset values of multiple key points, a brightness offset mapping table is calculated, including: based on the brightness offset values of the four key points of each image, a bilinear interpolation algorithm is used to calculate the brightness offset values of the pixels in each image to obtain a brightness offset mapping table.

[0010] Furthermore, based on the brightness offset values of multiple key points, a brightness offset mapping table is calculated, including: when the maximum value of the brightness offset values of multiple key points in the image is less than a first threshold, the offset value of the pixel point in the image is set to 0 to obtain a brightness offset mapping table.

[0011] Furthermore, based on the brightness offset values of multiple key points, a brightness offset mapping table is calculated, including: calculating the difference between the maximum and minimum brightness offset values of multiple key points in the image, and when the difference is less than a second threshold, calculating the average value of the brightness offset values of the multiple key points; setting the brightness offset values of all pixel points in the image to the average value of the brightness offset values of the multiple key points to obtain a brightness offset mapping table.

[0012] Furthermore, the brightness offset values of multiple key points corresponding to each image are calculated, and a brightness offset mapping table is calculated based on the brightness offset values of the multiple key points, including: after a first preset time interval, the brightness offset values of the multiple key points corresponding to the image are recalculated to obtain new brightness offset values of the multiple key points; when the new brightness offset values of the multiple key points meet the preset conditions, a new brightness offset mapping table is calculated based on the new brightness offset values of the multiple key points, the brightness offset value mapping table is updated based on the new brightness offset value mapping table, and the first preset time is shortened; when the new brightness offset values of the multiple key points do not meet the preset conditions, the first preset time is extended.

[0013] To solve the above-mentioned technical problems, the present application also proposes a panoramic image stitching device, which includes: a first acquisition module, used to collect video streams and obtain stitching parameters, the stitching parameters including stitching lines and edge lines of the panoramic image; a second acquisition module, used to obtain a brightness offset mapping table, wherein the brightness offset mapping table is calculated by selecting multiple key points from the intersections between stitching lines and the intersections between stitching lines and edge lines; a stitching module, used to stitch and adjust the brightness of multiple images of the video stream according to the stitching parameters and the brightness offset mapping table to obtain a panoramic image.

[0014] To solve the above technical problems, the present application also proposes a panoramic image stitching device, which includes a memory and a processor coupled to each other, and the processor is used to execute program instructions stored in the memory to implement the panoramic image stitching method of any of the above embodiments.

[0015] To solve the above technical problems, the present application further proposes a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implements the panoramic image stitching method of any of the above embodiments.

[0016] The beneficial effects of the present application are as follows: different from the prior art, the panoramic image stitching method of the present application stitches multiple images of a video stream through stitching parameters and a brightness offset mapping table to form a panoramic image. The brightness offset mapping table in the present application is calculated by selecting multiple key points from the intersections between stitching lines and the intersections between stitching lines and edge lines. A pixel-level brightness offset mapping table can be obtained through multiple key points, so that the brightness of the panoramic image is more balanced, and the image quality of the panoramic image after brightness adjustment is less different from the original image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is a flowchart of an embodiment of a panoramic image stitching method provided by the present application;

[0019] Figure 2 yes Figure 1 A flow chart of an embodiment of step S12;

[0020] Figure 3 yes Figure 2 A flow chart of an embodiment of step S122;

[0021] Figure 4 yes Figure 2 A flow chart of an embodiment of step S121;

[0022] Figure 5 This is a structural diagram of an embodiment of a panoramic image and multiple key points of a panoramic image provided by the present application;

[0023] Figure 6 This is a structural diagram of another embodiment of a panoramic image and multiple key points of a panoramic image provided by the present application;

[0024] Figure 7 This is a structural diagram of an embodiment of the four key points and other pixel points of a panoramic image provided by this application;

[0025] Figure 8 yes Figure 2 A flow chart of another embodiment of step S122;

[0026] Figure 9 This is a structural diagram of an embodiment of the live broadcast system provided by the present application;

[0027] Figure 10 This is a schematic diagram of a framework of an embodiment of a panoramic image stitching device provided by the present application;

[0028] Figure 11 This is a schematic diagram of a framework of an embodiment of a panoramic image stitching device provided by the present application;

[0029] Figure 12 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by this application. DETAILED DESCRIPTION

[0030] The present invention will be described in detail below with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by those skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0031] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0032] This application first provides a panoramic image stitching method that can be applied to panoramic image stitching. For example, this stitching method can be applied to a live panoramic video generated by stitching together multiple captured images. Specifically, the method captures video streams from various cameras and stitches multiple images in the video streams to produce a panoramic video. The panoramic image stitching method provided in this application can achieve a more balanced brightness in the panoramic image, and the image quality of the panoramic image after brightness adjustment is not much different from the original image quality.

[0033] like Figure 1 As shown, Figure 1 1 is a flow chart of an embodiment of a panoramic image stitching method provided by the present application. Specifically, the stitching method includes:

[0034] S11: Capture video streams and obtain stitching parameters, where the stitching parameters include stitching lines and edges of the panoramic image.

[0035] When performing panoramic image stitching, firstly, video streams from multiple cameras are obtained, and stitching parameters for stitching multiple images of the video streams are obtained.

[0036] Video stream acquisition methods include, but are not limited to, acquisition through a capture card, network stream acquisition, USB (Universal Serial Bus) acquisition, video file input, etc. After acquiring the video streams from each camera, a corresponding buffer area can be set up for each video stream to store the video stream data for splicing.

[0037] If there is an SDK (Software Development Kit) interface for directly controlling the brightness of each video stream, it can be applied to the present splicing device and called when the brightness of the panoramic image needs to be adjusted.

[0038] Furthermore, stitching parameters include data such as the stitching lines, edges, projection parameters, and feature points of the panoramic image to be stitched. Because the number, focal length, and relative spatial position of the cameras capturing the video streams are fixed, the stitching parameters can be calculated in advance and saved in a template. When stitching panoramic images, these parameters can be reused directly, rather than recalculating them for each frame, improving image stitching efficiency.

[0039] S12: Obtaining a brightness offset mapping table, wherein the brightness offset mapping table is obtained by selecting a plurality of key points from the intersections between stitching lines and the intersections between stitching lines and edge lines.

[0040] Obtain a brightness offset mapping table, which includes pixels in multiple images in a video stream and brightness offset values corresponding to the pixels. The brightness of the multiple images in the video stream can be adjusted based on the brightness offset mapping table. Here, the pixels in the image refer to the pixels to be projected in the image.

[0041] Specifically, the brightness offset table is calculated by selecting a number of key points from the intersections between stitching lines and the intersections between stitching lines and edge lines, thereby making the brightness of the panoramic image more balanced.

[0042] In a specific embodiment, Figure 2 As shown, the process of obtaining the brightness offset mapping table includes:

[0043] S121: Calculate the brightness offset values of multiple key points corresponding to each image.

[0044] S122: Calculate a brightness offset mapping table based on the brightness offset values of multiple key points.

[0045] After selecting multiple key points from the intersections between stitching lines and the intersections between stitching lines and edge lines, the brightness offset value of the key point is calculated. Then, based on the brightness offset values of the multiple key points, the brightness offset value of the pixel point to be projected in each image is calculated to obtain a brightness offset mapping table.

[0046] In a specific embodiment, after obtaining the brightness offset values of multiple key points, the maximum value among the brightness offset values of the multiple key points is compared to see if it is less than a first threshold. If the brightness offset values of the multiple key points are less than the first threshold, it can be considered that the brightness difference of the image is not significant and the impact is not significant, so the brightness of the image can be adjusted. In other words, the offset value of the pixel to be projected in the image can be set to 0 to obtain a brightness offset mapping table.

[0047] In another embodiment, Figure 3 As shown, the process of obtaining the brightness offset mapping table includes:

[0048] S111: Calculate the difference between the maximum and minimum values of the brightness offset values of multiple key points in the image.

[0049] The brightness offset values of multiple key points in the image are obtained, and then the difference between the maximum and minimum brightness offset values of the multiple key points is calculated.

[0050] S112: When the difference is less than a second threshold, an average value of the brightness offset values of the multiple key points is calculated.

[0051] S113: Setting the brightness offset values of the pixels in the image to the average value of the brightness offset values of the multiple key points to obtain a brightness offset mapping table.

[0052] When the difference between the maximum and minimum brightness offset values of multiple key points is less than a second threshold, the brightness offset values of all pixels to be projected in the image may be the same, and the average brightness offset value of the multiple key points in each image may be calculated. The brightness offset value of all pixels to be projected in the image may be set to the average brightness offset value of the multiple key points.

[0053] This method can make the acquisition process of the brightness offset mapping table simpler, simplify the calculation process, and improve performance.

[0054] In another specific embodiment, Figure 4 As shown, the process of obtaining the brightness offset mapping table includes:

[0055] S211: Select four key points from the intersection points between stitching lines and the intersection points between stitching lines and edge lines, wherein the four key points are respectively distributed in four quadrants of the panoramic image.

[0056] Four key points are selected from the intersections of the stitching lines and the intersections of the stitching lines and the edge lines. These four key points are distributed in the four quadrants of the panoramic image, so that the area of the quadrilateral formed by the four key points is as large as possible, thereby reducing the calculation error of the brightness offset value of other pixels.

[0057] For example, Figure 5 As shown, four key points can be selected from the intersections of the stitching line and the edge line, namely point A, point B, point C and point D. Point A, point B, point C and point D are located in four quadrants of the panoramic image respectively.

[0058] Further, if Figure 6 As shown in the figure, when there are two or more intersection points in a quadrant, the distance from each intersection point in the quadrant to the diagonal of the corresponding quadrant is calculated, and the point with the closest distance in each quadrant is taken as the key point. For example, the intersection points between the five suture lines are point A, point B, point C, point D, and point E. According to the distance from the intersection point to the diagonal of the corresponding quadrant, the key points finally selected are: point A, point C, point D, and point E.

[0059] In another embodiment, when there are two or more intersection points in a quadrant, the intersection point closer to the corner may be selected as the key point.

[0060] S212: Based on the four key points, four sampling areas are divided around the four key points, and each sampling area does not exceed the overlapping area of multiple images.

[0061] Based on the four key points, four sampling regions are divided around them. Specifically, a sampling region can be taken with each key point as the center. The sampling region does not exceed the overlapping area of multiple images. For example, if the four key points are A, B, C, and D, four sampling regions are divided around them: region_A, region_B, region_C, and region_D.

[0062] The shape of the sampling area can be square, rectangular, circular, elliptical or irregular, etc. When the sampling area is square, the side length of the square is 5-10 pixels, and the boundary of the square does not exceed the overlapping area of multiple images.

[0063] S213: Calculate the average brightness of each sampling area of each image respectively.

[0064] After the sampling areas are obtained, each sampling area corresponds to multiple images. For example, if three images are to be spliced, each of the three images corresponds to four sampling areas.

[0065] The average brightness value of each sampling area in each image is calculated separately. Optionally, a brightness histogram of multiple pixel points within each sampling area in each image can be statistically analyzed, and the average brightness value of the sampling area can be calculated based on the brightness histogram. Furthermore, a certain percentage of the brightness values at the left and right extremes of the brightness histogram can be removed, for example, 5% or 10%, to avoid large errors caused by extremely dark or bright areas. The average brightness value of the sampling area is then calculated based on the brightness histogram.

[0066] S214: Calculating a brightness offset value of the same sampling area of each image based on the average brightness value of the same sampling area of the multiple images, so as to obtain brightness offset values of four sampling areas corresponding to each image.

[0067] After obtaining the average brightness of the sampling regions corresponding to multiple images, the brightness offset value of the same sampling region in each image is calculated based on the average brightness of the same sampling region in multiple images. For example, if sampling region region_A corresponds to multiple images, the brightness offset value of sampling region_A in each image is calculated based on the average brightness of region_A in multiple images. Using this method, the brightness offset values of the remaining sampling regions in each image can be obtained.

[0068] Specifically, the median value of the brightness averages of the same sampling region in multiple images is found. For example, if there are three images, for sampling region region_A, the median value of the brightness averages of the sampling region region_A corresponding to the three images is found. For example, if the brightness averages of region_A corresponding to the three images are 1, 2, and 3, respectively, the brightness average of 2 is the median value. The offset value of the sampling region in the image corresponding to the median value is then set to 0, meaning that no brightness adjustment is performed on the pixels in the image corresponding to the median value. The brightness averages of the sampling region in the other images are subtracted from the median value to obtain the brightness offset values of the sampling region in the other images. In this way, the brightness offset values of the four sampling regions corresponding to each image can be obtained.

[0069] S215: Using the brightness offset value of the sampling area as the brightness offset value of the key point corresponding to the sampling area, so as to obtain the brightness offset values of the four key points in each image.

[0070] The brightness offset values obtained for each sampling region are used as the brightness offset values for the key points corresponding to that sampling region to obtain the brightness offset values for the four key points in each image. For example, the brightness offset value for sampling region region_A in each image is calculated and used as the brightness offset value for key point A in that image. In this way, the brightness offset values for the four key points in each image can be obtained.

[0071] S216: Based on the brightness offset values of the four key points of each image, a bilinear interpolation algorithm is used to calculate the brightness offset values of the pixels in each image to obtain a brightness offset mapping table.

[0072] After obtaining the brightness offset values of the four key points, a bilinear interpolation algorithm is then used to calculate the brightness offset values of the pixels to be projected in each image to obtain a brightness offset mapping table, so that each image can adjust the brightness of each pixel according to the brightness offset mapping table.

[0073] Specifically, if Figure 7 As shown, the four key points selected are point A, point B, point C and point D, point F is the pixel point on the line connecting point A and point B, point G is the pixel point on the line connecting point C and point D, and pixel point E is located on the line connecting point G and point F.

[0074] According to the bilinear interpolation algorithm, the brightness offset value E_i of the pixel E in the image is calculated by the following formula:

[0075] E_i=F_i×(E_y-F_y) / (G_y-F_y)+G_i×(G_y-E_y) / (G_y-F_y);

[0076] Among them, F_i=A_i×(F_x-A_x) / (B_x-A_x)+B_i×(B_x-F_x) / (B_x-A_x),

[0077] G_i=C_i×(G_x-C_x) / (C_x-D_x)+D_i×(D_x-G_x) / (D_x-C_x),

[0078] Among them, F_i is the brightness offset value of point F, F_x is the x-coordinate value of point F, F_y is the y-coordinate value of point F, E_y is the y-coordinate value of pixel point E, G_i is the brightness offset value of point G, G_x is the x-coordinate value of point G, G_y is the y-coordinate value of point G, A_i is the brightness offset value of point A, A_x is the x-coordinate value of point A, B_i is the brightness offset value of point B, B_x is the x-coordinate value of point B, C_i is the brightness offset value of point C, C_x is the brightness value of point C, D_i is the brightness offset value of point D, and D_x is the x-coordinate value of point D.

[0079] The brightness offset value of each pixel to be projected in the image can be calculated according to the above formula. The calculation method is simple and has a small error.

[0080] S13: stitching and brightness adjustment are performed on multiple images of the video stream according to the stitching parameters and the brightness offset mapping table to obtain a panoramic image.

[0081] After obtaining the brightness offset table, multiple images in the video stream are stitched and brightness adjusted according to the stitching parameters and the obtained brightness offset table to obtain a panoramic image. This process can use parallel computing to improve the efficiency of image stitching.

[0082] In the above embodiment, a pixel-level brightness offset mapping table is used to adjust the brightness of each image individually, down to the individual pixel points. This results in smoother image brightness changes. Furthermore, the total offset value is smaller than in traditional solutions, resulting in a brightness-adjusted image that more closely resembles the original image quality.

[0083] In another embodiment, the brightness offset mapping table may be updated periodically through dynamic brightness inspection.

[0084] like Figure 8 As shown, the update process of the brightness offset mapping table includes:

[0085] S311: After a first preset time interval, recalculate the brightness offset values of the multiple key points corresponding to the image to obtain new brightness offset values of the multiple key points.

[0086] After obtaining the brightness offset mapping table, the brightness offset values need to be updated because the lighting environment of each video stream may change, such as the light source of a live scene. After a first preset time interval, the brightness offset values of multiple key points in the image are recalculated. Specifically, the brightness offset values of the key points can be calculated using the method described in the above embodiment. The specific calculation method is described in the above embodiment and is not repeated here.

[0087] S312: When the new brightness offset values of multiple key points meet the preset conditions, a new brightness offset mapping table of multiple key points is calculated based on the new brightness offset values of the multiple key points, and the brightness offset value mapping table is updated based on the new brightness offset value mapping table, and the first preset time is shortened.

[0088] After obtaining new brightness offset values for multiple key points, it is determined whether the new brightness offset values for the multiple key points meet a preset condition. The preset condition may be whether the difference between the average of the new brightness offset values for the multiple key points and the average of the original brightness offset values for the multiple key points exceeds a certain value. If the difference exceeds the certain value, the new brightness offset values for the multiple key points are considered to meet the preset condition.

[0089] The brightness offset values of other pixels to be projected in the image are calculated based on the new brightness offset values of the multiple key points to obtain a new brightness offset value mapping table, and the original brightness offset value mapping table is updated using the new brightness offset value mapping table.

[0090] When the new brightness offset values of multiple key points meet the preset conditions, it indicates that the brightness offset mapping table needs to be updated based on the new brightness offset values of the key points. At this point, the first preset time can be shortened to reduce the inspection interval, for example, by reducing the first preset time by 1 minute each time. A minimum time interval can also be set, and the first preset time interval can be shortened but must be greater than this minimum time interval. By shortening the first preset time, the inspection interval can be shortened and the frequency of brightness adjustment can be increased.

[0091] S313: When the new brightness offset values of the plurality of key points do not meet the preset condition, the first preset time is extended.

[0092] If the new brightness offset values for multiple key points do not meet preset conditions, for example, if the difference between the average of the new brightness offset values for the multiple key points and the average of the original brightness offset values for the multiple key points does not exceed a certain value, then the change in the brightness offset values for the multiple key points is not significant, and therefore, there is no need to update the brightness offset mapping table. In this case, the first preset time can be extended to increase the inspection interval, thereby reducing the overall computational complexity of the device and optimizing performance.

[0093] Furthermore, a maximum time interval may be set, and the first preset time after extension cannot be greater than the maximum time interval, so as not to affect the image splicing effect.

[0094] In the above embodiment, the brightness offset mapping table can be dynamically inspected, so that the brightness offset mapping table can be updated in a timely manner, making the brightness of the panoramic image more balanced and improving device performance.

[0095] For example, the above-mentioned panoramic image stitching method can be applied to a live broadcast system. Specifically, Figure 9 As shown, Figure 9 This is a structural diagram of an embodiment of the live broadcast system provided by the present application. The live broadcast system 90 includes an anchor terminal 91, a server 92 and an audience terminal 93. The anchor terminal 91 collects multiple video streams through multiple cameras and sends the multiple video streams to the server 92. The server 92 receives the multiple video streams and stitches multiple images in the video stream through the panoramic image stitching method of any of the above embodiments to obtain a panoramic image. The server 92 then sends the panoramic image to the audience terminal 93 so that the audience terminal 93 can display the panoramic image to complete the live broadcast of the panoramic picture.

[0096] The present application also provides a splicing device, such as Figure 10 As shown, Figure 10 It is a framework schematic diagram of an embodiment of the stitching device provided by the present application. Furthermore, the stitching device 70 includes: a first acquisition module 71, used to capture the video stream and obtain stitching parameters, the stitching parameters including the stitching lines and edge lines of the panoramic image; a second acquisition module 72, used to obtain a brightness offset mapping table, wherein the brightness offset mapping table is calculated by selecting multiple key points from the intersections between the stitching lines and the intersections between the stitching lines and the edge lines; a stitching module 73, used to stitch and adjust the brightness of multiple images of the video stream according to the stitching parameters and the brightness offset mapping table to obtain a panoramic image.

[0097] Furthermore, the second acquisition module 72 is also used to calculate a brightness offset mapping table by selecting multiple key points from the intersections between the stitching lines and the intersections between the stitching lines and the edge lines, including: calculating the brightness offset values of multiple key points corresponding to each image; and calculating the brightness offset mapping table based on the brightness offset values of the multiple key points.

[0098] Furthermore, the second acquisition module 72 is also used to select four key points from the intersections between the stitching lines and the intersections between the stitching lines and the edge lines, wherein the four key points are respectively distributed in the four quadrants of the panoramic image; based on the four key points, four sampling areas are divided around the four key points, and each sampling area does not exceed the overlapping area of multiple images; the brightness average of each sampling area of each image is calculated respectively; based on the brightness average of the same sampling area of multiple images, the brightness offset value of the same sampling area of each image is calculated to obtain the brightness offset values of the four sampling areas corresponding to each image, and the brightness offset values of the sampling areas are used as the brightness offset values of the key points corresponding to the sampling areas to obtain the brightness offset values of the four key points in each image.

[0099] Furthermore, the second acquisition module 72 is also used to calculate a brightness offset mapping table based on the brightness offset values of multiple key points, including, based on the brightness offset values of the four key points of each image, using a bilinear interpolation algorithm to calculate the brightness offset values of the pixel points in each image to obtain a brightness offset mapping table.

[0100] Furthermore, the second acquisition module 72 is also used to calculate a brightness offset mapping table based on the brightness offset values of multiple key points, including: when the maximum value of the brightness offset values of multiple key points in the image is less than the first threshold, the offset value of the pixel point in the image is set to 0 to obtain a brightness offset mapping table.

[0101] Furthermore, the second acquisition module 72 is also used to calculate a brightness offset mapping table based on the brightness offset values of multiple key points, including: calculating the difference between the maximum and minimum values of the brightness offset values of multiple key points in the image, and when the difference is less than the second threshold, calculating the average value of the brightness offset values of the multiple key points; setting the brightness offset values of all pixel points in the image to the average value of the brightness offset values of the multiple key points to obtain a brightness offset mapping table.

[0102] Furthermore, the second acquisition module 72 is also used to calculate the brightness offset values of multiple key points corresponding to each image, and calculate a brightness offset mapping table based on the brightness offset values of the multiple key points, including: after a first preset time interval, recalculating the brightness offset values of the multiple key points corresponding to the image to obtain new brightness offset values of the multiple key points; when the new brightness offset values of the multiple key points meet the preset conditions, a new brightness offset mapping table is calculated based on the new brightness offset values of the multiple key points, and the brightness offset value mapping table is updated based on the new brightness offset value mapping table, and the first preset time is shortened; when the new brightness offset values of the multiple key points do not meet the preset conditions, the first preset time is extended.

[0103] The stitching device 70 can perform panoramic stitching on multiple images of a video stream, so that the brightness of the panoramic image is more balanced, the image quality of the panoramic image is less different from the original image quality, and the performance is better.

[0104] This application also provides a panoramic image stitching device, see Figure 11 , Figure 11 FIG2 is a schematic diagram of a framework of an embodiment of a panoramic image stitching device provided in the present application. Stitching device 80 includes a memory 81 and a processor 82 coupled to each other. Processor 82 is configured to execute program instructions stored in memory 81 to implement the steps of the panoramic image stitching method of any of the above-described embodiments.

[0105] The processor 82 may also be referred to as a CPU (Central Processing Unit). The processor 82 may be an integrated circuit chip having signal processing capabilities. The processor 82 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor, or the processor 82 may be any conventional processor. In addition, the processor 82 may be implemented by an integrated circuit chip.

[0106] The above-mentioned stitching device 80 can stitch multiple images of the video stream to form a panoramic image. The brightness of the panoramic image stitched by the stitching device 80 of the present application is more balanced, the picture after brightness adjustment is closer to the original picture quality, and the performance is better.

[0107] This application also provides a computer-readable storage medium, see Figure 12 As shown, Figure 12 The computer-readable storage medium 10 stores program instructions 101 that can be executed by a processor, and the program instructions 101 are used to implement the steps of the panoramic image stitching method of any of the above embodiments.

[0108] The program instructions 101 may be stored in the computer-readable storage medium 10 in the form of a software product, including several instructions for enabling a device or processor to execute all or part of the steps of the methods of various embodiments of the present application.

[0109] Computer-readable storage medium 10 is a medium in a computer memory used to store discrete physical quantities. Computer-readable storage medium 10 includes various media capable of storing program instruction 101 code, such as a USB flash drive, a removable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0111] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0112] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0113] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the methods of each embodiment of the present application.

Claims

1. A panoramic image stitching method, characterized in that: The splicing method comprises: Capturing a video stream and obtaining stitching parameters, wherein the stitching parameters include stitching lines and side lines of the panoramic image; Obtaining a brightness offset mapping table, wherein the brightness offset mapping table is calculated by selecting a plurality of key points from intersections between the stitching lines and intersections between the stitching lines and the edge lines; splicing and adjusting the brightness of the multiple images of the video stream according to the splicing parameters and the brightness offset mapping table to obtain the panoramic image, The step of calculating the brightness offset mapping table by selecting a plurality of key points from the intersections between the stitching lines and the intersections between the stitching lines and the edge lines comprises: calculating brightness offset values of the plurality of key points corresponding to each of the images; and calculating the brightness offset mapping table based on the brightness offset values of the plurality of key points. Calculating the brightness offset values of the plurality of key points corresponding to each of the images includes: Selecting four key points from intersections between the stitching lines and intersections between the stitching lines and the edge lines, wherein the four key points are respectively distributed in four quadrants of the panoramic image; Based on the four key points, four sampling areas are divided around the four key points, and each sampling area does not exceed the overlapping area of the multiple images; Calculating the average brightness of each sampling area of each image respectively; Based on the average brightness of the same sampling area of the plurality of images, a brightness offset value of the same sampling area of each image is calculated to obtain brightness offset values of the four sampling areas corresponding to each image. The brightness offset value of the sampling area is used as the brightness offset value of the key point corresponding to the sampling area, so as to obtain the brightness offset values of the four key points in each image.

2. The splicing method according to claim 1, characterized in that: The brightness offset mapping table is calculated based on the brightness offset values of the plurality of key points, including: Based on the brightness offset values of the four key points of each image, a bilinear interpolation algorithm is used to calculate the brightness offset value of each pixel in the image to obtain the brightness offset mapping table.

3. The splicing method according to claim 1, characterized in that: The calculating and obtaining the brightness offset mapping table based on the brightness offset values of the plurality of key points includes: When the maximum value of the brightness offset values of the plurality of key points in the image is less than a first threshold, the offset value of the pixel point in the image is set to 0 to obtain the brightness offset mapping table.

4. The splicing method according to claim 1, characterized in that: The calculating and obtaining the brightness offset mapping table based on the brightness offset values of the plurality of key points includes: Calculate the difference between the maximum and minimum values of the brightness offset values of the plurality of key points in the image, When the difference is less than a second threshold, an average value of the brightness offset values of the plurality of key points is calculated; The brightness offset values of all pixels in the image are set to an average value of the brightness offset values of a plurality of key points to obtain the brightness offset mapping table.

5. The splicing method according to claim 1, characterized in that: The calculating the brightness offset values of the plurality of key points corresponding to each of the images, and obtaining the brightness offset mapping table based on the brightness offset values of the plurality of key points, comprises: After a first preset time interval, recalculating the brightness offset values of the plurality of key points corresponding to the image to obtain new brightness offset values of the plurality of key points; When the new brightness offset values of the plurality of key points meet a preset condition, a new brightness offset mapping table is calculated based on the new brightness offset values of the plurality of key points, the brightness offset mapping table is updated based on the new brightness offset mapping table, and the first preset time is shortened; When the new brightness offset values of the plurality of key points do not meet the preset condition, the first preset time is extended.

6. A panoramic image stitching device, characterized in that: The splicing device comprises: A first acquisition module is used to collect video streams and obtain stitching parameters, where the stitching parameters include stitching lines and side lines of the panoramic image; a second acquisition module, configured to acquire a brightness offset mapping table, wherein the brightness offset mapping table is calculated by selecting a plurality of key points from intersections between the stitching lines and intersections between the stitching lines and the edge lines; a stitching module, configured to stitch and adjust the brightness of the plurality of images of the video stream according to the stitching parameters and the brightness offset mapping table to obtain the panoramic image; The second acquisition module is further configured to calculate the brightness offset values of the plurality of key points corresponding to each of the images; and obtain the brightness offset mapping table based on the brightness offset values of the plurality of key points. The second acquisition module is further configured to select four key points from the intersections between the stitching lines and the intersections between the stitching lines and the edge lines, wherein the four key points are respectively distributed in the four quadrants of the panoramic image; based on the four key points, four sampling areas are divided around the four key points, each sampling area does not exceed the overlapping area of the multiple images; the brightness average of each sampling area of each image is calculated; based on the brightness average of the same sampling area of the multiple images, the brightness offset value of the same sampling area of each image is calculated to obtain the brightness offset values of the four sampling areas corresponding to each image, and the brightness offset values of the sampling areas are used as the brightness offset values of the key points corresponding to the sampling areas to obtain the brightness offset values of the four key points in each image.

7. A panoramic image stitching device, characterized in that: The method comprises a memory and a processor coupled to each other, wherein the processor is configured to execute program instructions stored in the memory to implement the panoramic image stitching method according to any one of claims 1 to 5.

8. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the panoramic image stitching method according to any one of claims 1 to 5 is implemented.

Images