Panoramic image optimization method and device based on space-time filtering, equipment and medium

Abstract

The invention discloses a panoramic image optimization method and device based on space-time filtering, equipment and a medium, and the method comprises the steps: splicing a to-be-processed original video based on an optimal suture line splicing method, and obtaining an initial panoramic video; aligning the target frame with other frames in a preset space-time window based on an optical flow estimation algorithm; calculating the distance between the pixel in the space-time window and the optimal suture line of the frame where the pixel is located, and obtaining the splicing reliability of the pixel based on the distance; obtaining the weight of each pixel in the space-time window based on the space distance, the time distance and the pixel distance between each pixel in the space-time window and the target pixel in the target frame stitching area and the splicing reliability of each pixel in the space-time window; and optimizing the target pixel in the target frame stitching area based on the weight of each pixel in the space-time window. According to the invention, the spliced image can be further optimized, so that the quality and stability of the panoramic video are improved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a panoramic image optimization method, device, equipment and medium based on spatiotemporal filtering. Background Art

[0002] Panoramic image generation technology is a technology that can obtain 360-degree full-view images, aiming to provide users with an immersive visual experience. In this technology, multiple cameras simultaneously capture a scene from different angles, and then stitch and fuse these images to ultimately generate a panoramic image or video. This technology has a wide range of applications in various fields, including virtual reality, augmented reality, video surveillance, and film and television production.

[0003] In virtual reality applications, panoramic image generation technology can create a highly immersive virtual environment, where users can view the environment from all angles through head-mounted devices, giving them an immersive experience. This is very important in areas such as gaming, education, and virtual exhibitions. Augmented reality technology enables users to perceive both virtual and real information by superimposing virtual images on the real-world field of view. Panoramic image generation technology provides a complete environmental context in the process, which helps to enhance the authenticity and interactivity of the reality content. In terms of video surveillance, panoramic cameras can cover a wide monitoring area, reduce monitoring blind spots, and improve safety and monitoring efficiency. They are often used in large public places such as airports, stations, shopping malls, etc., as well as smart home systems.

[0004] In the existing seam-based panoramic image stitching methods, there is a lack of utilization of temporal context information. It only considers the image information at the same time point, while ignoring the image context information at other time points. In the video, some areas may be near the seam at a certain moment, but not near the seam at other moments. Due to the failure to utilize this temporal context information, the stitching effect may be inconsistent between different time frames, resulting in unstable image quality. Secondly, this scheme may result in poor image quality at the seam. Even after correction, there may still be differences in brightness and color between the two images at the seam. Even if stitching is performed along the optimal seam, obvious stitching traces may still appear. Thirdly, this type of method considers a type of local optimization scheme, that is, independently searching for the optimal seam in each frame of the image, lacking a global perspective. The integrity of the panoramic video is ignored, and the relevant information between consecutive frames in the video cannot be fully utilized, thus failing to achieve the best stitching effect. Summary of the invention

[0005] The embodiments of the present application provide a panoramic image optimization method, device, equipment and storage medium based on spatiotemporal filtering to at least solve the technical problem of low quality of panoramic video images generated in related technologies.

[0006] A panoramic image optimization method based on spatiotemporal filtering provided in an embodiment of the present application includes:

[0007] The original video to be processed is stitched based on the optimal stitching line stitching method to obtain an initial panoramic video; the target frame is aligned with other frames in a preset space-time window based on an optical flow estimation algorithm; the distance of the pixel in the space-time window relative to the optimal stitching line of the frame in which it is located is calculated, and the stitching reliability of the pixel at this point is obtained based on the distance; the weight of each pixel in the space-time window is obtained based on the spatial distance, temporal distance, pixel distance between each pixel in the space-time window and the target pixel in the stitching area of ​​the target frame and the stitching reliability of each pixel in the space-time window; based on the weight of each pixel in the space-time window, the target pixel in the stitching area of ​​the target frame is optimized.

[0008] In one embodiment, the original videos to be processed are stitched together based on the optimal stitching line stitching method to obtain an initial panoramic video, including:

[0009] Extract feature points of each frame image in the original video; perform feature matching based on the feature points, and geometrically align the image pairs according to the matched feature points; construct an energy equation for solving the optimal stitching line based on the difference between the pixel value and the gradient value, and obtain the optimal stitching line of the image pair based on the energy equation; perform image stitching based on the optimal stitching line to obtain the initial panoramic video.

[0010] In one embodiment, before extracting the feature points of each frame of the original video, the method further includes:

[0011] Acquire multiple original videos to be processed; perform denoising, distortion correction and color correction on the original videos to obtain preprocessed original videos.

[0012] In one embodiment, aligning the target frame with other frames in a preset spatiotemporal window based on an optical flow estimation algorithm includes:

[0013] Based on an optical flow algorithm for panoramic videos, an optical flow from a target frame in the initial panoramic video to other frames in a preset spatiotemporal window is estimated; based on optical flow mapping, pixels of other frames in the spatiotemporal window are aligned with an optimal stitching line to the target frame.

[0014] In one embodiment, calculating the distance between the pixel in the spatiotemporal window and the optimal stitching line of the frame in which the pixel is located includes:

[0015] Select the points of the optimal stitching line aligned by optical flow mapping within the preset spatial window; perform linear fitting on the selected points based on the least squares method to obtain the local fitting line of the optimal stitching line; use the vertical distance between the pixel point and the local fitting line to characterize the distance of the pixel point relative to the optimal stitching line of its frame.

[0016] In one embodiment, based on the spatial distance, temporal distance, pixel distance between each pixel in the spatiotemporal window and the target pixel in the stitching area of ​​the target frame and the splicing reliability of each pixel in the spatiotemporal window, the weight of each pixel in the spatiotemporal window is obtained, including:

[0017] Through an exponential function with a coefficient of -1, the time distance, spatial distance, pixel distance between the target pixel in the stitching area and the pixels in its space-time window and the stitching reliability of the pixels in the space-time window are expressed as the weight factor of space-time filtering; the standard deviation coefficient is used to adjust the weight factor to obtain the weight of each pixel in the space-time window.

[0018] In one embodiment, based on the weight of each pixel in the spatiotemporal window, the target pixel in the stitching area of ​​the target frame is optimized, including:

[0019] Based on the weight of each pixel in the spatiotemporal window, a weighted average is performed to obtain the optimized pixel value of the target pixel in the stitching area.

[0020] According to one aspect of an embodiment of the present application, a panoramic image optimization device based on spatiotemporal filtering is provided, comprising:

[0021] An initial stitching module is used to stitch the original video to be processed based on the optimal stitching line stitching method to obtain an initial panoramic video; an optical flow alignment module is used to align the target frame with other frames in a preset space-time window based on an optical flow estimation algorithm; a stitching reliability calculation module is used to calculate the distance of the pixel in the space-time window relative to the optimal stitching line of the frame in which it is located, and obtain the stitching reliability of the pixel at this point based on the distance; a weight calculation module is used to obtain the weight of each pixel in the space-time window based on the spatial distance, temporal distance, pixel distance between each pixel in the space-time window and the target pixel in the stitching area of ​​the target frame and the stitching reliability of each pixel in the space-time window; an optimization module is used to optimize the target pixel in the stitching area of ​​the target frame based on the weight of each pixel in the space-time window.

[0022] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned panoramic image optimization method based on spatiotemporal filtering through the computer program.

[0023] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned panoramic image optimization method based on spatiotemporal filtering when running.

[0024] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0025] The solution provided in the embodiment of the present application is optimized through spatiotemporal filtering after obtaining the initially spliced ​​panoramic video. When filtering a certain frame, the optical flow between it and other frames in the spatiotemporal neighborhood is solved, and other frames are aligned to the frame through optical flow mapping. For a pixel to be filtered that is to be optimized and updated, the filtering weight is obtained by comprehensively considering the splicing reliability of the time distance, spatial distance, pixel distance and reference pixel between the reference pixel and the pixel to be filtered in the spatiotemporal neighborhood. The present application makes full use of the time correlation of the panoramic video, considers the image context information at other time points, and effectively improves the quality and stability of the spliced ​​image. Furthermore, the present application makes full use of the relevant information between consecutive frames in the video, filters according to the information in the temporal and spatial neighborhood, considers the integrity of the panoramic video, and can reconstruct a high-quality panoramic video. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0027] Figure 1 is a flow chart of a panoramic image optimization method based on spatiotemporal filtering according to an embodiment of the present application;

[0028] Figure 2 is a flowchart of another panoramic image optimization method based on spatiotemporal filtering according to an embodiment of the present application;

[0029] Figure 3 is a schematic diagram of a spatiotemporal neighborhood of a pixel in a stitching area according to an embodiment of the present application;

[0030] Figure 4 is a schematic diagram of a pixel-to-seam line distance according to an embodiment of the present application.

[0031] Figure 5 is a schematic diagram of a panoramic image optimization device according to an embodiment of the present application;

[0032] Figure 6 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] For the initial panoramic video obtained based on the optimal stitching line solution, the present application comprehensively considers four factors: time distance, space distance, pixel distance and stitching credibility. For each pixel in the stitching area, based on the result of aligning other panoramic video frames to this frame, a spatiotemporal neighborhood is established and the weight of each pixel therein is calculated for filtering, so as to make full use of the temporal correlation of the panoramic video and thus improve the quality of the panoramic video.

[0036] The following is combined with Figure 1 The panoramic image optimization method based on spatiotemporal filtering in the embodiment of the present application is introduced in detail. Figure 1 As shown, the method mainly includes the following steps:

[0037] S101 stitches the original video to be processed based on the optimal stitching line stitching method to obtain an initial panoramic video.

[0038] In one embodiment, a plurality of original videos to be processed are first obtained, and denoising, distortion correction and color correction are performed on the original videos to obtain pre-processed original videos.

[0039] For example, two original videos to be stitched are obtained, assuming that the images are stitched horizontally. First, the two video sequences to be stitched are preprocessed. This includes operations such as denoising, distortion correction, and color correction to ensure the quality of the input images and the consistency of the images.

[0040] Furthermore, feature points of each frame image in the original video are extracted, feature matching is performed based on the feature points, and the image pairs are geometrically aligned according to the matched feature points.

[0041] You can use algorithms such as SIFT to detect and describe feature points. For each frame of the image, perform feature matching and use algorithms such as KNN to find matching feature points between the two images. Based on the matched feature points, calculate the homography matrix or other geometric transformation model to geometrically align the two images. This ensures that the images are aligned in the same spatial coordinate system.

[0042] Furthermore, an energy equation for solving the optimal stitching line is constructed based on the difference between the pixel value and the gradient value, and the optimal stitching line of the image pair is obtained based on the energy equation; the images are stitched based on the optimal stitching line to obtain the initial panoramic video.

[0043] On the basis of geometric alignment, an energy equation for solving the optimal stitching line is constructed based on the difference between pixel values ​​and gradient values, and solved by dynamic programming or graph cut schemes to obtain the optimal stitching line {s1, s2, ..., s n}, where s1, s2, ..., s n It is the set of points in the overlapping area of ​​the images after the overlapping parts of the image pairs are stitched together. For the optimal stitching line, each vertical coordinate of the image corresponds to a point on the stitching line. Based on the above optimal stitching line, stitching can be performed to obtain the initial panoramic video, that is, the initial panoramic image sequence. Wherein x=(x, y), x and y correspond to the horizontal and vertical coordinates respectively.

[0044] According to this step, a stitched initial panoramic image can be obtained.

[0045] S102 aligns the target frame with other frames in a preset spatiotemporal window based on an optical flow estimation algorithm.

[0046] In one embodiment, the target frame is aligned with other frames in a preset spatiotemporal window based on an optical flow estimation algorithm. First, an optical flow from the target frame in the initial panoramic video to other frames in the preset spatiotemporal window is estimated based on an optical flow algorithm for panoramic videos. The target frame represents the frame currently being processed.

[0047] For example, it is currently planned to optimize and improve the quality of the kth frame in the panoramic video, such as Figure 3 As shown, the selected time window radius is w, then the panoramic video frame {I k-w (x), ..., I k-1 (x), I k+1 (x), ..., I k+w(x)} are aligned to the kth frame of the panoramic video at the moment. In order to complete the alignment operation, the optical flow algorithm for panoramic video can be used to solve I k The optical flow field sequence aligned to the above panoramic image sequence {F k，k-w (x), ..., F k，k-1 (x), F k，k+1 (x), ..., F k，k+w (x)}, where F k，k-w (x) represents the optical flow of the k-th panoramic image aligned to the kw-th panoramic image.

[0048] Furthermore, the pixels of other frames in the spatiotemporal window are aligned to the target frame with the optimal stitching line based on the optical flow mapping.

[0049] According to the above optical flow, through optical flow mapping, the panoramic video frame in the time window aligned to the kth frame can be obtained Taking the k-1th frame as an example, the alignment of the optical flow map is as follows:

[0050]

[0051] The optimal stitching line of the panoramic video frame subjected to the above alignment operation is also updated through the above optical flow mapping to obtain the optimal stitching line after alignment. Still taking the k-1th frame as an example, its optical flow mapping can be expressed as follows:

[0052]

[0053] This step first determines the time window required to process the frame. Within the time window, the optical flow from the frame where the pixel to be processed is located to other frames is estimated, and the other frames are aligned to the frame where the pixel to be processed is located through optical flow mapping; through optical flow mapping, the stitching lines of other frames are aligned to the frame where the pixel to be processed is located through optical flow mapping.

[0054] Through this step, images can be aligned according to the optical flow so that images in the spatiotemporal neighborhood are in the same coordinate system.

[0055] S103 calculates the distance between the pixel in the spatiotemporal window and the optimal stitching line of the frame in which the pixel is located, and obtains the stitching reliability of the pixel at the point based on the distance.

[0056] In one embodiment, calculating the distance of the pixel in the spatiotemporal window relative to the optimal stitching line of the frame in which it is located includes: selecting points of the optimal stitching line aligned by optical flow mapping in a preset spatial window; performing linear fitting on the selected points based on the least squares method to obtain a local fitting line of the optimal stitching line.

[0057] It is still assumed that the pixels in the stitching area of ​​the kth frame are subjected to spatiotemporal filtering. According to the aligned optimal stitching line calculated in step S102, all pixels in the spatiotemporal window are calculated.

[0058] There is the distance of the pixel relative to the optimal stitching line of the frame in which it is located. Assume that the radius of the spatial window is r, that is, the pixel value I at the pixel (x0, y0) of the original panoramic image of the kth frame k For (x0, y0), its spatiotemporal window can be expressed as follows:

[0059] Θ k (x0, y0) = {I t (x,y),t∈[kw,k+w],x∈[xr,x+r],y∈[yr,y+r]}

[0060] For the pixel with coordinate position (x1, y1) in frame t1, to calculate its stitching credibility, it is necessary to first calculate its local stitching line first-order fitting line y=A(t1, x1, y1)x+B(t1, x1, y1). The coefficient of the first-order fitting line can be obtained by the following equation.

[0061]

[0062] in, They are The vertical and horizontal coordinates of c is the reference range radius when calculating the suture line fitting curve.

[0063] The above formula is the linear fit of discrete points and can be solved by the least squares method.

[0064] Furthermore, the vertical distance between the pixel point and the local fitting line is used to characterize the distance of the pixel point relative to the optimal stitching line of the frame in which it is located.

[0065] For the coordinate position (x1, y1) of the t1th frame, the distance between the point and the stitching line can be expressed by the following formula.

[0066]

[0067] like Figure 4 As shown, the curve in the overlapped area is the stitching line aligned by optical flow mapping. Linear fitting is performed on a part of the stitching line, and the obtained dotted straight line is the local fitting line. The distance from the pixel to be processed to the local fitting line is the stitching line distance.

[0068] The distance D(t1, x1, y1) between the point and the stitching line can be used to represent the stitching reliability of the point. The larger the distance, the lower the probability of poor quality caused by stitching, that is, the higher the reliability.

[0069] S104 obtains the weight of each pixel in the spatiotemporal window based on the spatial distance, temporal distance, pixel distance between each pixel in the spatiotemporal window and the target pixel in the stitching area of ​​the target frame and the splicing reliability of each pixel in the spatiotemporal window.

[0070] like Figure 4 As shown, the stitching area is the overlapping area of ​​the two original images, and the target pixel refers to the pixel point being optimized in the stitching area.

[0071] In one embodiment, the time distance, spatial distance, pixel distance and splicing reliability of the target pixel in the stitching area and the pixels in its spatiotemporal window are expressed as weight factors of spatiotemporal filtering through an exponential function with a coefficient of -1. The weight factor is regulated by the standard deviation coefficient, for example, the square of its reciprocal is used as the coefficient to obtain the weight of each pixel in the spatiotemporal window.

[0072] Calculate the weight of each pixel in the spatiotemporal window. For the value I of the original panoramic image at the pixel (x0, y0) of the kth frame k For (x0, y0), calculate its spatiotemporal window Θ k For the pixel with coordinate position (x1, y1) in frame t1, the distance between each pixel in the spatiotemporal window and the stitching line in the frame where it is located calculated in step S103 contributes to I k The weight of the spatiotemporal filtering of (x0, y0) can be expressed as follows.

[0073]

[0074] Among them, exp represents exponential operation, represents the l2 norm, x1=(x1,y1), x0=(x1,y1), σ s , σ i , σ v and σ d They are the weights for controlling spatial distance, temporal distance, pixel distance and stitching reliability, and can be set according to actual conditions.

[0075] S105 optimizes the target pixels in the stitching area of ​​the target frame based on the weights of the pixels in the spatiotemporal window.

[0076] In one embodiment, a weighted average is performed based on the weight of each pixel in the spatiotemporal window to obtain an optimized pixel value of the target pixel in the stitching area.

[0077] Specifically, the updated value of the original panoramic image of the kth frame at pixel (x0, y0) It can be expressed as follows:

[0078]

[0079] According to this step, by optimizing all the pixels in the stitching area of ​​the target frame, the image quality in the stitching area can be improved.

[0080] In order to facilitate understanding of the method of the present application, Figure 2 As shown in the figure, the method includes:

[0081] Specifically, firstly, the video sequences 1 and 2 to be processed are obtained, and the video sequences 1 and 2 are preprocessed and feature extracted respectively to obtain feature points, and feature matching and geometric alignment are performed. Then, the optimal stitching seam is calculated and the image is fused to obtain a preliminary stitched panoramic video.

[0082] Furthermore, for each frame of the image, the optical flow is solved and other frames are aligned to the frame. Then the splicing reliability of the spatiotemporal neighborhood of each pixel in the splicing area is calculated. The weight of the spatiotemporal filtering in each pixel neighborhood is calculated based on the temporal distance, spatial distance, pixel distance and splicing reliability. Spatiotemporal filtering is performed based on the weight of each pixel and the result is output. The final output panoramic video is obtained.

[0083] The present application proposes to use the temporal correlation of panoramic videos to improve the quality of panoramic image generation. Considering that panoramic videos may have problems such as artifacts and missing details at the seams, and the content near the seams in a certain frame may be located in a more reliable area near the non-seams in other frames, the present application proposes to perform spatiotemporal three-dimensional filtering of panoramic videos based on the reference of multiple factors to fully utilize the temporal correlation of panoramic videos. First, the two videos to be spliced ​​are preprocessed and the feature points of each frame are extracted. For each frame, the corresponding frames in the two videos are feature matched and geometrically aligned. On this basis, the optimal stitching line is obtained according to the energy equation and the image is stitched to obtain an initialized panoramic video. For the initialized video, for each frame, if Figure 3 As shown, for the pixels in the stitching area, filtering is performed according to the information in its spatiotemporal neighborhood, and when filtering a certain frame, the optical flow between it and other frames in the spatiotemporal neighborhood is solved and other frames are aligned to the frame through optical flow mapping. For a pixel to be filtered that is to be optimized and updated, the filtering weight is obtained by integrating the spatiotemporal distance between the reference pixel in the spatiotemporal neighborhood and the pixel to be filtered, the pixel distance and the stitching reliability of the reference pixel, where the stitching reliability is determined by the distance between the reference pixel and the stitching line. Based on the above factors, the weights of each pixel in the spatiotemporal neighborhood can be obtained, and filtering is performed according to these weights to reconstruct a higher quality panoramic video.

[0084] This application makes full use of the temporal correlation of panoramic videos, considers the image context information at other time points, and effectively improves the quality and stability of the stitched image. Furthermore, this application makes full use of the relevant information between consecutive frames in the video, performs filtering based on the information in the temporal and spatial neighborhood, considers the integrity of the panoramic video, and can reconstruct a high-quality panoramic video.

[0085] According to another aspect of the embodiments of the present application, a panoramic image optimization device based on spatiotemporal filtering is provided for implementing the above-mentioned panoramic image optimization method based on spatiotemporal filtering. Figure 5 As shown, the device comprises:

[0086] An initial stitching module 501 is used to stitch the original videos to be processed based on an optimal stitching line stitching method to obtain an initial panoramic video;

[0087] An optical flow alignment module 502, configured to align a target frame with other frames within a preset spatiotemporal window based on an optical flow estimation algorithm;

[0088] A stitching reliability calculation module 503 is used to calculate the distance between the pixel in the spatiotemporal window and the optimal stitching line of the frame in which it is located, and obtain the stitching reliability of the pixel at the point based on the distance;

[0089] A weight calculation module 504 is used to obtain the weight of each pixel in the spatiotemporal window based on the spatial distance, temporal distance, pixel distance between each pixel in the spatiotemporal window and the target pixel in the stitching area of ​​the target frame and the splicing reliability of each pixel in the spatiotemporal window;

[0090] The optimization module 505 is used to optimize the target pixels in the stitching area of ​​the target frame based on the weight of each pixel in the spatiotemporal window.

[0091] It should be noted that the panoramic image optimization device based on spatiotemporal filtering provided in the above embodiment only uses the division of the above functional modules as an example when executing the panoramic image optimization method based on spatiotemporal filtering. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the panoramic image optimization device based on spatiotemporal filtering provided in the above embodiment and the panoramic image optimization method based on spatiotemporal filtering belong to the same concept. The implementation process is detailed in the method embodiment, which will not be repeated here.

[0092] According to another aspect of the embodiments of the present application, an electronic device corresponding to the panoramic image optimization method based on spatiotemporal filtering provided in the aforementioned embodiments is also provided to execute the aforementioned panoramic image optimization method based on spatiotemporal filtering.

[0093] Please refer to Figure 6 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 6 As shown, the electronic device includes: a processor 600, a memory 601, a bus 602 and a communication interface 603, and the processor 600, the communication interface 603 and the memory 601 are connected via the bus 602; the memory 601 stores a computer program that can be run on the processor 600, and when the processor 600 runs the computer program, it executes the panoramic image optimization method based on spatiotemporal filtering provided in any of the aforementioned embodiments of the present application.

[0094] The memory 601 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 603 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0095] The bus 602 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 601 is used to store programs, and the processor 600 executes the program after receiving the execution instruction. The panoramic image optimization method based on spatiotemporal filtering disclosed in any of the embodiments of the present application may be applied to the processor 600, or implemented by the processor 600.

[0096] The processor 600 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 600. The above processor 600 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 601, and the processor 600 reads the information in the memory 601 and completes the steps of the above method in combination with its hardware.

[0097] The electronic device provided in the embodiment of the present application and the panoramic image optimization method based on spatiotemporal filtering provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0098] According to another aspect of the embodiments of the present application, a computer-readable storage medium corresponding to the panoramic image optimization method based on spatiotemporal filtering provided in the aforementioned embodiments is also provided, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the panoramic image optimization method based on spatiotemporal filtering provided in any of the aforementioned embodiments.

[0099] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0100] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the panoramic image optimization method based on spatiotemporal filtering provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0101] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0102] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the attached claims.

Claims

1. A panoramic image optimization method based on spatiotemporal filtering, It is characterized in that include: The original videos to be processed are stitched together based on the optimal stitching line stitching method to obtain an initial panoramic video; Align the target frame with other frames in a preset spatiotemporal window based on the optical flow estimation algorithm; Calculating the distance between the pixel in the spatiotemporal window and the optimal stitching line of the frame in which the pixel is located, and obtaining the stitching reliability of the pixel at the point based on the distance; Based on the spatial distance, temporal distance, pixel distance between each pixel in the spatiotemporal window and the target pixel in the stitching area of ​​the target frame and the splicing reliability of each pixel in the spatiotemporal window, the weight of each pixel in the spatiotemporal window is obtained; Based on the weight of each pixel in the spatiotemporal window, the target pixels in the stitching area of ​​the target frame are optimized.

2. The method according to claim 1, It is characterized in that The original videos to be processed are stitched together based on the optimal stitching line stitching method to obtain an initial panoramic video, including: Extracting feature points of each frame of the original video; Perform feature matching based on the feature points, and geometrically align the image pairs according to the matched feature points; An energy equation for solving an optimal stitching line is constructed based on the difference between the pixel value and the gradient value, and an optimal stitching line for the image pair is obtained based on the energy equation; Image stitching is performed based on the optimal stitching line to obtain the initial panoramic video.

3. The method according to claim 2, It is characterized in that Before extracting the feature points of each frame of the original video, the method further includes: Obtain multiple original videos to be processed; The original video is subjected to denoising, distortion correction and color correction to obtain a preprocessed original video.

4. The method according to claim 1, It is characterized in that The target frame is aligned with other frames in a preset spatiotemporal window based on the optical flow estimation algorithm, including: Based on an optical flow algorithm for panoramic videos, the optical flow from the target frame in the initial panoramic video to other frames in a preset spatiotemporal window is estimated; Pixels of other frames within the spatiotemporal window are aligned to the target frame with the optimal stitching line based on optical flow mapping.

5. The method according to claim 1, It is characterized in that Calculating the distance of the pixel in the spatiotemporal window relative to the optimal stitching line of the frame in which the pixel is located, including: Select the point of the optimal seam line aligned by optical flow mapping within the preset spatial window; Perform linear fitting on the selected points based on the least square method to obtain the local fitting line of the optimal suture line; The vertical distance between the pixel point and the local fitting line is used to characterize the distance of the pixel point relative to the optimal stitching line of the frame in which it is located.

6. The method according to claim 1, It is characterized in that Based on the spatial distance, temporal distance, pixel distance between each pixel in the spatiotemporal window and the target pixel in the stitching area of ​​the target frame and the splicing reliability of each pixel in the spatiotemporal window, the weight of each pixel in the spatiotemporal window is obtained, including: The time distance, spatial distance, pixel distance and splicing reliability of the target pixel in the stitching area and the pixels in its spatiotemporal window are expressed as the weight factor of spatiotemporal filtering through an exponential function with a coefficient of -1; The standard deviation coefficient is used to adjust the weight factor to obtain the weight of each pixel in the space-time window.

7. The method according to claim 1, It is characterized in that Based on the weight of each pixel in the spatiotemporal window, the target pixels in the stitching area of ​​the target frame are optimized, including: Based on the weight of each pixel in the spatiotemporal window, a weighted average is performed to obtain the optimized pixel value of the target pixel in the stitching area.

8. A panoramic image optimization device based on spatiotemporal filtering, It is characterized in that include: An initial stitching module, used for stitching the original video to be processed based on the optimal stitching line stitching method to obtain an initial panoramic video; An optical flow alignment module, used to align the target frame with other frames in a preset spatiotemporal window based on an optical flow estimation algorithm; A splicing reliability calculation module, used to calculate the distance between the pixel in the spatiotemporal window and the optimal stitching line of the frame in which it is located, and obtain the splicing reliability of the pixel at the point based on the distance; A weight calculation module is used to obtain the weight of each pixel in the spatiotemporal window based on the spatial distance, temporal distance, pixel distance between each pixel in the spatiotemporal window and the target pixel in the stitching area of ​​the target frame and the splicing reliability of each pixel in the spatiotemporal window; The optimization module is used to optimize the target pixels in the stitching area of ​​the target frame based on the weights of each pixel in the spatiotemporal window.

9. An electronic device, It is characterized in that The system comprises a processor and a memory storing program instructions, wherein the processor is configured to execute the panoramic image optimization method based on spatiotemporal filtering as claimed in any one of claims 1 to 7 when executing the program instructions.

10. A computer readable medium, It is characterized in that Computer-readable instructions are stored thereon, and the computer-readable instructions are executed by a processor to implement a panoramic image optimization method based on spatiotemporal filtering as described in any one of claims 1 to 7.

Images