Video stitching method, apparatus, device, medium and program product

Abstract

The present disclosure relates to the technical field of image processing, and provides a video splicing method, device, equipment, medium and program product. The video splicing method comprises: acquiring source video frames; performing correction preprocessing on each source video frame to obtain a corrected video frame and a corrected video frame group; sequentially acquiring an overlapping area of a first corrected video frame and a second corrected video frame of each corrected video frame group; acquiring an optical flow of the overlapping area according to a deep learning model; sequentially acquiring a non-overlapping area of the first corrected video frame and the second corrected video frame of each corrected video frame group; completing the optical flow of the non-overlapping area to obtain an overall optical flow; sequentially mapping the first corrected video frame in each corrected video frame group according to the overall optical flow to obtain a mapped video frame; and splicing the first corrected video frame and the mapped video frame in each corrected video frame group to obtain a spliced video frame. The video splicing method can improve the image splicing quality.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and further relates to a video stitching method and device, equipment, medium and program product. BACKGROUND

[0002] Video stitching technology is one of the most important and rapidly developing technologies in modern digital image processing. With the development of video processing technologies such as face recognition and target tracking, a single field of view cannot meet the application requirements in a large-angle scene. Video stitching effectively expands the application range of video processing related technologies, and its widening of the field of view widely serves application fields such as video monitoring, intelligent transportation, and driving assistance.

[0003] The quality of image stitching directly affects the overall video stitching effect. Most current image stitching methods are based on matching feature points, and the stitching quality is not ideal. Especially in the face of weak texture, repeated texture, noise and other scenes, there are often problems such as few feature points or many interfering feature points, and it is difficult to obtain accurate matching. SUMMARY

[0004] Therefore, the purpose of the present disclosure is to provide a video stitching method, device, equipment, medium and program product to solve at least one of the above technical problems.

[0005] According to an aspect of the present disclosure, a video stitching method is provided, comprising:

[0006] Obtaining m source video frames at the same time point under different viewing angles;

[0007] Correcting and preprocessing each source video frame to obtain m corrected video frames, the m corrected video frames including m-1 different corrected video frame groups, and each corrected video frame group including a first corrected video frame and a second corrected video frame;

[0008] Obtaining the overlapping area of the first corrected video frame and the second corrected video frame of each corrected video frame group in sequence, to obtain m-1 overlapping areas;

[0009] Obtaining the optical flow of the overlapping area according to a deep learning model, to obtain m-1 overlapping area optical flows;

[0010] Obtaining the non-overlapping area of the first corrected video frame and the second corrected video frame of each corrected video frame group in sequence, to obtain m-1 non-overlapping areas;

[0011] Completing the optical flow of the non-overlapping area to obtain m-1 non-overlapping area optical flows, and further obtaining m-1 overall optical flows;

[0012] According to the overall optical flow, the first rectified video frame in each rectified video frame group is sequentially mapped to obtain m-1 mapped video frames;

[0013] The first rectified video frame in each rectified video frame group is spliced with the mapped video frame to obtain a spliced video frame.

[0014] According to an embodiment of the present disclosure, the optical flow of the overlapping region is obtained according to a deep learning model, which comprises:

[0015] The overlapping region is subjected to a grayscale processing;

[0016] The first rectified video frame and the second rectified video frame in the same rectified video frame group are merged to obtain a merged video frame;

[0017] The merged video frame is input into the deep learning model;

[0018] The optical flow of the overlapping region is obtained through the deep learning model.

[0019] According to an embodiment of the present disclosure, the backbone network of the deep learning model is set as a residual network.

[0020] According to an embodiment of the present disclosure, the optical flow of the non-overlapping region is obtained, which comprises:

[0021] The non-overlapping region is completed according to an adaptive optical flow completion algorithm along the edge of the overlapping region to obtain the optical flow of the non-overlapping region.

[0022] According to an embodiment of the present disclosure, the first rectified video frame is mapped according to the overall optical flow to obtain the mapped video frame, which comprises:

[0023] The overall optical flow is unfolded into a first direction displacement and a second direction displacement consistent with the size direction of the video frame;

[0024] The first rectified video frame is added with the first direction displacement and the second direction displacement according to the pixel precision to obtain the mapped video frame.

[0025] According to an embodiment of the present disclosure, the first rectified video frame in each rectified video frame group is spliced with the mapped video frame to obtain the spliced video frame, which comprises:

[0026] The best splicing path between the first rectified video frame and the mapped video frame is searched according to an energy minimum search strategy;

[0027] The first rectified video frame and the mapped video frame are spliced along the best splicing path.

[0028] According to an embodiment of the present disclosure, the video splicing method further comprises:

[0029] The first rectified video frame in each rectified video frame group and the mapped video frame are subjected to image fusion along the best splicing path.

[0030] According to an embodiment of the present disclosure, the image fusion comprises:

[0031] setting a fusion region along the optimal stitching path;

[0032] setting a weight fusion mask;

[0033] setting a linear weight of the weight fusion mask;

[0034] fusing the first rectified video frame and the mapped video frame in the fusion region according to the linear weight.

[0035] According to an embodiment of the present disclosure, the rectification preprocessing comprises distortion rectification and illumination adjustment.

[0036] According to an embodiment of the present disclosure, the distortion rectification adopts a chessboard calibration method, and the illumination adjustment adopts gamma transformation to make the image brightness of each rectified video frame consistent.

[0037] A second aspect of the present disclosure provides a video stitching device, comprising:

[0038] a source video frame acquisition module configured to acquire m source video frames at the same time point under different perspectives;

[0039] a rectification preprocessing module configured to perform rectification preprocessing on each source video frame to obtain m rectified video frames, the m rectified video frames comprising m-1 different rectified video frame groups, and each rectified video frame group comprising a first rectified video frame and a second rectified video frame;

[0040] an overlapping region acquisition module configured to sequentially acquire regions where the first rectified video frame and the second rectified video frame of each rectified video frame group overlap with each other to obtain m-1 overlapping regions;

[0041] an overlapping region optical flow acquisition module configured to acquire optical flow of the overlapping regions according to a deep learning model to obtain m-1 overlapping region optical flows;

[0042] a non-overlapping region acquisition module configured to sequentially acquire regions where the first rectified video frame and the second rectified video frame of each rectified video frame group do not overlap with each other to obtain m-1 non-overlapping regions;

[0043] a non-overlapping region optical flow completion module configured to complete the optical flow of the non-overlapping regions to obtain m-1 non-overlapping region optical flows, and further obtain m-1 overall optical flows;

[0044] a mapping module configured to sequentially map the first rectified video frame in each rectified video frame group according to the overall optical flow to obtain m-1 mapped video frames; and

[0045] The splicing module is configured to splice the first rectified video frame in each rectified video frame group and the mapped video frame to obtain a spliced video frame.

[0046] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the video splicing method described above.

[0047] A fourth aspect of the present disclosure further provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the video splicing method described above.

[0048] A fifth aspect of the present disclosure further provides a computer program product comprising a computer program that, when executed by a processor, implements the video splicing method described above. BRIEF DESCRIPTION OF DRAWINGS

[0049] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure, taken in conjunction with the accompanying drawings, in which:

[0050] Figure 1 An application scenario diagram of the video splicing method, device, equipment, medium and program product according to embodiments of the present disclosure is schematically shown;

[0051] Figure 2 A flowchart of the video splicing method according to embodiments of the present disclosure is schematically shown;

[0052] Figure 3 A flow of obtaining the optical flow of the overlapping area according to the deep learning model of operation S204 is schematically shown;

[0053] Figure 4 A flow of obtaining the optical flow of the non-overlapping area of operation S206 is schematically shown;

[0054] Figure 5 A flow of mapping the first rectified video frame to obtain the mapped video frame according to the overall optical flow of operation S207 is schematically shown;

[0055] Figure 6 A flow of splicing the first rectified video frame in each rectified video frame group and the mapped video frame to obtain a spliced video frame of operation S208 is schematically shown;

[0056] Figure 7 A flowchart of the video splicing method according to another embodiment of the present disclosure is schematically shown;

[0057] Figure 8 A flow of image fusion of operation S309 is schematically shown;

[0058] Figure 9a An example image of a first rectified video frame is schematically shown;

[0059] Figure 9b An example image of a second rectified video frame is schematically shown;

[0060] Figure 9c An example image of a first rectified video frame is schematically shown; Figure 9a overlapping with Figure 9b An example image of a first rectified video frame is schematically shown;

[0061] Figure 9d An example image of a first rectified video frame is schematically shown;

[0062] Figure 9e An example image of a first rectified video frame is schematically shown; Figure 9a overlapping with Figure 9b An example image of a first rectified video frame is schematically shown;

[0063] Figure 10 A structural block diagram of a video stitching device according to an embodiment of the present disclosure is schematically shown; and

[0064] Figure 11 A block diagram of an electronic device suitable for implementing the video stitching method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0065] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It is to be understood, however, that the description is merely exemplary of the present disclosure, and is not intended to limit the scope of the present disclosure. In the following detailed description of the embodiments of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it would be apparent to one skilled in the art that the present disclosure can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring aspects of the present disclosure.

[0066] The terms used herein are merely used to describe specific embodiments, and are not intended to limit the present disclosure. The terms "include" and "have" and the like used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0067] All terms used herein, including technical and scientific terms, have the same meanings as those generally understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings consistent with the context of the present description, and should not be interpreted in an idealized or excessively formal manner.

[0068] In the case of using expressions like "at least one of A, B, and C", it will be understood that such phrases are meant to encompass the selections of "A only", "B only", "C only", "A and B only", "A and C only", "B and C only", "A, B, and C", and so on, as well as any other combination of two or more of the alternatives.

[0069] Video stitching technology is one of the most important and rapidly developing technologies in modern digital image processing. With the development of video processing technologies such as face recognition and target tracking, a single field of view cannot meet the application requirements in a large-angle field of view. Video stitching effectively expands the application range of video processing related technologies, and its widening of the field of view widely serves application fields such as video monitoring, intelligent transportation, and driving assistance.

[0070] The quality of image stitching directly affects the overall video stitching effect. Most current image stitching methods extract the invariance features of two images through traditional SIFT, SURF, or ORB feature descriptors, match the corresponding feature points through corresponding feature matching algorithms such as Brute-force matcher or Flann-based matcher, and finally obtain the homography matrix according to the matching point pairs to map and align the images. However, the traditional feature point-based method has high requirements for the matching quality of feature points. In the face of weak texture, repeated texture, noise, and other scenes, there are often problems such as few feature points or many interference feature points, making it difficult to obtain accurate matching to calculate homography. Meanwhile, homography is generally a linear mapping for the overall plane, and cannot effectively correct nonlinear local distortion.

[0071] Embodiments of the present disclosure provide a video stitching method, comprising: obtaining m source video frames at the same time point under different angles of view; performing rectification preprocessing on each source video frame to obtain m rectified video frames, the m rectified video frames comprising m-1 different rectified video frame groups, and each rectified video frame group comprising a first rectified video frame and a second rectified video frame; sequentially obtaining the overlapping regions of the first rectified video frame and the second rectified video frame of each rectified video frame group to obtain m-1 overlapping regions; obtaining the optical flow of the overlapping regions according to a deep learning model to obtain m-1 overlapping region optical flows; sequentially obtaining the non-overlapping regions of the first rectified video frame and the second rectified video frame of each rectified video frame group to obtain m-1 non-overlapping regions; completing the optical flow of the non-overlapping regions to obtain m-1 non-overlapping region optical flows, and further obtaining m-1 overall optical flows; sequentially mapping the first rectified video frame in each rectified video frame group according to the overall optical flow to obtain m-1 mapped video frames; and stitching the first rectified video frame and the mapped video frame in each rectified video frame group to obtain a stitched video frame.

[0072] According to the technical scheme of the embodiment of the present disclosure, the optical flow of the overlapping area obtained through the deep learning model and the overall optical flow obtained further are taken as the pixel displacement amount, which is accurate to the pixel granularity, and compared with the matching mode of a small number of feature points in the prior art, the robust image feature extraction and matching can be realized. Meanwhile, the mapping video frame obtained through the optical flow mapping still has the robust splicing capability in the special application scene of weak texture, and can be applied to various application scenes, and has better splicing quality.

[0073] Figure 1 An application scenario diagram of a video splicing method according to an embodiment of the present disclosure is schematically shown.

[0074] As shown in Figure 1 the application scenario 100 according to the embodiment can include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a communication link medium between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0075] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0076] The terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc.

[0077] The server 105 can be a server providing various services, such as a background management server supporting the website browsed by the user using the terminal devices 101, 102, 103 (only as an example). The background management server can analyze and process the received user request data, etc., and feed back the processing result (such as a webpage, information or data obtained or generated according to the user request, etc.) to the terminal device.

[0078] It should be noted that the video stitching method provided by the embodiments of the present disclosure can generally be executed by the server 105. Accordingly, the video stitching apparatus provided by the embodiments of the present disclosure can generally be arranged in the server 105. The video stitching method provided by the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the terminal device 101, 102, 103 and / or the server 105. Accordingly, the video stitching apparatus provided by the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the terminal device 101, 102, 103 and / or the server 105.

[0079] It should be understood that Figure 1 The number of terminal devices, networks and servers in the above-mentioned scenario is only illustrative. Any number of terminal devices, networks and servers can be provided according to the implementation needs.

[0080] The video stitching method of the embodiments of the present disclosure will be described in detail below based on Figure 1 the scenario described above. Figures 2 to 9e The video stitching method of the embodiments of the present disclosure will be described in detail below based on the scenario described above.

[0081] Figure 2 A flowchart of the video stitching method according to the embodiments of the present disclosure is schematically shown.

[0082] As shown in Figure 2 , the video stitching method 200 of this embodiment includes operations S201-S208.

[0083] In operation S201, m source video frames at the same time point under different perspectives are obtained.

[0084] In operation S202, each source video frame is subjected to rectification preprocessing to obtain m rectified video frames, the m rectified video frames including m-1 different rectified video frame groups, a rectified video frame group including a first rectified video frame and a second rectified video frame.

[0085] In operation S203, the first rectified video frame and the second rectified video frame of each rectified video frame group are obtained in sequence, and the regions where the first rectified video frame and the second rectified video frame of each rectified video frame group overlap with each other are obtained, to obtain m-1 overlapping regions.

[0086] In operation S204, the optical flow of the overlapping regions is obtained according to a deep learning model, to obtain m-1 overlapping region optical flows.

[0087] In operation S205, the regions where the first rectified video frame and the second rectified video frame of each rectified video frame group do not overlap with each other are obtained in sequence, to obtain m-1 non-overlapping regions.

[0088] In operation S206, the optical flow of the non-overlapping area is completed to obtain m-1 non-overlapping area optical flows, and further obtain m-1 overall optical flows.

[0089] In operation S207, the first rectified video frame in each rectified video frame group is mapped according to the overall optical flow to obtain m-1 mapped video frames.

[0090] In operation S208, the first rectified video frame in each rectified video frame group and the mapped video frame are spliced to obtain a spliced video frame.

[0091] For video data, the number of images per second is the frame number. The "video splicing" in the embodiment of the present disclosure refers to the splicing of each video image under each view (at least two different views). The single view video in the prior art cannot meet the application scenarios such as video monitoring, intelligent transportation, driving assistance, etc. Therefore, the splicing method in the embodiment of the present disclosure is based on the characteristics that the images under different views have overlapping parts, and it should be understood that the overlapping parts of the images under different views can be used as a reference for splicing.

[0092] The splicing method 200 in the embodiment of the present disclosure is implemented by code execution. Specifically, the corresponding code and video image data can be input by using computer vision and machine learning platform tools such as openCV for splicing.

[0093] It should be understood that in the technical solution of the embodiment of the present disclosure, the video splicing is definitely based on the images of at least two video frames. Therefore, in operation S201, "m source video frames" are at least two source video frames.

[0094] In operation S202, "rectification preprocessing" refers to the preprocessing required to achieve and improve the subsequent splicing effect. The object of splicing is the source video frame. Each source video frame is an image obtained by an image acquisition device such as a camera. The image has various parameters such as brightness and pixels. Various deviations may occur in the image acquisition device itself and the image acquisition process, and further cause the image data of the source video frame to deviate. In order to ensure the subsequent splicing effect, distortion correction and illumination correction can be performed to obtain a rectified rectified video frame. The rectified video frame is at least uniform in illumination and has reduced distortion.

[0095] The video splicing method of the embodiments of the present disclosure can splice video frame images of multiple perspectives. The specific splicing principle is to sequentially group the video frame images two by two, and splice the two video frame images in each group. Therefore, in operation S202, the m source video frames are divided into m-1 rectified video frame groups, and each rectified video frame group includes two different rectified video frames, i.e., a first rectified video frame and a second rectified video frame. For example, under three perspectives, three rectified video frames I1, I2 and I3 can be obtained, and I1 and I2 form one rectified video frame group, and I2 and I3 form another rectified video frame group. The principle will be described below by taking two rectified video frames under two perspectives, i.e., one rectified video frame group as an example.

[0096] In operation S204, the "optical flow" refers to the instantaneous speed of the pixel motion of a spatial moving object on an observation imaging plane, and is also equal to the displacement of the target point when the time interval is very small. Taking two source video frames obtained under two perspectives as an example, the two source video frames are rectified into rectified video frame I1 and rectified video frame I2. The optical flow in the embodiments of the present disclosure refers to the displacement of each pixel point of I2 relative to I1. The image change information of the two rectified video frames I1 and I2 can be compared with pixel precision by the optical flow, and the robustness is higher.

[0097] In operation S207, "mapping" refers to applying the generated overall optical flow to an image, i.e., obtaining an image corresponding to the overall optical flow, and the image is a mapping video frame.

[0098] According to the technical scheme of the embodiments of the present disclosure, the optical flow of the overlapping region obtained by the deep learning model and the further obtained overall optical flow are used as the pixel displacement amount, which is accurate to the pixel granularity. Compared with the matching mode of a small number of feature points in the prior art, the robust image feature extraction and matching can be realized. Meanwhile, the mapping video frame obtained by the optical flow mapping still has robust splicing capability in special application scenarios such as weak texture, and can be applied to various application scenarios, and has better splicing quality.

[0099] In combination with Figure 2 , Figure 3 , the specific process of operation S204 of obtaining the optical flow of the overlapping region according to the deep learning model includes operations S41-S44.

[0100] In operation S41, the overlapping region is subjected to grayscale processing.

[0101] In operation S42, the first rectified video frame and the second rectified video frame of the same rectified video frame group are merged to obtain a merged video frame.

[0102] In operation S43, the merged video frame is input into the deep learning model.

[0103] In operation S44, optical flow in the overlapping region is obtained through a deep learning model.

[0104] In the technical solution of this embodiment, in operation S41, "grayscale processing" refers to the process of converting the overlapping area of ​​a generally color image into a grayscale image. The color of each pixel in a color image is determined by three components: R, G, and B, and each component has 256 values ​​ranging from 0 to 255. A grayscale image is a special type of color image where the R, G, and B components are identical. "Merging the first and second corrected video frames of the same corrected video frame group" refers to merging the first and second corrected video frames of the same corrected video frame group at the channel layer to obtain a merged video frame. This merged video frame serves as input to a deep learning model, and the optical flow of the overlapping area can be obtained through processing by the deep learning model.

[0105] It should be understood that during the initial training of this deep learning model, a large number of video frames merged at the channel layer were used as input data for training, and the final output data obtained was a deep learning model with overlapping region optical flow that met the error target.

[0106] According to the technical solution of the present disclosure, the optical flow of the overlapping area can be obtained automatically, accurately and quickly through a deep learning model, which reduces the time of stitching single video frames and meets the requirements of real-time performance. Specifically, this is achieved through an algorithm for calculating optical flow.

[0107] For example, the backbone network of a deep learning model can be configured as a residual network. A residual network is a type of convolutional neural network, characterized by ease of optimization and the ability to improve accuracy by significantly increasing its depth. Its internal residual blocks utilize skip connections to mitigate the vanishing gradient problem caused by increasing depth in deep neural networks. In this embodiment, video frame images are stitched together at the pixel level. Feature extraction and matching in the deep learning model are both numerous and complex. Increasing the depth of the deep learning model facilitates accurate feature extraction and matching, but it also leads to increased bias. The residual network, through its internal residual blocks and skip connections, avoids vanishing gradients and increased bias, ensuring good stitching results.

[0108] Combination Figure 2 , Figure 4 The acquisition of optical flow in non-overlapping regions by operation S206 may include operation S61.

[0109] In operation S61, the non-overlapping region is completed along the edge of the overlapping region using an adaptive optical flow completion algorithm to obtain the optical flow of the non-overlapping region.

[0110] Specifically, assuming the object to be mapped is I1, and the overlapping region is located on the right half of I1, the specific operation to complete the non-overlapping region is as follows: starting from the upper left boundary of the overlapping region, a sliding window of size k*k is set. Each pixel P(i, j) in the non-overlapping region is taken as the lower left corner of the sliding window, and the mean completion strategy is used to gradually complete the region in the order from right to left and from top to bottom. That is, the mean value of the existing optical flow in the sliding window is used to smoothly fill the unknown optical flow.

[0111]

[0112] Where P(i,j) is the optical flow of the non-overlapping region to be completed, and P(m,n) is the optical flow of the neighborhood of P(i,j) within the sliding window. After completion using this strategy, the optical flow of the non-overlapping region with a more natural and smooth connection between the overlapping regions can be estimated, and finally the overall optical flow F for the current first corrected video frame I1 and the second corrected video frame I2 can be obtained.

[0113] In the technical solution of this disclosure embodiment, the optical flow in the non-overlapping region is calculated by averaging the optical flow in the overlapping region with reference to the optical flow in the overlapping region. This results in smoother and more seamless transitions between video frames, leading to better stitching effects.

[0114] Combination Figure 2 , Figure 5 As shown, according to an embodiment of this disclosure, operation S207 maps the first corrected video frame according to the overall optical flow to obtain the mapped video frame, including operations S71 to S72.

[0115] In operation S71, the overall optical flow is unfolded into a first direction displacement and a second direction displacement that are consistent with the size direction of the video frame.

[0116] In operation S72, the first corrected video frame is summed with the first direction displacement and the second direction displacement according to the pixel precision to obtain the mapped video frame.

[0117] It should be understood that optical flow is a displacement quantity, and displacement is a physical quantity with magnitude and direction. Therefore, the optical flow of each pixel can be decomposed and unfolded along two perpendicular directions of the plane, ultimately yielding a two-dimensional array corresponding to the size of a video frame. Specifically:

[0118] We can first define the initial coordinates (C) of I1 of the first corrected video frame. i C j The overall optical flow F is expanded into displacements F in the x and y directions, respectively. x With F v :

[0119]

[0120]

[0121] Where w and h are the width and height of the current frame, respectively. Then the corresponding coordinates (C') of the mapped video frame I' are... i C' j )for:

[0122] (C i ′, C j ′)=(C i +Δx ij C j ′+Δy ij )

[0123] Combination Figure 2 , Figure 6 As shown, according to an embodiment of this disclosure, operation S208 splicing the first corrected video frame and the mapped video frame in each corrected video frame group to obtain a spliced ​​video frame includes operations S81 to S82.

[0124] During operation S81, the optimal splicing path between the first corrected video frame and the mapped video frame is found according to the minimum energy search strategy.

[0125] In operation S82, the first corrected video frame and the mapped video frame are spliced ​​along the optimal splicing path.

[0126] In the technical solution of this disclosure embodiment, the "energy" of operation S81 refers to the sum of the squares of the differences between the image pixels on both sides of the stitching region (derived from the overlapping area and overlap rate of the image), or the squares of the pixel differences for a grayscale image; for example, R, G, and B represent the pixel values ​​of the three channels at a certain point in the stitching region, and ΔR, ΔG, and ΔB represent the difference between the image pixels on one side and the image pixels on the other side. The value of "energy" is E = ΔR*ΔR + ΔG*ΔG + ΔB*ΔB. Specifically, the total energy E can be obtained by calculating the difference between I' and I1 in the stitching region. The initial stitching path L is composed of the index of the column where the minimum energy value of each row is located.

[0127]

[0128] Where i and j are the indices of the stitching region, respectively. Then, the path is iteratively smoothed in each round:

[0129]

[0130] After multiple iterations, a relatively smooth and fluid stitching path was finally obtained, which is the optimal stitching path.

[0131] like Figure 7 As shown, according to another embodiment of this disclosure, the video stitching method 300 may further include operation S309.

[0132] In operation S309, the first rectified video frame in each rectified video frame group and the mapped video frame are image fused along the optimal stitching path.

[0133] It should be understood that the two video frame images stitched are the first rectified video frame I1 and the mapped video frame I' mapped by the overall optical flow between the two rectified video frames, and the two have formed a complete stitched image after stitching. The image stitched along the optimal stitching path may have relatively large pixel value jumps at the optimal stitching path and on both sides, and therefore, to obtain a better stitching effect, the video stitching method 300 of the embodiments of the present disclosure can make the image stitched along the optimal stitching path visually present a natural fade-in and fade-out fusion effect at the optimal stitching path and on both sides through image fusion.

[0134] In combination with FIGS. 9 to 10, Figure 7 Figure 8 As shown in FIGS. 9 to 10, according to the embodiments of the present disclosure, the image fusion of operation S309 can include operations S91 to S94.

[0135] In operation S91, a fusion region is set along the optimal stitching path.

[0136] In operation S92, a weight fusion mask is set.

[0137] In operation S93, a linear weight of the weight fusion mask is set.

[0138] In operation S94, the first rectified video frame and the mapped video frame in the fusion region are fused according to the linear weight.

[0139] According to the technical solution of the embodiments of the present disclosure, a selected image, figure or object is used to shield the processed image (all or part) to control the region or process of image processing, and the specific image or object used for covering is a mask. The mask can be used for: extracting a region of interest, that is, multiplying a pre-prepared region of interest mask with a to-be-processed image to obtain an image of interest, and the image values in the image of interest remain unchanged, and the image values outside the region are all 0; the mask is also used for shielding, that is, shielding some regions of the image with the mask so that they do not participate in processing or calculation of processing parameters, or only the shielded regions are processed and counted.

[0140] ​In the embodiments of the present disclosure, a mask in the form of a two-dimensional matrix array is provided, which is equivalent to being overlaid on the first rectified video frame I1. Firstly, the mask is used to shield the area in the first rectified video frame I1 that does not need to be fused, and the value of the mask corresponding to each pixel in the area is 0. Secondly, a linear weight is set in the fusion area of the first rectified video frame I1 and the mapping video frame I', and the value of each pixel in the fusion area of the first rectified video frame I1 is added to the product of the weight value of each pixel in the fusion area of the mapping video frame I' and the value of the corresponding pixel when calculating the pixel value of the area during fusion. Finally, the value of the mask corresponding to each pixel in the area of the mapping video frame I' that does not need to be fused is 1, which is equivalent to extracting the area of the mapping video frame I' to achieve image fusion. The specific process is as follows:

[0141] Firstly, a mask M of the same size is initialized in the fusion area (p, q), and a fusion interval b is set (the area formed in the fusion interval is the fusion area described above). Then, a linear weight is set on both sides of the optimal stitching path:

[0142]

[0143] It should be understood that the fusion area refers to an area determined along the optimal stitching path, and the contour of the area is consistent with the trend of the optimal stitching path.

[0144] According to the technical scheme of the embodiments of the present disclosure, the image fusion through the weight fusion mask can be based on two original images (the first rectified video frame I1 and the mapping video frame I') to perform fusion with pixel precision and obtain a good image fusion effect.

[0145] According to the embodiments of the present disclosure, the rectification preprocessing can include distortion correction and illumination adjustment.

[0146] According to the embodiments of the present disclosure, the distortion correction adopts a chessboard calibration method, and the illumination adjustment adopts gamma transformation to make the image brightness of each rectified video frame consistent.

[0147] Due to factors such as inaccuracy of optical instruments in the production process, the image acquisition device such as a camera is not standard, and therefore the accuracy of camera imaging will have errors, i.e., distortion occurs. The process of rectifying the deviation needs to calibrate the distortion parameters, intrinsic matrix and extrinsic matrix of the camera.

[0148] The chessboard calibration method uses multi-view shooting of a chessboard, uses two-dimensional images instead of three-dimensional images for calibration, greatly reduces the calculation amount of calibration, and the obvious characteristics of the chessboard corner points greatly enhance the accuracy of the calibration results.

[0149] According to the chessboard calibration method, the pixel point coordinates before and after distortion correction exist the following relationship: wherein the distortion correction adopts the chessboard calibration method, and for a two-dimensional plane, the pixel point coordinates before and after correction exist the following relationship:

[0150]

[0151]

[0152] wherein r 2 =x 2 +y 2 , k1, k2, k3, k4, k5, and k6 are radial distortion, p1 and p2 are axial distortion, and the two-dimensional point (u, v) after final imaging is represented as: u = f x *x’ + c x , and v = f y *y’ + c y , wherein (c x , c y ) is the principal point, f x and f y are focal lengths. Thus, the related parameters and distortion matrix of the camera and other image acquisition devices can be obtained according to the chessboard corner point coordinates, and the original image can be corrected for distortion.

[0153] In the technical scheme of the embodiments of the present disclosure, the light adjustment process is: first, the brightness average values L1 and L2 of the overlapping areas of the two images are calculated, respectively, the brightness ratio is determined as the gamma correction coefficient γ based on one of the images as a reference, and the image brightness is made consistent by using gamma transformation.

[0154] In the following, the video stitching method 300 of the embodiments of the present disclosure will be described with reference to the examples. Figures 9a to 9e

[0155] Figure 9a and Figure 9b are different view corrected preprocessed source video frames, that is, the first corrected video frame I1 and the second corrected video frame I2, forming a corrected video frame group, and the overlapping areas of the two are as shown in Figure 9c After obtaining the optical flow of the overlapping area by using the deep learning model and the optical flow of the non-overlapping area and the overall optical flow by using the completion algorithm, the Figure 9a ​The first corrected video frame I1 is mapped to obtain a mapped video frame, resulting in two stitched images: the first corrected video frame I1 and the mapped video frame. Their overlapping areas coincide. The optimal stitching path is found using a minimum energy search strategy, and the images are stitched together according to this optimal path. After stitching, the images are processed using... Figure 9d The weighted fusion mask shown fuses the image of the first corrected video frame I1 with the image of the mapped video frame, ultimately obtaining the image shown. Figure 9e The stitched image shown. It should be noted that... Figure 9d Shown in image form Figure 9a and Figure 9b The weighted fusion mask in the figure shows that the white area represents the area with a value of 0, which is the area of ​​the first corrected video frame I1 that does not need to be fused, and the black area represents the area with a value of 1, which is the area of ​​the corrected video frame that does not need to be fused. The area between the two is the fusion area, and its value is the weight between 0 and 1.

[0156] Figure 10 A schematic block diagram of a video splicing apparatus according to an embodiment of the present disclosure is shown.

[0157] like Figure 10 As shown, the video stitching device 400 of this embodiment includes a source video frame acquisition module 410, a correction preprocessing module 420, an overlapping region acquisition module 430, an overlapping region optical flow acquisition module 440, a non-overlapping region acquisition module 450, a non-overlapping region optical flow completion module 460, a mapping module 470, and a stitching module 480.

[0158] The source video frame acquisition module 410 is used to acquire m source video frames at the same time point from different viewpoints. In one embodiment, the source video frame acquisition module 410 can be used to perform the operation S201 described above, which will not be repeated here.

[0159] The correction preprocessing module 420 is used to perform correction preprocessing on each source video frame to obtain m corrected video frames. The m corrected video frames include m-1 different groups of corrected video frames, and each group of corrected video frames includes a first corrected video frame and a second corrected video frame. In one embodiment, the correction preprocessing module 420 can be used to perform the operation S202 described above, which will not be repeated here.

[0160] The overlapping region acquisition module 430 is used to sequentially acquire the overlapping regions between the first and second corrected video frames of each corrected video frame group, obtaining m-1 overlapping regions. In one embodiment, the overlapping region acquisition module 430 can be used to perform the operation S203 described above, which will not be repeated here.

[0161] The overlapping area optical flow obtaining module 440 is configured to obtain the optical flow of the overlapping area according to the deep learning model, to obtain m-1 optical flows of the overlapping area. In an embodiment, the overlapping area optical flow obtaining module 440 can be configured to perform the operation S204 described above, and details are not repeated here.

[0162] The non-overlapping area obtaining module 450 is configured to sequentially obtain the area of the first rectified video frame and the second rectified video frame of each rectified video frame group that does not overlap with each other, to obtain m-1 non-overlapping areas. In an embodiment, the non-overlapping area obtaining module 450 can be configured to perform the operation S205 described above, and details are not repeated here.

[0163] The non-overlapping area optical flow completing module 460 is configured to complete the optical flow of the non-overlapping area, to obtain m-1 optical flows of the non-overlapping area, and to further obtain m-1 overall optical flows. In an embodiment, the non-overlapping area optical flow completing module 460 can be configured to perform the operation S206 described above, and details are not repeated here.

[0164] The mapping module 470 is configured to sequentially map the first rectified video frame in each rectified video frame group according to the overall optical flow, to obtain m-1 mapped video frames. In an embodiment, the mapping module 470 can be configured to perform the operation S207 described above, and details are not repeated here.

[0165] The splicing module 480 is configured to splice the first rectified video frame and the mapped video frame in each rectified video frame group, to obtain a spliced video frame. In an embodiment, the splicing module 480 can be configured to perform the operation S208 described above, and details are not repeated here.

[0166] According to an embodiment of the present disclosure, any of the source video frame obtaining module 410, the rectification preprocessing module 420, the overlapping region obtaining module 430, the overlapping region optical flow obtaining module 440, the non-overlapping region obtaining module 450, the non-overlapping region optical flow complementing module 460, the mapping module 470 and the stitching module 480 can be combined in one module, or any of the modules can be split into multiple modules. Alternatively, at least part of the function of one or more of the modules can be combined with at least part of the function of other modules, and implemented in one module. According to an embodiment of the present disclosure, at least one of the source video frame obtaining module 410, the rectification preprocessing module 420, the overlapping region obtaining module 430, the overlapping region optical flow obtaining module 440, the non-overlapping region obtaining module 450, the non-overlapping region optical flow complementing module 460, the mapping module 470 and the stitching module 480 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable manner of integrating or packaging a circuit, etc. or implemented by hardware or firmware, or implemented in any one of software, hardware and firmware or in a proper combination of any of them. Alternatively, at least one of the source video frame obtaining module 410, the rectification preprocessing module 420, the overlapping region obtaining module 430, the overlapping region optical flow obtaining module 440, the non-overlapping region obtaining module 450, the non-overlapping region optical flow complementing module 460, the mapping module 470 and the stitching module 480 can be at least partially implemented as a computer program module which can perform corresponding functions when the computer program module is run.

[0167] Figure 11 A block diagram of an electronic device suitable for implementing the video stitching method according to an embodiment of the present disclosure is schematically shown.

[0168] As shown in Figure 11 The electronic device 500 according to an embodiment of the present disclosure includes a processor 501 which can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 502 or loaded from a storage portion 508 into a random access memory (RAM) 503. The processor 501 can include, for example, a general purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special purpose microprocessor (such as an application specific integrated circuit (ASIC)), etc. The processor 501 can also include an on-board memory for cache use. The processor 501 can include a single processing unit or multiple processing units for performing different actions of the method processes according to an embodiment of the present disclosure.

[0169] In the RAM 503, various programs and data required for the operation of the electronic device 500 are stored. The processor 501, the ROM 502, and the RAM 503 are connected to each other via the bus 504. The processor 501 performs various operations of the method flow according to the embodiments of the present disclosure by executing the programs in the ROM 502 and / or the RAM 503. It should be noted that the programs can also be stored in one or more memories other than the ROM 502 and the RAM 503. The processor 501 can also perform various operations of the method flow according to the embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0170] According to an embodiment of the present disclosure, the electronic device 500 can further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 can further include one or more of the following components connected to the I / O interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as necessary. A removable medium 511 such as a magnetic disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 510 as necessary, so that a computer program read out therefrom is installed in the storage portion 508 as necessary.

[0171] The present disclosure also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments; or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, when the one or more programs are executed, the method according to the embodiments of the present disclosure is implemented.

[0172] According to an embodiment of the present disclosure, the computer readable storage medium can be a nonvolatile computer readable storage medium, for example, can include but not limited to: a portable computer diskette, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), a portable compact disc read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, the computer readable storage medium can include one or more memories of the ROM 502 and / or the RAM 503 described above and / or in addition to the ROM 502 and the RAM 503.

[0173] Embodiments of the present disclosure also include a computer program product, which includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the video stitching method provided by the embodiments of the present disclosure.

[0174] The above functions defined in the system / apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 501. According to an embodiment of the present disclosure, the system, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0175] In one embodiment, the computer program can rely on tangible storage media such as optical storage media, magnetic storage media, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of signals on network media. The computer program containing program codes can be transmitted by any appropriate network media, including but not limited to wireless, wired, etc., or any suitable combination of the foregoing.

[0176] In such an embodiment, the computer program can be downloaded and installed from the network by the communication part 509 and / or installed from the detachable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiments of the present disclosure are performed. According to an embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0177] According to embodiments of the present disclosure, program code of the computer program for performing the methods provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages, and specifically, can be implemented using high-level procedural and / or object-oriented programming language, and / or assembly / machine language. Programming languages include, but are not limited to, Java, C++, python, “C” language, or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, and partly on a remote computing device, or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0178] The computer program product of the present disclosure can be a computer program product, which is a machine-readable medium (or computer readable medium) having stored therein a sequence of instructions readable, by one or more processors of a computer system, the instructions being executable by the one or more processors to cause the computer system to execute the method of the present disclosure. The instructions can be software instructions stored in memory (e.g., memory 120 of the computer system) and implemented as software programs to perform the method of the present disclosure. The computer program product can be propagated to and executed by one or more computer systems and / or apparatuses by way of one or more computer readable media.

[0179] Those skilled in the art will understand that features of the various embodiments and / or claims of the present disclosure can be combined or / and integrated with one another, even though such combinations or integrations are not expressly disclosed in the present disclosure. In particular, features of the various embodiments and / or claims of the present disclosure can be combined and / or integrated with one another, without departing from the spirit and teachings of the present disclosure. All such combinations and / or integrations are within the scope of the present disclosure.

[0180] The above describes embodiments of the present disclosure. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present disclosure. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present disclosure, and these substitutions and modifications should all fall within the scope of the present disclosure.

Claims

1. A video stitching method, characterized in that, include: Acquire m source video frames at the same time point from different perspectives; Each of the source video frames is subjected to correction preprocessing to obtain m corrected video frames. The m corrected video frames include m-1 different corrected video frame groups, and each corrected video frame group includes a first corrected video frame and a second corrected video frame. The overlapping areas of the first and second corrected video frames of each corrected video frame group are obtained sequentially to obtain m-1 overlapping areas. The optical flow of the overlapping region is obtained based on the deep learning model, resulting in m-1 optical flows of the overlapping region. The regions where the first and second corrected video frames of each corrected video frame group do not overlap are sequentially obtained to obtain m-1 non-overlapping regions. Complete the optical flow of the non-overlapping regions to obtain m-1 non-overlapping region optical flows, and then obtain m-1 overall optical flows; Based on the overall optical flow, the first corrected video frame in each corrected video frame group is mapped sequentially to obtain m-1 mapped video frames. By splicing the first corrected video frame in each of the corrected video frame groups with the mapped video frame, a spliced ​​video frame is obtained.

2. The video stitching method according to claim 1, characterized in that, The step of obtaining the optical flow of the overlapping region based on the deep learning model includes: The overlapping areas are then converted to grayscale. The first and second corrected video frames in the same corrected video frame group are merged to obtain a merged video frame. The merged video frames are input into the deep learning model; The optical flow in the overlapping region is obtained through the deep learning model.

3. The video stitching method according to claim 2, characterized in that, The backbone network of the deep learning model is configured as a residual network.

4. The video stitching method according to claim 1, characterized in that, Obtaining optical flow in non-overlapping regions includes: The non-overlapping region is completed along the edge of the overlapping region using an adaptive optical flow completion algorithm to obtain the optical flow of the non-overlapping region.

5. The video stitching method according to claim 1, characterized in that, The first corrected video frame is mapped based on the overall optical flow to obtain the mapped video frames, including: The overall optical flow is unfolded into a first directional displacement and a second directional displacement that are consistent with the size direction of the video frame; The mapped video frame is obtained by summing the first corrected video frame with the first directional displacement and the second directional displacement according to the pixel precision.

6. The video stitching method according to claim 1, characterized in that, The step of stitching together the first corrected video frame in each corrected video frame group with the mapped video frame to obtain the stitched video frame includes: The optimal splicing path between the first corrected video frame and the mapped video frame is found according to the minimum energy search strategy; The first corrected video frame and the mapped video frame are spliced ​​together along the optimal splicing path.

7. The video stitching method according to claim 6, characterized in that, Also includes: The first corrected video frame and the mapped video frame in each of the corrected video frame groups are image fused along the optimal stitching path.

8. The video stitching method according to claim 7, characterized in that, The image fusion includes: The fusion region is set along the optimal splicing path; Set a weighted fusion mask; Set the linear weights of the weighted fusion mask; The first corrected video frame and the mapped video frame within the fusion region are fused according to linear weights.

9. The video stitching method according to claim 1, characterized in that, The correction preprocessing includes distortion correction and illumination adjustment.

10. The video stitching method according to claim 9, characterized in that, The distortion correction uses a checkerboard calibration method, and the illumination adjustment uses gamma transformation to ensure that the image brightness of each corrected video frame is consistent.

11. A video splicing device, comprising: The source video frame acquisition module is used to acquire m source video frames at the same time point from different perspectives; The correction preprocessing module is used to perform correction preprocessing on each of the source video frames to obtain m corrected video frames. The m corrected video frames include m-1 different corrected video frame groups, and the corrected video frame groups include a first corrected video frame and a second corrected video frame. The overlapping region acquisition module is used to sequentially acquire the overlapping regions of the first and second corrected video frames of each corrected video frame group, and obtain m-1 overlapping regions. The overlapping region optical flow acquisition module is used to acquire the optical flow of the overlapping region according to the deep learning model, and obtain m-1 overlapping region optical flows; The non-overlapping region acquisition module is used to sequentially acquire the regions where the first and second corrected video frames of each corrected video frame group do not overlap, resulting in m-1 non-overlapping regions. The non-overlapping region optical flow completion module is used to complete the optical flow of the non-overlapping region to obtain m-1 non-overlapping region optical flows, and then obtain m-1 overall optical flows; The mapping module is used to sequentially map the first corrected video frame in each of the corrected video frame groups according to the overall optical flow, to obtain m-1 mapped video frames; and The splicing module is used to splice the first corrected video frame and the mapped video frame in each of the corrected video frame groups to obtain a spliced ​​video frame.

12. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images