Multi-camera video frame automatic alignment method

Abstract

The invention discloses a multi-camera video frame automatic alignment method, which belongs to the technical field of computer vision, and comprises the following steps: acquiring video frames shot by each camera, and measuring the similarity among the video frames captured by different cameras based on a preset scoring function, finding a standard frame in the video frames as an alignment reference frame; constructing a time stream model to model the change of the video frame along with time; estimating a time offset of each camera based on a time offset theorem by using a time flow model; and adjusting the video frame of each camera according to the time offset so that the video frame is aligned with the standard frame. According to the method, the accuracy and efficiency of video three-dimensional reconstruction are remarkably improved. The method can adapt to changes of different visual angles, dynamic scenes and illumination conditions, and realizes rapid and accurate video frame alignment.

Description

Technical Field

[0001] The invention relates to the technical field of computer vision, and in particular to a method for automatically aligning video frames of multiple cameras. Background Art

[0002] With the rapid development of information technology, multi-camera systems have been widely used in various fields. By shooting the same scene from different angles, multi-camera systems can provide more comprehensive and richer visual information and are widely used in monitoring systems, autonomous driving, sports event broadcasting and many other fields.

[0003] Video 4D reconstruction refers to the restoration of 3D scenes and their dynamic information over time from video sequences shot by multiple cameras. Unlike image 3D reconstruction, video 4D reconstruction not only needs to restore the geometric structure of the scene, but also needs to capture the dynamic changes of the scene. This technology has a wide range of applications in virtual reality, movie special effects, motion analysis and other fields. However, a major challenge facing multi-camera video 4D reconstruction is the time synchronization problem of video frames. Since the shooting time of different cameras is not synchronized, the time misalignment between video frames seriously affects the accuracy and effect of reconstruction.

[0004] In existing multi-camera systems, the time synchronization problem of videos is mainly solved by the following methods: (1) Using dedicated hardware devices, such as synchronization triggers, to ensure that all cameras start shooting at the same time. Although this method can provide high synchronization accuracy, it is expensive and has poor flexibility; (2) Using simple software algorithms to time-correct the captured video frames. Common methods include synchronization based on timestamps and synchronization based on image content. However, due to the difference in viewing angles of different cameras, it may increase the difficulty of aligning video frames. In particular, in the case of large viewing angle differences, the accuracy and reliability of feature point matching will be affected. In dynamic scenes, the movement and changes of objects will also affect the detection and matching of feature points, increasing the complexity of alignment. For example, in crowded scenes, moving pedestrians and vehicles will cause significant differences between videos. In addition, the different lighting conditions of different cameras will cause differences in brightness and contrast of video frames, affecting the alignment effect.

[0005] In summary, the existing multi-camera video frame alignment technology has the following shortcomings: (1) Perspective difference: The perspective difference of different cameras may make the alignment between video frames more difficult; (2) Dynamic scenes: In dynamic scenes, the movement and change of objects will also affect the detection and matching of feature points, increasing the complexity of alignment; (3) Lighting conditions: The different lighting conditions of different cameras will lead to differences in brightness and contrast of video frames, affecting the alignment effect; (3) Computational complexity: The existing alignment methods usually have high computational complexity and are difficult to achieve real-time processing, which limits their performance in real-time applications; (4) Lack of robustness: The existing alignment methods have poor robustness when facing complex environments and dynamic scenes and are easily affected by external interference. Summary of the invention

[0006] The present invention provides a method for automatic alignment of multi-camera video frames to solve the technical problems of low alignment accuracy, poor robustness and high computational complexity in the prior art when dealing with the problems of different camera viewing angles, dynamic scenes, lighting changes and the like.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] On the one hand, the present invention provides a method for automatically aligning video frames of multiple cameras, comprising:

[0009] Obtain the video frames captured by each camera, and measure the similarity between the video frames captured by different cameras based on a preset scoring function, so as to find a standard frame in the video frames as an alignment reference frame;

[0010] Construct a temporal flow model to model the changes of video frames over time; the changes of video frames over time include changes of image content over time and changes of image quality over time; changes of image content over time include changes of object moving speed and direction; changes of image quality over time include changes of light intensity and blur degree;

[0011] Using the time flow model and based on the time offset theorem, the time offset of each camera is estimated;

[0012] The video frames of each camera are adjusted to align with the standard frame according to the time offset.

[0013] Furthermore, the scoring function measures the similarity between video frames captured by different cameras through the matching degree of feature points, color histogram similarity and depth information consistency between video frames captured by different cameras.

[0014] Furthermore, the expression of the scoring function is:

[0015]

[0016] in, Represents the video frame captured by the jth camera at time t1 and the video frame captured by the kth camera at time t2 The similarity between Indicates the matching score of feature points; represents the color histogram similarity score; represents the depth information consistency score; w1, w2 and w3 are preset weight parameters, and the value of each weight parameter is adjusted according to the actual application scenario.

[0017] Further, Among them, N matches express and The number of matching point pairs between feature points; N total express and The total number of feature points detected in ;

[0018] The value of The color histogram of The similarity between the color histograms of

[0019] The value of The depth map and The mean absolute error between the depth maps.

[0020] Furthermore, the method of measuring the similarity between video frames captured by different cameras based on a preset scoring function to find a standard frame in the video frames as an alignment reference frame includes:

[0021] Select the first n frames in a video frame captured by a camera as candidate standard frames; wherein n is a preset value;

[0022] Calculate the scoring function value between each video frame captured by the current camera and each candidate standard frame;

[0023] The candidate standard frame corresponding to the largest score function value is selected as the final standard frame.

[0024] Furthermore, the expression of the time flow model is:

[0025] I(t)=I c (t)+I q (t)

[0026] Where I(t) represents the video frame data at time t; I cI(t) represents the part of the image content that changes over time; I(t) q (t) represents the part of the image quality of I(t) that changes over time.

[0027] Furthermore, I c (t) is a multi-order polynomial, expressed as:

[0028]

[0029] Where N is the polynomial order; a n are the coefficients of the polynomial;

[0030] I q (t) is a Fourier series, expressed as:

[0031]

[0032] Where L is the number of Fourier series terms; b l is the Fourier coefficient; ω l is the frequency; φ l It's the phase.

[0033] Furthermore, the estimating the time offset of each camera by using the time flow model based on the time offset theorem includes:

[0034] Set the polynomial order N and the number of Fourier series terms L;

[0035] Collect time series data of each camera around the standard frame;

[0036] Using the collected time series data, the least squares method is used to fit I c (t) and I q (t) expression;

[0037] Based on the fitting results, the time offset of each camera is estimated.

[0038] On the other hand, the present invention further provides an electronic device, comprising a processor and a memory; wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the above method.

[0039] In yet another aspect, the present invention further provides a computer-readable storage medium, wherein at least one instruction is stored in the storage medium, and the instruction is loaded and executed by a processor to implement the above method.

[0040] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0041] The technical solution provided by the present invention significantly improves the accuracy and efficiency of the multi-camera system in 3D reconstruction by introducing efficient image processing and computer vision algorithms, combined with deep learning technology. It can adapt to changes in different viewing angles, dynamic scenes and lighting conditions, and achieve fast and accurate video frame alignment; it improves the accuracy and reliability of multi-camera video frame alignment, especially in dynamic scenes with large viewing angle differences; it improves the robustness of the system in scenes with changing lighting, and can effectively handle differences in lighting conditions of different cameras; and it reduces computational complexity, improves processing efficiency, and can achieve real-time alignment of multi-camera video frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 Schematic diagram of the principle of the multi-camera video frame automatic alignment method provided by an embodiment of the present invention;

[0044] Figure 2 Schematic diagram of the execution flow of the multi-camera video frame automatic alignment method provided by an embodiment of the present invention;

[0045] Figure 3 It is a system block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0047] First of all, it should be noted that in the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplarily" is intended to present the concept in a concrete way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0048] First embodiment

[0049] This embodiment provides a method for automatically aligning video frames of multiple cameras, and its implementation principle is as follows: Figure 1 As shown, the method can be implemented by an electronic device, which can be a terminal or a server.

[0050] Specifically, the execution process of this method is as follows Figure 2 As shown, the following steps are included:

[0051] S1, obtaining video frames captured by each camera, and based on a preset scoring function, measuring the similarity between the video frames captured by different cameras, so as to find a standard frame in the video frames as an alignment reference frame;

[0052] It should be noted that, since the starting frames taken by each camera are not synchronized, and there are sampling noise or frame errors, such as Figure 1 As shown, The maximum difference is n frames. The input of each camera is sent to the time alignment module. A scoring function is set here to measure the distance between the pictures and find the relationship between the two pictures. The scoring function is constructed based on multiple features such as feature point matching, color histogram, depth information, etc. Through the scoring function, you can search in the first n frames and find a standard frame as a reference point, so that the frames before the standard frame are discarded at the least, and the frames after the standard frame are compensated at the least.

[0053] Specifically, the implementation process of the above S1 is as follows:

[0054] S11, Design of scoring function

[0055] In order to overcome the problems caused by asynchronous input and sampling noise, this embodiment designs a scoring function This function is used to measure the video frames captured by different cameras j and k at a certain time t. With reference frame The scoring function comprehensively considers the characteristics of feature point matching, color histogram similarity and depth information consistency, as follows:

[0056] Feature point matching: Use the Scale Invariant Feature Transform (SIFT) or Speeded Up Robust Features (SURF) algorithm to detect key points in the video frame, and use the Random Sample Consensus Algorithm (RANSAC) to calculate the homography matrix H between the two video frames to evaluate the quality of feature point matching.

[0057] Color histogram similarity: Calculate the color histograms of two video frames and use Chi-Square Distance or other similarity metrics to evaluate the similarity of color distribution.

[0058] Depth Information Consistency: If the video frames contain depth information (e.g., data from a depth camera), the depth consistency can be evaluated by comparing the depth maps of two video frames.

[0059] S12, the form of the scoring function

[0060] Scoring function It can be formalized as:

[0061]

[0062] in:

[0063] is the feature point matching score.

[0064] is the color histogram similarity score.

[0065] is the depth information consistency score.

[0066] w1, w2, w3 are weight parameters, which can be adjusted according to the actual application scenario. S13, feature point matching score S feat

[0067] Feature point matching score S feat The calculation method is as follows:

[0068] Detection in video frames using SIFT or SURF algorithm and The key points in the video. Use the RANSAC algorithm to calculate the homography matrix H between the two video frames. Calculate the number of matching point pairs N matches .

[0069] Calculate the feature point matching score:

[0070]

[0071] Among them, N total is the total number of keypoints detected in the two video frames.

[0072] S14, color histogram similarity score S color

[0073] Color histogram similarity score S color The calculation method is as follows:

[0074] Video Frame and Convert to grayscale image.

[0075] Calculate the color histogram H of two video frames j and H k .

[0076] Compute the similarity of color histograms using chi-square distance:

[0077]

[0078] Where B is the number of bins in the histogram.

[0079] S15, depth information consistency score S depth

[0080] Depth information consistency score S depth The calculation method is as follows:

[0081] Get video frame and The depth map D j and D k .

[0082] Calculate the mean absolute error (MAE) between two depth maps:

[0083]

[0084] Where N is the number of pixels in the depth map.

[0085] S16, Optimization of scoring function

[0086] To find the best standard frame We need to search in the first n frames of each camera and select the frame with the highest total score as the standard frame of the standard camera (there is only one standard frame of a standard camera, and all other cameras are aligned to this camera. If there are multiple frames with the same highest score, select the earliest frame. If the time is the same, select randomly). The specific steps are as follows:

[0087] Initialize the variable max_score = 0,

[0088] For each candidate frame

[0089] Calculate the scoring function

[0090] if Update max_score and best_frame.

[0091] Finally, the bestframe is selected as the standard frame.

[0092] Through the above scoring function and optimization algorithm, we can effectively find a standard frame in a standard camera, so that the frames before the standard frame are discarded as little as possible, and the frames after the standard frame are compensated as little as possible, thereby achieving accurate alignment of multi-camera video frames.

[0093] Furthermore, after finding the standard frame, considering that the time of the video stream is continuous, but the sampling time is discontinuous, this embodiment proposes to generate a curve of k cameras near the standard frame by fitting, and then estimate the initial point of each camera relative to the standard frame. Curve fitting can more accurately determine the behavior mode of each camera near the standard frame and improve the accuracy of alignment.

[0094] S2, constructs a temporal flow model to model the changes of video frames over time;

[0095] It should be noted that after finding the standard frame, the alignment of subsequent frames needs to ensure that the video frames of each camera are consistent with the standard frame in time. To achieve this, this embodiment assumes that the time stream of each camera is the same, and uses a time stream model to describe the changes of video frames over time. The changes of video frames over time include changes in image content over time and changes in image quality over time; changes in image content over time include changes in object movement speed and direction; changes in image quality over time include changes in light intensity and blur. The model includes two parts: a polynomial part and a Fourier series part.

[0096] Polynomial part: describes the changing trend of image content over time, such as the moving speed and direction of the object.

[0097] Fourier series part: describes the changes in image quality over time, such as light intensity and blur.

[0098] Based on the above, the time flow model can be expressed by the following formula:

[0099] I(t)=I c (t)+I q (t)

[0100] in:

[0101] I(t) represents the video frame data at a certain time t.

[0102] I c (t) represents the part of the image content that changes over time.

[0103] I q (t) represents the part where the image quality changes over time.

[0104] Polynomials Part I c (t) can be expressed as an N-order polynomial:

[0105]

[0106] Among them, a n are the coefficients of the polynomial, which can be obtained by least squares fitting.

[0107] Fourier Series Part I q (t) can be expressed as a Fourier series of L terms:

[0108]

[0109] Among them, b l is the Fourier coefficient, ω l is the frequency, φ l is the phase, and these parameters can also be obtained by least squares fitting.

[0110] S3, using the time flow model and based on the time offset theorem, estimates the time offset of each camera;

[0111] Assume that there is a time offset τ, so that the video frame data I(t) can be aligned by the time offset τ. Construct the objective function E(τ), which represents the error between the video frame data after time offset and the reference standard frame. Through the above optimization method, minimize the objective function E(τ) and solve the time offset τ. The objective function is defined as follows:

[0112]

[0113] Among them, I ref (t) is the data of the standard frame; I(t-τ) is the video frame data after time shift; t is the time variable, indicating the timestamp of the video frame.

[0114] It should be noted that the time shift theorem is a basic property of the Fourier transform, which describes how the time shift of a signal in the time domain affects its frequency domain performance. Suppose there is a signal f(t), and its Fourier transform is F(ω). If the signal is time shifted (that is, the new signal is f(t-τ)), then the Fourier transform of this new signal is:

[0115] F(ω)e -iωτ

[0116] Here, i is the imaginary unit, ω is the frequency component, and τ is the time offset.

[0117] When the signal moves forward (τ<0) or backward (τ>0) on the time axis, the corresponding Fourier transform spectrum does not change the amplitude of the signal, but introduces a phase factor e in the frequency domain. -iωτ The phase factor is proportional to the offset τ, and for each frequency component ω, the phase change caused by the offset is -ωτ.

[0118] S4, according to the time offset, adjusts the video frame of each camera to align it with the standard frame.

[0119] Based on the above, the subsequent frame alignment steps are as follows:

[0120] 1. Initialization parameters: set the polynomial order N and the number of Fourier series terms L.

[0121] 2. Data preparation: Collect time series data of each camera near the standard frame. Ensure the integrity and accuracy of the data. This step provides a reliable basis for subsequent model fitting and time offset estimation.

[0122] 3. Polynomial Fitting: Fitting Polynomials Using Least Squares Method Part I c (t).

[0123] 4. Fourier series fitting: Fitting the Fourier series using the least squares method Part I q (t).

[0124] 5. Time offset estimation: Based on the fitting results, estimate the time offset τ of each camera k .

[0125] 6. Frame alignment: According to the time offset τ k , adjust the video frames of each camera to align them with the standard frame.

[0126] Through the above steps, we can achieve the subsequent alignment of multi-camera video frames and ensure that the video frames of each camera are consistent with the standard frames in time, thereby improving the accuracy and robustness of video frame alignment.

[0127] In summary, this embodiment provides a multi-camera video frame automatic alignment method, which significantly improves the accuracy and efficiency of the multi-camera system in 3D reconstruction by introducing efficient image processing and computer vision algorithms and combining deep learning technology. It can adapt to changes in different viewing angles, dynamic scenes and lighting conditions, and achieve fast and accurate video frame alignment. The following are the main beneficial effects brought by the solution of the present invention:

[0128] Improving the accuracy and reliability of 3D reconstruction

[0129] 3D reconstruction relies on accurate alignment of video frames acquired from multiple perspectives. This invention ensures accurate alignment of video frames captured by different cameras by comprehensively utilizing multiple features such as feature point matching, color histogram similarity, and depth information consistency. Specifically:

[0130] Feature point matching: Use the scale-invariant feature transform (SIFT) or speeded up robust features (SURF) algorithm to detect key points in the video frame, and use the random sampling consensus algorithm (RANSAC) to calculate the homography matrix, which improves the accuracy and reliability of feature point matching. This is crucial for geometric alignment in 3D reconstruction, ensuring that video frames from different perspectives can be correctly aligned, thereby improving the accuracy of the 3D model.

[0131] Scoring function: The scoring function comprehensively considers multiple features such as feature point matching, color histogram similarity, and depth information consistency, and finds the best standard frame through the scoring function to ensure that video frames from different perspectives can be accurately aligned. This step is particularly important in 3D reconstruction because any alignment error will directly affect the quality of the final model.

[0132] Color histogram similarity: By calculating the similarity of color histograms, the differences in lighting conditions of different cameras can be effectively handled to improve the alignment effect. In 3D reconstruction, changes in lighting conditions may lead to significant differences between video frames, affecting the reconstruction accuracy. The present invention ensures the alignment accuracy of video frames under different lighting conditions by calculating the similarity of color histograms.

[0133] Depth information consistency: The consistency of depth information is used to further improve the alignment accuracy under different lighting conditions. Depth information provides important geometric constraints in 3D reconstruction. By checking the consistency of depth information, the correct alignment of video frames in 3D space can be ensured, thereby improving the geometric accuracy of the reconstructed model.

[0134] Improving the robustness and adaptability of 3D reconstruction

[0135] Three-dimensional reconstruction requires not only high precision, but also good robustness and adaptability to cope with various complex environments and dynamic scenes. The present invention significantly improves the robustness and adaptability of the system through the following methods:

[0136] Temporal flow model: The polynomial part and the Fourier series part are used to model the changes of video frames over time, which can effectively handle the movement and changes of objects in dynamic scenes. The polynomial part describes the changing trend of image content over time, such as the speed and direction of the object's movement; the Fourier series part describes the changes of image quality over time, such as the light intensity and blur level. This temporal flow model enables the system to flexibly adapt to different dynamic scenes, improving the robustness and adaptability of 3D reconstruction.

[0137] Time offset estimation: Using the time offset theorem, the time offset of each camera is accurately estimated to ensure that the video frames in dynamic scenes can be accurately aligned. Time offset estimation is implemented through the least squares method, which can complete the estimation of time offset in a short time to ensure the real-time processing capability of the system. In 3D reconstruction, accurate estimation of time offset is particularly important for the processing of dynamic scenes, which can significantly improve the stability and reliability of the reconstructed model.

[0138] Multiple features of the scoring function: The scoring function comprehensively considers multiple features, which can reduce the impact of external interference in complex environments and improve the robustness of the system. In three-dimensional reconstruction, external interference (such as lighting changes, occlusions, etc.) may cause alignment errors between video frames, affecting the reconstruction accuracy. The present invention ensures that the most suitable alignment reference frame can be found under various conditions through the multiple features of the scoring function. This ensures the robustness and stability of the system in complex environments.

[0139] Reduce computational complexity and improve processing efficiency

[0140] Three-dimensional reconstruction usually involves a lot of calculations, which places high demands on real-time processing capabilities. The present invention significantly reduces the computational complexity and improves processing efficiency through the following method.

[0141] Efficiency of polynomials and Fourier series: Polynomial fitting and Fourier series fitting are implemented through the least squares method, which has low computational complexity and can quickly find the optimal solution. In 3D reconstruction, efficient calculation methods can significantly shorten the reconstruction time and improve the real-time processing capability of the system.

[0142] Efficiency of the scoring function: The scoring function is implemented through efficient algorithms such as feature point matching, color histogram similarity, and depth information consistency, and can complete the scoring calculation in a short time. In 3D reconstruction, fast scoring calculation can ensure the efficiency and real-time performance of the system when processing a large number of video frames.

[0143] Fast time offset estimation: Time offset estimation is implemented through the least squares method, which can complete the estimation of time offset in a short time, ensuring the real-time processing capability of the system. In 3D reconstruction, fast time offset estimation can significantly improve the real-time processing capability and reconstruction efficiency of the system.

[0144] Expanding the application scope of 3D reconstruction

[0145] The present invention significantly expands the application scope of multi-camera systems in three-dimensional reconstruction by improving the accuracy and efficiency of video frame alignment. Specifically:

[0146] Monitoring system: Through accurate alignment of video frames, the monitoring system can obtain more comprehensive information from multiple perspectives and improve monitoring effects. In 3D reconstruction, multi-view monitoring can provide richer geometric information and improve the accuracy and integrity of the 3D model.

[0147] Autonomous driving: Through accurate alignment of video frames, autonomous driving systems can better fuse multi-sensor data, improve perception accuracy and decision-making capabilities. In 3D reconstruction, the fusion of multi-sensor data can provide more detailed environmental information and improve the accuracy and reliability of 3D maps.

[0148] Sports event broadcasting: Through accurate alignment of video frames, sports event broadcasting can provide richer visual information from multiple perspectives and improve the audience experience. In 3D reconstruction, multi-perspective broadcasting can provide more comprehensive game information and improve the accuracy and viewing experience of 3D reconstruction models.

[0149] Virtual reality and movie special effects: Through accurate alignment of video frames, 3D reconstruction in virtual reality and movie special effects is more accurate and improves visual effects. In 3D reconstruction, alignment of multi-view video frames can provide richer geometry and texture information, improving the realism and visual effects of 3D models.

[0150] In summary, the present invention significantly improves the accuracy and efficiency of multi-camera systems in 3D reconstruction by comprehensively applying a variety of advanced image processing and computer vision technologies, improves the robustness and adaptability of the system, reduces computational complexity, and expands the application scope of 3D reconstruction. These improvements not only improve the quality and efficiency of 3D reconstruction, but also provide strong support for the application of multi-camera systems in various fields.

[0151] Second embodiment

[0152] This embodiment provides an electronic device, such as Figure 3 As shown, the electronic device includes: a processor and a memory; wherein the processor and the memory can be connected via a communication bus; the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method of the first embodiment. In addition, the electronic device may also include a transceiver, the processor and the transceiver can be connected via a communication bus, and the transceiver is used to communicate with other devices.

[0153] Next, combine Figure 3 The following is a detailed introduction to the various components of the electronic device:

[0154] Among them, the processor is the control center of the electronic device, and the electronic device may include multiple processors, each of which may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may be a processor or a general term for multiple processing elements. For example, the processor is one or more central processing units (CPUs), or other general-purpose processors, application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement an embodiment of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor may execute various functions of the electronic device by running or executing software programs stored in the memory and calling data stored in the memory.

[0155] In a specific implementation, as an embodiment, the processor may include one or more CPUs, such as Figure 3 The CPU0 and CPU1 shown in the figure are, of course, only exemplary.

[0156] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0157] Optionally, the memory may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be integrated with the processor or exist independently and accessed through the interface circuit ( Figure 3 (not shown) is coupled to the processor, which is not specifically limited in this embodiment of the present invention.

[0158] The transceiver may include a receiver and a transmitter ( Figure 3 The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function. The transceiver can be integrated with the processor or exist independently and communicate with the electronic device through the interface circuit ( Figure 3 (not shown) is coupled to the processor, which is not specifically limited in this embodiment of the present invention.

[0159] In addition, it should be noted that Figure 3 The structure of the electronic device shown in the figure does not constitute a limitation on the device, and the actual device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In addition, the technical effects achieved by the electronic device when executing the method of the first embodiment above can refer to the technical effects described in the first embodiment above, so they are not repeated here.

[0160] Third embodiment

[0161] This embodiment provides a computer-readable storage medium, which stores at least one instruction, and the instruction is loaded and executed by a processor to implement the method of the first embodiment. The computer-readable storage medium may be a ROM, a random access memory, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc. The instructions stored therein may be loaded by a processor in a terminal to execute the method.

[0162] In addition, it should be noted that the present invention can be provided as a method, an apparatus or a computer program product. Therefore, the embodiment of the present invention can be in the form of a full or partial hardware embodiment, a full or partial software embodiment or an embodiment combining software and hardware. Moreover, when implemented using software, the embodiment of the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program codes. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center containing one or more available media sets. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium. The semiconductor medium may be a solid state hard disk.

[0163] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0164] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0165] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements. In addition, the term "and / or" is only an association relationship describing the associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone, wherein A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding. "At least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can be represented by: a, b, c, ab, ac, bc or abc, where a, b, c can be single or plural.

[0166] In addition, it can be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0167] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0168] In several embodiments provided by the present invention, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of functional modules / units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0169] If the method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0170] Finally, it should be noted that the above is only a preferred embodiment of the present invention. It should be pointed out that although the preferred embodiment of the present invention has been described, for ordinary technicians in this technical field, once the basic creative concept of the present invention is known, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. Therefore, the attached claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the embodiments of the present invention.

Claims

1. A method for automatic alignment of multi-camera video frames, characterized in that: include: Obtain the video frames captured by each camera, and measure the similarity between the video frames captured by different cameras based on a preset scoring function, so as to find a standard frame in the video frames as an alignment reference frame; Construct a temporal flow model to model the changes of video frames over time; the changes of video frames over time include changes of image content over time and changes of image quality over time; changes of image content over time include changes of object moving speed and direction; changes of image quality over time include changes of light intensity and blur degree; Using the time flow model and based on the time offset theorem, the time offset of each camera is estimated; The video frames of each camera are adjusted to align with the standard frame according to the time offset.

2. The multi-camera video frame automatic alignment method as claimed in claim 1, characterized in that: The scoring function measures the similarity between video frames captured by different cameras through the matching degree of feature points, the similarity of color histograms and the consistency of depth information between the video frames captured by different cameras.

3. The multi-camera video frame automatic alignment method as described in claim 2, characterized in that: The expression of the scoring function is: in, Represents the video frame captured by the jth camera at time t1 and the video frame captured by the kth camera at time t2 The similarity between Indicates the matching score of feature points; represents the color histogram similarity score; represents the depth information consistency score; w1, w2 and w3 are preset weight parameters, and the value of each weight parameter is adjusted according to the actual application scenario.

4. The multi-camera video frame automatic alignment method as claimed in claim 3, characterized in that: Among them, N matches express and The number of matching point pairs between feature points; N total express and The total number of feature points detected in ; The value of The color histogram of The similarity between the color histograms of The value of The depth map and The mean absolute error between the depth maps.

5. The multi-camera video frame automatic alignment method as claimed in claim 1, characterized in that: The method of measuring the similarity between video frames captured by different cameras based on a preset scoring function to find a standard frame in the video frames as an alignment reference frame includes: Select the first n frames in a video frame captured by a camera as candidate standard frames; wherein n is a preset value; Calculate the scoring function value between each video frame captured by the current camera and each candidate standard frame; The candidate standard frame corresponding to the largest score function value is selected as the final standard frame.

6. The multi-camera video frame automatic alignment method as claimed in claim 1, characterized in that: The expression of the time flow model is: I(t)=I c (t)+I q (t) Where I(t) represents the video frame data at time t; I c I(t) represents the part of the image content that changes over time; I(t) q (t) represents the part of the image quality of I(t) that changes over time.

7. The multi-camera video frame automatic alignment method as claimed in claim 6, characterized in that: I c (t) is a multi-order polynomial, expressed as: Where N is the polynomial order; a n are the coefficients of the polynomial; I q (t) is a Fourier series, expressed as: Where L is the number of Fourier series terms; b l is the Fourier coefficient; ω l is the frequency; φ l It's the phase.

8. The multi-camera video frame automatic alignment method as claimed in claim 7, characterized in that: The method of using the time flow model to estimate the time offset of each camera based on the time offset theorem includes: Set the polynomial order N and the number of Fourier series terms L; Collect time series data of each camera around the standard frame; Using the collected time series data, the least squares method is used to fit I c (t) and I q (t) expression; Based on the fitting results, the time offset of each camera is estimated.

Images