Method, apparatus, storage medium and program product for multi-path video alignment

Abstract

The application provides a multi-path video alignment method and device, a storage medium and a program product. The method comprises the following steps: constructing image feature vectors of a first video and a second video into a first time sequence and a second time sequence respectively; determining a target path according to the difference in the change rate of the first time sequence and the second time sequence; performing DTW on a first matrix formed by the first time sequence and the second time sequence by taking the target path as a shape reference to obtain an alignment path; updating elements exceeding the path boundary in the alignment path by using a dynamic penalty coefficient to obtain an optimized path, the dynamic penalty coefficient being related to the distance between the punished elements and the target path; and obtaining a first alignment sequence and a second alignment sequence in time alignment according to the optimized path. The application introduces a dynamic penalty coefficient and an abstract level shape feature to perform sequence alignment, which can significantly improve the accuracy of video sequence alignment, effectively improve the video alignment effect and reduce the device cost.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a method, apparatus, storage medium, and program product for aligning multiple video streams. Background Technology

[0002] Due to the massive amount of video data, compression is typically required to support long-distance transmission. During the encoding, network transmission, and decoding display processes, a certain degree of latency is unavoidable. This latency fluctuates not only with changes in encoding configuration and network conditions but is also affected by the video content itself.

[0003] To meet the needs of large-view scene reconstruction, digital twins, virtual reality, and other scenario construction, it is often necessary to transform and stitch multiple video streams from different perspectives. However, because the latency of each video stream during transmission is not consistent, directly transforming and stitching multiple untime-aligned video streams will result in a severely degraded composite video quality.

[0004] While using cameras with time synchronization capabilities can solve this problem to some extent, such devices are expensive, and there are already a large number of surveillance video cameras without time synchronization capabilities in use.

[0005] Therefore, there is an urgent need for a technical solution that can make full use of existing monitoring resources to align multiple video streams, thereby improving the video synthesis effect while reducing equipment costs. Summary of the Invention

[0006] To address the aforementioned issues, this application provides a multi-channel video alignment method, apparatus, storage medium, and program product, which can significantly improve the accuracy of multi-channel video alignment and reduce equipment costs while enhancing video synthesis effects.

[0007] In a first aspect, this application provides a method for aligning multiple video streams, the method comprising:

[0008] S1. Extract the image feature vectors of the first video and the second video respectively, and construct the first time sequence of the first video and the second time sequence of the second video according to the time order of the image feature vectors.

[0009] S2. Determine the target path based on the difference in the rate of change between the first time series and the second time series; the target path characterizes the shape features of the first time series and the second time series;

[0010] S3. Using the target path as a shape reference, perform dynamic time warping on the first matrix composed of the first time series and the second time series to obtain the alignment path.

[0011] S4. Update the penalized elements that exceed the path boundary in the alignment path using a dynamic penalty coefficient to obtain an optimized path. The smaller the distance between the penalized element and the target path, the smaller the dynamic penalty coefficient used.

[0012] S5. Based on the corresponding elements of each element in the optimized path in the first time series and the corresponding elements in the second time series, obtain the first alignment sequence and the second alignment sequence after time alignment.

[0013] In one possible implementation, step S2 includes:

[0014] S21. Calculate the difference between the first derivative of the first time series and the first derivative of the second time series to obtain the first matrix D′. x,y The first matrix D′ x,t The values ​​of the elements in the middle represent the differences in the rates of change between the two time series;

[0015] S22, Select the first matrix D′ x,t The extreme elements in the matrix construct the second matrix D. x,t The element d in the second matrix x[i],y[j] The value represents the distance between the i-th element of the first time series and the j-th element of the second time series, where i and j are positive integers;

[0016] S23. Perform dynamic time warping on the second matrix to obtain the target path.

[0017] In one possible implementation, step S21 includes:

[0018] S211. Using the first time series X as the horizontal axis data and the second time series Y as the vertical axis data, construct a coordinate system containing the elements of the first matrix;

[0019] S212. Perform first-order differences on the first time series and the second time series respectively to obtain the first difference sequence and the second difference sequence;

[0020] S213. Calculate the difference between corresponding elements of the first difference sequence and the second difference sequence to obtain the numerical value of the corresponding element in the first matrix.

[0021] In one possible implementation, step S23 includes:

[0022] The target path with the minimum cumulative distance is determined by performing dynamic time warping on each element in the second matrix. The dynamic time warping process is expressed as follows:

[0023] R(i,j)=d(i,j)+min(R(i-1,j-1),R(i-1,j),R(i,j-1));

[0024] d(i,j) represents the distance between the i-th element of the first time series and the j-th element of the second time series; R(i,j) represents the distance from the starting element d. 0,0 to element d x[i],y[j] The cumulative distance;

[0025] Step S3 includes: using multiple path elements d that constitute the target path x[i],y[j] The corresponding multiple extreme elements extreme(d′) x[i],y[j] The first matrix is ​​divided into multiple segments. Dynamic time warping is performed on each segment to determine the sub-path with the smallest cumulative distance for each segment. The sub-paths corresponding to multiple segments form the alignment path.

[0026] In one possible implementation, the coordinate system containing the elements of the first matrix uses the first time series as the horizontal axis data and the second time series as the vertical axis data. The method prior to step S4 further includes:

[0027] Step A: Calculate the farthest distance from the diagonal of the coordinate system in the alignment path, where the diagonal is a ray originating from the origin with a slope of 1;

[0028] Step B: Determine the path boundary as two straight lines that have the same slope as the diagonal of the coordinate axis and are half the furthest distance from the diagonal of the coordinate axis.

[0029] The update process for the penalized element in step S4 is performed according to the following formula:

[0030] D′ x,y =D′ x,y *(1+λd′*(d-Mar));D′ x,y Let d' represent the penalized element, d' represent the distance from the penalized element to the nearest extreme element, d represent the straight-line distance from the penalized element to the diagonal, Mar represent the straight-line distance from the path boundary to the diagonal, and λ represent the given base penalty coefficient.

[0031] In one possible implementation, step S1 includes:

[0032] S11. Using a given human pose detection algorithm, perform human pose detection on the first video and the second video to obtain human pose information for each frame of the first video and human pose information for each frame of the second video.

[0033] S12. Convert the human pose information of each frame of each video into a feature vector, and construct the feature vectors corresponding to each frame of each video in chronological order to obtain the first time series and the second time series respectively.

[0034] In one possible implementation, step S5 includes:

[0035] S51, regarding the optimized path D′ x[i],y[j] Perform DTW to determine the optimal path with the minimum cumulative distance;

[0036] S52, The element D″ in the optimal path is... x[i],y[j] In the first time series, the corresponding element X[i] is determined as an element in the first alignment sequence, and the optimal path D″ is... x[i],y[j] In the second time series, the corresponding element Y[j] is determined to be an element in the second alignment sequence.

[0037] In a second aspect, a multi-channel video alignment apparatus is provided, which includes multiple functional modules for performing corresponding steps in the multi-channel video alignment method provided in the first aspect.

[0038] Thirdly, a computer-readable storage medium is provided, wherein at least one program is stored therein, the at least one program being executed by a processor to implement the multi-channel video alignment method as provided in the first aspect.

[0039] Fourthly, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the multi-channel video alignment method as provided in the first aspect.

[0040] Fifthly, a computing device is provided, comprising a memory and a processor, the memory storing at least one program, the at least one program being executed by the processor to implement the multi-channel video alignment method as provided in the first aspect.

[0041] The technical solution provided in this application includes at least the following technical effects:

[0042] (1) By using human posture information in the video as the feature vector of the time series, the alignment operation can be guaranteed based on the correlation of the content of multiple videos.

[0043] (2) By constructing a matrix using the distance between time series, the high-level features “shape” between time series can be abstracted, thereby further improving the accuracy of path planning and the effect of sequence alignment.

[0044] (3) During the DTW path search process, the extreme value elements of the second matrix are matched separately first, and then the first matrix is ​​segmented by the extreme value elements, so as to accurately match each segment and obtain the aligned path.

[0045] (4) Introduce the dynamic penalty coefficient of the associated extreme value element to optimize the alignment path, so that the shape feature always constrains the sequence alignment process, and the alignment effect can be guaranteed even when there are large differences between the sequences to be aligned.

[0046] In summary, this application introduces dynamic penalty coefficients and shape features of abstract levels for sequence alignment, which can significantly improve the accuracy of video sequence alignment. The efficient algorithm not only improves the video alignment effect, but can also be seamlessly embedded into existing devices, effectively reducing device costs. Attached Figure Description

[0047] Figure 1 This is a flowchart of a multi-channel video alignment method provided in an embodiment of this application;

[0048] Figure 2 This is a schematic diagram illustrating a path optimization method provided in an embodiment of this application;

[0049] Figure 3 This is a flowchart of another multi-channel video alignment method provided in the embodiments of this application;

[0050] Figure 4 This is a schematic diagram of a multi-channel video alignment device provided in an embodiment of this application;

[0051] Figure 5 This is a schematic diagram of the hardware structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0052] To further illustrate the various embodiments, this application provides accompanying drawings. These drawings are part of the disclosure of this application and are mainly used to illustrate the embodiments, and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of this application. Components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components. In this application, the term "at least one" means one or more, and the term "multiple" means two or more.

[0053] This application will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0054] In this application, the term "element" is used to describe a single item / value in a sequence. In other alternative implementations, "element" and "point" are often used interchangeably when describing a sequence, and this application does not limit this usage. A sequence is an ordered collection of elements, which can be a basic data type (such as integers, floating-point numbers, Boolean values, etc.) or other sequences or objects. Each element or point in a sequence has a position or index, which can be accessed starting from 0 using an integer index.

[0055] The multi-channel video alignment method provided in this application is described below.

[0056] For example, the multi-channel video alignment method provided in this application can be executed by any computing device to perform precise temporal alignment of multiple video streams. The computing device can be a terminal device such as a personal computer, tablet computer, or smartphone; or it can be a server, a server cluster consisting of multiple physical servers, or a distributed file system; or it can be a cloud server cluster providing basic cloud computing services such as cloud storage, cloud services, cloud databases, cloud computing, cloud functions, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), big data, and artificial intelligence platforms. This application does not limit the scope of the application to these specific methods.

[0057] Example 1

[0058] This application provides a method for aligning multiple video streams. Figure 1 A flowchart of a multi-channel video alignment method provided in this application is shown below. Figure 1 As shown, the method includes at least the following steps S1 to S5, which can be executed by any computing device.

[0059] S1. Extract the image feature vectors of the first video and the second video respectively, and construct the first time sequence X of the first video and the second time sequence Y of the second video according to the time order of the image feature vectors.

[0060] The video can be a real-time video stream or a video clip; this application does not limit the scope of the video.

[0061] In this embodiment, the image feature vector can be determined based on human posture information, or based on the speed characteristics of dynamic objects in the video, video optical flow, and other feature information.

[0062] In one possible implementation, the image feature vector is determined based on human pose information, and step S1 includes the following steps S11 and S12.

[0063] S11. Using the given human pose detection algorithm, perform human pose detection on the first video and the second video to obtain the human pose information of each frame of the first video and the human pose information of each frame of the second video.

[0064] For example, the human pose detection algorithm used in this application involves multiple steps, including deep learning, feature extraction, and pose estimation. First, features are extracted from video frames using deep learning models such as convolutional neural networks (CNNs). These features include, for example, edges, textures, and shapes. Next, pose estimation algorithms such as OpenPose and MoveNet are used to identify key points of the human body in the video frames (e.g., head, shoulders, knees), and to determine the relative positions and pose relationships between these key points. Pose estimation includes key point detection and connection, describing the human pose through the spatial relationships of these key points. In this example, human pose information can include key points and the spatial relationships between them.

[0065] S12. Convert the human pose information of each frame of each video into a feature vector, and construct the feature vectors corresponding to each frame of each video in chronological order to obtain the first time series and the second time series respectively.

[0066] For example, each element in the time series is a feature value used to describe the human posture in the video at a certain moment, and the elements are arranged in the order of their appearance in the video.

[0067] This application utilizes human posture information in videos as feature vectors for time series, which can ensure the effectiveness of alignment operations based on the correlation of image content between multiple videos.

[0068] S2. Determine the target path based on the difference in the rate of change between the first time series and the second time series.

[0069] The target path represents the shape characteristics of the first time series and the second time series.

[0070] In this embodiment of the application, shape features are abstracted by calculating the difference of the first derivative between two sequences. This step S2 includes S21 to S23.

[0071] S21. Calculate the difference between the first derivative of the first time series and the first derivative of the second time series to obtain the first matrix D′. x,y The first matrix D′ x,y The values ​​of the elements in the middle represent the differences in the rates of change between the two time series.

[0072] For example, step S21 includes steps S211 to S213.

[0073] S211. Using the first time series X as the horizontal axis data and the second time series Y as the vertical axis data, construct the coordinate system of the elements of the first matrix.

[0074] Figure 2 This is a schematic diagram of a path optimization provided in an embodiment of this application, such as... Figure 2 As shown, the data points on the horizontal axis of the coordinate system represent one video sequence to be aligned, and the data points on the horizontal axis of the coordinate system represent another video sequence to be aligned.

[0075] S212. Perform first-order differences on the first time series and the second time series respectively to obtain the first difference sequence and the second difference sequence.

[0076] In this embodiment, the first derivative is used to describe the slope of a function at a certain point, and the slope represents the rate of change of the function. For discrete sequences (such as time series and signal sequences), the difference is usually used to approximate the first derivative. The difference is the difference between adjacent elements of the sequence. For example, for a sequence [x1,x2,x3,...], its first difference sequence is represented as [x2-x1,x3-x2,x4-x3,...].

[0077] S213. Calculate the difference between corresponding elements of the first difference sequence and the second difference sequence to obtain the numerical values ​​of the corresponding elements in the first matrix.

[0078] For example, the elements of the first time series X and the second time series Y are represented as follows:

[0079] X=[x1,x2,x3,...]; Y=[y1,y2,y3,...];

[0080] The first difference sequence obtained by performing a first-order difference on the first time series is represented as follows:

[0081] ΔX=[x2-x1,x3-x2,x4-x3,...];

[0082] The second difference sequence obtained by performing a first-order difference on the second time series is expressed as:

[0083] ΔY=[y2-y1,y3-y2,y4-y3,...];

[0084] Based on this, the difference of the first derivatives of all elements in the two time series can be expressed as:

[0085] ΔX-ΔY=[(x2-x1)-(y2-y1), (x3-x2)-(y3-y2), (x4-x3)-(y4-y3),...].

[0086] The embodiments of this application use the difference of the first derivative to accurately compare the dynamic changes of two time series, which helps to extract higher-level correlations or differences between the two time series.

[0087] S22. Select the first matrix D′ x,y The extreme elements in the matrix construct the second matrix D. x,y The element d in the second matrix x[i],y[j] The value represents the distance between the i-th element of the first time series and the j-th element of the second time series, where i and j are positive integers;

[0088] S23. Perform Dynamic Time Warping (DTW) on the second matrix to obtain the target path.

[0089] Time series dynamics (DTW) is a method for measuring the similarity between two time series, particularly suitable for series of unequal length. DTW aligns the time series by non-linearly adjusting their time axes, thereby calculating the minimum cumulative distance between them. The basic idea of ​​DTW is to stretch and compress the time series to find the optimal alignment between the two series. This alignment is achieved by calculating a cumulative distance matrix between the two time series, which records the cumulative distance between all possible pairs of points between the two series. A path is found from this matrix that connects the start and end points of the two series by selecting the pair of points with the minimum cumulative distance.

[0090] In one possible implementation, step S23 includes: performing Direct Time-Distance Written (DTW) on each element of the second matrix to determine the target path with the minimum cumulative distance. The starting element (origin) of the DTW is the origin of the coordinate system containing the second matrix, denoted as d. 0,0 .

[0091] For example, the dynamic time warping process is represented as:

[0092] R(i,j)=d(i,j)+min(R(i-1,j-1),R(i-1,j),R(i,j-1));

[0093] Where d(i,j) represents the distance between the i-th element of the first time series and the j-th element of the second time series; R(i,j) represents the distance from the starting element d. 0,0 to element d x[i],y[j] The cumulative distance. The distance described in the embodiments of this application refers to the Euclidean distance.

[0094] This application embodiment constructs a first matrix by utilizing the distance between time series, and then constructs a second matrix by selecting extreme value elements, abstracting the high-level feature "shape" between time series, thereby further improving the accuracy of path planning and the sequence alignment effect.

[0095] S3. Using the target path as the shape reference, perform dynamic time warping on the first matrix composed of the first time series and the second time series to obtain the alignment path.

[0096] In this embodiment of the application, the coordinate system containing the elements of the first matrix uses a first time series as the horizontal axis data and a second time series as the vertical axis data. In this example, the process of constructing the first matrix includes the following steps A and B.

[0097] Step A: Calculate the farthest distance from the coordinate system diagonal in the alignment path. The coordinate system diagonal is a ray that starts from the origin and has a slope of 1.

[0098] Figure 2 This is a schematic diagram illustrating a path optimization method provided in an embodiment of this application. Figure 2 The ray (the light gray line in the middle) that originates from the origin of the coordinate system and has a slope of 1 is the diagonal of the coordinate system.

[0099] Step B: Determine the path boundary as two straight lines that have the same slope as the diagonal of the coordinate axis and are half the furthest distance from the diagonal of the coordinate axis.

[0100] Figure 2 This is a schematic diagram illustrating a path optimization method provided in an embodiment of this application. Figure 2 The two lines parallel to the diagonal indicated by the margin legend in the middle are the path boundaries.

[0101] In one possible implementation, step S3 includes: using multiple path elements d that constitute the target path. x[i],y[j] The corresponding multiple extreme elements extreme(d′) x[i],y[j] The first matrix is ​​divided into multiple segments. Dynamic time warping is performed on each segment to determine the sub-path with the smallest cumulative distance for each segment. The sub-paths corresponding to multiple segments form an alignment path.

[0102] For example, the second matrix D' is divided into P segments, where P is a positive integer greater than 1; for the P-th segment D'[p], find D'[p]. 0,0 To D'[p] i,j The sub-path P with the smallest cumulative distance; and so on, we can get P sub-paths. The P sub-paths can be combined to get the aligned path path.

[0103] In the DTW path search process of this application embodiment, the extreme value elements are first matched separately for the second matrix, and then the first matrix is ​​segmented by the extreme value elements, so as to accurately match each segment and obtain the aligned path.

[0104] S4. Update the penalized elements that exceed the path boundary in the alignment path using a dynamic penalty coefficient to obtain the optimized path.

[0105] The smaller the distance between the penalized element and the target path, the smaller the dynamic penalty coefficient used. The distance between the penalized element and the target path refers to the straight-line distance.

[0106] For example, the update process for the penalized element in step S4 is performed according to the following formula:

[0107] D′ x,y =D′ x,y *(1+λd′*(d-Mar));

[0108] Among them, D′ x,y Let d' represent the penalized element, d' represent the distance from the penalized element to the nearest extreme value element, and d represent the straight-line distance from the penalized element to the diagonal (see [link to documentation]). Figure 2 Mar represents the straight-line distance from the path boundary to the diagonal; λ is the given base penalty coefficient. That is, the smaller the distance between the penalized element and the nearest extreme element on the target path, the smaller the dynamic penalty coefficient used. If the path boundary is not exceeded, the element is not updated.

[0109] For example, the straight-line distance d from the penalized element to the diagonal can be calculated using the following formula:

[0110]

[0111] Where k is the length of sequence Y, l is the length of sequence Y; abs is the operation for calculating the absolute value, and math.sqrt is the operation for calculating the square root.

[0112] This application's embodiments introduce a dynamic penalty coefficient for associated extreme value elements to optimize the alignment path, ensuring that shape features always constrain the sequence alignment process, and guaranteeing alignment results even when there are significant differences between the sequences to be aligned.

[0113] S5. Based on the corresponding elements in the first time series and the corresponding elements in the second time series of each element in the optimized path, obtain the first alignment sequence and the second alignment sequence after time alignment.

[0114] In this embodiment of the application, step S5 includes steps S51 and S52.

[0115] S51. Optimize path D′x[i],y[j] Perform DTW to determine the optimal path with the minimum cumulative distance.

[0116] S52, extract element D″ from the optimal path. x[i],y[j] In the first time series, the corresponding element X[i] is determined as the element in the first alignment sequence, and the optimal path D″ is determined. x[i],y[j] In the second time series, the corresponding element Y[j] is determined to be an element in the second alignment sequence.

[0117] In one possible implementation, the first matrix and the second matrix described above can be stored in a "table" format, denoted as table D′ and table D, respectively.

[0118] To facilitate understanding of the logical relationships between the various possible implementations described above, this application provides a flowchart of another multi-channel video alignment method, see [link to flowchart]. Figure 3 :

[0119] First, human pose detection is performed on the first and second videos to be aligned. Based on the human pose information, a video time series is modeled, resulting in a first time series X and a second time series Y. Then, a table D' (first matrix) is constructed with X as the horizontal axis and Y as the vertical axis. Next, a table D (second matrix) is constructed using the XY Euclidean distances corresponding to the extreme elements in table D'. Finally, a search is performed in table D to find the source of the source from D'. 0,0 To D i,j The target path with the minimum cumulative distance (DTW applied to the second matrix); then, table D' is segmented using the extreme value elements corresponding to the target path; D'[p] is calculated for each of the P segments in table D'. 0,0 To D'[p] i,j The subpath with the smallest cumulative distance is the aligned path; a path boundary Mar is introduced to optimize the aligned path. If the path exceeds the boundary, a dynamic penalty coefficient associated with the extreme element is introduced to update the corresponding position; if the path does not exceed the boundary, the element remains unchanged; the updated D is then processed. 'i,j Perform DTW to determine the optimal path with the minimum cumulative distance, and output the first alignment sequence and the second alignment sequence based on the optimal path.

[0120] This application introduces dynamic penalty coefficients and shape features of abstract levels for sequence alignment, which can significantly improve the accuracy of video sequence alignment. The efficient algorithm not only improves the video alignment effect, but can also be seamlessly embedded into existing devices, effectively reducing device costs.

[0121] Example 2

[0122] This application provides a method for aligning multiple video streams, which is described below in code form.

[0123]

[0124] In the code example above, the two videos v1 and v2 correspond to the first and second videos in Example 1, respectively; the first derivative difference matrix D_(m×n)^' is also the first matrix in Example 1; the cost matrix D_(m×n) is also the second matrix in Example 1; dist represents the calculation of Euclidean distance; min_trace represents the path with the minimum cumulative distance determined by DTW; len indicates the sequence length; path represents the alignment path, and the initial path'[i] represents the target path determined from the second matrix; in subsequent processes, path' is iteratively updated, and finally updated to the final optimal matrix through the "min_trace" operation. The remaining parameters are all explained in Example 1 and will not be repeated here. It should be noted that the above code is only a code example of the method described in Example 1, and it can be implemented in other embodiments using other types of coding languages, which is not limited in this application.

[0125] This application also provides a schematic diagram of a multi-channel video alignment device, see reference. Figure 4 The device includes:

[0126] The sequence conversion module is used to extract the image feature vectors of the first video and the second video respectively, and construct the first time sequence of the first video and the second time sequence of the second video according to the time order of the image feature vectors.

[0127] An alignment module is used to determine a target path based on the difference in the rate of change between the first time series and the second time series; the target path characterizes the shape features of the first time series and the second time series.

[0128] The alignment module is further configured to use the target path as a shape reference to perform dynamic time warping on the first matrix composed of the first time series and the second time series to obtain the alignment path.

[0129] The alignment module is also used to update the penalized elements that exceed the path boundary in the alignment path with a dynamic penalty coefficient to obtain an optimized path. The smaller the distance between the penalized element and the target path, the smaller the dynamic penalty coefficient used.

[0130] The alignment module is further configured to obtain a first alignment sequence and a second alignment sequence after time alignment based on the corresponding elements of each element in the optimized path in the first time sequence and the corresponding elements in the second time sequence.

[0131] It should be noted that the multi-channel video alignment device provided in the above embodiments is only illustrated by the division of the above functional modules when implementing the corresponding steps. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the multi-channel video alignment device provided in the above embodiments and the multi-channel video alignment method described above belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0132] This application provides a computing device that can perform operations. Figure 5 This is a schematic diagram of the hardware structure of a computing device provided in an embodiment of this application, such as... Figure 5 As shown, the computing device includes a processor 501, a memory 502, a bus 503, and a computer program stored in the memory 502 and executable on the processor 501. The processor 501 includes one or more processing cores. The memory 502 is connected to the processor 501 via the bus 503 and is used to store program instructions. When the processor executes the computer program, it implements all or part of the steps in the above-described method embodiments provided in this application.

[0133] Furthermore, as an executable solution, the aforementioned computing device can be a computer unit, which may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer unit may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above-described computer unit structure is merely an example and does not constitute a limitation on the computer unit. It may include more or fewer components, or combine certain components, or different components. For example, the computer unit may also include input / output devices, network access devices, buses, etc., which are not limited in this application embodiment.

[0134] Furthermore, as an executable solution, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The processor is the control center of the computer unit, connecting various parts of the entire computer unit via various interfaces and lines.

[0135] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer unit by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0136] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the multi-channel video alignment method described above in the embodiments of this application.

[0137] If the modules / units integrated in the computer unit are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0138] Although this application has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to this application without departing from the spirit and scope of this application as defined by the appended claims, and all such changes shall be within the scope of protection of this application.

Claims

1. A method for aligning multiple video streams, characterized in that, The method includes: S1. Extract the image feature vectors of the first video and the second video respectively, and construct the first time sequence of the first video and the second time sequence of the second video according to the time order of the image feature vectors. S2. Determine the target path based on the difference in the rate of change between the first time series and the second time series; the target path characterizes the shape features of the first time series and the second time series; S3. Using the target path as a shape reference, perform dynamic time warping on the first matrix composed of the first time series and the second time series to obtain the alignment path. S4. Update the penalized elements that exceed the path boundary in the alignment path using a dynamic penalty coefficient to obtain an optimized path. The smaller the distance between the penalized element and the target path, the smaller the dynamic penalty coefficient used. S5. Based on the corresponding elements of each element in the optimized path in the first time series and the corresponding elements in the second time series, obtain the first alignment sequence and the second alignment sequence after time alignment. Step S2 includes: S21. Calculate the difference between the first derivative of the first time series and the first derivative of the second time series to obtain the first matrix. The first matrix The values ​​of the elements in the middle represent the differences in the rates of change between the two time series; S22, Select the first matrix Construct the second matrix from the extreme elements in The elements in the second matrix The numerical value represents the first time series. The element and the second time series The distance between elements and It is a positive integer; S23. Perform dynamic time warping on the second matrix to obtain the target path; Step S23 includes: The target path with the minimum cumulative distance is determined by performing dynamic time warping on each element in the second matrix. The dynamic time warping process is expressed as follows: ； Represents the first time series The element and the second time series The distance between elements; Indicates starting from the element To element The cumulative distance; Step S3 includes: using multiple path elements that constitute the target path. Corresponding multiple extreme elements The elements of the first matrix are divided into multiple segments. Dynamic time warping is performed on each segment to determine the sub-path with the smallest cumulative distance for each segment. The sub-paths corresponding to multiple segments form the alignment path. The coordinate system containing the elements of the first matrix uses the first time series as the horizontal axis and the second time series as the vertical axis. Prior to step S4, the method further includes: Step A: Calculate the farthest distance from the diagonal of the coordinate system in the alignment path, where the diagonal is a ray originating from the origin with a slope of 1; Step B: Determine the path boundary as two straight lines that have the same slope as the diagonal of the coordinate system and are half the furthest distance from the diagonal of the coordinate system. The update process for the penalized element in step S4 is performed according to the following formula: = ( ); Indicates the element being penalized. This represents the distance from the penalized element to the nearest extreme element. This represents the straight-line distance from the penalized element to the diagonal. This represents the straight-line distance from the path boundary to the diagonal; It is the given base penalty coefficient.

2. The multi-channel video alignment method according to claim 1, characterized in that, Step S21 includes: S211, using the first time series The second time series was used as the data on the horizontal axis. For the vertical axis data, construct the coordinate system containing the elements of the first matrix; S212. Perform first-order differences on the first time series and the second time series respectively to obtain the first difference sequence and the second difference sequence; S213. Calculate the difference between corresponding elements of the first difference sequence and the second difference sequence to obtain the numerical value of the corresponding element in the first matrix.

3. The multi-channel video alignment method according to claim 1, characterized in that, Step S1 includes: S11. Using a given human pose detection algorithm, perform human pose detection on the first video and the second video to obtain human pose information for each frame of the first video and human pose information for each frame of the second video. S12. Convert the human pose information of each frame of each video into a feature vector, and construct the feature vectors corresponding to each frame of each video in chronological order to obtain the first time series and the second time series respectively.

4. The multi-channel video alignment method according to claim 1, characterized in that, Step S5 includes: S51, regarding the optimized path Perform DTW to determine the optimal path with the minimum cumulative distance; S52, elements in the optimal path The corresponding element in the first time series The element identified as the first alignment sequence is the optimal path. Corresponding elements in the second time series It is determined to be an element in the second alignment sequence.

5. A multi-channel video alignment device, characterized in that, The device includes: The sequence conversion module is used to extract the image feature vectors of the first video and the second video respectively, and construct the first time sequence of the first video and the second time sequence of the second video according to the time order of the image feature vectors. An alignment module is used to determine a target path based on the difference in the rate of change between the first time series and the second time series; the target path characterizes the shape features of the first time series and the second time series. The alignment module is further configured to use the target path as a shape reference to perform dynamic time warping on the first matrix composed of the first time series and the second time series to obtain the alignment path. The alignment module is also used to update the penalized elements that exceed the path boundary in the alignment path with a dynamic penalty coefficient to obtain an optimized path. The smaller the distance between the penalized element and the target path, the smaller the dynamic penalty coefficient used. The alignment module is further configured to obtain a first alignment sequence and a second alignment sequence after time alignment based on the corresponding elements of each element in the optimized path in the first time sequence and the corresponding elements in the second time sequence. The step of determining the target path based on the difference in the rate of change between the first time series and the second time series includes: S21. Calculate the difference between the first derivative of the first time series and the first derivative of the second time series to obtain the first matrix. The first matrix The values ​​of the elements in the middle represent the differences in the rates of change between the two time series; S22, Select the first matrix Construct the second matrix from the extreme elements in The elements in the second matrix The numerical value represents the first time series. The element and the second time series The distance between elements and It is a positive integer; S23. Perform dynamic time warping on the second matrix to obtain the target path; Step S23 includes: The target path with the minimum cumulative distance is determined by performing dynamic time warping on each element in the second matrix. The dynamic time warping process is expressed as follows: ； Represents the first time series The element and the second time series The distance between elements; Indicates starting from the element To element The cumulative distance; The step of using the target path as a shape reference to perform dynamic time warping on the first matrix composed of the first time series and the second time series to obtain the alignment path includes: Using multiple path elements that make up the target path Corresponding multiple extreme elements The elements of the first matrix are divided into multiple segments. Dynamic time warping is performed on each segment to determine the sub-path with the smallest cumulative distance for each segment. The sub-paths corresponding to multiple segments form the alignment path. The coordinate system containing the elements of the first matrix uses the first time series as the horizontal axis and the second time series as the vertical axis. Before updating the penalized elements in the alignment path that exceed the path boundary using a dynamic penalty coefficient to obtain the optimized path, the alignment module is also used to perform the following steps: Step A: Calculate the farthest distance from the diagonal of the coordinate system in the alignment path, where the diagonal is a ray originating from the origin with a slope of 1; Step B: Determine the path boundary as two straight lines that have the same slope as the diagonal of the coordinate system and are half the furthest distance from the diagonal of the coordinate system. The update process for the penalized element is performed according to the following formula: = ( ); Indicates the element being penalized. This represents the distance from the penalized element to the nearest extreme element. This represents the straight-line distance from the penalized element to the diagonal. This represents the straight-line distance from the path boundary to the diagonal; It is the given base penalty coefficient.

6. A computer-readable storage medium, characterized in that, The storage medium stores at least one program, which is executed by a processor to implement the multi-channel video alignment method as described in any one of claims 1 to 4.

7. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the multi-channel video alignment method according to any one of claims 1 to 4.

Images