A video alignment method, apparatus, device, and storage medium

Abstract

This disclosure provides a video alignment method, apparatus, device, and storage medium, relating to the field of image processing technology, and particularly to the fields of computer vision and multimedia. The specific implementation scheme is as follows: A video alignment method includes: acquiring multiple first images to be processed from a first video to be aligned, and acquiring multiple second images to be processed from a second video to be aligned; calculating the similarity between each first image to be processed and each second image to be processed; mapping each similarity to the element value of a corresponding element; and determining an aligned image sequence in the first and second videos to be aligned based on the element values ​​of all elements. This disclosure provides a general video alignment method.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to the fields of computer vision and multimedia. Background Technology

[0002] With the development of computer vision and multimedia information technology, the application scope of video processing is becoming more and more extensive. However, in general, video processing requires a prerequisite: video alignment. Summary of the Invention

[0003] This disclosure provides a video alignment method, apparatus, device, and storage medium.

[0004] According to a first aspect of this disclosure, a video alignment method is provided, comprising:

[0005] Obtain multiple first images to be processed from the first video to be aligned, and obtain multiple second images to be processed from the second video to be aligned;

[0006] Calculate the similarity between each first image to be processed and each second image to be processed;

[0007] Map each of the aforementioned similarities to the element value of the corresponding element;

[0008] Based on the element values ​​of all elements, the alignment image sequence in the first video to be aligned and the second video to be aligned is determined.

[0009] According to a second aspect of this disclosure, a video alignment apparatus is provided, comprising:

[0010] The acquisition module is used to acquire multiple first images to be processed in the first video to be aligned, and to acquire multiple second images to be processed in the second video to be aligned;

[0011] The calculation module is used to calculate the similarity between each first image to be processed and each second image to be processed;

[0012] A mapping module is used to map each similarity to the element value of the corresponding element;

[0013] The determination module is used to determine the alignment image sequence in the first video to be aligned and the second video to be aligned based on the element values ​​of all elements.

[0014] According to a third aspect of this disclosure, an electronic device is provided, comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0018] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method according to the first aspect.

[0019] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method according to the first aspect.

[0020] This disclosure provides a general method for video alignment.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0022] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0023] Figure 1 This is a flowchart of the video alignment method provided in the embodiments of this disclosure;

[0024] Figure 2 This is a schematic diagram illustrating the video alignment method provided in the embodiments of this disclosure;

[0025] Figure 3 This is a schematic diagram of the structure of the video alignment device provided in the embodiments of this disclosure;

[0026] Figure 4 This is a block diagram of an electronic device used to implement the video alignment method of the embodiments of this disclosure. Detailed Implementation

[0027] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0028] Video alignment has applications in many fields. For example, in video quality assessment, changes and quality of videos with identical content are typically measured and evaluated using both subjective and objective methods. A crucial prerequisite for this evaluation is aligning the content of corresponding video frames in the damaged video and the source video. Similarly, in video super-resolution, recovering high-resolution images from low-resolution images improves image quality and yields clearer pictures; this technique also requires aligning consecutive video frames. Furthermore, in video plagiarism detection, in the era of widespread social media, determining whether a video infringes on copyright involves finding matching segments between the source and target video. All of these fields require video alignment as a foundation for subsequent research.

[0029] This disclosure provides a video alignment method, which may include:

[0030] Obtain multiple first images to be processed from the first video to be aligned, and obtain multiple second images to be processed from the second video to be aligned;

[0031] Calculate the similarity between each first image to be processed and each second image to be processed;

[0032] Map each similarity score to the element value of the corresponding element;

[0033] Based on the element values ​​of all elements, determine the alignment image sequence in the first and second videos to be aligned.

[0034] In this embodiment, the similarity between each first image to be processed and each second image to be processed is calculated; each similarity is mapped to the element value of the corresponding element; based on the element values ​​of all elements, the alignment image sequence in the first and second videos to be aligned can be determined, which can support video alignment of any type of video and provides a general video alignment method.

[0035] Figure 1 This is a flowchart of the video alignment method provided in an embodiment of this disclosure. (Refer to...) Figure 1 The video alignment method provided in this disclosure may include:

[0036] S101, acquire multiple first images to be processed from the first video to be aligned, and acquire multiple second images to be processed from the second video to be aligned.

[0037] In this embodiment of the disclosure, all images in the video to be aligned can be processed, or the images to be processed in each video to be aligned can be obtained by extracting frames from the video.

[0038] In one possible implementation, S101 may include:

[0039] Obtain the first frame rate of the first video to be aligned; obtain the second frame rate of the second video to be aligned; obtain a preset interval; use the ratio of the first frame rate to the preset interval as the first frame extraction interval, and extract frames from the first video to be aligned according to the first frame extraction interval to obtain multiple first images to be processed; use the ratio of the second frame rate to the preset interval as the second frame extraction interval, and extract frames from the second video to be aligned according to the second frame extraction interval to obtain multiple second images to be processed.

[0040] The preset interval can be determined according to the actual situation.

[0041] For example, the preset interval is t. t can be flexibly set according to the actual situation. For example, the larger t is set, the smaller the frame extraction interval, the more accurate the result, and the higher the amount of computation. t can be set based on the accuracy of the result and the amount of computation.

[0042] Obtain the frame rate (FPS1) of the first video to be aligned: Video 1, and obtain the frame rate (FPS2) of the second video to be aligned: Video 2. Extract frames from Video 1 at an interval of FPS1 / t to obtain multiple images to be processed, such as an ordered image set img1; extract frames from Video 2 at an interval of FPS2 / t to obtain multiple images to be processed, such as an ordered image set img2.

[0043] In this embodiment of the disclosure, the frame rate of the video to be aligned is used to extract frames from the video to be aligned by a ratio of the frame rate to a preset interval, which can make the frame extraction more uniform and facilitate subsequent video alignment.

[0044] S102, calculate the similarity between each first image to be processed and each second image to be processed.

[0045] This disclosure does not limit the method of calculating similarity. Any method that can calculate the similarity between two images is within the protection scope of this disclosure.

[0046] In practical applications, the image similarity algorithm for calculating the similarity between each first image to be processed and each second image to be processed can be flexibly selected according to the usage requirements. For example, if the goal is to align the damaged video and the source video, i.e., the first and second images to be processed are images from the damaged video and the source video respectively, the Structural Similarity (SSIM) algorithm or the Peak Signal-to-Noise Ratio (PSNR) algorithm can be selected. In other cases, hash algorithms such as ahash (mean hash algorithm), dhash (image difference hash algorithm), and phash (image perceptual hash algorithm) can be selected according to the usage requirements; histogram, oriented keypoint detection and Oriented Fast and Rotated BRIEF (ORB) feature extraction, cosine similarity algorithm, and other image similarity algorithms can also be selected.

[0047] In one possible implementation, after extracting frames to obtain an image set img1 containing multiple first images to be processed and an image set img2 containing multiple second images to be processed, each pixel cell and its value in each first image to be processed in img1 can be traversed and saved to a three-dimensional array corresponding to the three channels of the Blue Green Red (BGR) pattern. Then, the BGR array of all pixels in each image in image set img1 is saved to array 1. Similarly, for each second image to be processed in img2, each pixel cell and its value in each image cell can be traversed and saved to a three-dimensional array corresponding to the three channels of the BGR pattern. Then, the BGR array of all pixels in each image in image set img2 is saved to array 2. Thus, when calculating the similarity between each first image to be processed and each second image to be processed, the BGR array of all pixels of the first image to be processed and the BGR array of all pixels of the second image to be processed can be taken from array 1 and array 2 respectively. For a first image to be processed and a second image to be processed, the similarity between the first image to be processed and the second image to be processed can be calculated based on the BGR array of the first image to be processed and the BGR array of the second image to be processed.

[0048] S103, map each similarity to the element value of the corresponding element.

[0049] A similarity can be calculated between a first image to be processed and a second image to be processed. The element corresponding to this similarity is the element corresponding to the first image to be processed and the second image to be processed.

[0050] For each similarity score, the similarity score can be directly used as the element value, or the similarity score can be mapped to the element value according to a preset mapping rule. For example, if the similarity score is less than a preset threshold, the element value of the element corresponding to the similarity score is 0; if the similarity score is not less than the preset threshold, the element value of the element is determined based on the element values ​​of the elements corresponding to the preceding first image of the first image to be processed and the preceding second image of the second image to be processed. For example, the element value of the element corresponding to the preceding first image of the first image to be processed and the preceding second image of the second image to be processed is increased by a preset value to obtain the element value of the element.

[0051] S104, Based on the element values ​​of all elements, determine the alignment image sequence in the first and second videos to be aligned.

[0052] Each element corresponds to an image in the first video to be aligned and an image in the second video to be aligned. The alignment image sequence in the first and second videos to be aligned can be determined based on the relationship between the element values ​​of multiple elements. Specifically, the process of determining the alignment image sequence in the first and second videos to be aligned based on the relationship between the element values ​​of multiple elements will be described in detail in the following embodiments, and will not be repeated here.

[0053] The aligned image sequence may include a first aligned image sequence and a second aligned image sequence. The first aligned image sequence is an image sequence in the first video to be aligned that is aligned with the second video to be aligned, and the second aligned image sequence is an image sequence in the second video to be aligned that is aligned with the first video to be aligned. That is, the first aligned image sequence and the second video to be aligned are aligned with each other.

[0054] The first and second aligned image sequences can be understood as the video alignment results. Alternatively, the times of each image in the first aligned image sequence within the first video to be aligned, and the times of each image in the second aligned image sequence within the second video to be aligned, can be obtained, and the times of each image in the first and second aligned image sequences within the second video to be aligned can be used as the video alignment results. Specifically, the time of the starting frame in the first aligned image sequence can be used as the first starting frame time, i.e., the starting time in the first alignment sequence, and the time of the ending frame in the first aligned image sequence can be used as the first ending frame time, i.e., the ending time in the first alignment sequence; similarly, the time of the starting frame in the second aligned image sequence can be used as the second starting frame time, i.e., the starting time in the second alignment sequence, and the time of the ending frame in the second aligned image sequence can be used as the second ending frame time, i.e., the ending time in the second aligned image sequence.

[0055] Specifically, after obtaining each first image to be processed, the time of each first image to be processed in the first video to be aligned can be obtained, and after obtaining each second image to be processed, the time of each second image to be processed in the second video to be aligned can be obtained. The times of each first image to be processed in the first video to be aligned and the times of each second image to be processed in the second video to be aligned are stored. For example, the time of each first image to be processed in the first video to be aligned can be stored in hash table map1, and the time of each second image to be processed in the second video to be aligned can be stored in hash table map2. Thus, after obtaining the first aligned image sequence and the second aligned image sequence, the times of the first images to be processed in the first aligned image sequence and the times of the second images to be processed in the second aligned image sequence can be obtained in the second video to be aligned. The times of each image in the first aligned image sequence and the times of each image in the second aligned image sequence in the second video to be aligned are used as the video alignment result.

[0056] This disclosure provides a general video alignment method that can support the alignment of any two videos, such as aligning a damaged video and the source video, aligning videos before and after compression, aligning the beginning and end of similar videos, etc. Its application areas include, but are not limited to, video super-resolution, video restoration, video quality assessment, etc., and it can support the alignment of videos with different resolutions and frame rates without additional processing.

[0057] In one optional embodiment, S103 may include:

[0058] Image groups are selected sequentially according to the order of each first image to be processed in the first video to be aligned, and the order of each second image to be processed in the second video to be aligned. Each image group includes one first image to be processed and one second image to be processed, and each image group corresponds to one element. For each image group, the element value of the element corresponding to the image group is determined based on the similarity between the first image to be processed and the second image to be processed.

[0059] It is understandable that, according to the order of each first image to be processed in the first video to be aligned, and the order of each second image to be processed in the second video to be aligned, each first image to be processed and each second image to be processed are traversed, and one first image to be processed and one second image to be processed are selected in turn as an image group. Based on the similarity between the first image to be processed and the second image to be processed, the element value of the corresponding element of the image group is determined.

[0060] In this way, the element value of a first image to be processed and a second image to be processed can be conveniently determined as an element of an image group, according to the order of each first image to be processed in the first video to be aligned and the order of each second image to be processed in the second video to be aligned.

[0061] In one alternative embodiment, the elements are matrix elements in the image matching matrix.

[0062] Based on the similarity between the first and second images to be processed, the element values ​​of the corresponding elements in the image group are determined, including:

[0063] Based on the order of the first image to be processed in the first video to be aligned, the row number of the matrix element corresponding to the image group in the image matching matrix is ​​determined; based on the order of the second image to be processed in the second video to be aligned, the column number of the matrix element corresponding to the image group in the image matching matrix is ​​determined; based on the similarity between the first image to be processed and the second image to be processed, the element value of the matrix element corresponding to the row number and column number is determined.

[0064] In this way, multiple image matching matrices can be constructed corresponding to the first and second images to be processed. Each element of the image matching matrix corresponds to a first image to be processed and a second image to be processed. The image matching matrix is ​​a two-dimensional array.

[0065] Simply put, a first image to be processed and a second image to be processed correspond to a matrix element in the image matching matrix. The number of rows in the image matching matrix is ​​determined based on the number of first images to be processed, and the number of columns is determined based on the number of second images to be processed. For example, the size of the image matching matrix can be len1*len2 or (len1+1)*(len2+1), where len1 is the number of first images to be processed and len2 is the number of second images to be processed.

[0066] For example, for the nth first image to be processed, map this first image to be processed to each of the second images to be processed to the matrix elements in the (n-1)th row of the image matching matrix, and map the first image to be processed to the mth second image to be processed to the matrix elements in the (n-1)th row and (m-1)th column of the image matching matrix. Here, n ranges from 1 to N, where N is the number of first images to be processed, and m ranges from 1 to M, where M is the number of second images to be processed. The size of the image matching matrix obtained in this way is len1*len2. Alternatively, for the nth first image to be processed, map this first image to be processed to each of the second images to be processed to the matrix elements in the nth row of the image matching matrix, and map the first image to be processed to the mth second image to be processed to the matrix elements in the nth row and (m)th column of the image matching matrix. The size of the image matching matrix obtained in this way is (len1+1)*(len2+1). In this method, the element values ​​of the matrix elements in the 0th row and 0th column of the image matching matrix are 0.

[0067] In one example, there are two first images to be processed and three second images to be processed. For the first image to be processed, this first image to be processed is mapped to each of the second images to be processed, and then to the matrix elements in the 0th row of the image matching matrix. For example, the first image to be processed and the first second image to be processed are mapped to matrix element a. 00 And determine a based on the similarity between the first image to be processed and the first image to be processed. 00 The element values; mapping the first image to be processed and the second image to be processed to matrix element a. 01 And determine a based on the similarity between the first image to be processed and the second image to be processed. 01 The element values; mapping the first image to be processed and the third image to be processed to matrix element a. 02 And determine a based on the similarity between the first image to be processed and the third image to be processed. 02 The element values; mapping the second first image to be processed and the first second image to be processed to matrix element a. 10 And based on the similarity between the second first image to be processed and the first second image to be processed, a is determined. 10 The element values; mapping the second first image to be processed and the second second image to be processed to matrix element a. 11 And based on the similarity between the second first image to be processed and the second second image to be processed, a is determined. 11 The element values; mapping the second first image to be processed and the third second image to be processed to matrix element a. 12And based on the similarity between the second first image to be processed and the third second image to be processed, a is determined. 12 The element value.

[0068] Alternatively, set the element values ​​of the 0th row and 0th column of the image matching matrix to 0, i.e., a 00 a 01 a 02 a 10 a 11 a 12 The element value is 0, which maps the first image to be processed and the first image to be processed to matrix element a. 11 And determine a based on the similarity between the first image to be processed and the first image to be processed. 11 The element values; mapping the first image to be processed and the second image to be processed to matrix element a. 12 And determine a based on the similarity between the first image to be processed and the second image to be processed. 12 The element values; mapping the first image to be processed and the third image to be processed to matrix element a. 13 And determine a based on the similarity between the first image to be processed and the third image to be processed. 13 The element values; mapping the second first image to be processed and the first second image to be processed to matrix element a. 21 And based on the similarity between the second first image to be processed and the first second image to be processed, a is determined. 21 The element values; mapping the second first image to be processed and the second second image to be processed to matrix element a. 22 And based on the similarity between the second first image to be processed and the second second image to be processed, a is determined. 22 The element values; mapping the second first image to be processed and the third second image to be processed to matrix element a. 23 And based on the similarity between the second first image to be processed and the third second image to be processed, a is determined. 23 The element value.

[0069] The matrix elements can intuitively and conveniently represent each first image to be processed and each second image to be processed. The matrix elements can be used to associate each first image to be processed with each second image to be processed, so as to conveniently determine the element value corresponding to each matrix element.

[0070] In one optional embodiment, determining the element values ​​of the matrix elements corresponding to the row and column numbers based on the similarity between the first image to be processed and the second image to be processed includes:

[0071] The similarity between the first image to be processed and the second image to be processed is compared with a preset similarity threshold. If the similarity is less than the preset similarity threshold, the element value of the matrix element corresponding to the row number and column number is the first preset value. If the similarity is not less than the preset similarity threshold, the element value of the matrix element corresponding to the row number and column number is determined by using the element value of the matrix element corresponding to the row number obtained by subtracting 1 from the row number and the column number obtained by subtracting 1 from the column number.

[0072] The preset similarity threshold can be determined based on actual needs or experience.

[0073] The first preset value can be determined according to the actual situation. In one possible implementation, the first preset value is 0.

[0074] Specifically, the element value of the matrix element corresponding to the row number obtained by subtracting 1 from the row number and the column number obtained by subtracting 1 from the column number is used to determine the element value of the matrix element corresponding to the row number and the column number. This can also be understood as determining the element value of the matrix element based on the element value of the matrix element corresponding to the first image to be processed before the first image to be processed corresponding to the first image to be processed and the element value of the matrix element corresponding to the second image to be processed before the second image to be processed corresponding to the second image to be processed.

[0075] For example, the image matching matrix is ​​obtained by mapping the nth first image to be processed to each of the matrix elements in the nth row of the image matching matrix, and mapping the first image to be processed to the mth second image to the matrix element in the mth column of the nth row. For the first image to be processed (image i) and the second image to be processed (image j), the row number of the matrix element is the same as the index of the first image to be processed in the first video to be aligned, and the column number of the matrix element is the same as the index of the second image to be processed in the second video to be aligned. Images i and j correspond to the matrix element array[i][j] in the image matching matrix.

[0076] In determining the element of a matrix, a comparison with a preset similarity threshold is made, and different element value determination methods are used for different comparison results. This allows for convenient and rapid determination of the element values ​​of each matrix element. Furthermore, for cases where the similarity is not less than the preset similarity threshold, a correlation is established between the element values ​​of different matrix elements to facilitate the subsequent determination of the image alignment sequence.

[0077] In an optional embodiment, the element values ​​of the matrix elements corresponding to the row number and column number are determined using the element values ​​of the matrix elements corresponding to the row number minus 1 and the column number minus 1, respectively. This includes:

[0078] The element values ​​of the matrix elements corresponding to the row number (minus 1) and column number (minus 1) are increased by a second preset value to obtain the element values ​​of the matrix elements corresponding to the row number and column number.

[0079] The second preset value can be determined according to the actual situation. For example, the second preset value is a natural number. In one possible implementation, the second preset value is 1.

[0080] For example, determining the element values ​​of a matrix element based on a formula;

[0081]

[0082] Here, threshold represents the preset similarity threshold, and Similar(i,j) represents the similarity between image i and image j.

[0083] If the similarity between image i and image j is less than the preset similarity threshold, then array[i][j] = 0; if the similarity between image i and image j is not less than the preset similarity threshold, then array[i][j] = array[i-1][j-1] + 1.

[0084] For cases where the similarity is not less than a preset threshold, the element value of the element can be quickly and easily determined based on the element value of the element corresponding to the element in the preceding first image of the first image to be processed and the element corresponding to the element in the preceding second image of the second image to be processed.

[0085] In one alternative embodiment, S104 may include two methods:

[0086] The first method is to arrange all elements to obtain multiple element sequences, each containing at least two elements; select the element sequence in which multiple element values ​​are consecutively greater than a third preset value, and take the element sequence with the most elements in the multiple element sequences where multiple element values ​​are consecutively greater than the third preset value as the target element sequence; based on the target element sequence, determine the alignment image sequence in the first and second videos to be aligned.

[0087] The third preset value can be determined based on actual needs or experience. In one possible implementation, the third preset value is the same as the first preset value, which is 0.

[0088] The second method is to arrange all elements to obtain multiple element sequences, each containing at least two elements; select the element sequence with the most element values ​​greater than a third preset value as the target element sequence; and determine the alignment image sequence in the first and second videos to be aligned based on the target element sequence.

[0089] The process of arranging all elements to obtain multiple element sequences may include: firstly constructing multiple first image sequences corresponding to first images to be processed and multiple second image sequences corresponding to second images to be processed, each first image sequence and second image sequence including at least two images; then, for each first image sequence and each second image sequence, selecting one first image to be processed from the first image sequence and determining the order of the first image to be processed in the first image sequence; selecting a second image to be processed in the same order from the second image sequence; and constructing an element sequence from the elements corresponding to the first image to be processed and the second image to be processed.

[0090] For example, given two first images to be processed and three second images to be processed, the first image sequence corresponding to the two first images to be processed includes A(1,2), and the second image sequence corresponding to the three second images to be processed includes B(1,2), B(2,3), and B(1,2,3). For A(1,2) and B(1,2), the element ARRAY corresponding to the first first image to be processed and the first second image to be processed is selected. 11 The ARRAY element corresponding to the second first image to be processed and the second second image to be processed. 22 For A(1,2) and B(2,3), select the element ARRAY corresponding to the first image to be processed and the second image to be processed. 12 The ARRAY element corresponding to the second first image to be processed and the third second image to be processed. 23 .

[0091] After obtaining the element sequence, you can select the element sequence in which multiple element values ​​are continuously greater than the third preset value, and take the element sequence with the most elements in the element sequence in which multiple element values ​​are continuously greater than the third preset value as the target element sequence; and / or select the element sequence in which the number of element values ​​is greater than the third preset value in all element sequences as the target element sequence. After obtaining the target element sequence, you can determine the alignment image sequence in the first video to be aligned and the second video to be aligned based on the target element sequence.

[0092] In this embodiment of the disclosure, the alignment image sequence in the first and second videos to be aligned can be conveniently determined based on the element values ​​of the elements.

[0093] The first method can be understood as precise alignment, and the aligned image sequence determined using the first method can be called a precisely aligned image sequence. The second method can be understood as fuzzy alignment, and the aligned image sequence determined using the second method can be called a fuzzy aligned image sequence.

[0094] In this embodiment of the disclosure, for the first video to be aligned and the second video to be aligned, the alignment image sequence can be determined using a first method and a second method respectively, resulting in a precisely aligned image sequence and a fuzzy aligned image sequence. Alternatively, the precisely aligned image sequence can be obtained using only the first method. Alternatively, the fuzzy aligned image sequence can be obtained using only the second method.

[0095] In one possible implementation, it can be determined whether the first and second videos to be aligned have dropped frames, skipped frames, and / or pauses. If the first and second videos to be aligned do not have dropped frames, skipped frames, and / or pauses, then a first method can be used to precisely align the first and second videos to be aligned. If the first and second videos to be aligned do have dropped frames, skipped frames, and / or pauses, then a first method can be used to perform a blurry alignment. Thus, video alignment can be achieved even when the first and second videos to be aligned have dropped frames, skipped frames, and / or pauses, expanding the application scope of video alignment. Furthermore, precise alignment can be performed when the first and second videos to be aligned do not have dropped frames, skipped frames, and / or pauses, improving the accuracy of video alignment.

[0096] In an optional embodiment, for the case where the elements are matrix elements in the image matching matrix, the first method selects an element sequence from all element sequences where multiple element values ​​are consecutively greater than a third preset value, and takes the element sequence with the most elements among the multiple element sequences where multiple element values ​​are consecutively greater than the third preset value as the target element sequence, including:

[0097] Find the maximum element value among all matrix elements; determine the first target matrix element diagonal line containing the maximum element value; find a continuous element sequence in the element sequence corresponding to the first target matrix element diagonal line where multiple element values ​​are continuously greater than a third preset value, and the continuous element sequence is the target element sequence.

[0098] The third preset value can be determined according to actual needs. In one possible implementation, the third preset value is 0.

[0099] A matrix element slash can be understood as a slash connecting matrix elements. Matrix element slashes can include lines connecting matrix elements with the same row and column numbers; lines connecting matrix elements starting from each element in the first row (excluding the first element), with row numbers increasing sequentially and column numbers greater than 1; and lines connecting matrix elements starting from each element in the first column (excluding the first element), with row numbers increasing sequentially and column numbers less than 1.

[0100] The first starting frame, the first ending frame, the second starting frame, and the second ending frame are determined based on the target element sequence; the image sequence from the first starting frame to the first ending frame in multiple first images to be processed is taken as the first aligned image sequence; the image sequence from the second starting frame to the second ending frame in multiple second images to be processed is taken as the second aligned image sequence.

[0101] By determining the first start frame, the first end frame, the second start frame, and the second end frame, and taking the image sequence from the first start frame to the first end frame as the first aligned image sequence, and the image sequence from the second start frame to the second end frame as the second aligned image sequence, not only can the start frame and end frame of the aligned image sequence be determined, but the aligned image sequence can also be determined quickly.

[0102] The first method, which determines the first starting frame, the first ending frame, the second starting frame, and the second ending frame based on the target element sequence, includes:

[0103] The first image to be processed corresponding to the first matrix element with a value greater than the third preset value in the continuous element sequence is taken as the first starting frame, and the first image to be processed corresponding to the matrix element with the largest element value is taken as the first ending frame; the second image to be processed corresponding to the first matrix element with a value greater than the third preset value in the continuous element sequence is taken as the second starting frame, and the second image to be processed corresponding to the matrix element with the largest element value is taken as the second ending frame.

[0104] The image sequence from the first starting frame to the first ending frame in a plurality of first images to be processed is taken as the first aligned image sequence; the image sequence from the second starting frame to the second ending frame in a plurality of second images to be processed is taken as the image sequence in the second video to be aligned that is aligned with the first aligned image sequence.

[0105] For example, frames are extracted to obtain an image set img1 containing multiple first images to be processed and an image set img2 containing multiple second images to be processed. The element values ​​of the matrix are determined by the following method: if the similarity between image i and image j is less than a preset similarity threshold, then array[i][j] = 0; if the similarity between image i and image j is not less than the preset similarity threshold, then array[i][j] = array[i-1][j-1] + 1, so as to obtain the image matching matrix array for img1 and img2.

[0106] If no image in either img1 or img2 has a similarity score that meets the preset threshold (i.e., all elements in the array have a value of 0), then the two videos are completely inconsistent and cannot be aligned.

[0107] If not all elements in the array are zero, find the largest element value, `max`. The longest consecutive sequence of similar images for `img1` and `img2` lies within the diagonal line of the array containing the maximum value `max`. Specifically, the rows and columns of the array containing values ​​1 (from 1 to `max`) on this diagonal line correspond to the starting frames of `img1` and `img2`, respectively, while the rows and columns containing values ​​of `max` correspond to the ending frames of `img1` and `img2`, respectively.

[0108] like Figure 2 As shown, the maximum value in array is 13. The diagonal line of the matrix element containing array 13, i.e., the diagonal line of the first target matrix element, is shown as diagonal line 201 in the figure. The selected frames from circle 1 to circle 13 on this diagonal line are the consecutively aligned frames of the two video image sets, which can also be understood as the longest consecutive similar image sequence. The row and column of the array containing circle 1 are 8 and 7 respectively, and the row and column of the array containing circle 13 are 20 and 19 respectively. That is, the result of the precise alignment of image set img1 and image set img2 is: the 8th image in img1 is the starting alignment image, the 7th image in img2 is the starting alignment image, the 20th image in img1 is the tail alignment image, and the 19th image in img2 is the tail alignment image. Here, the starting alignment image is also the starting frame, and the tail alignment image is also the tail frame. The sequence of images 8 to 20 in img1 and images 7 to 19 in img2 are the precise alignment image sequences of the two image sets.

[0109] By identifying the diagonal line of the first target matrix element containing the maximum element value, and then finding a continuous sequence of elements in the element sequence corresponding to the diagonal line of the first target matrix element where multiple element values ​​are consecutively greater than a third preset value, a continuous element sequence can be quickly determined, reducing computational load compared to traversing all element sequences. Furthermore, by identifying a continuous sequence of elements where multiple element values ​​are consecutively greater than the third preset value as the target element sequence, a continuous alignment image sequence can be determined, enabling rapid and accurate alignment.

[0110] In an optional embodiment, for the case where the elements are matrix elements in the image matching matrix, the second method selects the element sequence with the largest number of element values ​​greater than a third preset value as the target element sequence, including:

[0111] The second target matrix element diagonal line with the most element values ​​greater than the third preset value is identified. The element sequence corresponding to the second target matrix element diagonal line is taken as the longest similar element sequence, and the longest similar element sequence is taken as the target element sequence.

[0112] The first starting frame, the first ending frame, the second starting frame, and the second ending frame are determined based on the target element sequence; the image sequence from the first starting frame to the first ending frame in multiple first images to be processed is taken as the first aligned image sequence; the image sequence from the second starting frame to the second ending frame in multiple second images to be processed is taken as the image sequence in the second video to be aligned and aligned with the first aligned image sequence.

[0113] The second method determines the first starting frame, the first ending frame, the second starting frame, and the second ending frame based on the target element sequence, including:

[0114] Find the matrix element whose first element value is greater than the third preset value in the longest similar element sequence, and the matrix element whose last element value is greater than the third preset value in the longest similar element sequence; take the first image to be processed corresponding to the matrix element whose first element value is greater than the third preset value in the longest similar element sequence as the first starting frame, and take the first image to be processed corresponding to the matrix element whose last element value is greater than the third preset value in the longest similar element sequence as the first ending frame; take the second image to be processed corresponding to the matrix element whose first element value is greater than the third preset value in the longest similar element sequence as the second starting frame, and take the second image to be processed corresponding to the matrix element whose last element value is greater than the third preset value in the longest similar element sequence as the second ending frame.

[0115] The image sequence from the first starting frame to the first ending frame in a plurality of first images to be processed is taken as the first aligned image sequence; the image sequence from the second starting frame to the second ending frame in a plurality of second images to be processed is taken as the image sequence in the second video to be aligned that is aligned with the first aligned image sequence.

[0116] For example, frames are extracted to obtain an image set img1 containing multiple first images to be processed and an image set img2 containing multiple second images to be processed. The element values ​​of the matrix are determined by the following method: if the similarity between image i and image j is less than a preset similarity threshold, then array[i][j] = 0; if the similarity between image i and image j is not less than the preset similarity threshold, then array[i][j] = array[i-1][j-1] + 1, so as to obtain the image matching matrix array for img1 and img2.

[0117] If no image in either img1 or img2 has a similarity score that meets the preset threshold (i.e., all elements in the array have a value of 0), then the two videos are completely inconsistent and cannot be aligned.

[0118] If not all values ​​in array are 0, the longest similarity sequence of img1 and img2 lies on the diagonal line with the most positive values ​​in the matrix array. The row and column of the first positive number in the diagonal line with the most positive values ​​correspond to the starting frames of img1 and img2, respectively, and the row and column of the last positive number correspond to the ending frames of img1 and img2, respectively. The image sequences from the starting frame to the ending frame of each of the two image sets are then the blurred alignment sequences of img1 and img2.

[0119] like Figure 2 As shown, the diagonal line with the most values ​​greater than 0 in the array, i.e., the diagonal line of the second target matrix element (diagonal line 201 in the figure), represents the longest similar image sequence between img1 and img2, which is the part of the two videos that are blurred and aligned. The row and column of the array containing the first box are 2 and 1 respectively, and the row and column of the array containing the second box are 24 and 23 respectively. That is, the result of blur alignment between img1 and the image set img2 is as follows: the second image in img1 is the starting alignment image, the first image in img2 is the starting alignment image, the 24th image in img1 is the tail alignment image, and the 23rd image in img2 is the tail alignment image. Here, the starting alignment image is also the starting frame, and the tail alignment image is also the tail frame. The image sequence of img1 from the 2nd to the 24th and the image sequence of img2 from the 1st to the 23rd are the blurred alignment image sequences of the two image sets.

[0120] By finding the sequence of elements corresponding to the diagonal line of the second target matrix element with the most element values ​​greater than a third preset value, the longest similar element sequence can be quickly determined, reducing computational cost compared to traversing all element sequences. Furthermore, finding the longest similar element sequence as the target element sequence allows for determination of the target element sequence in any situation, enabling the determination of aligned image sequences based on the target element sequence. For example, it can also be used to align videos in cases of dropped frames, skipped frames, and / or pauses.

[0121] This disclosure provides two methods for video alignment: precise alignment and fuzzy alignment. Alignment can also be achieved in cases of dropped frames, skipped frames, and / or stagnation, thus expanding the scope of video alignment.

[0122] Corresponding to the video alignment method provided in the above embodiments, this disclosure also provides a video alignment device, such as... Figure 3 As shown, it may include:

[0123] The acquisition module 301 is used to acquire multiple first images to be processed in the first video to be aligned, and to acquire multiple second images to be processed in the second video to be aligned;

[0124] The calculation module 302 is used to calculate the similarity between each first image to be processed and each second image to be processed;

[0125] The mapping module 303 is used to map each similarity to the element value of the corresponding element;

[0126] The determination module 304 is used to determine the alignment image sequence in the first and second videos to be aligned based on the element values ​​of all elements.

[0127] Optionally, the mapping module 303 is specifically used to select image groups sequentially according to the order of each first image to be processed in the first video to be aligned, and the order of each second image to be processed in the second video to be aligned. Each image group includes a first image to be processed and a second image to be processed, and each image group corresponds to an element. For each image group, the element value of the element corresponding to the image group is determined based on the similarity between the first image to be processed and the second image to be processed.

[0128] Optionally, the elements are matrix elements in the image matching matrix;

[0129] The mapping module 303 is specifically used to determine the row number of the matrix element corresponding to the image group in the image matching matrix based on the order of the first image to be processed in the first video to be aligned; to determine the column number of the matrix element corresponding to the image group in the image matching matrix based on the order of the second image to be processed in the second video to be aligned; and to determine the element value of the matrix element corresponding to the row number and column number based on the similarity between the first image to be processed and the second image to be processed.

[0130] Optionally, the mapping module 303 is specifically used to compare the similarity between the first image to be processed and the second image to be processed with a preset similarity threshold; if the similarity is less than the preset similarity threshold, the element value of the matrix element corresponding to the row number and column number is the first preset value; if the similarity is not less than the preset similarity threshold, the element value of the matrix element corresponding to the row number and column number is determined by using the element value of the matrix element corresponding to the row number obtained by subtracting 1 from the row number and the column number obtained by subtracting 1 from the column number.

[0131] Optionally, the determining module 304 is specifically used to arrange all elements to obtain multiple element sequences, each containing at least two elements; select an element sequence from all element sequences whose multiple element values ​​are consecutively greater than a third preset value, and take the element sequence with the most elements among the multiple element sequences whose multiple element values ​​are consecutively greater than the third preset value as the target element sequence; and / or, select the element sequence from all element sequences whose element values ​​are greater than the third preset value the most, and take it as the target element sequence; and based on the target element sequence, determine the alignment image sequence in the first video to be aligned and the second video to be aligned.

[0132] Optionally, the elements are matrix elements in the image matching matrix;

[0133] The determination module 304 is specifically used to find the maximum element value among all matrix element values; determine the first target matrix element diagonal line where the maximum element value is located; and find a continuous element sequence in the element sequence corresponding to the first target matrix element diagonal line where multiple element values ​​are continuously greater than a third preset value, and the continuous element sequence is the target element sequence.

[0134] Optionally, the determining module 304 is specifically used to determine the second target matrix element diagonal with the largest number of element values ​​greater than a third preset value; and to take the element sequence corresponding to the second target matrix element diagonal as the longest similar element sequence, and the longest similar element sequence as the target element sequence.

[0135] Optionally, the determining module 304 is specifically used to determine a first starting frame, a first ending frame, a second starting frame, and a second ending frame based on the target element sequence; to take the image sequence from the first starting frame to the first ending frame in a plurality of first images to be processed as a first aligned image sequence; and to take the image sequence from the second starting frame to the second ending frame in a plurality of second images to be processed as a second aligned image sequence.

[0136] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0137] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0138] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0139] like Figure 4As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 402 or a computer program loaded from storage unit 408 into random access memory (RAM) 403. RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0140] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0141] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as video alignment methods. For example, in some embodiments, the video alignment method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the video alignment method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the video alignment method by any other suitable means (e.g., by means of firmware).

[0142] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0144] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0146] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0147] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0148] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0149] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video alignment method, comprising: Get the first frame rate of the first video to be aligned; Obtain the second frame rate of the second video to be aligned; Get the preset interval; The ratio of the first frame rate to the preset interval is used as the first frame extraction interval, and the first video to be aligned is frame-sampling according to the first frame extraction interval to obtain multiple first images to be processed; the ratio of the second frame rate to the preset interval is used as the second frame extraction interval, and the second video to be aligned is frame-sampling according to the second frame extraction interval to obtain multiple second images to be processed. Calculate the similarity between each first image to be processed and each second image to be processed; Image groups are selected sequentially according to the order of each first image to be processed in the first video to be aligned, and the order of each second image to be processed in the second video to be aligned. Each image group includes one first image to be processed and one second image to be processed. Each image group corresponds to one element, which is a matrix element in the image matching matrix. For each image group, the row number of the matrix element corresponding to the image group in the image matching matrix is ​​determined based on the order of the first image to be processed in the first video to be aligned. Based on the order of the second image to be processed in the second video to be aligned, determine the column number of the matrix element corresponding to the image group in the image matching matrix; The similarity between the first image to be processed and the second image to be processed is compared with a preset similarity threshold; If the similarity is less than the preset similarity threshold, then the element values ​​of the matrix elements corresponding to the row number and the column number are the first preset values; If the similarity is not less than the preset similarity threshold, then the element values ​​of the matrix elements corresponding to the row number obtained by subtracting 1 from the row number and the column number obtained by subtracting 1 from the column number are increased by a second preset value to obtain the element values ​​of the matrix elements corresponding to the row number and the column number. Arrange all elements to obtain multiple element sequences, each containing at least two elements; Find the maximum element value among all matrix element values; determine the first target matrix element diagonal line where the maximum element value is located; find a continuous element sequence in the element sequence corresponding to the first target matrix element diagonal line where multiple element values ​​are continuously greater than a third preset value, the continuous element sequence being the target element sequence; and / or, select the element sequence with the most element values ​​greater than the third preset value among all element sequences as the target element sequence; Based on the target element sequence, the alignment image sequence in the first video to be aligned and the second video to be aligned is determined.

2. The method according to claim 1, wherein, The step of selecting the element sequence with the most element values ​​greater than a third preset value from all element sequences as the target element sequence includes: The diagonal line of the second target matrix element whose element value is greater than the third preset value is determined; The sequence of elements corresponding to the diagonal lines of the second target matrix is ​​taken as the longest similar element sequence, and the longest similar element sequence is the target element sequence.

3. The method according to claim 1, wherein, The step of determining the alignment image sequence in the first video to be aligned and the second video to be aligned based on the target element sequence includes: The first starting frame, the first ending frame, the second starting frame, and the second ending frame are determined based on the target element sequence. The image sequence from the first starting frame to the first ending frame in a plurality of first images to be processed is taken as the first aligned image sequence; The image sequence from the second starting frame to the second ending frame in a plurality of second images to be processed is used as the second aligned image sequence.

4. A video alignment device, comprising: The acquisition module is used to acquire the first frame rate of the first video to be aligned; Obtain the second frame rate of the second video to be aligned; Get the preset interval; The ratio of the first frame rate to the preset interval is used as the first frame extraction interval, and the first video to be aligned is frame-sampling according to the first frame extraction interval to obtain multiple first images to be processed; the ratio of the second frame rate to the preset interval is used as the second frame extraction interval, and the second video to be aligned is frame-sampling according to the second frame extraction interval to obtain multiple second images to be processed. The calculation module is used to calculate the similarity between each first image to be processed and each second image to be processed; The mapping module is used to sequentially select image groups according to the order of each first image to be processed in the first video to be aligned and the order of each second image to be processed in the second video to be aligned. Each image group includes a first image to be processed and a second image to be processed. Each image group corresponds to an element, which is a matrix element in the image matching matrix. For each image group, the row number of the matrix element corresponding to the image group in the image matching matrix is ​​determined based on the order of the first image to be processed in the first video to be aligned. Based on the order of the second image to be processed in the second video to be aligned, determine the column number of the matrix element corresponding to the image group in the image matching matrix; The similarity between the first image to be processed and the second image to be processed is compared with a preset similarity threshold; If the similarity is less than the preset similarity threshold, then the element values ​​of the matrix elements corresponding to the row number and the column number are the first preset values; If the similarity is not less than the preset similarity threshold, then the element values ​​of the matrix elements corresponding to the row number obtained by subtracting 1 from the row number and the column number obtained by subtracting 1 from the column number are increased by a second preset value to obtain the element values ​​of the matrix elements corresponding to the row number and the column number. A determining module is used to arrange all elements to obtain multiple element sequences, wherein the element sequences contain at least two elements; Find the maximum element value among all matrix elements; determine the first target matrix element diagonal line where the maximum element value is located; find a continuous element sequence in the element sequence corresponding to the first target matrix element diagonal line where multiple element values ​​are continuously greater than a third preset value, the continuous element sequence being the target element sequence; and / or, select the element sequence with the most element values ​​greater than the third preset value among all element sequences as the target element sequence; based on the target element sequence, determine the alignment image sequence in the first video to be aligned and the second video to be aligned.

5. The apparatus according to claim 4, wherein, The determining module is specifically used to determine the second target matrix element diagonal line with the largest number of element values ​​greater than a third preset value; and to take the element sequence corresponding to the second target matrix element diagonal line as the longest similar element sequence, wherein the longest similar element sequence is the target element sequence.

6. The apparatus according to claim 4, wherein, The determining module is specifically used to determine a first starting frame, a first ending frame, a second starting frame, and a second ending frame based on the target element sequence. The image sequence from the first starting frame to the first ending frame in a plurality of first images to be processed is taken as the first aligned image sequence; The image sequence from the second starting frame to the second ending frame in a plurality of second images to be processed is used as the second aligned image sequence.

7. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-3.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-3.

9. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-3.

Images