Method, device and computer-readable storage medium for video processing

Abstract

The present disclosure provides a method, device and computer-readable storage medium for video processing. The method includes: obtaining a first video and a second video shot by a first device and a second device for the same content respectively; determining a first start frame and a first end frame in the first video corresponding to the start and end of the movement of the first device respectively based on the similarity between adjacent frames of the first video; comparing the first start frame and the first end frame of the first video with each frame of the second video respectively to determine a second start frame and a second end frame in the second video that match the first start frame and the first end frame respectively; and determining a plurality of matching video frame pairs from the first video and the second video based on the frame numbers of the first start frame, the first end frame, the second start frame and the second end frame. According to the video processing method provided by the present disclosure, the efficiency and accuracy of matching two videos with different shooting qualities can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision, and more specifically, to a method, a device, and a computer-readable storage medium for video processing. Background Art

[0002] In recent years, video matching technology has been widely used in the field of image processing, especially for matching two videos shot separately in the same environment. By matching two videos to find the correspondence between their video frames, various evaluation analyses and subsequent processing can be performed on the matched video frames. However, the existing video matching technology only relies on the similarity between the images of the video frames involved in the matching, and its evaluation factors are too single and the matching process is not efficient.

[0003] On the other hand, video enhancement tasks have also made significant progress in recent years. For example, video super-resolution tasks can use machine learning to reconstruct corresponding high-resolution videos from low-resolution videos, which has important application value in the fields of medical imaging, satellite images, and video surveillance. However, when constructing training data sets for machine learning, existing video super-resolution technologies only resize high-resolution video frames to low-resolution video frames to construct pairs of matching video frames, resulting in unsatisfactory learning effects. Although real-world training data sets can be constructed by shooting two videos at different resolutions, the two videos are not aligned in shooting timing and cannot be directly used for machine learning. Therefore, how to match the two videos shot so that they can be used for machine learning is also a real problem faced by those skilled in the art.

[0004] Therefore, there is a need for a technology that can effectively match two videos, especially effectively match two videos with different shooting qualities. Summary of the invention

[0005] According to one aspect of the present disclosure, a method for video processing is provided, including: obtaining a first video and a second video shot by a first device and a second device respectively of the same content; determining a first start frame and a first end frame in the first video corresponding to the start and the end of the movement of the first device respectively based on the similarity between adjacent frames of the first video; comparing the first start frame and the first end frame of the first video with each frame of the second video respectively to determine a second start frame and a second end frame in the second video that match the first start frame and the first end frame respectively; and determining a plurality of matching video frame pairs from the first video and the second video based on the frame numbers of the first start frame, the first end frame, the second start frame, and the second end frame.

[0006] According to another aspect of the present disclosure, a device for video processing is provided, comprising: a processor; and a memory storing computer program instructions, wherein when the computer program instructions are executed by the processor, the processor performs the following steps: obtaining a first video and a second video shot by a first device and a second device of the same content, respectively; determining a first start frame and a first end frame in the first video corresponding to the start and end of the movement of the first device, respectively, based on the similarity between adjacent frames of the first video; comparing the first start frame and the first end frame of the first video with each frame of the second video, respectively, to determine a second start frame and a second end frame in the second video that match the first start frame and the first end frame, respectively; and determining a plurality of matching video frame pairs from the first video and the second video, based on the frame numbers of the first start frame, the first end frame, the second start frame, and the second end frame.

[0007] According to another aspect of the present disclosure, a device for video processing is provided, including: a video receiving unit, configured to obtain a first video and a second video shot by a first device and a second device for the same content, respectively; an intra-video motion determination unit, configured to determine a first start frame and a first end frame in the first video corresponding to the start and the end of the motion of the first device, respectively, based on the similarity between adjacent frames of the first video; an inter-video matching unit, configured to compare the first start frame and the first end frame of the first video with each frame of the second video, respectively, to determine a second start frame and a second end frame in the second video that match the first start frame and the first end frame, respectively; and a matching frame pair determination unit, configured to determine a plurality of matching video frame pairs from the first video and the second video based on the frame numbers of the first start frame, the first end frame, the second start frame, and the second end frame.

[0008] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the following steps when executed: obtaining a first video and a second video shot by a first device and a second device respectively of the same content; determining a first start frame and a first end frame in the first video corresponding to the start and the end of the movement of the first device, respectively, based on the similarity between adjacent frames of the first video; comparing the first start frame and the first end frame of the first video with each frame of the second video, respectively, to determine a second start frame and a second end frame in the second video that match the first start frame and the first end frame, respectively; and determining a plurality of matching video frame pairs from the first video and the second video based on the frame numbers of the first start frame, the first end frame, the second start frame, and the second end frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] These and / or other aspects and advantages of the present disclosure will become more clear and easier to understand from the following detailed description of the embodiments of the present disclosure in conjunction with the accompanying drawings, in which:

[0010] Figure 1 A flowchart of a method for video processing according to an embodiment of the present disclosure is shown.

[0011] Figure 2a A schematic diagram is shown of calculating a start frame and an end frame of a first video based on a mutual information value when the first video is a low-resolution video in a method for video processing according to an embodiment of the present disclosure.

[0012] Figure 2b A schematic diagram is shown of calculating a start frame and an end frame of a first video based on a mutual information value when the first video is a high-resolution video in a method for video processing according to an embodiment of the present disclosure.

[0013] Figure 3a A schematic diagram showing the result of the number of matching feature points between a first starting frame of a first video and each video frame of a second video in a method for video processing according to an embodiment of the present disclosure is shown.

[0014] Figure 3b A schematic diagram showing the result of the number of matching feature points between a first termination frame of a first video and each video frame of a second video in a method for video processing according to an embodiment of the present disclosure is shown.

[0015] Figure 4 A schematic diagram showing a common area between two video frames included in each matched video frame pair in a method for video processing according to an embodiment of the present disclosure is shown.

[0016] Figure 5 A schematic diagram is shown of dividing the matched video frames and / or the determined common areas into image blocks in a method for video processing according to an embodiment of the present disclosure.

[0017] Figure 6 A schematic diagram showing a method for video processing according to an embodiment of the present disclosure, using a trained neural network to perform video enhancement processing.

[0018] Figure 7 A schematic hardware block diagram of a device for video processing according to an embodiment of the present disclosure is shown.

[0019] Figure 8 A schematic structural block diagram of a device for video processing according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the present disclosure, the present disclosure is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0021] First, the basic idea of ​​the image processing technology of the present invention is briefly summarized. For two videos to be matched, and usually two videos with different shooting qualities, the image processing technology of the present invention considers the similarity between adjacent video frames in a single video to determine the motion state of the shooting device of the video, and then locates the start frame and the end frame corresponding to the start and end of the motion of the shooting device in the video with the help of the determined motion state of the shooting device, and then matches the start frame and the end frame in the video with each video frame of another video to determine the starting point and the end point of the match between the two videos, so as to select the video frames that match each other from the two videos according to the matching results to form a matching video frame pair. In other words, the present invention can determine the motion state of the shooting device based on the inter-frame similarity of a single video, and match the video frames of different videos with the help of the determined motion state, which can effectively improve the efficiency and accuracy of the video matching process.

[0022] As discussed above, existing video super-resolution technologies, when constructing training data sets for machine learning, simply resize high-resolution video frames to low-resolution video frames in order to form matching video frame pairs, resulting in unsatisfactory learning results. In view of this, in order to improve the effect of machine learning, the present invention shoots two videos with different resolutions, and matches the two videos to solve the problem of misaligned shooting timing, so as to construct a real-world training data set in this way, so that the trained machine learning model has the expected performance. As an example, the image processing technology described above can be used to match the two videos shot, so that the obtained paired matching video frames are used as a real-world training data set for machine learning.

[0023] For the sake of convenience of explanation, the image processing method is described here and below by taking the first video and the second video shot of the same content as an example, which are respectively a high-resolution video and a low-resolution video. However, this is merely an example and not a limitation of the present disclosure. The image processing technology of the present disclosure has a wide range of application scenarios, and it can be used for matching any videos with different qualities. For example, the image processing method of the present disclosure can be applied to matching between videos with different color saturation, matching between videos with different degrees of fineness, and matching between different videos that differ in any other dimension. The following is a detailed description of the method and device for video processing of the present disclosure in conjunction with the accompanying drawings and examples.

[0024] First, refer to Figure 1To describe a flowchart of a method for image processing according to an embodiment of the present disclosure. The image processing method of the present disclosure can be used to match a first video and a second video that shoot the same content, so as to determine paired video frames (or video frame pairs) that match each other from the first video and the second video. The image processing method of the present disclosure can be applied to matching videos shot in any indoor and outdoor environment, including street scenes such as buildings, and videos shot in indoor environments such as offices. In an embodiment of the present disclosure, a video frame may refer to a picture of a video shot by a shooting device at each moment, and may be used interchangeably with terms such as image, image frame, and picture throughout the claims and / or the specification. It can be understood that the shooting device mentioned in the present disclosure may be any device with a video shooting function, including a mobile phone, a SLR, a sports camera, etc., and different shooting devices generally have different shooting parameters, such as field of view, exposure, frame rate, resolution, color saturation, and the like.

[0025] like Figure 1 As shown, in step S101, a first video and a second video shot by a first device and a second device for the same content are obtained. It can be understood that in step S101, the first video and the second video can be shot by the same shooting device or different shooting devices for the same environment content, so that they have a common shooting content and can be matched between the two.

[0026] In an embodiment of the present disclosure, the first video and the second video may be two videos to be matched with different shooting qualities, and the above different shooting qualities may be evaluated from multiple dimensions. For example, the first video may be a high-resolution video and the second video may be a low-resolution video; or the first video may be a low-resolution video and the second video may be a high-resolution video. It is understood that the above high resolution and low resolution are relative concepts, and those skilled in the art may select according to actual needs. As an illustrative example, a high-resolution video may be a 4k resolution video, and a low-resolution video may be a 2k resolution video. For another example, the first video may be a high color saturation video and the second video may be a low color saturation video; or the first video may be a low color saturation video and the second video may be a high color saturation video. As an illustrative example, a high color saturation video may be a color video, and a low color saturation video may be a grayscale video. It is understood that the examples of dimensions of different shooting qualities described above are only illustrative examples, and the present disclosure is not limited thereto.

[0027] In addition, it can be understood that the first video and the second video can be obtained by shooting in a variety of ways. For example, the first video can be obtained by sweeping the captured environment from left to right while the user holds the first device, and the second video can be obtained by sweeping the same captured environment from left to right in a similar manner while the same or different users hold the second device in the same time period or different time periods. The shooting angle, sweep range and sweep speed of the two shots should be kept as similar as possible to facilitate subsequent matching. It should be noted that the above-mentioned handheld shooting method and sweeping camera movement method are only illustrative examples, and the shooting device can also be set up on a moving object such as a vehicle and shot with any suitable shooting parameters, shooting methods and shooting device movement methods. It can also be understood by those skilled in the art that the present disclosure does not require the captured environment to remain completely consistent during the shooting process, but allows the captured environment to change to a certain extent during the same shooting period and / or during different shooting periods. For example, in the case of street scene shooting, different pedestrians can be allowed to appear in the picture during the same shooting process and / or different shooting processes, without affecting the subsequent matching of the common picture (e.g., building) in the captured street scene.

[0028] Those skilled in the art can also understand that the first device and the second device described above can also be the same shooting device, but the shooting device shoots the first video and the second video respectively in two consecutive shots with different shooting parameters. It can be understood that since the first video and the second video may be shot by the same or different devices at the same or different times, the scenes aimed at by the shooting devices at the beginning of the two shots may not be the same, and the still time between pressing the start shutter and the shooting device actually starting to move and shoot (similarly, the still time between the camera actually stopping moving and pressing the stop shutter) may not be consistent. In this case, there is a problem that the videos obtained from the two shots are not aligned in shooting timing. In view of this, there is a need to match the first video and the second video obtained from the two shots in order to find the corresponding relationship between their video frames.

[0029] In step S102, based on the similarity between adjacent frames of the first video, a first start frame and a first end frame corresponding to the start and end of the motion of the first device are determined in the first video. The inventor of the present disclosure has noticed that when actually shooting a video, there is a certain delay between when the user presses the start shutter and when the shooting device starts to move, and there is also a certain delay between when the shooting device stops moving and when the user presses the stop shutter, and the video frames in these delay time periods belong to motion still frames rather than effective shooting time periods, and they may not contain useful information for the matching process. In view of this, compared with the prior art that only relies on the similarity between the video frames of different videos for matching, the present disclosure additionally introduces an evaluation factor of the motion state of the shooting device in the matching process, and determines the motion state of the corresponding shooting device based on the similarity between adjacent video frames of any video in the first video and the second video, that is, determines the start and end of the motion of the shooting device, and determines the start frame and the end frame corresponding to the start and end of the motion of the shooting device based on the determined motion state, thereby eliminating the motion still frames between pressing the shutter and the effective shooting period.

[0030] Various methods can be used to determine the start frame and end frame of a video based on the similarity between adjacent frames of the video. As illustrative examples, methods based on mutual information values, methods based on histogram information, and methods based on SSIM (structural similarity), etc., can be used, and the present disclosure does not limit this. For the sake of completeness, the mutual information value is used as an example of a measure for evaluating the similarity between frames of a video. In this example, each mutual information value between adjacent frames of a first video can be calculated, and each mutual information value can be compared with a predetermined mutual information threshold in sequence to determine the first start frame and the first end frame in the first video. The following is combined with Figure 2a and Figure 2b How to determine the start frame and the end frame of the first video will be described in detail.

[0031] Figure 2a The figure shows a schematic diagram of calculating the start frame and the end frame of the first video based on the mutual information value when the first video is a low-resolution video in a method for video processing according to an embodiment of the present disclosure. Mutual information is a measure used in information theory to evaluate the degree of dependence between two random variables. The more similar the two variables are, the greater the mutual information between them, and vice versa. In this example, mutual information is used as a metric to evaluate the difference between two video frames. It can be understood that the video frames during the period between pressing the shutter and starting the motion shooting are almost still, so during this period, for example Figure 2aAs shown, the mutual information value between adjacent frames is relatively large between frame number 0 and frame number 14. Subsequently, as the camera starts to move and the captured scene changes, the mutual information value begins to decrease and remains at a relatively low value during the effective shooting process, such as Figure 2a As shown in about frame number 14 to frame number 88 in the image. Afterwards, when the user intends to end the shooting and thus stops moving, the mutual information value starts to increase again until the user presses the stop shutter, as shown in FIG. Figure 2a As shown in about frame number 88 to frame number 110 in . In other words, in the next few frames after pressing the shutter (corresponding to frame number 0), the mutual information value is large, indicating that the camera is not moving at this moment, and when the mutual information value changes sharply (corresponding to about frame number 14), it indicates that the camera has moved. For a period of time thereafter, the camera continues to shoot motion and the mutual information remains at a low value, and until about frame number 88, the mutual information value rises significantly again, indicating that the camera has stopped moving but has not yet pressed the shutter, and the user presses the shutter to stop shooting until about 110 frames later. In this case, we can pay attention to the moment when the mutual information value exceeds the predetermined mutual information threshold and falls below the predetermined mutual information threshold, and use the video frame corresponding to the moment of such significant change as its first start frame and first end frame.

[0032] From another perspective, by observing the trend of each mutual information value changing over time during the entire shooting process, it can be seen that during the effective shooting period (i.e., the period between the start and end of the movement), the mutual information value is maintained at a relatively low value, so for example, the predetermined mutual information threshold can be set to half of the maximum mutual information value (or any suitable proportion thereof), and the moment when the mutual information value first drops to the threshold is determined as the start frame of the first video, and the moment when the mutual information value last remains below the threshold is determined as the end frame of the first video. In practice, the mutual information value between each adjacent frame of the first video can be calculated, and each mutual information value can be compared with the predetermined mutual information threshold in sequence, wherein the first video frame in the first video whose mutual information value is less than the predetermined mutual information threshold can be determined as the first start frame, and the last video frame in the first video whose mutual information value is less than the predetermined mutual information threshold can be determined as the first end frame. For example, in combination with Figure 2a It can be determined that the actual start movement and actual end movement of the first device shooting the first video correspond to approximately frame numbers 14 and 88 respectively, so the first start frame and the first end frame are determined to be frame numbers 14 and 88.

[0033] It should be noted that, for the setting of the predetermined mutual information threshold, those skilled in the art can set it according to actual needs. For example, the predetermined mutual information threshold can be set to half of the maximum mutual information value or any suitable ratio. Accordingly, the moment when the mutual information value falls / exceeds the predetermined mutual information threshold can be used as the start frame and end frame of the first video, and / or the first / last video frame whose mutual information value is less than the predetermined mutual information threshold can be used as the start frame and end frame of the first video. For another example, the change in the mutual information value of each adjacent video frame within a predetermined time range can be calculated, and the moment when the absolute value of the change exceeds the predetermined mutual information threshold can be used as the start frame and end frame of the first video.

[0034] Figure 2b The figure shows a schematic diagram of calculating the start frame and the end frame based on the mutual information value in the method for video processing according to an embodiment of the present disclosure, when the first video is a high-resolution video. Similar to the case where the first video is a low-resolution video, when the first video is a high-resolution video, the mutual information between adjacent frames will change similarly during the entire shooting process. Figure 2b It can be seen that for the first video, the frame numbers corresponding to the time when the start shutter is pressed, the time when the camera starts to move, the time when the camera stops moving, and the time when the stop shutter is pressed are approximately 0, 110, 620, and 700. In this case, the time when the first device starts to move and the time when the movement ends can be determined as frame numbers 110 and 620 based on the comparison of each mutual information value with the predetermined mutual information threshold.

[0035] In other words, in the present disclosure, two positions where the mutual information value changes sharply and / or two positions that cross a predetermined mutual information threshold are used as video frames corresponding to the start and end of the motion of the shooting device, and are used as the starting point and end point for matching, so that the video frames falling between the first start frame and the first end frame in the first video are used for the subsequent matching process, rather than using the starting point and end point of the direct shooting result (for example, which includes a certain number of motion still frames) as the matching starting point and end point, thereby avoiding the waste of computing resources and poor matching efficiency caused by the presence of motion still frames in the matching process. Therefore, by introducing the determination of the motion state of the shooting device in the video matching process, a better matching process can be achieved.

[0036] In addition, although the above describes the situation where the user shoots the first video and the second video continuously, in practice, the user may pause between the initial start of the shooting device and the final stop of the shooting device, and the shooting device also stops moving, but the shooting does not stop. Figure 2a and Figure 2b The difference shown is that in the frame range where the mutual information remains stable and relatively low (e.g. Figure 2a Between frame numbers 14 and 88, Figure 2a In the example shown in FIG. 1 , the mutual information value may rise sharply, then maintain a high value, and then drop sharply. The above-mentioned change trend corresponds to the pause of the shooting device. Similarly, the embodiment of the present application can also identify the video frames corresponding to the pause of the shooting device in the shot video based on the above-mentioned change trend of the mutual information value, and similarly remove these video frames to avoid affecting the subsequent matching process.

[0037] It should be noted that in the above step S102, the adjacent video frames in the first video for inter-frame similarity comparison may refer to directly adjacent video frames in the first video, or may refer to adjacent frames taken every few frames according to a predetermined sampling interval (for example, sampling every 10 frames for inter-frame similarity comparison). It is understood that the sampling granularity of the adjacent frames in the first video may be flexibly selected by those skilled in the art according to requirements of computing speed and accuracy.

[0038] return Figure 1 , in step S103, the first start frame and the first end frame of the first video are respectively compared with each frame of the second video to determine the second start frame and the second end frame in the second video that match the first start frame and the first end frame respectively. As discussed above, although the first video and the second video are obtained by shooting the same environmental content, the scene aimed at when starting to shoot may not be exactly the same, the pause time after pressing the shutter is not exactly the same, and the frame rates of the two shooting devices are not the same. Therefore, in this step, it is necessary to perform feature matching between the video frames of the two videos, so as to find the start frame and the end frame corresponding to the start frame and the end frame of the first video in the second video, so as to determine the corresponding relationship between the two videos in time. That is, through this step S103, it is determined which period of time of the first video and which period of time of the second video The video frames are corresponding in time. In this example, after determining the motion state of the first device that shot the first video, the actual starting point and end point of the shooting can be determined, so that the above starting point and end point can be used as a reference in the matching process to find the corresponding starting point and end point in the second video, and then it can be determined that the video frames in the time period between the start point and end point of the first video correspond to the video frames in the time period between the start point and end point of the second video, which can help to match the remaining video frames falling between the starting point and end point.

[0039] It should be noted that in this step, although the two shooting devices shoot the same environmental content, their different shooting parameters may lead to a certain degree of difference in their imaging effects. Therefore, it is necessary to use the same image features that can be maintained across different conditions to match videos with different shooting parameters. For example, various matching algorithms in the art can be used to find the corresponding start frame in the second video frame for the start frame of the first video as the starting point for the match between the two; and find the corresponding end frame in the second video frame for the end frame of the first video as the end point for the match between the two. As an illustrative example, feature matching algorithms such as SIFT, SURF, and ORB can be used to determine the matching feature points between the first start frame of the first video and each frame of the second video, as well as the matching feature points between the first end frame of the first video and each frame of the second video, and determine the starting point and end point of the match between the two videos based on the results of the matching feature points.

[0040] Specifically, the matching process according to the embodiment of the present disclosure may include the following steps: matching the first start frame of the first video with each frame of the second video to determine the number of matching feature points between the first start frame of the first video and each frame of the second video; and matching the first end frame of the first video with each frame of the second video to determine the number of matching feature points between the first end frame of the first video and each frame of the second video; and based on the number of matching feature points, determining the second start frame and the second end frame in the second video that match the first start frame and the first end frame, respectively. For example, the second start frame corresponds to the video frame in the second video that has the most matching feature points with the first start frame, and the second end frame corresponds to the video frame in the second video that has the most matching feature points with the first end frame.

[0041] Figure 3a A schematic diagram showing the result of comparing the number of matching feature points between a first starting frame of a first video and each video frame of a second video in a method for video processing according to an embodiment of the present disclosure. Figure 3a As shown, it can be determined that the frame number with the largest number of matching feature points between the second video frame and the first starting frame is approximately 160, so the video frame corresponding to frame number 160 in the second video can be determined as the second starting frame of the second video as the starting point for matching between the two videos. Figure 3b A schematic diagram showing the result of comparing the number of matching feature points between the first termination frame of a first video and each video frame of a second video in a method for video processing according to an embodiment of the present disclosure. Figure 3bAs shown, it can be determined that the frame number with the largest number of matching feature points between the second video frame and the first end frame is approximately 535, so the video frame corresponding to frame number 535 in the second video can be determined as the second end frame of the second video, as the end point of the match between the two videos.

[0042] It should be noted that in order to be able to search for the matching starting point and end point between the two videos in the second video, preferably, the shooting range of the shooting device of the second video can be larger than the shooting range of the first video. For example, the first device can have a sweep shooting range of -60° to 60°, and the second device can have a sweep shooting range of -90° to 90°, that is, the shooting span of the second device is larger than the shooting span of the first device. It can be understood that, at least in part due to the difference in the shooting spans of the two, the scene aimed at when the second device starts / ends movement may be different from the scene aimed at when the first device starts / ends movement, so the second start frame and the second end frame determined from the second video may be different from the video frames corresponding to the actual start and actual end of the movement of the second device.

[0043] Optionally, a variety of measures can be adopted to reduce the amount of calculation for matching in step S103. For example, while determining the first start frame and the first end frame of the first video based on the inter-frame similarity in step S102, the video frames in the second video corresponding to the start and end of the movement of the second device can also be determined at the same time, whereby, when comparing the start frame and the end frame of the first video frame with each video frame of the second video, the start frame of the first video frame can be matched with the video frame near the moment corresponding to the start movement of the second device in the second video, and the end frame of the first video frame can be matched with the video frame near the moment corresponding to the end movement of the second device in the second video, without matching it with the video frames in the entire range captured by the second device. For another example, in step S103, the start frame and the end frame of the first video are compared with each frame of the second video, which can be compared with each video frame of the second video; or it can be compared with the video frames selected at a predetermined sampling interval in the second video. In addition, when the start motion state / end motion state of the second device is determined as described above, video frames can be selected in the second video near the determined start motion state / end motion state according to a predetermined sampling interval for matching. It can be understood that the granularity of selecting the video frames to be matched from the second video in the above matching process can be flexibly selected by those skilled in the art according to the requirements of computing speed and accuracy.

[0044] return Figure 1In step S104, based on the frame numbers of the first start frame, the first end frame, the second start frame, and the second end frame, a plurality of matching video frame pairs are determined from the first video and the second video. In this step, each matching video frame pair is a pair of video frames, which includes a video frame from the first video and a corresponding video frame from the second video.

[0045] In step 104, after determining the frame numbers of the first start frame, the first end frame, the second start frame, and the second end frame, it can be determined that the video frames between the first start frame and the first end frame of the first video and the video frames between the second start frame and the second end frame of the second video are corresponding in time sequence, and then the frame numbers of the above start frame and the end frame can be used to select video frames from the first video and the second video according to a predetermined rule to form a matching video frame pair. For example, for any video frame in the first video, a matching video frame corresponding to any video frame in the first video can be selected from the second video based on the corresponding relationship between the frame numbers of the first start frame and the first end frame and the frame numbers of the second start frame and the second end frame. For example, the first start frame and the second start frame can be regarded as a matching video frame pair, and the first end frame and the second end frame can be regarded as a matching video frame pair. In addition, for those video frames between the first start frame and the first end frame in the first video and those video frames between the second start frame and the second end frame in the second video, the corresponding video frames can be found in other videos according to their respective frame numbers based on the corresponding relationship between the frame numbers. For example, when the first number of video frames between the first start frame and the first end frame and the second number of video frames between the second start frame and the second end frame are the same, the video frames of the first video can be corresponded in sequence to the video frames of the second video according to the correspondence between the frame numbers.

[0046] However, the inventors note that, in actual situations, due to the different frame rates of the two shooting devices, the moving speeds during shooting, etc., the first number of video frames between the first start frame and the first end frame and the second number of video frames between the second start frame and the second end frame may not be the same, so it may not be possible to find the corresponding relationship between the video frames in a sequential manner. In addition, in this case, a relatively large number of video frames in a certain video may constitute redundant information to a certain extent. Accordingly, the redundant information can be reduced by removing a part of the number of video frames, and at the same time, it is easy to find the corresponding relationship between the video frames of the first video and the second video. For example, in this case, the above-mentioned first number and second number can be fine-tuned to ensure that there is an integer ratio relationship of 1:n between them. In this case, for any video frame in the first video, a video frame is taken from the second video every n frames according to the corresponding relationship between the frame numbers and according to the ratio relationship, so that the two video frames constitute a matching video frame pair, and so on, thereby determining multiple matching video frame pairs.

[0047] In addition, the inventors of the present disclosure also noticed that the field of view of the two shooting devices is also different. Therefore, even if a one-to-one correspondence between the video frames is found in the two videos, the shooting content of the two video frames in the matching video frame pair may be different, and there may be a problem of spatial misalignment of the two video frames. For example, in the case where the first video and the second video are both taken by the user holding the shooting device to shoot the street scene, the two video frames in the matching video frame pair determined from the first video and the second video may both include a building in the street scene, but the location of the building in the picture may be different, and its background may also have a large difference. In view of this, further cropping the common picture portion including the area of ​​interest from the two video frames in the determined matching video frame pair is extremely beneficial for its various analysis, evaluation and subsequent processing.

[0048] In practice, for any video frame in the first video, its corresponding matching video frame can be selected from the second video to form a matching video frame pair with the two video frames. Then, for the two video frames, the matching feature points between the two can be determined, and the common area between the two can be determined based on the matching feature points. The so-called common area refers to the region of interest shared by the two video frames. Next, the parts corresponding to the region of interest can be segmented out from the two video frames, respectively, and the segments from the two video frames are used as matching video frame pairs. Similarly, in this embodiment, similar processing can be adopted for each of the remaining matching video frame pairs, so that the two video frames in each matching video frame pair only include the common area determined based on the matching of feature points, thereby reducing the efficiency and accuracy of various subsequent analysis, evaluation and processing affected by factors such as background differences and spatial misalignment.

[0049] Figure 4 A schematic diagram of a common area between two video frames contained in each matched video frame pair in a method for video processing according to an embodiment of the present disclosure is shown. In this example, the first video and the second video are captured in an office environment, but the segmentation of the common area in the present disclosure is not limited thereto, and the region of interest in any captured environment can also be segmented. Figure 4 As shown, the left and right sides of the dotted line are video frames from the first video and the second video in a matched video frame pair, respectively. By performing a feature point matching process, matching feature points can be determined between the two video frames, such as Figure 4 , which is shown as a line segment with the matching feature points of the two video frames as endpoints. In this example, the region of interest of concern can be determined based on the one-to-one correspondence between the matching points of the two video frames, for example, the region of interest is a computer and the furnishings around it. In this case, the portions of the picture corresponding to the region of interest can be cropped out from the two video frames respectively, so that the cropped portions of the two video frames can be used as an updated matching video frame pair. Alternatively, in the case where the fields of view of the two video frames in the matching video frame pair are different, the region corresponding to the video frame with a smaller field of view can be directly used as the region of interest, and based on the feature point matching results between the two video frames, the portion shared with the video frame with a smaller field of view can be cropped out from the video frame with a larger field of view, and the video frame with a smaller field of view itself and the cropped portion of the video frame with a larger field of view can be used as an updated matching video frame pair. As shown in Figure 4 As shown, the parts in the boxes of the two video frames can be used as the updated matching video frame pair, and it can be seen that the scene ranges contained in the boxes are roughly the same. According to the above method of the embodiment of the present disclosure, on the basis of solving the problem of time sequence misalignment of the two videos, the problem of spatial misalignment can be further solved, thereby facilitating various subsequent evaluation analyses and subsequent processing.

[0050] As discussed above, the matched video pairs obtained according to the video processing technology of the embodiment of the present disclosure can be used in various analysis evaluations and subsequent processing. For example, the obtained multiple matched video frame pairs can be applied to machine learning models such as video enhancement tasks. As an example, the following describes the processing of using the real-world training data set obtained according to the embodiment of the present disclosure for the training phase of machine learning, and the processing of using the trained machine learning model for the use phase of the super-resolution task from the perspective of using multiple matched video frame pairs for the machine learning model of the super-resolution task, thereby solving the problem that the existing super-resolution technology cannot obtain a training model with the desired performance.

[0051] For example, for training datasets for video super-resolution tasks, existing SOTA datasets (such as Vimeo, REDS, etc.) simply downsample high-resolution videos to obtain corresponding low-resolution videos to generate high / low resolution matching frames, which obviously cannot achieve the learning effect obtained based on real-world training datasets. Moreover, for some special cameras (such as RICOH-THETA), their own resolution is not very high and there is no corresponding high-resolution video to match it. Therefore, for such cameras, in order to perform video super-resolution tasks, it is necessary to match the fisheye video taken by it with the perspective video taken by an ordinary camera to generate pairs of matching video frames for model training for super-resolution tasks. It can be understood that the above combined with Figure 1-Figure 4 The described method for video processing is to obtain a training dataset for training the machine learning model.

[0052] The following is a brief introduction to an example of learning based on multiple matching video frame pairs obtained. As an illustrative example, the machine learning model to be trained may be, for example, various types of neural networks, such as a fully connected neural network, a convolutional neural network, a recurrent neural network, and the like. In an embodiment of the present disclosure, an initial neural network may be constructed, and the parameters of the neural network may be initially set or obtained through a certain degree of learning. In order to make the neural network have stronger performance in video super-resolution tasks, a training data set may be constructed using multiple matching video frame pairs obtained and input into the neural network so as to continuously train and learn the neural network based on the input training data set. In this method, the following may be obtained: Figure 1The plurality of matched video frame pairs obtained in step S104 are obtained, and a training data set is constructed therefrom. For example, the training data set may include a low-resolution video composed of each low-resolution video frame in the plurality of matched video frame pairs obtained, and a high-resolution video composed of each high-resolution video frame in the plurality of matched video frame pairs obtained, so as to serve as the low-resolution video and the high-resolution video in the training data set. Then, the predicted value of the high-resolution video based on the low-resolution video in the training data set is calculated by a neural network, and the loss between the predicted value and the true value of the high-resolution video in the training data set is calculated, so as to update the parameters of the neural network based on the loss. For example, the neural network can process and analyze the low-resolution video through each neural network layer inside it on the one hand, and obtain the predicted value of the high-resolution video; on the other hand, the true value of the high-resolution video can be obtained from the training data set, and the predicted value can be compared with the true value, and the loss between the predicted value and the true value can be calculated accordingly, and the various parameters of the neural network can be updated based on the loss. It can be understood that in the present disclosure, the neural network can be continuously and iteratively updated for a large amount of training data, and the training process is not terminated until a predetermined condition is met, so as to obtain a trained neural network, so as to meet the expected performance required for the super-resolution task. Various conditions can be used as training termination conditions for machine learning models. For example, the values ​​of the parameters of the neural network when the loss is less than a predetermined threshold can be determined as the values ​​of the parameters of the trained neural network; or the values ​​of the parameters when the number of training times of the neural network reaches a predetermined threshold can be determined as the values ​​of the parameters of the trained neural network.

[0053] It can be understood that in the training process of the machine learning model for super-resolution tasks, various loss functions can be used to evaluate the learning and performance of the neural network, such as loss functions used to measure pixel-level differences, loss functions used to measure feature-level differences, loss functions used to measure texture-level differences, and any other known loss functions and loss functions designed in the future. The present disclosure does not limit the type of loss function. In the learning process of the neural network, the updated parameters include parameters such as the weights of each neural network layer. In addition, a large amount of data including the photographed environment of various types of scenes can be used to train the neural network, so that the trained model can perform video enhancement on the video to be processed involving any scene. The method for training a neural network is well known in the art and will not be described in detail here. In addition, it should be noted that the neural network described above is only an illustrative example of various machine learning models that can be used in the present disclosure, and those skilled in the art can also use other machine learning models for training.

[0054] It should be noted that in the training process of the machine learning model described above, considering the Figure 1 The size of the matching video frame pairs and / or the cropped common areas directly obtained in step S104 may be large. If low-resolution videos and high-resolution videos are directly constructed therefrom and the constructed two-resolution videos are used for machine learning, it may cause a large computational burden and memory consumption on the hardware device. Preferably, after obtaining the matching video frame pairs or the cropped common areas, they can be segmented to obtain smaller paired video frame blocks, and the segmented paired video frame blocks are used as training data sets required for the super-resolution task.

[0055] Figure 5 FIG. 1 is a schematic diagram showing a method for video processing according to an embodiment of the present disclosure, in which the matched video frames and / or the determined common areas are divided into image blocks. Figure 4 The examples shown in FIG. 1 are different for capturing different quality videos of an office environment. Figure 5 , a schematic diagram of image block segmentation after respectively shooting a high-resolution video and a low-resolution video of a sky street scene and determining a matching video frame pair and / or a cropped common area between the two videos. In this example, the high-resolution video frame or the low-resolution video frame in the paired matching video frames can be segmented into video frame blocks of suitable sizes, such as 32*32, 64*64, or the common area further determined in the high-resolution video frame or the low-resolution video frame can be segmented into image blocks of suitable sizes. For example, Figure 5 , a schematic result of segmenting a common area with sky content in a low-resolution video frame into 30 image blocks is shown. It can be understood that by obtaining image blocks of a size suitable for machine learning through segmentation processing, rather than directly inputting the matched video frame pairs into the machine learning model, the computational burden and memory consumption caused to the hardware device can be reduced. In addition, it can be understood that in order to use the segmented image blocks for machine model training of super-resolution tasks, the granularity of image block segmentation for high-resolution video frames / common areas and low-resolution video frames / common areas can be different. For example, the high-resolution video frame / common area can be segmented into 64*64 image blocks, and the corresponding low-resolution video frame / common area can be segmented into 32*32 image blocks. It can be understood that the image block sizes of 32*32, 64*64, etc. and the number of 30 image blocks mentioned above are only schematic examples. Those skilled in the art can segment the video frame or the common area into any suitable size and any suitable number of image blocks depending on various needs and computational considerations.

[0056] As described above, the training can be terminated when a predetermined condition is met during the training process, and a trained neural network model is obtained, which can then be used for various video enhancement tasks, such as super-resolution tasks. As an illustrative example of the use of the trained model, in an embodiment of the present disclosure, a third video shot by a third device can be obtained, and the third video can be input into a trained neural network to output a fourth video corresponding to the third video. It can be understood that in the case of a super-resolution task, the third video is a low-resolution video, and the fourth video is a high-resolution video. It should be noted that the third device can be any device that is the same or different from the first device and the second device, and the third video obtained therefrom needs to be processed for resolution enhancement. By utilizing a trained machine learning model obtained through machine learning, a high-resolution video can be reconstructed from a low-resolution video. Figure 6 FIG. 1 is a schematic diagram showing a method for video processing according to an embodiment of the present disclosure, in which a trained neural network is used to perform video enhancement processing. Figure 6 It can be seen that the neural network can receive the video to be processed as input, and process it through each neural network layer within the neural network, and use the processed video as output, for example, reconstructing a low-resolution video into a high-resolution video.

[0057] It should be noted that the training and use process of the neural network is described above using the super-resolution task as an example, but the present disclosure is not limited thereto, and can be used in any field where video enhancement is required. For example, the neural network can learn a dataset of videos with different color saturation to reconstruct a video with high color saturation from a video with low color saturation. For another example, the neural network can learn a dataset of videos with different levels of detail to reconstruct a video with high detail from a video with low detail.

[0058] The method for video processing according to the embodiment of the present disclosure has been described above in conjunction with the accompanying drawings. In this method, the similarity between adjacent video frames in a single video is taken into account to determine the motion state of the shooting device of the video, and then with the help of the determined motion state of the shooting device, the start frame and the end frame corresponding to the start and end of the motion of the shooting device are located in the video, and then the start frame and the end frame in the video are matched with each video frame of another video to determine the starting point and the end point of the match between the two videos, so as to select the video frames that match each other from the two videos according to the matching results to form a matching video frame pair. In other words, in the present disclosure, the motion state of the shooting device can be determined based on the inter-frame similarity of a single video, and the matching between video frames of different videos can be performed with the help of the determined motion state, which can effectively improve the efficiency and accuracy of the video matching process.

[0059] In addition, the matched video frame pairs obtained by the video processing method of the present invention can be used in various analysis and evaluation and subsequent processing, for example, they can be used in machine learning models for video enhancement tasks. Different from the prior art in which two videos of different shooting qualities are artificially constructed from only one video of shooting quality, the present invention can match videos of different shooting qualities obtained by two shots, so that the obtained matched video frame pairs can be used as real-world training data sets for training machine learning models, thereby achieving the desired machine learning model performance. Then, the trained machine learning model can be used in various video enhancement tasks to reconstruct enhanced videos from the video to be processed, playing an important role in the fields of medical imaging, satellite images, and video surveillance.

[0060] Image processing equipment

[0061] According to another aspect of the present disclosure, a device for video processing is provided, Figure 7 The apparatus 700 for video processing is described in detail.

[0062] Figure 7 FIG. 4 shows a hardware block diagram of a device for video processing according to an embodiment of the present disclosure. Figure 7 As shown, the device 700 includes a processor U701 and a memory U702.

[0063] Processor U701 can be any device with processing capabilities that can implement the functions of the various embodiments of the present disclosure, for example, it can be a general-purpose processor designed to perform the functions described herein, a digital signal processor (DSP), an ASIC, a field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof.

[0064] Memory U702 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may also include other removable / non-removable, volatile / non-volatile computer system memory, such as a hard drive, floppy disk, CD-ROM, DVD-ROM or other optical storage media.

[0065] In this embodiment, the memory U702 stores computer program instructions, and the processor U701 can execute the instructions stored in the memory U702. When the computer program instructions are executed by the processor, the processor executes the method for video processing of the embodiment of the present disclosure. Figure 1-6 The description is basically the same, so in order to avoid repetition, it will not be repeated.

[0066] According to another aspect of the present disclosure, a device for video processing is provided, Figure 8 The apparatus 800 for video processing is described in detail.

[0067] Figure 8 FIG. 2 shows a structural block diagram of a device for video processing according to an embodiment of the present disclosure. Figure 8 As shown, the device 800 includes a video receiving unit U801, an intra-video motion determination unit U802, an inter-video matching unit U803, and a matching frame pair determination unit U804. The various components can respectively perform the above combined Figure 1-6 The various steps / functions of the video processing method described herein are described, and therefore, in order to avoid repetition, only a brief description of the device is given below, and a detailed description of the same details is omitted.

[0068] The video receiving unit U801 can obtain the first video and the second video shot by the first device and the second device respectively for the same content. The first video and the second video obtained by the video receiving unit U801 can be obtained by shooting the same environment content by the same shooting device or different shooting devices, so that they have the same shooting content and can be matched between them.

[0069] It can be understood that the first video and the second video can be two videos to be matched with different shooting qualities, and the above different shooting qualities can be evaluated from multiple dimensions. For example, the first video can be a high-resolution video and the second video can be a low-resolution video; or the first video is a low-resolution video and the second video is a high-resolution video, and so on. In addition, the first video and the second video can be obtained by shooting in a variety of ways, but the shooting angle, scanning range and scanning speed of the two shots should be kept as similar as possible to facilitate subsequent matching. In addition, the first device and the second device can also be the same shooting device, but the shooting parameters are different.

[0070] The intra-video motion determination unit U802 determines the first start frame and the first end frame in the first video that correspond to the start and end of the motion of the first device, respectively, based on the similarity between adjacent frames of the first video. Compared with the prior art that only relies on the similarity between video frames of different videos for matching, the embodiment of the present disclosure introduces an additional evaluation factor of the motion state of the shooting device in the matching process by setting the intra-video motion determination unit U802 in the video processing device 800, thereby determining the motion state of the corresponding shooting device based on the similarity between adjacent video frames of any video in the first video and the second video, that is, determining the start and end of the motion of the shooting device, and determining the start frame and the end frame corresponding to the start and end of the motion of the shooting device, respectively, based on the determined motion state, thereby eliminating the motion still frames between pressing the shutter and the effective shooting period.

[0071] The video intra-motion determination unit U802 can use various methods to determine the start frame and the end frame of the video based on the similarity between adjacent frames of the first video. For example, in the case where the mutual information value is used as a measure of the inter-frame similarity of a certain video, the video intra-motion determination unit U802 can calculate each mutual information value between each adjacent frame of the first video, and compare each mutual information value with a predetermined mutual information threshold in sequence to determine the first start frame and the first end frame in the first video. For example, the video intra-motion determination unit U802 can determine the moment when the mutual information value in the first video exceeds / falls below the predetermined mutual information threshold, and use the video frame corresponding to the moment of such significant change as its first start frame and the first end frame. In addition, the video intra-motion determination unit U802 can also calculate the mutual information value between each adjacent frame of the first video, and compare each mutual information value with a predetermined mutual information threshold in sequence, so that the first video frame whose mutual information value in the first video is less than the predetermined mutual information threshold can be determined as the first start frame, and the last video frame whose mutual information value in the first video is less than the predetermined mutual information threshold can be determined as the first end frame. The video motion determination unit U802 can avoid the waste of computing resources and poor matching efficiency caused by the existence of motion still frames during the matching process by determining the video frames corresponding to the start and end of the motion of the shooting device and using them as the starting and end points for matching.

[0072] The video matching unit U803 compares the first start frame and the first end frame of the first video with each frame of the second video, respectively, to determine the second start frame and the second end frame in the second video that match the first start frame and the first end frame, respectively. For example, the video matching unit U803 can perform feature matching between the video frames of the two videos, so as to find the start frame and the end frame corresponding to the start frame and the end frame of the first video in the second video, so as to determine the temporal correspondence between the two videos, that is, to determine which period of time the video frames of the first video correspond to which period of time the video frames of the second video in time. The video matching unit U803 can use various matching algorithms in the art to match video frames of different videos, for example, feature matching algorithms such as SIFT, SURF, and ORB can be used to determine the starting point and end point of the match between the two videos.

[0073] Specifically, the video matching unit U803 can calculate the number of matching feature points between the first start frame of the first video and each frame of the second video, and calculate the number of matching feature points between the first end frame of the first video and each frame of the second video, and based on the number of matching feature points, determine the second start frame and the second end frame in the second video that match the first start frame and the first end frame, respectively. For example, the second start frame corresponds to the video frame in the second video that has the most matching feature points with the first start frame, and the second end frame corresponds to the video frame in the second video that has the most matching feature points with the first end frame.

[0074] The matching frame pair determination unit U804 determines multiple matching video frame pairs from the first video and the second video based on the frame numbers of the first start frame, the first end frame, the second start frame and the second end frame, wherein each matching video frame pair is a pair of video frames, including a video frame from the first video and a corresponding video frame from the second video.

[0075] Specifically, the matching frame pair determination unit U804 can use the frame numbers of the above-mentioned start frame and the end frame to select video frames from the first video and the second video respectively according to a predetermined rule to form a matching video frame pair. For example, for any video frame in the first video, a matching video frame corresponding to any video frame in the first video can be selected from the second video based on the correspondence between the frame numbers of the first start frame and the first end frame and the frame numbers of the second start frame and the second end frame. For example, when the first number of video frames between the first start frame and the first end frame and the second number of video frames between the second start frame and the second end frame are the same, the matching frame pair determination unit U804 can correspond each video frame of the first video to each video frame of the second video in sequence according to the correspondence between the frame numbers; when the first number is different from the second number, the matching frame pair determination unit U804 can fine-tune the above-mentioned first number and the second number to ensure an integer ratio of 1:n, and for any video frame in the first video, a video frame is taken from the second video every n frames according to the correspondence between the frame numbers and the ratio, so that the two video frames form a matching video frame pair. Optionally, the matching frame pair determination unit U804 can also determine the matching feature points between the two video frames in the matching video frame pair, and determine the common area between the two based on the matching feature points, and then segment out the parts corresponding to the area of ​​interest from the two video frames respectively, and use the segmented parts as the matching video frame pair, thereby reducing the efficiency and accuracy of various subsequent analysis, evaluation and processing affected by factors such as background differences.

[0076] In addition, after the matching frame pair determination unit U804 determines the matching video pair, it can be used in various analysis evaluations and subsequent processing. For example, the obtained multiple matching video frame pairs can be applied to machine learning models such as video enhancement tasks. For example, a real-world training data set can be constructed from the matching video pairs so that the video processing device 800 can learn from them; or the video processing device 800 sends the matching video pairs and / or the real-world training data set constructed therefrom to another computing device for the other computing device to learn. The process of machine learning is similar to that described above in conjunction with the video processing method and will not be repeated here.

[0077] By learning from a real-world training data set, a trained neural network model can be obtained, which can then be used for various video enhancement tasks, such as super-resolution tasks. For example, the video processing device 800 and / or the other computing device can obtain a third video shot by a third device, and process the third video as an input to output a fourth video corresponding to the third video, wherein the third video is a low-resolution video and the fourth video is a high-resolution video. By using a trained machine learning model obtained through machine learning, a high-resolution video can be reconstructed from a low-resolution video.

[0078] The above has been described in conjunction with the accompanying drawings about the device for video processing according to the embodiment of the present disclosure. The device for video processing considers the similarity between adjacent video frames in a single video to determine the motion state of the shooting device of the video, and then locates the start frame and the end frame corresponding to the start and end of the motion of the shooting device in the video by means of the determined motion state of the shooting device, and then matches the start frame and the end frame in the video with each video frame of another video to determine the starting point and the end point of the match between the two videos, so as to select the video frames that match each other from the two videos according to the matching results to form a matching video frame pair, which can effectively improve the efficiency and accuracy of the video matching process.

[0079] In addition, the matched video frame pairs obtained by the video processing device can be used in various analysis and evaluation and subsequent processing, for example, they can be used in machine learning models for video enhancement tasks. In the present disclosure, the obtained matched video frame pairs can be used as real-world training data sets for training machine learning models, and the trained machine learning models can be used in various video enhancement tasks to reconstruct enhanced videos from the videos to be processed, which plays an important role in the fields of medical imaging, satellite images, and video surveillance.

[0080] Computer readable storage medium

[0081] The method / device for video processing according to the present disclosure may also be implemented by providing a computer program product containing a program code for implementing the method or device, or by any storage medium storing such a computer program product.

[0082] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0083] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0084] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0085] It should also be noted that in the device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0086] For those of ordinary skill in the art, it is understood that all or any part of the methods and devices disclosed herein can be implemented in hardware, firmware, software, or a combination thereof in any computing device (including processors, storage media, etc.) or a network of computing devices. The hardware can be a general-purpose processor, digital signal processor (DSP), ASIC, field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor can be a microprocessor, but as an alternative, the processor can be any commercially available processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration. The software can be present in any form of computer-readable tangible storage medium. By way of example and not limitation, such computer-readable tangible storage media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other tangible media that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer. As used herein, disc includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc.

[0087] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0088] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0089] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for video processing, comprising: Obtain a first video and a second video shot by the first device and the second device respectively of the same content; Based on the similarity between adjacent frames of the first video, determining a first start frame and a first end frame in the first video corresponding to the start and end of the movement of the first device, respectively; Comparing a first start frame and a first end frame of the first video with respective frames of the second video to determine a second start frame and a second end frame in the second video that match the first start frame and the first end frame respectively; as well as Determine a plurality of matching video frame pairs from the first video and the second video based on the frame numbers of the first start frame, the first end frame, the second start frame, and the second end frame; The first video is a high-resolution video, and the second video is a low-resolution video; or The first video is a low-resolution video, and the second video is a high-resolution video; The method further comprises: Build a neural network; Constructing a training data set, wherein the training data set includes a low-resolution video consisting of low-resolution video frames in the plurality of matching video frame pairs and a high-resolution video consisting of high-resolution video frames in the plurality of matching video frame pairs; Calculate a prediction value of the high-resolution video based on the low-resolution video in the training data set; Calculating the loss between the predicted value and the true value of the high-resolution video in the training dataset; and Based on the loss, parameters of the neural network are updated.

2. The method according to claim 1, wherein: The determining, based on the similarity between adjacent frames of the first video, a first start frame and a first end frame respectively corresponding to the start and end of the movement of the first device in the first video comprises: Calculating mutual information values ​​between adjacent frames of the first video; comparing each mutual information value with a predetermined mutual information threshold in sequence; Determine the first video frame in the first video whose mutual information value is less than the predetermined mutual information threshold as the first starting frame; and The last video frame in the first video whose mutual information value is less than the predetermined mutual information threshold is determined as the first termination frame.

3. The method according to claim 1, wherein: The step of comparing the first start frame and the first end frame of the first video with the frames of the second video to determine the second start frame and the second end frame in the second video that match the first start frame and the first end frame respectively comprises: Calculating the number of matching feature points between a first starting frame of the first video and each frame of the second video; Calculating the number of matching feature points between the first end frame of the first video and each frame of the second video; and Based on the number of the matching feature points, the second start frame and the second end frame that match the first start frame and the first end frame respectively are determined in the second video.

4. The method according to claim 3, wherein: The determining a plurality of matching video frame pairs from the first video and the second video based on the frame numbers of the first start frame, the first end frame, the second start frame, and the second end frame comprises: For any video frame in the first video, based on the correspondence between the frame numbers of the first start frame and the first end frame and the frame numbers of the second start frame and the second end frame, selecting a matching video frame corresponding to the any video frame in the first video from the second video; Determining matching feature points between any one of the video frames and the matching video frame; Based on the matching feature points, determining a common area between any one of the video frames and the matching video frame, wherein the common area is a region of interest shared by any one of the video frames and the matching video frame; Segmenting a portion corresponding to the region of interest from the any one video frame and the matching video frame; and The segmented portion of any one of the video frames and the matching video frame is used as a matching video frame pair.

5. The method according to claim 1, further comprising: Obtaining a third video shot by a third device; as well as The third video is input into the updated neural network to output a fourth video corresponding to the third video, wherein: The third video is a low-resolution video, and the fourth video is a high-resolution video.

6. A device for video processing, comprising: processor; as well as a memory storing computer program instructions, Wherein, when the computer program instructions are executed by the processor, the processor is caused to perform the following steps: Obtain a first video and a second video shot by the first device and the second device respectively of the same content; Based on the similarity between adjacent frames of the first video, determining a first start frame and a first end frame in the first video corresponding to the start and end of the movement of the first device, respectively; Comparing a first start frame and a first end frame of the first video with respective frames of the second video to determine a second start frame and a second end frame in the second video that match the first start frame and the first end frame, respectively; and Determine a plurality of matching video frame pairs from the first video and the second video based on the frame numbers of the first start frame, the first end frame, the second start frame, and the second end frame; The first video is a high-resolution video, and the second video is a low-resolution video; or The first video is a low-resolution video, and the second video is a high-resolution video; Wherein, when the computer program instructions are executed by the processor, the processor is further caused to perform the following steps: Build a neural network; Constructing a training data set, wherein the training data set includes a low-resolution video consisting of low-resolution video frames in the plurality of matching video frame pairs and a high-resolution video consisting of high-resolution video frames in the plurality of matching video frame pairs; Calculate a prediction value of the high-resolution video based on the low-resolution video in the training data set; Calculating the loss between the predicted value and the true value of the high-resolution video in the training dataset; and Based on the loss, parameters of the neural network are updated.

7. A device for video processing, comprising: A video receiving unit is configured to obtain a first video and a second video shot by a first device and a second device respectively for the same content; an intra-video motion determination unit configured to determine, based on similarities between adjacent frames of the first video, a first start frame and a first end frame in the first video corresponding to the start and end of a motion of the first device, respectively; a video matching unit, configured to compare a first start frame and a first end frame of the first video with respective frames of the second video, so as to determine a second start frame and a second end frame in the second video that match the first start frame and the first end frame respectively; as well as a matching frame pair determining unit, configured to determine a plurality of matching video frame pairs from the first video and the second video based on frame numbers of the first start frame, the first end frame, the second start frame, and the second end frame; The first video is a high-resolution video, and the second video is a low-resolution video; or The first video is a low-resolution video, and the second video is a high-resolution video; Wherein, the device for video processing is further configured as: Build a neural network; Constructing a training data set, wherein the training data set includes a low-resolution video consisting of low-resolution video frames in the plurality of matching video frame pairs and a high-resolution video consisting of high-resolution video frames in the plurality of matching video frame pairs; Calculate a prediction value of the high-resolution video based on the low-resolution video in the training data set; Calculating the loss between the predicted value and the true value of the high-resolution video in the training dataset; and Based on the loss, parameters of the neural network are updated.

8. A computer-readable storage medium having computer program instructions stored thereon, wherein: The computer program instructions, when executed, implement the following steps: Obtain a first video and a second video shot by the first device and the second device respectively of the same content; Based on the similarity between adjacent frames of the first video, determining a first start frame and a first end frame in the first video corresponding to the start and end of the movement of the first device, respectively; Comparing a first start frame and a first end frame of the first video with respective frames of the second video to determine a second start frame and a second end frame in the second video that match the first start frame and the first end frame respectively; as well as Determine a plurality of matching video frame pairs from the first video and the second video based on the frame numbers of the first start frame, the first end frame, the second start frame, and the second end frame; The first video is a high-resolution video, and the second video is a low-resolution video; or The first video is a low-resolution video, and the second video is a high-resolution video; Wherein, the computer program instructions further implement the following steps when executed: Build a neural network; Constructing a training data set, wherein the training data set includes a low-resolution video consisting of low-resolution video frames in the plurality of matching video frame pairs and a high-resolution video consisting of high-resolution video frames in the plurality of matching video frame pairs; Calculate a prediction value of the high-resolution video based on the low-resolution video in the training data set; Calculating the loss between the predicted value and the true value of the high-resolution video in the training dataset; and Based on the loss, parameters of the neural network are updated.

Images