Video processing method, device and equipment, computer readable storage medium and product

Abstract

The embodiment of the invention provides a video processing method and device, equipment, a computer readable storage medium and a product, and the method comprises the steps: obtaining a to-be-processed video, and recognizing at least one to-be-repaired defect corresponding to a current video frame in the to-be-processed video; determining motion information corresponding to the current video frame according to the current video frame and at least one associated video frame corresponding to the current video frame; repairing at least one to-be-repaired defect corresponding to the current video frame based on the motion information to obtain a repaired first video frame; and performing visual effect restoration on the first video frame according to a preset restoration algorithm to obtain a target video frame, and obtaining a restored target video based on at least one target video frame. Therefore, the repaired target video does not comprise a defect area, a better visual effect is presented, and the video quality of the repaired target video is improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of image processing technologies, and in particular, to a video processing method, apparatus, device, computer-readable storage medium, and product. Background Art

[0002] With the continuous development of audio-visual technologies, when playing old video works using current playback tools, there may often be problems such as scratches, dropped frames, decolorization, etc., which affect the visual experience of the video works. Therefore, how to repair the defects of old video works to make them present a better visual effect has become an urgent problem to be solved. Summary of the Invention

[0003] Embodiments of the present disclosure provide a video processing method, apparatus, device, computer-readable storage medium, and product for solving the technical problems of scratches, dropped frames, and decolorization in old videos.

[0004] In a first aspect, embodiments of the present disclosure provide a video processing method, including:

[0005] Obtain a video to be processed, and identify at least one defect to be repaired corresponding to the current video frame in the video to be processed;

[0006] Determine the motion information corresponding to the current video frame according to the current video frame and at least one associated video frame corresponding to the current video frame;

[0007] Perform a repair operation on at least one defect to be repaired corresponding to the current video frame based on the motion information to obtain a first repaired video frame;

[0008] Perform visual effect repair on the first video frame according to a preset repair algorithm to obtain a target video frame, and obtain a repaired target video based on at least one target video frame.

[0009] In a second aspect, embodiments of the present disclosure provide a video processing apparatus, including:

[0010] An acquisition module, configured to obtain a video to be processed, and identify at least one defect to be repaired corresponding to the current video frame in the video to be processed;

[0011] A determination module, configured to determine the motion information corresponding to the current video frame according to the current video frame and at least one associated video frame corresponding to the current video frame;

[0012] A repair module, configured to perform a repair operation on at least one defect to be repaired corresponding to the current video frame based on the motion information to obtain a first repaired video frame;

[0013] A processing module, configured to perform visual effect repair on the first video frame according to a preset repair algorithm to obtain a target video frame, and obtain a repaired target video based on at least one target video frame.

[0014] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0015] The memory stores computer-executable instructions;

[0016] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the video processing method described in the first aspect and various possible designs of the first aspect as above.

[0017] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When the processor executes the computer-executable instructions, the video processing method described in the first aspect and various possible designs of the first aspect as above is implemented.

[0018] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, and when the computer program is executed by a processor, the video processing method described in the first aspect and various possible designs of the first aspect as above is implemented.

[0019] The video processing method, device, equipment, computer-readable storage medium and product provided in this embodiment, after obtaining the video to be processed, determine the defect to be repaired corresponding to the current video frame, and perform defect repair on the defect to be repaired based on the motion information corresponding to the current video frame. Thus, the motion trend of the content main body in the current video frame can be determined based on at least one associated video frame corresponding to the current video frame, and then the defect can be accurately repaired based on this motion trend, improving the accuracy of defect repair. In addition, after completing the defect repair of the current video frame, the visual effect of the first video frame can also be repaired, so that the repaired target video does not include the defect area and presents a better visual effect, improving the video quality of the repaired target video. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0021] Figure 1 It is a schematic flowchart of the video processing method provided by the embodiment of the present disclosure;

[0022] Figure 2 Schematic diagram of video frame repair provided by an embodiment of the present disclosure;

[0023] Figure 3 Flow schematic diagram of a video processing method provided by another embodiment of the present disclosure;

[0024] Figure 4 Flow schematic diagram of a video processing method provided by another embodiment of the present disclosure;

[0025] Figure 5 Flow schematic diagram of a video processing method provided by another embodiment of the present disclosure;

[0026] Figure 6 Schematic diagram of the structure of a video processing apparatus provided by an embodiment of the present disclosure;

[0027] Figure 7 Schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Detailed implementation manners

[0028] To make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are some but not all of the embodiments of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present disclosure without creative efforts shall fall within the scope of protection of the present disclosure.

[0029] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved in the present disclosure shall be informed to the user and the user's authorization shall be obtained by appropriate means in accordance with relevant laws and regulations.

[0030] For example, when responding to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested by the user will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, an application program, a server, or a storage medium that executes the operation of the technical solution of the present disclosure according to the prompt message.

[0031] As an optional but non-limiting implementation manner, the manner of sending a prompt message to the user in response to receiving an active request from the user may be, for example, in the form of a pop-up window, and the prompt message may be presented in text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0032] It should be understood that the above notification and the process of obtaining user authorization are only illustrative and do not limit the implementation manner of the present disclosure. Other methods that comply with relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0033] To solve the technical problems of scratches, dropped frames, and decolorization in old videos, the present disclosure provides a video processing method, apparatus, device, computer-readable storage medium, and product.

[0034] It should be noted that the video processing method, apparatus, device, computer-readable storage medium, and product provided by the present disclosure can be applied to any video restoration scenario.

[0035] Due to their long age, many classic old movies have problems such as scratches, poor details, decolorization, and dropped frames when played on current video playback tools, which affect the visual experience. Current old movie restoration methods generally only focus on restoring the scratches in old movies. However, old movies generally have problems such as decolorization and dropped frames, and only restoring the scratches in old movies will result in poor restoration effects and affect the viewing effect of old movies.

[0036] In the process of solving the above technical problems, the inventors found through research that in order to ensure the restoration effect of old movies, the scratches and noises corresponding to each video frame can be restored based on the optical flow information corresponding to each video frame. In addition, the color and resolution of each video frame can also be restored to improve the visual effect of the video frame. To improve the smoothness of the target video after restoration, frame interpolation operations can also be performed on video parts with a low frame rate. Thus, the old movies can be restored comprehensively, and the video quality of the target video after restoration can be improved.

[0037] The system architecture on which the present disclosure is based at least includes a terminal device and a server. Among them, the terminal device is communicatively connected to the server, so that the server can obtain a restoration request sent by the terminal device, obtain the video to be processed based on the restoration request, and perform restoration operations on the defects, color, resolution, and frame rate of the video to be processed based on at least one preset restoration algorithm to improve the video quality of the target video after restoration.

[0038] Figure 1 For the flowchart of the video processing method provided by the embodiments of the present disclosure, as Figure 1 shown, the method includes:

[0039] Step 101, obtain the video to be processed, and identify at least one defect to be restored corresponding to the current video frame in the video to be processed.

[0040] The execution subject of this embodiment is a video processing device. The video processing device can be coupled to a server. The server can communicate with a terminal device. Thus, it can obtain a repair request sent by the terminal device, and obtain a video to be processed based on the repair request. Perform repair operations on defects, color, resolution, frame rate, etc. of the video to be processed, so as to improve the video quality of the repaired target video.

[0041] Alternatively, the video processing device can be coupled to a terminal device, and a variety of repair algorithms can be preset in the terminal device. Thus, it can obtain a video to be processed in response to a video repair request triggered by a user on the terminal device. Perform repair operations on defects, color, resolution, frame rate, etc. of the video to be processed, so as to improve the video quality of the repaired target video.

[0042] In practical applications, there may be a large number of videos with poor video quality that need to be repaired. For example, old movie works, videos shot in scenes with poor lighting, etc. In order to implement video repair and optimization operations, a user can trigger a video repair request on the terminal device.

[0043] Correspondingly, the video processing device can obtain the video repair request and obtain the video to be processed based on the video repair request. The video to be processed can be obtained by the user from a preset storage path.

[0044] Furthermore, in order to implement the repair operation on the video to be processed, each video frame in the video to be processed can be sequentially determined as the current video frame, and at least one defect to be repaired corresponding to the current video frame can be identified. Among them, the defects to be repaired include but are not limited to scratch defects, noise defects, etc.

[0045] Step 102: Determine the motion information corresponding to the current video frame according to the current video frame and at least one associated video frame corresponding to the current video frame.

[0046] In this embodiment, after at least one defect to be repaired is identified, the motion information corresponding to the current video frame can be determined according to the current video frame and at least one associated video frame corresponding to the current video frame. Among them, the motion information includes but is not limited to optical flow information or any other information that can characterize the motion of the displayed content in the current video frame.

[0047] In practical applications, if there is a scratch on the current video frame, and there may be no scratch on the adjacent associated video frame with similar displayed content. Therefore, the repair operation on the current video frame can be implemented based on the associated video frame.

[0048] By identifying the motion information corresponding to the current video frame, it is possible to determine the position in the associated video frame that matches the defect of the current video frame based on this motion information. Furthermore, it is possible to perform a scratch repair operation on the current video frame based on the display content in the associated video frame, improving the accuracy of video repair.

[0049] Step 103: Perform a repair operation on at least one defect to be repaired corresponding to the current video frame based on the motion information, and obtain a first repaired video frame.

[0050] In this embodiment, after determining the motion information corresponding to the current video frame, a repair operation can be performed on at least one defect to be repaired corresponding to the current video frame based on the motion information, and a first repaired video frame can be obtained. Among them, a defect repair model can be preset in advance, and the repair operation on the current video frame can be implemented through this defect repair model.

[0051] Step 104: Perform a visual effect repair on the first video frame according to a preset repair algorithm to obtain a target video frame, and obtain a repaired target video based on at least one target video frame.

[0052] In this embodiment, after completing the repair of defects such as scratches and noise in the video to be processed, in order to improve the visual effect of the video to be processed, repair operations can also be performed on the color and resolution corresponding to the video to be processed.

[0053] Optionally, after obtaining the first video frame with defects removed, a visual effect repair can be performed on the first video frame to obtain a target video frame. Among them, this visual effect restoration includes, but is not limited to, operations such as color repair and resolution repair.

[0054] Furthermore, after obtaining at least one target video frame, the at least one target video frame can be combined according to the timestamp information of the at least one target video frame to obtain a repaired target video.

[0055] Figure 2 Schematic diagram of video frame repair provided by an embodiment of the present disclosure, as Figure 2 shown, there are at least one scratch 22 in video frame 21. After obtaining video frame 21, a repair operation can be performed on at least one scratch 22 based on the motion information corresponding to this video frame 21 to obtain a first repaired video frame 23. Furthermore, the color and resolution of the first video frame 23 can be adjusted to obtain a target video frame 24.

[0056] The video processing method provided in this embodiment performs defect repair on the content to be repaired based on the motion information corresponding to the current video frame after obtaining the video to be processed, so that the defect repair operation can be accurately performed based on the motion trend of the content main body in the current video frame, improving the accuracy of defect repair. In addition, after completing the defect repair of the current video frame, the visual effect of the first video frame can also be repaired, so that on the basis of realizing defect repair, the repaired target video can present a better visual effect and improve the video quality of the repaired target video.

[0057] Optionally, based on any of the above embodiments, the defects to be repaired include scratch defects and noise defects. Step 101 includes:

[0058] Input the current video frame into a preset defect recognition model, and obtain the recognition result output by the defect recognition model, where the recognition result includes the position information of scratch defects and / or noise defects.

[0059] Determine the recognition result as the at least one defect to be repaired.

[0060] In this embodiment, the defects to be repaired include, but are not limited to, scratch defects and noise defects.

[0061] In order to realize the recognition of the defects to be repaired in the current video frame, a defect recognition model can be preset. Among them, the defect recognition model can be obtained by iteratively training based on a large amount of training data in advance. The training data can include multiple training data pairs, and each training data pair includes an image with at least one scratch and / or noise, and an image without scratch and / or noise. The preset model to be trained can be iteratively trained through the training data pairs until it meets the preset convergence condition to obtain the defect recognition model. Thus, the defect recognition model can accurately realize the recognition operation of scratch and / or noise defects.

[0062] Therefore, after obtaining the video to be processed, each video frame can be sequentially determined as the current video frame, the current video frame is input into the preset defect recognition model, and the recognition result output by the defect recognition model is obtained, where the recognition result includes the position information of scratch defects and / or noise defects. The recognition result is determined as at least one defect to be repaired.

[0063] The video processing method provided in this embodiment pre-sets a defect recognition model, so that after obtaining the current video frame, the scratch and / or noise in the current video frame can be accurately recognized based on the defect recognition model, and then the video repair operation can be accurately performed based on the recognition result, improving the accuracy of video repair.

[0064] Figure 3The flowchart of the video processing method provided by another embodiment of the present disclosure. Based on any of the above embodiments, as Figure 3 shown, step 102 includes:

[0065] Step 301: According to the timestamp information corresponding to the video frames in the video to be processed, determine at least one video frame before the current video frame and at least one video frame after the current video frame as the associated video frames.

[0066] Step 302: Calculate the optical flow information corresponding to the current video frame based on the current video frame and the at least one associated video frame, and determine the optical flow information as the motion information.

[0067] In this embodiment, in order to implement the repair operation on the video to be processed, for each video frame in the video to be processed, it is necessary to sequentially determine each video frame as the current video frame and determine at least one associated video frame corresponding to the current video frame.

[0068] Optionally, in the video to be processed, each video frame corresponds to timestamp information. Therefore, at least one video frame before the current video frame and at least one video frame after the current video frame can be determined as associated video frames based on this timestamp information. And calculate the optical flow information corresponding to the current video frame based on the current video frame and the at least one associated video frame, and determine the optical flow information as the motion information. Among them, any optical flow algorithm can be used to implement the calculation of the optical flow information, and the present disclosure does not limit this.

[0069] The video processing method provided by this embodiment can accurately determine the motion information of the displayed content in the current video frame based on the optical flow information by calculating the optical flow information based on the current video frame and at least one video frame before and after it. Furthermore, the position in the associated video frames that matches the defect of the current video frame can be determined based on the optical flow information, and then the current video frame can be repaired for scratches based on the displayed content in the associated video frames, improving the accuracy of video repair.

[0070] Further, based on any of the above embodiments, step 103 includes:

[0071] Perform a picture alignment operation on the current video frame and the at least one associated video frame according to the optical flow information to obtain multiple aligned video frames.

[0072] Input the multiple aligned video frames into a preset defect repair model to obtain the first video frame.

[0073] In this embodiment, in order to implement the defect repair operation for the current video frame, a defect repair model can be preset. Among them, the defect repair model can be obtained after iterative training on preset training data until the model converges. Among them, the training data can include multiple training data pairs, and each training data pair respectively includes multiple video frames with defects after alignment and multiple video frames without defects after alignment. By performing iterative training based on the training data, the defect repair model can accurately implement the defect repair operation.

[0074] Due to the changes in the content of the video to be processed, there will also be slight differences between consecutive video frames (such as the displacement of objects, the change of perspectives, etc.). Therefore, after determining the optical flow information corresponding to the current video frame and at least one associated video frame, the current video frame and at least one associated video frame can be subjected to a video alignment operation based on the optical flow information to obtain multiple aligned video frames, so that the content / objects in the multiple aligned video frames are in the same position.

[0075] Further, after completing the alignment operation on multiple video frames, the multiple aligned video frames can be input into a preset defect repair model, and the defect repair model can implement the defect repair operation to obtain a first video frame without defects.

[0076] The video processing method provided in this embodiment can accurately perform a repair operation on multiple aligned video frames based on the preset defect repair model, thereby improving the efficiency and accuracy of video repair.

[0077] Figure 4 As shown in the flowchart of the video processing method provided in another embodiment of the present disclosure, based on any of the above embodiments, Figure 4 Step 104 includes:

[0078] Step 401: Perform a color repair operation on the first video frame to obtain a second video frame with color adjustment.

[0079] Step 402: Perform a resolution adjustment operation on the second video frame to obtain the target video frame.

[0080] In this embodiment, after completing the repair of defects such as scratches and noises in the video to be processed, in order to improve the visual effect of the video to be processed, color and resolution repair operations can also be performed on the video to be processed.

[0081] Optionally, after obtaining the first video frame with defects removed, a color restoration operation can be performed on the first video frame to obtain a second video frame with color adjusted. Thus, the second video frame can present a better and more beautiful color effect. Further, a resolution adjustment operation can be performed on the second video frame, so that a target video frame with clear and beautiful color can be obtained.

[0082] The video processing method provided in this embodiment can obtain a target video frame with beautiful color and high resolution by performing a color restoration operation on the first video frame and a resolution adjustment operation on the second video frame after color restoration, thereby improving the visual effect of the target video.

[0083] Further, based on any of the above embodiments, step 401 includes:

[0084] If the video to be processed is a color video, identify the color parameters corresponding to the first video frame, and perform a color adjustment operation on the first video frame according to the color parameters and a preset color adjustment algorithm to obtain the second video frame.

[0085] If the video to be processed is a black-and-white video, input the restored video frame into a preset coloring model to obtain a color video frame, identify the color parameters corresponding to the color video frame, and perform a color adjustment operation on the color video frame according to the color parameters and a preset color adjustment algorithm to obtain the second video frame.

[0086] In this embodiment, the video to be processed can be a color video or a black-and-white video. For different video types, different color restoration methods can be used to perform restoration operations on the video to be processed.

[0087] Optionally, when the video to be processed is a color video, over time, the video to be processed may fade or the like, resulting in a poor visual effect of the video to be processed. Therefore, the color parameters corresponding to the first video frame can be identified, where the color parameters include but are not limited to parameters such as color, saturation, contrast, brightness, and light sense. After determining the color parameters, a color adjustment operation can be performed on the first video frame according to the color parameters and a preset color adjustment algorithm to obtain the second video frame. The color adjustment algorithm can be a color adjustment model pre-trained based on a large amount of training data.

[0088] As an implementable manner, multiple groups of relatively beautiful color matching information can be preset in advance. After identifying the color information corresponding to the first video frame, the target color matching information closest to the color information can be determined from the preset color matching information, and a color adjustment operation is performed on the first video frame based on the target color matching information. The present disclosure does not limit this.

[0089] Optionally, when the video to be processed is a black-and-white video, first, the black-and-white video to be processed can be colored. Then, color restoration operation can be performed on the colored video to be processed. Among them, the restored video frame can be input into a preset coloring model to obtain a colored video frame, and the color parameters corresponding to the colored video frame can be identified. The color parameters include, but are not limited to, parameters such as color, saturation, contrast, brightness, and light sense. According to the color parameters and a preset color adjustment algorithm, color adjustment operation is performed on the colored video frame to obtain a second video frame. Among them, the color adjustment scheme for the colored video frame is the same as the above-mentioned color adjustment scheme for the first video frame, and will not be elaborated here.

[0090] The video processing method provided in this embodiment can make the color effect of the colored video more beautiful by performing color adjustment on the first video frame for the colored video, coloring the first video frame for the black-and-white video, and performing color adjustment operation on the colored video frame after coloring. Moreover, on the basis of realizing the adjustment of the black-and-white video to a colored video, the color effect of the colored video can be made more beautiful.

[0091] Further, on the basis of any of the above embodiments, step 402 includes:

[0092] Performing super-resolution processing on the second video frame through a preset super-resolution algorithm to obtain a third video frame.

[0093] Identifying the target area in the third video frame, and performing repair operation on the target area through a preset image repair algorithm to obtain the target video frame.

[0094] In this embodiment, the video to be processed from an earlier era generally has a low resolution, resulting in a poor viewing experience. Therefore, in order to further improve the repair effect of the video to be processed, after obtaining the second video frame after color restoration, the second video frame can also be subjected to super-resolution processing through a preset super-resolution algorithm to obtain a third video frame.

[0095] Further, the third video frame may include a target area. The target area may be a facial area, a limb area, a text area, etc. In order to improve the display effect of the third video frame and reduce the computational complexity during the repair process, targeted repair operation can be performed on the target area.

[0096] Optionally, the target area in the third video frame can be identified. For different target areas, different identification algorithms can be selected for the identification operation. For example, when the target area is a facial area, a preset facial recognition model can be used to implement the identification operation of the target area.

[0097] Further, the target area can be repaired through a preset image repair algorithm to obtain the target video frame. Continuing with the above example, after identifying the facial area, a preset facial repair model can be used to implement the repair operation on the facial area.

[0098] The video processing method provided in this embodiment can further improve the content quality of the target video by performing super-resolution processing on the second video frame and optimizing the target area of the third video frame.

[0099] Figure 5 It is a schematic flowchart of the video processing method provided in another embodiment of the present disclosure. Based on any of the above embodiments, as Figure 5 described, step 104 includes:

[0100] Step 501: Combine the at least one target video frame according to the timestamp information corresponding to the at least one target video frame to obtain a video to be processed.

[0101] Step 502: Determine the frame rate information corresponding to the video to be processed.

[0102] Step 503: If the frame rate information is lower than a preset frame rate threshold and any two target video frames meet the preset conditions, perform an interpolation operation on the video to be processed to obtain the target video.

[0103] Frame rate is the frequency (rate) at which bitmap images in frames appear continuously on the display. A high frame rate can result in a smoother and more realistic visual effect.

[0104] Taking the video to be processed as an old movie as an example, old movies generally have a low frame rate, resulting in a poor viewing experience for users when playing.

[0105] Therefore, after completing the defect repair and visual effect repair of the current video frame, in order to further improve the video quality of the target video, an interpolation operation can also be performed to increase the frame rate of the target video.

[0106] Optionally, after completing the repair operation on each video frame corresponding to the video to be processed, the at least one target video frame can be combined according to the timestamp information corresponding to the at least one target video frame to obtain a video to be processed. This video to be processed has the same frame rate as the video to be processed.

[0107] To implement the repair operation on the video to be processed, the frame rate information corresponding to the video to be processed can be determined. Among them, any method for determining the frame rate can be used to determine the frame rate information of the video to be processed, and the present disclosure does not limit this.

[0108] Further, if the frame rate information is lower than a preset frame rate threshold and any two target video frames meet a preset condition, an interpolation operation is performed on the video to be processed to obtain a target video. Wherein, the preset condition may be that the inter-frame change is less than a preset change threshold.

[0109] Optionally, an interpolation model can be preset, which can automatically calculate optical flow information based on two consecutive frames of images in the video to be processed, and then insert video frames between these two frames based on the optical flow information to implement the interpolation operation. Alternatively, the interpolation operation on the video to be processed can also be implemented by any other interpolation method, and the present disclosure does not limit this.

[0110] The video processing method provided in this embodiment can improve the smoothness of the target video and further improve the content quality of the target video by performing an interpolation operation on the video to be processed.

[0111] Figure 6 is a schematic structural diagram of a video processing device provided in an embodiment of the present disclosure, as Figure 6 shown, the device includes: an acquisition module 61, a determination module 62, a repair module 63, and a processing module 64. Among them, the acquisition module 61 is used to acquire a video to be processed and identify at least one defect to be repaired corresponding to the current video frame in the video to be processed. The determination module 62 is used to determine the motion information corresponding to the current video frame according to the current video frame and at least one associated video frame corresponding to the current video frame. The repair module 63 is used to perform a repair operation on at least one defect to be repaired corresponding to the current video frame based on the motion information to obtain a first video frame after repair. The processing module 64 is used to perform a visual effect repair on the first video frame according to a preset repair algorithm to obtain a target video frame, and obtain a repaired target video based on at least one target video frame.

[0112] Further, based on any of the above embodiments, the defects to be repaired include scratch defects and noise defects. The acquisition module is used to: input the current video frame into a preset defect recognition model, and obtain the recognition result output by the defect recognition model, where the recognition result includes the position information of scratch defects and / or noise defects. Determine the recognition result as the at least one defect to be repaired.

[0113] Further, based on any of the above embodiments, the determination module is used to: determine at least one video frame before the current video frame and at least one video frame after the current video frame as the associated video frames according to the timestamp information corresponding to the video frames in the video to be processed. Calculate the optical flow information corresponding to the current video frame based on the current video frame and the at least one associated video frame, and determine the optical flow information as the motion information.

[0114] Further, based on any of the above embodiments, the repair module is configured to: perform picture alignment operations on the current video frame and the at least one associated video frame according to the optical flow information to obtain a plurality of aligned video frames. Input the plurality of aligned video frames into a preset defect repair model to obtain the first video frame.

[0115] Further, based on any of the above embodiments, the processing module is configured to: perform color restoration operations on the first video frame to obtain a second video frame with adjusted color. Perform resolution adjustment operations on the second video frame to obtain the target video frame.

[0116] Further, based on any of the above embodiments, the processing module is configured to: if the video to be processed is a color video, identify the color parameters corresponding to the first video frame, and perform color adjustment operations on the first video frame according to the color parameters and a preset color adjustment algorithm to obtain the second video frame. If the video to be processed is a black-and-white video, input the repaired video frame into a preset colorization model to obtain a color video frame, identify the color parameters corresponding to the color video frame, and perform color adjustment operations on the color video frame according to the color parameters and a preset color adjustment algorithm to obtain the second video frame.

[0117] Further, based on any of the above embodiments, the processing module is configured to: perform super-resolution processing on the second video frame through a preset super-resolution algorithm to obtain a third video frame. Identify the target area in the third video frame, and perform repair operations on the target area through a preset image repair algorithm to obtain the target video frame.

[0118] Further, based on any of the above embodiments, the processing module is configured to: combine the at least one target video frame according to the timestamp information corresponding to the at least one target video frame to obtain a video to be processed. Determine the frame rate information corresponding to the video to be processed. If the frame rate information is lower than a preset frame rate threshold and any two target video frames meet a preset condition, perform frame interpolation operations on the video to be processed to obtain the target video.

[0119] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments, and its implementation principles and technical effects are similar, which will not be elaborated here in this embodiment.

[0120] To implement the above embodiments, the present disclosure also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the video processing method as described in any of the above embodiments.

[0121] To implement the above embodiments, an embodiment of the present disclosure also provides a computer program product, including a computer program which, when executed by a processor, implements the video processing method described in any of the above embodiments.

[0122] To implement the above embodiments, an embodiment of the present disclosure also provides an electronic device, including: a processor and a memory;

[0123] The memory stores computer-executable instructions;

[0124] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the video processing method described in any of the above embodiments.

[0125] Figure 7 FIG. is a schematic structural diagram of the electronic device provided by an embodiment of the present disclosure. The electronic device 700 may be a terminal device or a server. Among them, the terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The illustrated electronic device is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present disclosure.

[0126] As Figure 7 shown, the electronic device 700 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 701, which may perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0127] Typically, the following devices can be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 can allow the electronic device 700 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 7 an electronic device 700 with various devices is shown, it should be understood that it is not required to implement or have all the shown devices. Instead, more or fewer devices can be implemented or had.

[0128] In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program product that includes a computer program carried on a computer-readable medium, and the computer program contains program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above functions defined in the methods of the embodiments of the present disclosure are executed.

[0129] It should be noted that the computer-readable medium described above in the present disclosure can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0130] The above computer-readable medium can be included in the above electronic device; or it can exist separately and not be assembled into the electronic device.

[0131] The above computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device is caused to execute the method shown in the above embodiments.

[0132] Computer program code for performing the operations of this disclosure may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, execute as a stand-alone software package, execute partially on the user's computer and partially on a remote computer, or execute entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0134] The units involved in the embodiments described in this disclosure may be implemented in software or in hardware. Among them, the name of the unit does not constitute a limitation to the unit itself in some cases. For example, the first acquisition unit may also be described as "the unit for acquiring at least two Internet protocol addresses".

[0135] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), system on a chip (SOC), complex programmable logic devices (CPLD), and so on.

[0136] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0137] The above description is only a preferred embodiment of the present disclosure and an illustration of the applied technical principles. Those skilled in the art should understand that the scope of the disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosure concept. For example, the technical solutions formed by mutually replacing the above features with the (but not limited to) technical features having similar functions disclosed in the present disclosure.

[0138] In addition, although the operations are depicted in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments can also be implemented combinatorially in a single embodiment. Conversely, the various features described in the context of a single embodiment can also be implemented separately or in any suitable sub-combination in multiple embodiments.

[0139] Although the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A video processing method, characterized in that, comprising: obtaining a video to be processed, and identifying at least one defect to be repaired corresponding to the current video frame in the video to be processed; determining motion information corresponding to the current video frame according to the current video frame and at least one associated video frame corresponding to the current video frame; performing a repair operation on at least one defect to be repaired corresponding to the current video frame based on the motion information to obtain a first video frame after repair; performing visual effect repair on the first video frame according to a preset repair algorithm to obtain a target video frame, and obtaining a repaired target video based on at least one target video frame.

2. The method according to claim 1, characterized in that, the defects to be repaired include scratch defects and noise defects; the identifying at least one defect to be repaired corresponding to the current video frame in the video to be processed includes: inputting the current video frame into a preset defect recognition model, and obtaining an identification result output by the defect recognition model, where the identification result includes position information of scratch defects and / or noise defects; determining the identification result as the at least one defect to be repaired.

3. The method according to claim 1, characterized in that, the determining motion information corresponding to the current video frame according to the current video frame and at least one associated video frame corresponding to the current video frame includes: determining at least one video frame before the current video frame and at least one video frame after the current video frame as the associated video frames according to the timestamp information corresponding to the video frames in the video to be processed; calculating optical flow information corresponding to the current video frame based on the current video frame and the at least one associated video frame, and determining the optical flow information as the motion information.

4. The method according to claim 3, characterized in that, the performing a repair operation on at least one defect to be repaired corresponding to the current video frame based on the motion information to obtain a first video frame after repair includes: performing a picture alignment operation on the current video frame and the at least one associated video frame according to the optical flow information to obtain a plurality of aligned video frames; inputting the plurality of aligned video frames into a preset defect repair model to obtain the first video frame.

5. The method according to any one of claims 1-4, characterized in that, the performing visual effect repair on the first video frame according to a preset repair algorithm to obtain a target video frame includes: performing a color repair operation on the first video frame to obtain a second video frame after color adjustment; performing a resolution adjustment operation on the second video frame to obtain the target video frame.

6. The method according to claim 5, characterized in that, the performing a color repair operation on the repaired video frame to obtain a second video frame after color adjustment includes: if the video to be processed is a color video, identifying color parameters corresponding to the first video frame, and performing a color adjustment operation on the first video frame according to the color parameters and a preset color adjustment algorithm to obtain the second video frame; If the video to be processed is a black-and-white video, input the restored video frame into a preset colorization model to obtain a color video frame, identify the color parameters corresponding to the color video frame, and perform a color adjustment operation on the color video frame according to the color parameters and a preset color adjustment algorithm to obtain the second video frame.

7. The method according to claim 5, wherein, the operation of adjusting the resolution of the second video frame to obtain the target video frame includes: performing super-resolution processing on the second video frame through a preset super-resolution algorithm to obtain a third video frame; identifying a target area in the third video frame, and performing a repair operation on the target area through a preset image repair algorithm to obtain the target video frame.

8. The method according to any one of claims 1-4, wherein, obtaining the repaired target video based on at least one target video frame includes: combining the at least one target video frame according to the timestamp information corresponding to the at least one target video frame to obtain a video to be processed; determining the frame rate information corresponding to the video to be processed; if the frame rate information is lower than a preset frame rate threshold and any two target video frames meet a preset condition, then perform an interpolation operation on the video to be processed to obtain the target video.

9. A video processing device, wherein, it includes: an acquisition module, configured to acquire a video to be processed and identify at least one defect to be repaired corresponding to the current video frame in the video to be processed; a determination module, configured to determine the motion information corresponding to the current video frame according to the current video frame and at least one associated video frame corresponding to the current video frame; a repair module, configured to perform a repair operation on at least one defect to be repaired corresponding to the current video frame based on the motion information to obtain a first restored video frame; a processing module, configured to perform a visual effect repair on the first video frame according to a preset repair algorithm to obtain a target video frame, and obtain a repaired target video based on at least one target video frame.

10. An electronic device, wherein, it includes: a processor and a memory; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, so that the processor executes the video processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium, wherein, computer execution instructions are stored in the computer-readable storage medium, and when the processor executes the computer execution instructions, the video processing method according to any one of claims 1 to 8 is implemented.

Images