A video restoration method and device based on a computer vision algorithm

Abstract

The application relates to a video repairing method and device based on a computer vision algorithm. The method comprises the following steps: extracting a to-be-repaired area in each first image frame of a to-be-repaired video; deleting the to-be-repaired area from the corresponding first image frame to form a missing area in the first image frame; repairing the missing area to obtain a second image frame; and combining the second image frame into a video. The technical scheme provided by the application extracts to-be-repaired features through an image translation algorithm by using the recognized video, and finally obtains an original image through post-processing, thereby solving the problems of difficult original video acquisition, time and labor consumption of manual processing, reducing management complexity and personnel cost, and the scheme is targeted to specific to-be-repaired areas, has high repairing accuracy, strong generalization performance, high speed and the like.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a video restoration method and device based on computer vision algorithms. Background Art

[0002] With the development of artificial intelligence and 5G technologies, artificial intelligence is being used more and more widely in intelligent detection. Many videos will generate a series of recognition frames after being processed by the target detection algorithm. However, under certain conditions, the original video cannot be directly obtained. In this case, it is necessary to repair the video with the recognition frames after being processed by the target detection algorithm and remove the recognition frames to achieve the purpose of secondary utilization of the video image data.

[0003] Currently, most common algorithms support video resolution restoration or video super-resolution algorithms, but cannot effectively remove and repair other information on the video. There are also methods based on masks for image restoration, but they cannot automatically repair videos. Another method is to manually restore the identified video to the original video frame by frame through image editing, which is very time-consuming and labor-intensive. Summary of the Invention

[0004] In order to overcome the problems existing in the related art, the present invention provides a video restoration method and device based on computer vision algorithm.

[0005] According to a first aspect of an embodiment of the present invention, a video restoration method based on a computer vision algorithm is provided, comprising:

[0006] Extracting the area to be repaired in each first image frame of the video to be repaired;

[0007] Deleting the area to be repaired from the corresponding first image frame to form a missing area in the first image frame;

[0008] Repairing the missing area to obtain a second image frame;

[0009] The second image frames are merged into a video.

[0010] Furthermore, extracting the area to be repaired in each image frame of the video to be repaired specifically includes:

[0011] Performing frame processing on the video to be repaired to obtain a first image frame;

[0012] An image translation algorithm is used to extract the first image frame to obtain a first feature map of the area to be repaired.

[0013] Furthermore, the step of extracting the area to be repaired in each image frame of the video to be repaired further includes:

[0014] Grayscale, binarization, and dilation and corrosion processing are performed on the first feature map.

[0015] Furthermore, deleting the area to be repaired from the corresponding first image frame specifically includes:

[0016] Delete the pixel points at positions corresponding to the feature points of the first feature map in the area to be repaired.

[0017] Furthermore, repairing the missing area to obtain a second image frame specifically includes:

[0018] Finding pixels adjacent to the missing area in the first image frame;

[0019] The found pixel point is filled into the missing area closest to the pixel point to obtain a second image frame.

[0020] Furthermore, before deleting the area to be repaired from the corresponding first image frame, the method further includes:

[0021] The first image frame is reduced to obtain a third image frame, so that the size of the area to be repaired in the third image frame is the same as the size of the first feature map.

[0022] Furthermore, deleting the area to be repaired from the corresponding first image frame specifically includes:

[0023] Delete the pixel points at positions corresponding to the feature points of the first feature map in the area to be repaired in the third image frame.

[0024] Furthermore, repairing the missing area to obtain a second image frame specifically includes:

[0025] Finding pixels adjacent to the missing area in the third image frame;

[0026] Filling the found pixel point into the missing area closest to the pixel point to obtain a fourth image frame;

[0027] The fourth image frame is magnified and restored to the second image frame.

[0028] Further, the fourth image frame is magnified and restored to the second image frame, specifically comprising:

[0029] Performing bilinear interpolation to enlarge the fourth image frame and restore it to its original size to obtain a fifth image frame;

[0030] Perform bilinear interpolation and amplification on the first feature map to obtain a second feature map;

[0031] Deleting pixels in the first image frame at positions corresponding to feature points in the second feature map;

[0032] Overlay the pixel points in the fifth image frame corresponding to the feature points in the second feature map onto the first image frame to obtain a second image frame.

[0033] According to a second aspect of an embodiment of the present invention, a video restoration device based on a computer vision algorithm is provided, comprising:

[0034] An extraction module, configured to extract an area to be repaired in each first image frame of the video to be repaired;

[0035] a deleting module, configured to delete the area to be repaired from the corresponding first image frame, thereby forming a missing area in the first image frame;

[0036] A repair module, configured to repair the missing area to obtain a second image frame;

[0037] A merging module is used to merge the second image frames into a video.

[0038] Furthermore, the extraction module is specifically used to:

[0039] Performing frame processing on the video to be repaired to obtain a first image frame;

[0040] An image translation algorithm is used to extract the first image frame to obtain a first feature map of the area to be repaired.

[0041] Furthermore, the extraction module is further configured to:

[0042] Grayscale, binarization, and dilation and corrosion processing are performed on the first feature map.

[0043] Furthermore, the deletion module is specifically used to:

[0044] Delete the pixel points at positions corresponding to the feature points of the first feature map in the area to be repaired.

[0045] Furthermore, the repair module is specifically used to:

[0046] Finding pixels adjacent to the missing area in the first image frame;

[0047] The found pixel point is filled into the missing area closest to the pixel point to obtain a second image frame.

[0048] Furthermore, the device also includes:

[0049] A reduction module is used to reduce the first image frame to obtain a third image frame before the deletion module deletes the area to be repaired from the corresponding first image frame, so that the size of the area to be repaired in the third image frame is the same as the size of the first feature map.

[0050] Furthermore, the deletion module is specifically used to:

[0051] Delete the pixel points at positions corresponding to the feature points of the first feature map in the area to be repaired in the third image frame.

[0052] Furthermore, the repair module is specifically used to:

[0053] Finding pixels adjacent to the missing area in the third image frame;

[0054] Filling the found pixel point into the missing area closest to the pixel point to obtain a fourth image frame;

[0055] The fourth image frame is magnified and restored to the second image frame.

[0056] Furthermore, the restoration module magnifies and restores the fourth image frame to the second image frame, specifically including:

[0057] Performing bilinear interpolation to enlarge the fourth image frame and restore it to its original size to obtain a fifth image frame;

[0058] Perform bilinear interpolation and amplification on the first feature map to obtain a second feature map;

[0059] Deleting pixels in the first image frame at positions corresponding to feature points in the second feature map;

[0060] Overlay the pixel points in the fifth image frame corresponding to the feature points in the second feature map onto the first image frame to obtain a second image frame.

[0061] According to a third aspect of an embodiment of the present invention, a terminal device is provided, including:

[0062] processor; and

[0063] The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method described above.

[0064] According to a fourth aspect of an embodiment of the present invention, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method described above.

[0065] The technical solutions provided by the embodiments of the present invention may have the following beneficial effects:

[0066] This solution uses the recognized video to extract the features to be repaired through an image translation algorithm. After post-processing, the original image is finally obtained. This solves the problems of difficult original video acquisition and time-consuming and labor-intensive manual processing, reducing management complexity and personnel costs. In addition, this solution can identify specific areas to be repaired, with high repair accuracy, strong generalization performance, and fast speed.

[0067] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The above and other objects, features and advantages of the present invention will become more apparent through a more detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present invention.

[0069] Figure 1 is a flowchart illustrating a video restoration method based on a computer vision algorithm according to an exemplary embodiment of the present invention;

[0070] Figure 2 is a schematic diagram of the area to be repaired in the image;

[0071] Figure 3 is a schematic diagram of the first feature map of the area to be repaired;

[0072] Figure 4 It is a schematic diagram of the effect after deleting the pixels of the feature map corresponding to the area to be repaired;

[0073] Figure 5 It is a schematic diagram of generating pixel points of the area to be repaired;

[0074] Figure 6 This is a schematic diagram of the effect after the missing area is filled;

[0075] Figure 7 is a flowchart illustrating another video restoration method based on a computer vision algorithm according to an exemplary embodiment of the present invention;

[0076] Figure 8 is a schematic diagram of the second feature map after binarization;

[0077] Figure 9 is a schematic diagram of the fifth image frame after the missing area is filled and enlarged;

[0078] Figure 10 is a schematic diagram of the second image frame finally obtained after restoration;

[0079] Figure 11 It is the overall flow chart of the algorithm technical solution of the present invention;

[0080] Figure 12 4 is a structural block diagram of a video restoration device based on a computer vision algorithm according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0081] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to make the present invention more thorough and complete and to fully convey the scope of the present invention to those skilled in the art.

[0082] The terms used in this invention are for the purpose of describing specific embodiments only and are not intended to limit the invention. The singular forms "a," "the," and "the" used in this invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0083] It should be understood that although the terms "first", "second", "third", etc. may be used to describe various information in the present invention, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0084] The technical solutions of the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0085] Figure 1 The figure is a flowchart of a video restoration method based on a computer vision algorithm according to an exemplary embodiment of the present invention.

[0086] See also Figure 1 , the method comprising:

[0087] 100. Extract a region to be repaired in each first image frame of the video to be repaired.

[0088] Specifically, in this step, the video to be repaired needs to be divided into frames first, such as Figure 2As shown in the figure, the target detection algorithm will form a recognition frame in the image frame. The image translation algorithm is used to detect and extract the area to be repaired in the image frame, so that the recognition frame is extracted, as shown in the figure. Figure 3 shown.

[0089] 110. Delete the area to be repaired from the corresponding first image frame to form a missing area in the first image frame.

[0090] like Figure 4 The following figure shows the effect after deleting the identification box of the area to be repaired.

[0091] 120. Repair the missing area to obtain a second image frame;

[0092] Specifically, the image frame is used to obtain Figure 5 The pixel points of the area to be repaired are obtained by using the generated pixel points to fill the missing area formed after deleting the recognition frame, and a complete image frame can be obtained, as shown in FIG. Figure 6 shown.

[0093] 130. Merge the second image frames into a video.

[0094] Specifically, after the above processing, the processed image frames are merged to obtain a video without an identification frame.

[0095] Optionally, in this embodiment, step 100 specifically includes:

[0096] 1001. Perform frame processing on the video to be restored to obtain a first image frame;

[0097] 1002. Use an image translation algorithm to extract the first image frame to obtain a first feature map of the area to be repaired.

[0098] In order to remove noise points generated by algorithm misidentification, optionally, in this embodiment, step 100 further includes:

[0099] 1003. Perform grayscale conversion, binarization, and dilation and corrosion processing on the first feature map.

[0100] Optionally, in this embodiment, step 110 specifically includes:

[0101] Delete the pixel points at positions corresponding to the feature points of the first feature map in the area to be repaired.

[0102] Optionally, in this embodiment, step 120 specifically includes:

[0103] 1201. Searching for pixels adjacent to the missing area in the first image frame;

[0104] 1202. Fill the found pixel point into the missing area closest to the pixel point to obtain a second image frame.

[0105] Specifically, as shown in the figure, find the pixels on the top, bottom, left, and right sides of the missing area in the image frame, such as Figure 5 Then fill the missing area with the pixel closest to the pixel point. The effect after filling is as follows Figure 6 shown.

[0106] In step 1002, due to the limitation of some algorithms, the size of the feature map extracted by the algorithm will be smaller than the size of the identification box area to be repaired in the original image. In this case, the above method cannot be used to fill and repair directly. Therefore, the size of the original image needs to be reduced first. As an embodiment of the present invention, Figure 7 As shown, the method includes:

[0107] 200. Perform frame processing on the video to be restored to obtain a first image frame;

[0108] 210. Extract the first image frame using an image translation algorithm to obtain a first feature map of the area to be repaired;

[0109] 220. Reduce the first image frame to obtain a third image frame, so that the size of the area to be repaired in the third image frame is the same as the size of the first feature map;

[0110] 230. Delete the pixel points at positions corresponding to the feature points of the first feature map in the area to be repaired in the third image frame;

[0111] 240. Find pixels adjacent to the missing area in the third image frame;

[0112] 250. Fill the found pixel point into the missing area closest to the pixel point to obtain a fourth image frame;

[0113] 260. Enlarge the fourth image frame and restore it to the second image frame;

[0114] 270. Merge the second image frames into a video.

[0115] Optionally, in this embodiment, step 260 specifically includes:

[0116] 2601. Perform bilinear interpolation to enlarge the fourth image frame and restore it to its original size to obtain a fifth image frame.

[0117] 2602. Perform bilinear interpolation and amplification on the first feature map to obtain a second feature map;

[0118] 2603. Delete pixels in the first image frame at positions corresponding to feature points in the second feature map.

[0119] 2604. Overlay the pixel points in the fifth image frame at positions corresponding to the feature points in the second feature map onto the first image frame to obtain a second image frame.

[0120] Optionally, in step 2602, the second feature map is binarized in order to remove noise from the amplified feature map.

[0121] in, Figure 8 The second feature map after binarization is shown. Figure 9 is the fifth image frame, Figure 10 is the second image frame finally obtained after restoration.

[0122] The overall flow chart of the algorithm technical solution of the present invention is as follows Figure 11 shown.

[0123] Figure 12 4 is a structural block diagram of a video restoration device based on a computer vision algorithm according to an exemplary embodiment of the present invention.

[0124] See also Figure 12 , the system comprises:

[0125] An extraction module, configured to extract an area to be repaired in each first image frame of the video to be repaired;

[0126] a deleting module, configured to delete the area to be repaired from the corresponding first image frame, thereby forming a missing area in the first image frame;

[0127] A repair module, configured to repair the missing area to obtain a second image frame;

[0128] A merging module is used to merge the second image frames into a video.

[0129] Optionally, in this embodiment, the extraction module is specifically configured to:

[0130] Performing frame processing on the video to be repaired to obtain a first image frame;

[0131] An image translation algorithm is used to extract the first image frame to obtain a first feature map of the area to be repaired.

[0132] Optionally, in this embodiment, the extraction module is further configured to:

[0133] Grayscale, binarization, and dilation and corrosion processing are performed on the first feature map.

[0134] Optionally, in this embodiment, the deletion module is specifically configured to:

[0135] Delete the pixel points at positions corresponding to the feature points of the first feature map in the area to be repaired.

[0136] Optionally, in this embodiment, the repair module is specifically configured to:

[0137] Finding pixels adjacent to the missing area in the first image frame;

[0138] The found pixel point is filled into the missing area closest to the pixel point to obtain a second image frame.

[0139] Optionally, in this embodiment, the device further includes:

[0140] A reduction module is used to reduce the first image frame to obtain a third image frame before the deletion module deletes the area to be repaired from the corresponding first image frame, so that the size of the area to be repaired in the third image frame is the same as the size of the first feature map.

[0141] Optionally, in this embodiment, the deletion module is specifically configured to:

[0142] Delete the pixel points at positions corresponding to the feature points of the first feature map in the area to be repaired in the third image frame.

[0143] Optionally, in this embodiment, the repair module is specifically configured to:

[0144] Finding pixels adjacent to the missing area in the third image frame;

[0145] Filling the found pixel point into the missing area closest to the pixel point to obtain a fourth image frame;

[0146] The fourth image frame is magnified and restored to the second image frame.

[0147] Optionally, in this embodiment, the restoration module magnifies and restores the fourth image frame to the second image frame, specifically including:

[0148] Performing bilinear interpolation to enlarge the fourth image frame and restore it to its original size to obtain a fifth image frame;

[0149] Perform bilinear interpolation and amplification on the first feature map to obtain a second feature map;

[0150] Deleting pixels in the first image frame at positions corresponding to feature points in the second feature map;

[0151] Overlay the pixel points in the fifth image frame corresponding to the feature points in the second feature map onto the first image frame to obtain a second image frame.

[0152] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0153] A computing device according to an exemplary embodiment of the present invention includes a memory and a processor.

[0154] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0155] The memory may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even if the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as the permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory may store some or all of the instructions and data required by the processor during operation. In addition, the memory may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be used. In some embodiments, the memory may include a readable and / or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0156] The memory stores executable codes, which, when processed by the processor, can enable the processor to execute part or all of the above-mentioned methods.

[0157] In addition, the method according to the present invention may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present invention.

[0158] Alternatively, the present invention can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present invention.

[0159] The scheme of the present invention has been described in detail above with reference to the accompanying drawings. In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. Those skilled in the art should also be aware that the actions and modules involved in the description are not necessarily required by the present invention. In addition, it can be understood that the steps in the method of the embodiment of the present invention can be adjusted in order, merged and deleted according to actual needs, and the modules in the device of the embodiment of the present invention can be merged, divided and deleted according to actual needs.

[0160] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both.

[0161] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems and methods according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0162] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A video restoration method based on computer vision algorithm, characterized in that: include: Performing frame processing on the video to be repaired to obtain a first image frame; The first image frame is extracted using an image translation algorithm to obtain a first feature map of the area to be repaired, and the first feature map is grayscaled, binarized, and subjected to dilation and corrosion processing; Reducing the first image frame to obtain a third image frame, so that the size of the area to be repaired in the third image frame is the same as the size of the first feature map; Deleting pixels at positions corresponding to feature points of the first feature map in the area to be repaired in the third image frame, to form a missing area in the first image frame; Finding pixels adjacent to the missing area in the third image frame; Filling the found pixel point into the missing area closest to the pixel point to obtain a fourth image frame; Performing bilinear interpolation to enlarge the fourth image frame and restore it to its original size to obtain a fifth image frame; Perform bilinear interpolation and amplification on the first feature map to obtain a second feature map; Deleting pixels in the first image frame at positions corresponding to feature points in the second feature map; Overlaying the pixel points in the fifth image frame corresponding to the feature points in the second feature map onto the first image frame to obtain a second image frame; The second image frames are merged into a video.

2. A video restoration device based on computer vision algorithm, characterized in that: include: An extraction module, configured to perform frame processing on the video to be repaired to obtain a first image frame; The first image frame is extracted using an image translation algorithm to obtain a first feature map of the area to be repaired, and the first feature map is grayscaled, binarized, and subjected to dilation and corrosion processing; a reduction module, configured to reduce the first image frame to obtain a third image frame before the deletion module deletes the area to be repaired from the corresponding first image frame, so that the size of the area to be repaired in the third image frame is the same as the size of the first feature map; a deleting module, configured to delete pixels at positions corresponding to feature points of the first feature map in the area to be repaired in the third image frame, thereby forming a missing area in the first image frame; a restoration module configured to search for pixels adjacent to the missing region in the third image frame; fill the found pixels into the missing region closest to the pixels to obtain a fourth image frame; perform bilinear interpolation and amplification on the fourth image frame to restore it to its original size to obtain a fifth image frame; perform bilinear interpolation and amplification on the first feature map to obtain a second feature map; and delete pixels in the first image frame at positions corresponding to feature points in the second feature map; Overlaying the pixel points in the fifth image frame corresponding to the feature points in the second feature map onto the first image frame to obtain a second image frame; A merging module is used to merge the second image frames into a video.

Images