Video-based image restoration method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video-based image restoration method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence and financial science and technology. The method comprises the steps of obtaining a repairing object; selecting a reference video frame image and a damaged video frame image based on the restoration object; in the reference video frame image, extracting a plurality of image blocks containing the restoration object; performing image description generation on each image block to obtain a plurality of pieces of image description information; generating a repair prompt word according to the image description information; performing image generation according to the repair prompt word to obtain a reconstructed image; and repairing the damaged video frame image by using the reconstructed image. The restoration prompt word obtained by the embodiment of the invention comprises different levels of spatial information of the restoration object in the same time dimension, and more details of the surrounding environment of the restoration object can be provided for image generation, so that the generated reconstructed image can be better fused into the damaged video frame image, and the restoration effect of the target video frame image is improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and financial technology, and in particular to a video-based image restoration method and device, an electronic device, and a storage medium. Background Art

[0002] Image restoration technology aims to restore the damaged or missing parts of an image to make it as consistent as possible with the original image. In related technologies, image generation technology is used to reconstruct some areas of an image and then fill them into the original image. However, the generated reconstructed image may have significant differences from the original image in terms of texture, color, lighting, style, etc., and cannot be well integrated into the original image, resulting in poor image restoration effects. Therefore, how to improve the image restoration effect has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a video-based image restoration method and device, an electronic device and a storage medium, aiming to improve the image restoration effect.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a video-based image restoration method, the method comprising:

[0005] Obtain a video frame image set from the original video data;

[0006] Performing target detection on the video frame image set to obtain a repaired object;

[0007] Selecting a reference video frame image and a damaged video frame image from the video frame image set based on the repair object, wherein the reference video frame image and the damaged video frame image contain the same repair object;

[0008] Extracting at least two image blocks containing the repair object in the reference video frame image, wherein the spatial scales of any two of the image blocks are different;

[0009] Generating an image description for each of the image blocks to obtain a plurality of image description information;

[0010] Generate a repair prompt word according to the image description information;

[0011] Generate an image of the repair object according to the repair prompt word to obtain a reconstructed image;

[0012] The reconstructed image is used to replace the repaired object in the damaged video frame image to obtain a target video frame image.

[0013] In some embodiments, the at least two image blocks include a first image block and a second image block; and extracting the at least two image blocks containing the repaired object includes:

[0014] In the reference video frame image, extracting the contour of the repaired object to obtain the contour of the repaired object;

[0015] In the reference video frame image, extracting a region surrounded by the contour of the repaired object as the first image block;

[0016] In the reference video frame image, calculating the circumscribed rectangle of the repaired object;

[0017] In the reference video frame image, an area enclosed by the circumscribed rectangle is extracted as the second image block.

[0018] In some embodiments, the at least two image blocks further include a third image block; and the extracting the at least two image blocks containing the repaired object further includes:

[0019] In the reference video frame image, calculating an extended frame of the repaired object; wherein the extended frame coincides with the second image block and an area of ​​the extended frame is larger than an area of ​​the second image block;

[0020] In the reference video frame image, an area surrounded by the extended frame is extracted as a third image block.

[0021] In some embodiments, calculating the expansion box of the repair object includes:

[0022] Calculating a center point of the second image block;

[0023] Determining a center point of the extended frame according to a center point of the second image block;

[0024] Calculating the size of the second image block;

[0025] Calculating the size of the extended frame according to a preset scale factor and the size of the second image block;

[0026] The extended frame is generated according to the center point of the extended frame and the size of the extended frame.

[0027] In some embodiments, selecting a reference video frame image and a damaged video frame image from the video frame image set based on the repair object includes:

[0028] Extracting key frames from the video frame image set to obtain a key frame image set;

[0029] The reference video frame image and the damaged video frame image are selected in the key frame image set based on the repair object.

[0030] In some embodiments, the method further comprises:

[0031] Calculate the number of key frame images in the key frame image set to obtain the number of key frames; calculate the number of damaged video frame images to obtain the number of damaged video frames; calculate the ratio of the number of damaged video frames to the number of key frames to obtain a damage degree index;

[0032] If the damage index is less than or equal to a preset index threshold, in the key frame image set, a plurality of the damaged video frame images are replaced with corresponding target video frame images; intermediate frames are generated according to the key frame image set to obtain a target video frame image sequence;

[0033] If the damage degree index is greater than the index threshold, then in the video frame image set, the damaged video frame image is replaced with a corresponding target video frame image to obtain a target video frame image sequence.

[0034] In some embodiments, selecting a reference video frame image and a damaged video frame image from the video frame image set based on the repair object includes:

[0035] Constructing a mask for identifying the repair object according to the area where the repair object is located in the video frame image;

[0036] Performing image restoration on the video frame image set with the mask by using an image restoration model to obtain an intermediate video frame image set;

[0037] Extracting key frames from the intermediate video frame image set to obtain an intermediate key frame image set;

[0038] The reference video frame image and the damaged video frame image are selected in the intermediate key frame image set based on the repair object.

[0039] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a video-based image restoration device, the device comprising:

[0040] An image data acquisition module is used to acquire a video frame image set from the original video data;

[0041] An image block extraction module is used to perform target detection on the video frame image set to obtain a repair object; select a reference video frame image and a damaged video frame image from the video frame image set based on the repair object, wherein the reference video frame image and the damaged video frame image contain the same repair object; and extract at least two image blocks containing the repair object from the reference video frame image, wherein the spatial scales of any two of the image blocks are different;

[0042] A repair prompt word generation module, used to generate an image description for each of the image blocks to obtain a plurality of image description information; and to generate a repair prompt word according to the plurality of image description information;

[0043] The image restoration module is used to generate an image of the restoration object according to the restoration prompt word to obtain a reconstructed image; and to replace the restoration object in the damaged video frame image with the reconstructed image to obtain a target video frame image.

[0044] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0045] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method described in the first aspect is implemented.

[0046] The video-based image restoration method and device, electronic device and storage medium proposed in the present application select a reference video frame image from a video frame image set, the reference video frame image and the damaged video frame image are in the same time dimension and have a high correlation in image content, extract image blocks of different spatial scales from the reference video frame image to generate image descriptions, generate restoration prompt words based on the image description information, and the obtained restoration prompt words contain spatial information of the restoration object at different levels in the same time dimension, which can provide the image generation method with more details of the surrounding environment of the restoration object, so that the generated reconstructed image can be better integrated into the damaged video frame image, thereby improving the image restoration effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of a video-based image restoration method provided in an embodiment of the present application;

[0048] Figure 2 is another flow chart of the video-based image restoration method provided by an embodiment of the present application;

[0049] Figure 3 is another flow chart of the video-based image restoration method provided in an embodiment of the present application;

[0050] Figure 4 yes Figure 1 Flow chart of step S104 in FIG.

[0051] Figure 5 yes Figure 4 Flow chart of step S405 in FIG.

[0052] Figure 6 is a structural schematic diagram of a video-based image restoration device provided in an embodiment of the present application;

[0053] Figure 7 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0057] First, some nouns involved in this application are analyzed:

[0058] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0059] Object Detection: Object detection is an important task in the field of computer vision, which aims to identify and locate one or more target objects from an image or video. Object detection usually involves building a model to identify objects of a specific category in an image and determine their location. This process includes image preprocessing, feature extraction, classifier design, and bounding box localization. Object detection is very important in many applications, such as self-driving cars, video surveillance, face recognition, and medical image analysis.

[0060] Image Captioning: Generate natural language descriptions for images and use the generated descriptions to help applications understand the semantics expressed in the visual scene of the image. For example, image description can convert image retrieval into text retrieval, which is used to classify images and improve image retrieval results. People usually only need to take a quick glance to describe the details of the visual scene of an image, while automatically adding descriptions to images is a comprehensive and arduous computer vision task that requires converting the complex information contained in the image into natural language descriptions. Compared with ordinary computer vision tasks, image captioning requires not only identifying objects from images, but also associating the identified objects with natural semantics and describing them in natural language. Therefore, image description requires people to extract deep features of images, associate with semantic features and convert them for generating descriptions.

[0061] Image Generation: refers to the process of automatically creating new images using computer methods. It usually involves building models to generate realistic new images. Deep learning technology has performed well in the field of image generation, especially models such as Generative Adversarial Networks (GAN) and Variational Autoencoders (VAE), which have been widely used in image generation. Image generation can be used to create works of art, simulate scenes, and generate virtual environments in games and film production.

[0062] Keyframe: A frame in a video sequence that has significant features or important changes, usually marking the start or end of an action, or marking a different state of a scene transition. For example, in a 1-second animation video, the keyframes are usually the first and last video frames.

[0063] Based on this, the embodiments of the present application provide a video-based image restoration method and device, an electronic device and a storage medium, aiming to improve the video restoration effect.

[0064] The video-based image restoration method and device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the video-based image restoration method in the embodiments of the present application is described.

[0065] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0066] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0067] The video-based image restoration method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The video-based image restoration method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements a video-based image restoration method, etc., but is not limited to the above forms.

[0068] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0069] Figure 1is an optional flowchart of the video-based image restoration method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S108.

[0070] Step S101, obtaining a video frame image set in the original video data;

[0071] Step S102, performing target detection on the video frame image set to obtain a repair object;

[0072] Step S103, selecting a reference video frame image and a damaged video frame image from the video frame image set based on the repair object, wherein the reference video frame image and the damaged video frame image contain the same repair object;

[0073] Step S104, extracting at least two image blocks containing the repaired object in the reference video frame image, wherein the spatial scales of any two image blocks are different;

[0074] Step S105, generating an image description for each image block to obtain a plurality of image description information;

[0075] Step S106, generating a repair prompt word according to the image description information;

[0076] Step S107, generating an image of the repair object according to the repair prompt word to obtain a reconstructed image;

[0077] Step S108: using the reconstructed image to replace the repaired object in the damaged video frame image to obtain a target video frame image.

[0078] In steps S101 to S108 shown in the embodiment of the present application, a reference video frame image is selected from a video frame image set. The reference video frame image and the damaged video frame image are in the same time dimension and have a high correlation in image content. Image blocks of different spatial scales are extracted from the reference video frame image to generate image descriptions, and repair prompt words are generated according to the image description information. The obtained repair prompt words contain spatial information of the repair object at different levels in the same time dimension, which can provide the image generation method with more details of the surrounding environment of the repair object, so that the generated reconstructed image can be better integrated into the damaged video frame image, thereby improving the image repair effect.

[0079] In step S101 of some embodiments, the original video data may be a real video drawn or shot manually, or a fake video generated by artificial intelligence, but is not limited thereto.

[0080] In step S101 of some embodiments, taking the financial technology scenario as an example, the original video data may be a loan face-to-face signing video, or an operation instruction video guiding users on how to use financial services, or a financial product introduction video, but is not limited thereto.

[0081] Taking the operation instruction video for guiding users to use the insurance claim platform as an example, in some embodiments, the video-based image restoration method may include but is not limited to:

[0082] Obtain a video frame image set in the operation guidance video;

[0083] Performing target detection on the video frame image set to obtain the restoration object; for example, the target detection model outputs targets such as insurance agent, car, and road, and the insurance agent is taken as the restoration object;

[0084] Select a reference video frame image and a damaged video frame image containing an insurance agent from a video frame image set; for example, the video frame image set includes a first video frame and a second video frame, and an irregular pixel block with color error appears in the human body region of the insurance agent in the first video frame, and the irregular pixel block does not appear in the human body region of the insurance agent in the second video frame, then the first video frame is used as a damaged video frame image, and the second video frame is used as a reference video frame image;

[0085] In the reference video frame image, two image blocks containing the insurance agent are extracted, and the spatial scales of the two image blocks are different;

[0086] An image description is generated for the first image block to obtain first image description information; the first image description information may be the appearance, expression, clothing, actions, etc. of the insurance agent, such as the insurance agent wearing a suit, smiling, with his hands crossed and clasped in front of him and hanging naturally.

[0087] An image description is generated for the second image block to obtain second image description information; the second image description information may be the environment surrounding the insurance agent, the interactive relationship between the insurance agent and the environment, and the like, such as the insurance agent standing next to a desk, a black monitor is placed on the brown oak desk, and the background is an office environment;

[0088] According to the above image description information, repair prompt words are generated; for example, the generated repair prompt words include: the insurance agent is standing next to the desk, a black monitor is placed on the brown oak desk, and the background is an office environment; the insurance agent is wearing a suit, smiling, with his hands crossed and clasped in front of him and hanging naturally.

[0089] Generate an image of the repair object according to the repair prompt word to obtain a reconstructed image;

[0090] The reconstructed image is used to replace the repaired object in the damaged video frame image to obtain the target video frame image.

[0091] In step S102 of some embodiments, the repair object may be a person, an object, or a background, such as a staff member, a vehicle, an office, a road, etc., without limitation.

[0092] In some embodiments, step S102 may include, but is not limited to:

[0093] Extracting key frames from the video frame image set to obtain a key frame image set;

[0094] Perform target detection on the key frame image set to obtain the repaired object.

[0095] The key frames capture important scenes or actions in the original video data. The steps illustrated in the embodiment of the present application can extract the main repair objects that affect the video quality by extracting repair objects from the key frames. Based on the repair objects, damaged video frame images are selected for repair, which can reduce the repair workload and improve the real-time performance of the repair.

[0096] In some embodiments, key frame extraction can be performed by a key frame extraction method based on inter-frame differences, which identifies key frames by calculating the differences between consecutive frames; or by a key frame extraction method based on motion analysis, which identifies key frames by analyzing motion information in the video; or by a key frame extraction method based on clustering, which regards video frames as data points, and classifies similar frames into one category to obtain multiple clusters through a clustering method, and the central data point of each cluster is the key frame; but it is not limited to this.

[0097] In step S102 of some embodiments, target detection may be performed by a target detection method based on R-CNN, which utilizes a convolutional neural network to extract image features and uses a support vector machine or a fully connected layer for classification and regression; or target detection may be performed by a target detection method based on YOLO, which divides the image into grids of a fixed size, and predicts a fixed number of object confidences and bounding box coordinates for each grid; but it is not limited thereto.

[0098] See also Figure 2 In some embodiments, step S103 may include but is not limited to:

[0099] Step S201, extracting key frames from a video frame image set to obtain a key frame image set;

[0100] Step S202: selecting a reference video frame image and a damaged video frame image from a key frame image set based on the repair object, wherein the reference video frame image and the damaged video frame image contain the same repair object.

[0101] The key frames capture important scenes or actions in the original video data. Steps S201 to S202 shown in the embodiment of the present application select damaged video frame images from the key frame image set and only perform image repair on the key frames, which can reduce the repair workload and improve the real-time performance of the repair.

[0102] In step S202 of some embodiments, bad frames can be identified for the key frame image set based on factors such as inter-frame differences, and bad frames can be taken as damaged video frame images, and non-bad frames can be taken as reference video frame images; image quality scores can also be performed on each key frame image in the key frame image set, and the key frame image with the highest image quality score can be taken as the reference video frame image, and the key frame image with the lowest image quality score can be taken as the damaged video frame image; it is not limited thereto. The method for calculating the image quality score is not limited, such as the image quality score value of each key frame image can be calculated by an image quality scoring method based on structural similarity, or by an image quality evaluation method based on visual perception.

[0103] In some embodiments, step S202 may include, but is not limited to:

[0104] Traversing the key frame images in the key frame image set, determining whether the currently accessed key frame image is a bad frame, if the currently accessed key frame image is a bad frame, marking the currently accessed key frame image as a damaged video frame image and deleting the currently accessed key frame image from the key frame image set; otherwise, continuing to access the next key frame image until there is no unaccessed key frame image;

[0105] Calculate the similarity between each damaged video frame image and each key frame image in the key frame image set respectively;

[0106] For each damaged video frame image, the key frame image with the highest similarity is taken as the corresponding reference video frame image.

[0107] The steps illustrated in the embodiment of the present application calculate the similarity between the damaged video frame image and the non-damaged video frame image, and match each damaged video frame image with the non-damaged video frame image with the highest similarity as the reference video frame image, thereby providing a more suitable restoration basis for the image generation method, making the restored image more natural and of higher quality.

[0108] See also Figure 2 In some embodiments, after step S202, the video-based image restoration method further includes steps S203 to S207:

[0109] Step S203, calculating the number of key frame images in the key frame image set to obtain the number of key frames;

[0110] Step S204, calculating the number of damaged video frame images to obtain the number of damaged video frames;

[0111] Step S205, calculating the ratio of the number of damaged video frames to the number of key frames to obtain a damage degree index;

[0112] Step S206, presetting an index threshold; if the damage index is less than or equal to the index threshold, replacing multiple damaged video frame images with corresponding target video frame images in the key frame image set; generating intermediate frames according to the key frame image set to obtain a target video frame image sequence;

[0113] Step S207: if the damage index is greater than the index threshold, then in the video frame image set, the damaged video frame image is replaced with the corresponding target video frame image to obtain a target video frame image sequence.

[0114] In steps S203 to S207 shown in the embodiment of the present application, a damage index is constructed to determine the damage degree of the key frame image set, and intermediate frames are generated for the key frame image set with a lower degree of damage. This can obtain intermediate frames with better image quality than the original video frame images, thereby improving the consistency and smoothness of the target video frame image sequence.

[0115] In step S206 of some embodiments, the indicator threshold may be 10% or 15%, but is not limited thereto.

[0116] See also Figure 3 In some embodiments, step S103 may include but is not limited to:

[0117] Step S301, constructing a mask for identifying the repair object according to the area where the repair object is located in the damaged video frame image;

[0118] Step S302, using an image restoration model to perform image restoration on the video frame image set with the mask to obtain an intermediate video frame image set;

[0119] Step S303, extracting key frames from the intermediate video frame image set to obtain an intermediate key frame image set;

[0120] Step S304: Select reference video frame images and damaged video frame images from the intermediate key frame image set.

[0121] In steps S301 to S304 shown in the embodiment of the present application, all video frame images in the video frame image set are preliminarily repaired, and then the key frame images therein are repaired with high precision, thereby improving the repair effect and having better repair efficiency.

[0122] In step S301 of some embodiments, a mask can be constructed by a rule-based method, which presets rules to extract the region of interest in the mask, such as extracting the mask of a damaged video frame image with color errors according to the image color channel values; the mask can also be constructed by a user interaction-based method, in which the user indicates the region of interest in the mask by clicking, boxing or other forms of interaction; but it is not limited to this.

[0123] In step S302 of some embodiments, the image restoration model may be a context encoder (ContextEncoders), which learns the context information of the image through an encoder-decoder architecture and predicts the restoration object; it may also be a BrushNet model, which effectively processes the restoration object in the image by decomposing a dual-branch architecture; but it is not limited to this.

[0124] See also Figure 4 In some embodiments, the at least two image blocks include a first image block and a second image block; step S104 may include but is not limited to steps S401 to S404:

[0125] Step S401, extracting the contour of the repaired object in the reference video frame image to obtain the contour of the repaired object;

[0126] Step S402, extracting the area surrounded by the outline of the repaired object in the reference video frame image as the first image block;

[0127] Step S403, calculating the circumscribed rectangle of the repaired object in the reference video frame image;

[0128] Step S404: extracting an area surrounded by a circumscribed rectangle in the reference video frame image as a second image block.

[0129] In steps S401 to S404 shown in the embodiment of the present application, the area enclosed by the contour of the repair object is taken as the first image block, and the area enclosed by the circumscribed rectangle is taken as the second image block. The image description information generated by the first image block and the second image block is integrated to obtain the repair prompt word. The repair prompt word contains the spatial information of the contour of the repair object itself and the surrounding environment, which helps the image generation method to understand the repair object with a complex contour, so that it can be better integrated into the video frame image, and the repair effect of the target video frame image is improved.

[0130] In step S401 of some embodiments, the contour of the repaired object may be extracted by an edge detection method based on a Canny operator, or by an edge detection method based on a Hough transform, but the invention is not limited thereto.

[0131] In step S403 of some embodiments, the bounding rectangle may be a minimum bounding rectangle or a maximum bounding rectangle, but is not limited thereto.

[0132] See also Figure 4 In some embodiments, the at least two image blocks further include a third image block; after step S404, the video-based image restoration method further includes steps S405 to S406:

[0133] Step S405, calculating an extended frame of the repaired object in the reference video frame image; wherein the extended frame coincides with the second image block and the area of ​​the extended frame is larger than the area of ​​the second image block;

[0134] Step S406: extracting the area surrounded by the extended frame in the reference video frame image as the third image block.

[0135] In steps S405 to S406 shown in the embodiment of the present application, the area surrounded by the expansion box is extracted as the third image block, the expansion box overlaps with the second image block and the area of ​​the expansion box is larger than the area of ​​the second image block, and the image description information of the first image block, the second image block and the third image block are integrated to generate a repair prompt word, then the repair prompt word also includes the spatial information around the second image block, and the reconstructed image can be better integrated into the damaged video frame image, further improving the repair effect of the target video frame image.

[0136] See also Figure 5 In some embodiments, step S405 may include but is not limited to steps S501 to S505:

[0137] Step S501, calculating the center point of the second image block;

[0138] Step S502, determining the center point of the extended frame according to the center point of the second image block;

[0139] Step S503, calculating the size of the second image block;

[0140] Step S504, according to the preset scale factor and the size of the second image block, the size of the extended frame is calculated, which is expressed as:

[0141]

[0142] Where α is the scale factor, α>1; H and W are the length and width of the second image block respectively; H ′ , W ′ are the length and width of the third image block respectively.

[0143] Step S505: Generate an extended frame according to the center point of the extended frame and the size of the extended frame.

[0144] Steps S501 to S505 shown in the embodiment of the present application determine the size of the third image block by presetting the proportional relationship between the second image block and the third image block. The size of the third image block can be adaptively determined according to the size of the repaired object, thereby improving the consistency of the target video frame image in the video sequence.

[0145] In step S501 of some embodiments, the center point of the second image block may be the geometric center of the second image block or the centroid of the second image block, but is not limited thereto. For example, the pixel value is used as a weight to calculate the ratio of the weighted sum of all pixel coordinates in the second image block to the accumulated value of all pixel coordinates to obtain the centroid of the second image block.

[0146] In step S502 in some embodiments, the center point of the second image block can be directly used as the center point of the extended frame; the center point of the second image block can also be offset and the offset center point can be used as the center point of the extended frame; this is not limited to this.

[0147] In some embodiments, step S505 may include, but is not limited to:

[0148] Calculate the coordinates of the four vertices of the extended box according to the center point and size of the extended box;

[0149] It is determined whether the coordinates of the four vertices of the extended frame exceed the image boundary of the reference video frame image. If so, the size of the extended frame is modified so that the coordinates of the four vertices do not exceed the image boundary of the reference video frame image.

[0150] In step S105 of some embodiments, the image description information may describe the action of the object in the image, such as jumping, standing, sitting; may also describe the style of the image, such as cartoon, photo; or may describe the environment in the image, such as forest, ocean; but is not limited thereto.

[0151] In step S105 of some embodiments, a visual language model is used to generate an image description for each image block to obtain multiple image description information. Collect multiple damaged video frame images, extract the area surrounded by the outline of the repaired object for each damaged video frame image, and manually annotate the image description information of the area to obtain multiple negative samples. It should be noted that the manually annotated image description information contains the original state of the repaired object. Collect multiple reference video frame images, extract the area containing the repaired object for each reference video frame image, and manually annotate the image description information of the area to obtain multiple positive samples. In the training process of the visual language model, the distance between the image block and the image description information in the positive sample is maximized, while the distance between the image block and the image description information in the negative sample is minimized.

[0152] The embodiment of the present application extracts the repair object from the damaged video frame image and annotates the image description information to construct a negative sample. By training the visual language model with the negative sample, the visual language model can more accurately generate description information consistent with the image content and provide more precise repair prompt words.

[0153] In step S106 of some embodiments, multiple image description information may be directly combined to obtain the repair prompt word; or multiple image description information may be input into a large language model to generate the repair prompt word through the large language model; but the present invention is not limited thereto.

[0154] In some embodiments, step S105 may include but is not limited to: generating image descriptions for the first image block, the second image block, and the third image block, respectively, to obtain corresponding first image description information, second image description information, and third image description information. Step S106 may include but is not limited to: combining the image description information in the order of the third image description information, the second image description information, and the first image description information to obtain a repair prompt word. Step S107 may include but is not limited to: the image generation model weights each image description information in the repair prompt word, and the weight of the image description information decreases in order; the image generation model generates an image according to the weighted value of the repair prompt word to obtain a reconstructed image.

[0155] The embodiment of the present application weights each image description information according to the spatial scale, and sets the maximum weight for the image description information containing spatial information of the maximum spatial scale. The image generation model can more effectively process the overall structure of the image and improve the integrity and coherence of the target video frame image.

[0156] In step S107 of some embodiments, the image of the repaired object may be generated through a latent diffusion model (Latent Diffusion Models), or may be generated through a generative adversarial network (GAN), but is not limited thereto.

[0157] In step S107 of some embodiments, the size of the reconstructed image can be set according to actual conditions without limitation. For example, the size of the reconstructed image can be the same as the size of the second image block or the same as the size of the third image block.

[0158] After step S108 in some embodiments, the video-based image restoration method further includes: performing post-processing on the target video frame image to enhance the image quality of the target video frame image, the post-processing including: color correction, noise reduction, contrast adjustment, image fusion, image super-resolution. For example, smoothing the edge of the reconstructed image in the target video frame image to improve the fusion effect of the reconstructed image.

[0159] See also Figure 6The embodiment of the present application also provides a video-based image restoration device, which can implement the above-mentioned video-based image restoration method, and the device includes:

[0160] An image data acquisition module is used to acquire a video frame image set from the original video data;

[0161] The image block extraction module is used to perform target detection on the video frame image set to obtain the repair object; select a reference video frame image and a damaged video frame image from the video frame image set based on the repair object, wherein the reference video frame image and the damaged video frame image contain the same repair object; and extract at least two image blocks containing the repair object from the reference video frame image, wherein the spatial scales of any two image blocks are different;

[0162] A repair prompt word generation module is used to generate an image description for each image block to obtain a plurality of image description information; and to generate a repair prompt word according to the plurality of image description information;

[0163] The image restoration module is used to generate an image of the restoration object according to the restoration prompt word to obtain a reconstructed image; and to replace the restoration object in the damaged video frame image with the reconstructed image to obtain a target video frame image.

[0164] The specific implementation of the video-based image restoration device is basically the same as the specific implementation of the above-mentioned video-based image restoration method, and will not be repeated here.

[0165] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned video-based image restoration method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0166] See also Figure 7 , Figure 7 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0167] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, AAIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0168] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the video-based image restoration method of the embodiment of this application;

[0169] Input / output interface 903, used to implement information input and output;

[0170] The communication interface 904 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as UAB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);

[0171] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0172] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0173] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned video-based image restoration method is implemented.

[0174] The memory can be used to store non-transient software programs and non-transient computer executable programs as a non-transient computer readable storage medium. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0175] The video-based image restoration method, video-based image restoration device, electronic device and storage medium provided in the embodiments of the present application select a reference video frame image from a video frame image set, the reference video frame image and the damaged video frame image are in the same time dimension and have a high correlation in image content, extract image blocks of different spatial scales from the reference video frame image to generate image descriptions, generate restoration prompt words based on the image description information, and the obtained restoration prompt words contain spatial information of the restoration object at different levels in the same time dimension, which can provide the image generation method with more details of the surrounding environment of the restoration object, so that the generated reconstructed image can be better integrated into the damaged video frame image, thereby improving the image restoration effect.

[0176] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0177] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0178] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0180] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0181] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0182] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0183] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0184] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0185] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0186] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A video-based image restoration method, characterized in that: The method comprises: Obtain a video frame image set from the original video data; Performing target detection on the video frame image set to obtain a repaired object; Selecting a reference video frame image and a damaged video frame image from the video frame image set based on the repair object, wherein the reference video frame image and the damaged video frame image contain the same repair object; Extracting at least two image blocks containing the repair object in the reference video frame image, wherein the spatial scales of any two of the image blocks are different; Generating an image description for each of the image blocks to obtain a plurality of image description information; Generate a repair prompt word according to the image description information; Generate an image of the repair object according to the repair prompt word to obtain a reconstructed image; The reconstructed image is used to replace the repaired object in the damaged video frame image to obtain a target video frame image.

2. The method according to claim 1, characterized in that The at least two image blocks include a first image block and a second image block; and extracting the at least two image blocks containing the repair object includes: In the reference video frame image, extracting the contour of the repaired object to obtain the contour of the repaired object; In the reference video frame image, extracting a region surrounded by the contour of the repaired object as the first image block; In the reference video frame image, calculating the circumscribed rectangle of the repaired object; In the reference video frame image, an area enclosed by the circumscribed rectangle is extracted as the second image block.

3. The method according to claim 2, characterized in that The at least two image blocks also include a third image block; and the extracting of the at least two image blocks containing the repaired object further includes: In the reference video frame image, calculating an extended frame of the repaired object; wherein the extended frame coincides with the second image block and an area of ​​the extended frame is larger than an area of ​​the second image block; In the reference video frame image, an area surrounded by the extended frame is extracted as a third image block.

4. The method according to claim 3, characterized in that The calculating the extended frame of the repair object includes: Calculating a center point of the second image block; Determining a center point of the extended frame according to a center point of the second image block; Calculating the size of the second image block; Calculating the size of the extended frame according to a preset scale factor and the size of the second image block; The extended frame is generated according to the center point of the extended frame and the size of the extended frame.

5. The method according to any one of claims 1 to 4, characterized in that: The selecting a reference video frame image and a damaged video frame image from the video frame image set based on the repair object comprises: Extracting key frames from the video frame image set to obtain a key frame image set; The reference video frame image and the damaged video frame image are selected in the key frame image set based on the repair object.

6. The method according to claim 5, characterized in that The method further comprises: Calculate the number of key frame images in the key frame image set to obtain the number of key frames; Calculating the number of damaged video frame images to obtain the number of damaged video frames; Calculate the ratio of the number of damaged video frames to the number of key frames to obtain a damage degree index; If the damage index is less than or equal to a preset index threshold, in the key frame image set, a plurality of the damaged video frame images are replaced with corresponding target video frame images; intermediate frames are generated according to the key frame image set to obtain a target video frame image sequence; If the damage degree index is greater than the index threshold, then in the video frame image set, the damaged video frame image is replaced with a corresponding target video frame image to obtain a target video frame image sequence.

7. The method according to any one of claims 1 to 4, characterized in that: The selecting a reference video frame image and a damaged video frame image from the video frame image set based on the repair object comprises: Constructing a mask for identifying the repair object according to the area where the repair object is located in the video frame image; Performing image restoration on the video frame image set with the mask by using an image restoration model to obtain an intermediate video frame image set; Extracting key frames from the intermediate video frame image set to obtain an intermediate key frame image set; The reference video frame image and the damaged video frame image are selected in the intermediate key frame image set based on the repair object.

8. A video-based image restoration device, characterized in that: The device comprises: An image data acquisition module is used to acquire a video frame image set from the original video data; An image block extraction module is used to perform target detection on the video frame image set to obtain a repair object; based on the repair object, a reference video frame image and a damaged video frame image are selected from the video frame image set, wherein the reference video frame image and the damaged video frame image contain the same repair object; and at least two image blocks containing the repair object are extracted from the reference video frame image, wherein the spatial scales of any two of the image blocks are different; A repair prompt word generation module is used to generate an image description for each of the image blocks to obtain a plurality of image description information; and generate a repair prompt word according to the plurality of image description information; The image restoration module is used to generate an image of the restoration object according to the restoration prompt word to obtain a reconstructed image; and to replace the restoration object in the damaged video frame image with the reconstructed image to obtain a target video frame image.

9. An electronic device, characterized in that The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images