Video-based image inpainting method and device, electronic device, and storage medium

Abstract

Embodiments of the present application provide a video-based image inpainting method and device, an electronic device and a storage medium, and belong to the technical fields of artificial intelligence and financial technology. The method comprises: obtaining an inpainting object; selecting a reference video frame image and a damaged video frame image based on the inpainting object; extracting a plurality of image blocks containing the inpainting object in the reference video frame image; generating image descriptions for each image block to obtain a plurality of image description information; generating an inpainting prompt word according to the image description information; generating an image according to the inpainting prompt word to obtain a reconstructed image; and inpainting the damaged video frame image using the reconstructed image. The inpainting prompt word obtained by the embodiments of the present application contains different levels of spatial information of the inpainting object in the same time dimension, can provide more details of the surrounding environment of the inpainting object for image generation, and can make the generated reconstructed image better integrate into the damaged video frame image, thereby improving the inpainting effect of the target video frame image.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and financial technology, and in particular to video-based image restoration methods and apparatuses, electronic devices and storage media. Background Technology

[0002] Image restoration techniques aim to restore damaged or missing parts of an image, making them as consistent as possible with the original. Related techniques utilize image generation to reconstruct regions of the original image and then fill them into the original image. However, the reconstructed image may differ significantly from the original image in texture, color, lighting, and style, failing to blend well and resulting in poor restoration effects. Therefore, improving image restoration performance has become a pressing technical problem. Summary of the Invention

[0003] The main objective of this application is to propose a video-based image restoration method, apparatus, electronic device, and storage medium, aiming to improve the image restoration effect.

[0004] To achieve the above objectives, a first aspect of this application proposes a video-based image restoration method, the method comprising:

[0005] Obtain the set of video frame images from the raw video data;

[0006] Target detection is performed on the set of video frame images to obtain the object to be repaired;

[0007] Based on the repair object, a reference video frame image and a damaged video frame image are selected from the video frame image set, wherein the reference video frame image and the damaged video frame image contain the same repair object;

[0008] In the reference video frame image, at least two image blocks containing the object to be repaired are extracted, and the spatial scales of any two image blocks are different;

[0009] For each of the image blocks, an image description is generated to obtain multiple image description information;

[0010] Based on the image description information, generate repair prompts;

[0011] Based on the repair prompts, an image is generated for the object to be repaired to obtain a reconstructed image;

[0012] The target video frame image is obtained by replacing the repair object in the damaged video frame image with the reconstructed image.

[0013] In some embodiments, the at least two image blocks include a first image block and a second image block; the extraction of at least two image blocks containing the object to be repaired includes:

[0014] In the reference video frame image, the contour of the object to be repaired is extracted to obtain the contour of the object to be repaired;

[0015] In the reference video frame image, the region enclosed by the outline of the object to be repaired is extracted as the first image block;

[0016] Calculate the bounding rectangle of the repair object in the reference video frame image;

[0017] In the reference video frame image, the region enclosed by the circumscribed rectangle is extracted as the second image block.

[0018] In some embodiments, the at least two image blocks further include a third image block; the extraction of the at least two image blocks containing the repair object further includes:

[0019] In the reference video frame image, the expanded bounding box of the repair object is calculated; wherein the expanded bounding box coincides with the second image block and the area of ​​the expanded bounding box is larger than the area of ​​the second image block;

[0020] In the reference video frame image, the region enclosed by the expanded box is extracted as a third image block.

[0021] In some embodiments, calculating the expanded frame of the repair object includes:

[0022] Calculate the center point of the second image patch;

[0023] The center point of the expanded frame is determined based on the center point of the second image block;

[0024] Calculate the size of the second image patch;

[0025] The size of the expansion box is calculated based on the preset scaling factor and the size of the second image block;

[0026] The expansion box is generated based on its center point and size.

[0027] In some embodiments, selecting a reference video frame image and a damaged video frame image from the video frame image set based on the object to be repaired includes:

[0028] Keyframes are extracted from the video frame image set to obtain a keyframe image set;

[0029] Based on the object to be repaired, the reference video frame image and the damaged video frame image are selected from the keyframe image set.

[0030] In some embodiments, the method further includes:

[0031] Calculate the number of keyframe images in the keyframe image set to obtain the number of keyframes; calculate the number of damaged video frame images to obtain the number of damaged video frames; calculate the ratio of the number of damaged video frames to the number of keyframes to obtain the damage index.

[0032] If the damage index is less than or equal to a preset index threshold, then in the keyframe image set, multiple damaged video frame images are replaced with corresponding target video frame images; intermediate frames are generated based on the keyframe image set to obtain a target video frame image sequence;

[0033] If the damage index is greater than the index threshold, then in the video frame image set, the damaged video frame image is replaced with the corresponding target video frame image to obtain the target video frame image sequence.

[0034] In some embodiments, selecting a reference video frame image and a damaged video frame image from the video frame image set based on the object to be repaired includes:

[0035] Based on the region where the object to be repaired is located in the video frame image, a mask for identifying the object to be repaired is constructed;

[0036] The image restoration model is used to perform image restoration on the video frame image set with the mask to obtain an intermediate video frame image set.

[0037] Keyframe extraction is performed on the intermediate video frame image set to obtain an intermediate keyframe image set;

[0038] Based on the object to be repaired, the reference video frame image and the damaged video frame image are selected from the intermediate keyframe image set.

[0039] To achieve the above objectives, a second aspect of this application provides a video-based image restoration apparatus, the apparatus comprising:

[0040] The image data acquisition module is used to acquire the set of video frame images from the raw video data;

[0041] An image patch extraction module is used to perform target detection on the video frame image set to obtain a repair object; select a reference video frame image and a damaged video frame image in the video frame image set based on the repair object, wherein the reference video frame image and the damaged video frame image contain the same repair object; and extract at least two image patches containing the repair object from the reference video frame image, wherein the spatial scales of any two image patches are different.

[0042] The repair prompt word generation module is used to generate image descriptions for each image block to obtain multiple image description information; and to generate repair prompt words based on the multiple image description information.

[0043] The image restoration module is used to generate an image of the object to be restored based on the restoration prompt, thereby obtaining a reconstructed image; and to replace the object to be restored in the damaged video frame image with the reconstructed image, thereby obtaining a target video frame image.

[0044] To achieve the above objectives, a third aspect of the present application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0045] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0046] The video-based image restoration method, apparatus, electronic device, and storage medium proposed in this application select reference video frame images from a set of video frame images. The reference video frame images and the damaged video frame images are in the same time dimension and have a high correlation in image content. Image blocks of different spatial scales are extracted from the reference video frame images to generate image descriptions. Based on the image description information, restoration prompts are generated. The obtained restoration prompts contain different levels of spatial information of the restoration object in the same time dimension, which can provide more details of the surrounding environment of the restoration object to the image generation method, so that the generated reconstructed image can be better integrated into the damaged video frame image and improve the image restoration effect. Attached Figure Description

[0047] Figure 1 This is a flowchart of the video-based image restoration method provided in the embodiments of this application;

[0048] Figure 2 This is another flowchart of the video-based image restoration method provided in the embodiments of this application;

[0049] Figure 3 This is another flowchart of the video-based image restoration method provided in the embodiments of this application;

[0050] Figure 4 yes Figure 1 The flowchart of step S104 in the process;

[0051] Figure 5 yes Figure 4 The flowchart of step S405 in the document;

[0052] Figure 6 This is a schematic diagram of the structure of the video-based image restoration device provided in the embodiments of this application;

[0053] Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0057] First, let's analyze some of the terms used in this application:

[0058] Artificial Intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0059] Object detection is a crucial task in computer vision, aiming to identify and locate one or more target objects from images or videos. Object detection typically involves building models to identify objects of specific categories in an image and determine their locations. This process includes image preprocessing, feature extraction, classifier design, and bounding box localization. Object detection is essential in many applications, such as autonomous vehicles, video surveillance, face recognition, and medical image analysis.

[0060] Image captioning involves generating natural language descriptions for images and using these descriptions to help applications understand the semantics expressed in the visual scene. For example, image captioning can convert image retrieval into text retrieval, be used for image classification, and improve image retrieval results. While people can often describe the details of a visual scene with a quick glance, automatically adding descriptions to images is a comprehensive and challenging computer vision task, requiring the conversion of complex information contained in the image into natural language descriptions. Compared to ordinary computer vision tasks, image captioning not only requires identifying objects in an image but also associating the identified objects with natural semantics and describing them in natural language. Therefore, image captioning requires extracting deep features from the image, associating them with semantic features, and converting them to generate descriptions.

[0061] Image generation refers to the process of automatically creating new images using computer methods. It typically involves building models to generate realistic new images. Deep learning techniques have shown outstanding performance in image generation, particularly models such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), which are widely used in image generation. Image generation can be used to create works of art, simulate scenes, and generate virtual environments in game and film production.

[0062] Keyframes are frames in a video sequence that have significant features or show important changes. They typically mark the start or end of an action, or different states where a scene transitions. For example, in a 1-second animated video, keyframes are usually the first and last video frames.

[0063] Based on this, embodiments of this application provide a video-based image restoration method and apparatus, electronic device and storage medium, aiming to improve the video restoration effect.

[0064] The video-based image restoration method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the video-based image restoration method in this application is described.

[0065] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0066] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0067] The video-based image restoration method provided in this application relates to the field of artificial intelligence technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the video-based image restoration method, but is not limited to the above forms.

[0068] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0069] Figure 1This is an optional flowchart of the video-based image restoration method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S108.

[0070] Step S101: Obtain the set of video frame images from the original video data;

[0071] Step S102: Perform target detection on the video frame image set to obtain the object to be repaired;

[0072] Step S103: Select a reference video frame image and a damaged video frame image from the video frame image set based on the object to be repaired. The reference video frame image and the damaged video frame image contain the same object to be repaired.

[0073] Step S104: Extract at least two image patches containing the object to be repaired from the reference video frame image, wherein the spatial scales of any two image patches are different;

[0074] Step S105: Generate image descriptions for each image block to obtain multiple image description information;

[0075] Step S106: Generate repair prompts based on image description information;

[0076] Step S107: Generate an image of the object to be repaired based on the repair prompt words to obtain a reconstructed image;

[0077] Step S108: Replace the repair object in the damaged video frame image with the reconstructed image to obtain the target video frame image.

[0078] In the embodiments of this application, steps S101 to S108 involve selecting a reference video frame image from the video frame image set. The reference video frame image and the damaged video frame image are in the same time dimension and have a high correlation in image content. Image blocks of different spatial scales are extracted from the reference video frame image to generate image descriptions. Repair prompts are generated based on the image description information. The obtained repair prompts contain different levels of spatial information of the repair object in the same time dimension, which can provide more details of the surrounding environment of the repair object for the image generation method, so that the generated reconstructed image can be better integrated into the damaged video frame image and improve the image repair effect.

[0079] In step S101 of some embodiments, the original video data may be a real video drawn or filmed by humans, or a fake video generated by artificial intelligence, and is not limited thereto.

[0080] In step S101 of some embodiments, taking a fintech scenario as an example, the original video data may be a loan interview video, an operation guide video instructing users on how to use financial services, or a financial product introduction video, and is not limited to these.

[0081] Taking a user guide video on how to use an insurance claims platform as an example, in some embodiments, the video-based image restoration method may include, but is not limited to:

[0082] Obtain the set of video frame images from the operation guidance video;

[0083] The object to be repaired is obtained by performing object detection on a set of video frame images; for example, the object detection model outputs objects such as insurance agents, cars, and roads, and the insurance agent is selected as the object to be repaired.

[0084] Select a reference video frame image and a damaged video frame image that contain the insurance agent from the video frame image set; for example, if the video frame image set contains a first video frame and a second video frame, and the first video frame contains irregular pixel blocks with color errors in the human body area of ​​the insurance agent, while the second video frame does not contain such irregular pixel blocks in the human body area of ​​the insurance agent, then the first video frame is used as the damaged video frame image and the second video frame is used as the reference video frame image.

[0085] In the reference video frame image, two image blocks containing the insurance agent are extracted. The two image blocks have different spatial scales.

[0086] Image description is generated from the first image block to obtain first image description information; the first image description information may include the appearance, expression, clothing, and actions of the insurance agent, such as the insurance agent wearing a suit, smiling, with his hands clasped in front of him and hanging down naturally.

[0087] Image descriptions are generated from the second image block to obtain second image description information. The second image description information may include the environment around the insurance agent, the interaction between the insurance agent and the environment, etc. For example, the insurance agent is standing next to a desk with a black monitor on the brown oak desk and the background is the office environment.

[0088] Based on the above image description information, generate repair prompts; for example, the generated repair prompts include: the insurance agent is standing next to a desk with a black monitor on the brown oak desk, and the background is an office environment; the insurance agent is wearing a suit, smiling, with his hands clasped in front of him and hanging down naturally.

[0089] Based on the repair prompts, an image of the object to be repaired is generated to obtain a reconstructed image;

[0090] The target video frame image is obtained by replacing the damaged object in the reconstructed image with the object to be repaired.

[0091] In step S102 of some embodiments, the object to be repaired can be a person, an object, or a background, such as a worker, a vehicle, an office, a road, etc., without limitation.

[0092] In some embodiments, step S102 may include, but is not limited to:

[0093] Keyframes are extracted from the video frame image set to obtain a keyframe image set;

[0094] Target detection is performed on the keyframe image set to obtain the object to be repaired.

[0095] Keyframes capture important scenes or actions in the original video data. The steps shown in the embodiments of this application can extract the repair object from the keyframes, thereby extracting the main repair object that affects the video quality. Based on the repair object, the damaged video frame image is selected for repair, which can reduce the repair workload and improve the real-time performance of the repair.

[0096] In some embodiments, keyframe extraction can be performed using a keyframe extraction method based on inter-frame difference, which identifies keyframes by calculating the differences between consecutive frames; it can also be performed using a keyframe extraction method based on motion analysis, which identifies keyframes by analyzing motion information in the video; or it can be performed using a keyframe extraction method based on clustering, which treats video frames as data points and uses a clustering method to group similar frames into multiple clusters, with the central data point of each cluster being the keyframe; and it is not limited to these methods.

[0097] In step S102 of some embodiments, object detection may be performed by an R-CNN-based object detection method, which uses a convolutional neural network to extract image features and uses a support vector machine or a fully connected layer for classification and regression; or object detection may be performed by a YOLO-based object detection method, which divides the image into a grid of fixed size, and predicts a fixed number of object confidences and bounding box coordinates for each grid; it is not limited to these.

[0098] Please see Figure 2 In some embodiments, step S103 may include, but is not limited to:

[0099] Step S201: Extract keyframes from the video frame image set to obtain a keyframe image set;

[0100] Step S202: Select a reference video frame image and a damaged video frame image from the keyframe image set based on the object to be repaired. The reference video frame image and the damaged video frame image contain the same object to be repaired.

[0101] Keyframes capture important scenes or actions in the original video data. Steps S201 to S202 shown in the embodiments of this application, by selecting damaged video frame images from the keyframe image set and performing image repair only on the keyframes, can reduce the amount of repair work and improve the real-time performance of the repair.

[0102] In step S202 of some embodiments, bad frames can be identified in the keyframe image set based on factors such as inter-frame differences, with bad frames designated as damaged video frame images and non-bad frames as reference video frame images; alternatively, image quality scoring can be performed on each keyframe image in the keyframe image set, with the keyframe image with the highest image quality score designated as the reference video frame image and the keyframe image with the lowest image quality score designated as the damaged video frame image; however, this is not limited to these methods. The calculation method for the image quality score is not limited; for example, the image quality score of each keyframe image can be calculated using an image quality scoring method based on structural similarity, or it can be calculated using an image quality evaluation method based on visual perception.

[0103] In some embodiments, step S202 may include, but is not limited to:

[0104] Traverse the keyframe images in the keyframe image set, determine whether the currently accessed keyframe image is a bad frame. If the currently accessed keyframe image is a bad frame, mark the currently accessed keyframe image as a corrupted video frame image and delete the currently accessed keyframe image from the keyframe image set; otherwise, continue to access the next keyframe image until there are no unaccessed keyframe images.

[0105] Calculate the similarity between each damaged video frame image and each keyframe image in the keyframe image set;

[0106] For each damaged video frame image, the keyframe image with the highest similarity is taken as the corresponding reference video frame image.

[0107] The steps illustrated in this application's embodiments calculate the similarity between damaged and undamaged video frame images, and match the undamaged video frame image with the highest similarity for each damaged video frame image as a reference video frame image. This provides a more suitable basis for the image generation method, resulting in a more natural and higher-quality repaired image.

[0108] Please see Figure 2 In some embodiments, after step S202, the video-based image restoration method further includes steps S203 to S207:

[0109] Step S203: Calculate the number of keyframe images in the keyframe image set to obtain the number of keyframes;

[0110] Step S204: Calculate the number of damaged video frame images to obtain the number of damaged video frames;

[0111] Step S205: Calculate the ratio of the number of corrupted video frames to the number of keyframes to obtain the corruption index;

[0112] Step S206: Preset index threshold; if the damage index is less than or equal to the index threshold, replace multiple damaged video frame images with the corresponding target video frame images in the key frame image set; generate intermediate frames based on the key frame image set to obtain the target video frame image sequence.

[0113] Step S207: If the damage index is greater than the index threshold, then in the video frame image set, the damaged video frame image is replaced with the corresponding target video frame image to obtain the target video frame image sequence.

[0114] Steps S203 to S207 as shown in the embodiments of this application determine the degree of damage to the key frame image set by constructing a damage index, and generate intermediate frames for the key frame image set with a lower degree of damage. This can obtain intermediate frames with image quality better than the original video frame images, thereby improving the consistency and smoothness of the target video frame image sequence.

[0115] In step S206 of some embodiments, the indicator threshold may be 10% or 15%, and is not limited thereto.

[0116] Please see Figure 3 In some embodiments, step S103 may include, but is not limited to:

[0117] Step S301: Construct a mask to identify the object to be repaired based on the region where the object to be repaired is located in the damaged video frame image;

[0118] Step S302: Use an image inpainting model to perform image inpainting on the masked video frame image set to obtain an intermediate video frame image set;

[0119] Step S303: Extract keyframes from the intermediate video frame image set to obtain the intermediate keyframe image set;

[0120] Step S304: Select the reference video frame image and the damaged video frame image from the intermediate keyframe image set.

[0121] Steps S301 to S304 as shown in the embodiments of this application improve the repair effect and have better repair efficiency by performing preliminary repair on all video frame images in the video frame image set and then performing high-precision repair on the key frame images.

[0122] In some embodiments, step S301 can be performed by constructing a mask using a rule-based method, which extracts the region of interest in the mask by pre-setting rules, such as extracting a mask from a damaged video frame image with color errors based on the image color channel values; or by constructing a mask using a user interaction-based method, in which the user indicates the region of interest in the mask by clicking, selecting boxes, or other forms of interaction; it is not limited to these methods.

[0123] In step S302 of some embodiments, the image inpainting model can be a context encoder, which learns the context information of the image through an encoder-decoder architecture and predicts the object to be inpainted; it can also be a BrushNet model, which effectively processes the object to be inpainted in the image by decomposing a dual-branch architecture; it is not limited to these.

[0124] Please see Figure 4 In some embodiments, at least two image blocks include a first image block and a second image block; step S104 may include, but is not limited to, steps S401 to S404:

[0125] Step S401: Extract the contour of the object to be repaired from the reference video frame image to obtain the contour of the object to be repaired.

[0126] Step S402: In the reference video frame image, extract the region surrounded by the outline of the object to be repaired as the first image block;

[0127] Step S403: Calculate the bounding rectangle of the object to be repaired in the reference video frame image;

[0128] Step S404: In the reference video frame image, extract the region enclosed by the bounding rectangle as the second image block.

[0129] In the embodiments of this application, steps S401 to S404 involve using the region enclosed by the outline of the object to be repaired as a first image block and the region enclosed by the circumscribed rectangle as a second image block. By combining the image description information generated from the first and second image blocks, a repair prompt word is obtained. The repair prompt word contains spatial information about the outline of the object to be repaired and its surrounding environment, which helps the image generation method understand the object to be repaired with a complex outline, allowing it to be better integrated into the video frame image and improving the repair effect of the target video frame image.

[0130] In step S401 of some embodiments, the contour of the object to be repaired can be extracted by an edge detection method based on the Canny operator, or by an edge detection method based on the Hough transform, and is not limited thereto.

[0131] In step S403 of some embodiments, the circumscribed rectangle can be either the minimum circumscribed rectangle or the maximum circumscribed rectangle, and is not limited thereto.

[0132] Please see Figure 4 In some embodiments, at least two image blocks further include a third image block; after step S404, the video-based image inpainting method further includes steps S405 to S406:

[0133] Step S405: In the reference video frame image, calculate the expanded bounding box of the repair object; wherein the expanded bounding box coincides with the second image block and the area of ​​the expanded bounding box is greater than the area of ​​the second image block;

[0134] Step S406: In the reference video frame image, extract the region enclosed by the extended box as the third image block.

[0135] In steps S405 to S406 of the embodiments of this application, the region enclosed by the extended frame is extracted as the third image block. The extended frame coincides with the second image block and the area of ​​the extended frame is larger than the area of ​​the second image block. The image description information of the first image block, the second image block and the third image block are combined to generate a repair prompt word. The repair prompt word also includes the spatial information around the second image block. The reconstructed image can be better integrated into the damaged video frame image, further improving the repair effect of the target video frame image.

[0136] Please see Figure 5 In some embodiments, step S405 may include, but is not limited to, steps S501 to S505:

[0137] Step S501: Calculate the center point of the second image block;

[0138] Step S502: Determine the center point of the expansion box based on the center point of the second image block;

[0139] Step S503: Calculate the size of the second image block;

[0140] Step S504: Calculate the size of the expansion box based on the preset scaling factor and the size of the second image block, expressed by the formula:

[0141]

[0142] In the formula, α is a scaling factor, α > 1; H and W are the length and width of the second image patch, respectively; H ′ W ′ These are the length and width of the third image block, respectively.

[0143] Step S505: Generate the expansion box based on the center point and size of the expansion box.

[0144] In the embodiments of this application, steps S501 to S505 determine the size of the third image block by pre-setting the ratio between the second and third image blocks. The size of the third image block can be adaptively determined according to the size of the object to be repaired, thereby improving the consistency of the target video frame image in the video sequence.

[0145] In step S501 of some embodiments, the center point of the second image block can be the geometric center of the second image block or the centroid of the second image block, and is not limited thereto. For example, the centroid of the second image block can be obtained by calculating the ratio of the weighted sum of all pixel coordinates in the second image block to the cumulative sum of all pixel coordinates, using pixel values ​​as weights.

[0146] In some embodiments, step S502 can be performed by directly using the center point of the second image block as the center point of the expansion box; alternatively, the center point of the second image block can be offset and the offset center point can be used as the center point of the expansion box; this is not the only option.

[0147] In some embodiments, step S505 may include, but is not limited to:

[0148] Calculate the coordinates of the four vertices of the expanded box based on its center point and size.

[0149] Determine whether the coordinates of the four vertices of the expanded bounding box exceed the image boundary of the reference video frame. If they do, modify the size of the expanded bounding box so that the coordinates of the four vertices do not exceed the image boundary of the reference video frame.

[0150] In some embodiments, in step S105, the image description information may describe the action of an object in the image, such as jumping, standing, or sitting; it may also describe the style of the image, such as cartoon or portrait; or it may describe the environment in the image, such as forest or ocean; and it is not limited thereto.

[0151] In step S105 of some embodiments, an image description is generated for each image patch using a visual language model, resulting in multiple image description information. Multiple damaged video frame images are collected, and for each damaged video frame image, the region enclosed by the outline of the object to be repaired is extracted, and the image description information of this region is manually annotated, resulting in multiple negative samples. It should be noted that the manually annotated image description information includes the original state of the object to be repaired. Multiple reference video frame images are collected, and for each reference video frame image, the region containing the object to be repaired is extracted, and the image description information of this region is manually annotated, resulting in multiple positive samples. During the training of the visual language model, the distance between the image patch and the image description information in the positive samples is maximized, while the distance between the image patch and the image description information in the negative samples is minimized.

[0152] In this embodiment, the object to be repaired is extracted from the damaged video frame image and the image description information is labeled to construct negative samples. The visual language model is trained through the negative samples, so that the visual language model can generate description information that matches the image content more accurately and provide more precise repair prompts.

[0153] In step S106 of some embodiments, multiple image description information can be directly combined to obtain repair prompt words; or multiple image description information can be input into a large language model to generate repair prompt words; it is not limited to this.

[0154] In some embodiments, step S105 may include, but is not limited to, generating image descriptions for the first image block, the second image block, and the third image block respectively, to obtain corresponding first image description information, second image description information, and third image description information. Step S106 may include, but is not limited to, combining the image description information in the order of third image description information, second image description information, and first image description information to obtain repair prompt words. Step S107 may include, but is not limited to, weighting each image description information in the repair prompt words using an image generation model, with the weights of the image description information decreasing sequentially; and generating an image based on the weighted values ​​of the repair prompt words to obtain a reconstructed image.

[0155] In this embodiment, the image description information is weighted according to spatial scale, and the image description information containing the largest spatial scale is assigned the largest weight. The image generation model can more effectively process the overall structure of the image and improve the integrity and coherence of the target video frame image.

[0156] In step S107 of some embodiments, the image of the object to be repaired can be generated by a Latent Diffusion Model or by a Generative Adversarial Network (GAN), and is not limited thereto.

[0157] In step S107 of some embodiments, the size of the reconstructed image can be set according to actual conditions and is not limited. For example, the size of the reconstructed image can be the same as the size of the second image block or the same as the size of the third image block.

[0158] Following step S108 in some embodiments, the video-based image inpainting method further includes post-processing the target video frame image to enhance its image quality. Post-processing includes color correction, noise reduction, contrast adjustment, image fusion, and image super-resolution. For example, smoothing the edges of the reconstructed image in the target video frame image improves the fusion effect of the reconstructed image.

[0159] Please see Figure 6This application also provides a video-based image restoration apparatus, which can implement the above-described video-based image restoration method. The apparatus includes:

[0160] The image data acquisition module is used to acquire the set of video frame images from the raw video data;

[0161] The image patch extraction module is used to perform target detection on the video frame image set to obtain the repair object; select a reference video frame image and a damaged video frame image from the video frame image set based on the repair object, wherein the reference video frame image and the damaged video frame image contain the same repair object; and extract at least two image patches containing the repair object from the reference video frame image, wherein the spatial scales of any two image patches are different.

[0162] The repair prompt word generation module is used to generate image descriptions for each image block, obtaining multiple image description information; and to generate repair prompt words based on the multiple image description information.

[0163] The image restoration module is used to generate an image of the object to be restored based on restoration prompts, thereby obtaining a reconstructed image; and to replace the object to be restored in the damaged video frame image with the reconstructed image to obtain the target video frame image.

[0164] The specific implementation of this video-based image restoration device is basically the same as the specific embodiment of the video-based image restoration method described above, and will not be repeated here.

[0165] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described video-based image restoration method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0166] Please see Figure 7 , Figure 7 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0167] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application application-specific integrated circuit (AAIC), or one or more integrated circuits, and is used to execute relevant programs to achieve the technical solutions provided in the embodiments of this application.

[0168] The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the video-based image restoration method of the embodiments of this application.

[0169] The input / output interface 903 is used to implement information input and output;

[0170] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as UAB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0171] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0172] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0173] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described video-based image restoration method.

[0174] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0175] The video-based image restoration method, device, electronic device, and storage medium provided in this application select reference video frame images from a set of video frame images. The reference video frame images and the damaged video frame images are in the same time dimension and have a high correlation in image content. Image blocks of different spatial scales are extracted from the reference video frame images to generate image descriptions. Repair prompts are generated based on the image description information. The obtained repair prompts contain different levels of spatial information of the object to be repaired in the same time dimension, which can provide more details of the surrounding environment of the object to be repaired for the image generation method. This allows the generated reconstructed image to be better integrated into the damaged video frame image, thereby improving the image restoration effect.

[0176] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0177] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0178] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0179] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0180] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0181] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0182] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0183] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0184] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0185] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0186] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A video-based image restoration method, characterized in that, The method includes: Obtain the set of video frame images from the raw video data; Target detection is performed on the set of video frame images to obtain the object to be repaired; Based on the repair object, a reference video frame image and a damaged video frame image are selected from the video frame image set, wherein the reference video frame image and the damaged video frame image contain the same repair object; In the reference video frame image, at least two image blocks containing the repair object are extracted, wherein any two image blocks have different spatial scales. This includes: extracting the contour of the repair object in the reference video frame image to obtain the object's contour; extracting the region enclosed by the object's contour as a first image block in the reference video frame image; calculating the bounding rectangle of the repair object in the reference video frame image; extracting the region enclosed by the bounding rectangle as a second image block in the reference video frame image; calculating the expanded bounding box of the repair object in the reference video frame image, wherein the expanded bounding box coincides with the second image block and the area of ​​the expanded bounding box is larger than the area of ​​the second image block; and extracting the region enclosed by the expanded bounding box as a third image block in the reference video frame image. Image description generation is performed on each image block to obtain multiple image description information, including: image description generation is performed on the first image block, the second image block and the third image block respectively to obtain corresponding first image description information, second image description information and third image description information; Based on the image description information, a repair prompt word is generated, including: combining the image description information in the order of the third image description information, the second image description information, and the first image description information to obtain the repair prompt word; The process of generating an image of the object to be repaired based on the repair prompts to obtain a reconstructed image includes: weighting each image description information in the repair prompts, with the weights of the image description information decreasing in order, and generating an image based on the weighted values ​​of the repair prompts to obtain a reconstructed image. The target video frame image is obtained by replacing the repair object in the damaged video frame image with the reconstructed image.

2. The method according to claim 1, characterized in that, The calculation of the expanded frame of the repair object includes: Calculate the center point of the second image patch; The center point of the expanded frame is determined based on the center point of the second image block; Calculate the size of the second image patch; The size of the expansion box is calculated based on the preset scaling factor and the size of the second image block; The expansion box is generated based on its center point and size.

3. The method according to any one of claims 1 to 2, characterized in that, The step of selecting reference video frame images and damaged video frame images from the video frame image set based on the repair object includes: Keyframes are extracted from the video frame image set to obtain a keyframe image set; Based on the object to be repaired, the reference video frame image and the damaged video frame image are selected from the keyframe image set.

4. The method according to claim 3, characterized in that, The method further includes: Calculate the number of keyframe images in the keyframe image set to obtain the number of keyframes; Calculate the number of damaged video frame images to obtain the number of damaged video frames; The ratio of the number of corrupted video frames to the number of keyframes is calculated to obtain the corruption index; If the damage index is less than or equal to a preset index threshold, then in the keyframe image set, multiple damaged video frame images are replaced with corresponding target video frame images; intermediate frames are generated based on the keyframe image set to obtain a target video frame image sequence; If the damage index is greater than the index threshold, then in the video frame image set, the damaged video frame image is replaced with the corresponding target video frame image to obtain the target video frame image sequence.

5. The method according to any one of claims 1 to 2, characterized in that, The step of selecting reference video frame images and damaged video frame images from the video frame image set based on the repair object includes: Based on the region where the object to be repaired is located in the video frame image, a mask for identifying the object to be repaired is constructed; The image restoration model is used to perform image restoration on the video frame image set with the mask to obtain an intermediate video frame image set. Keyframe extraction is performed on the intermediate video frame image set to obtain an intermediate keyframe image set; Based on the object to be repaired, the reference video frame image and the damaged video frame image are selected from the intermediate keyframe image set.

6. A video-based image restoration device, characterized in that, The device includes: The image data acquisition module is used to acquire the set of video frame images from the raw video data; An image patch extraction module is used to perform target detection on the video frame image set to obtain a repair object; based on the repair object, a reference video frame image and a damaged video frame image are selected from the video frame image set, wherein the reference video frame image and the damaged video frame image contain the same repair object; in the reference video frame image, at least two image patches containing the repair object are extracted, wherein any two image patches have different spatial scales, including: in the reference video frame image, contour extraction of the repair object is performed to obtain the contour of the repair object; in the reference video frame image, the region enclosed by the contour of the repair object is extracted as a first image patch; in the reference video frame image, the bounding rectangle of the repair object is calculated; in the reference video frame image, the region enclosed by the bounding rectangle is extracted as a second image patch; in the reference video frame image, an expanded bounding box of the repair object is calculated; wherein the expanded bounding box coincides with the second image patch and the area of ​​the expanded bounding box is larger than the area of ​​the second image patch; in the reference video frame image, the region enclosed by the expanded bounding box is extracted as a third image patch; The repair prompt word generation module is used to generate image descriptions for each image block to obtain multiple image description information, including: generating image descriptions for the first image block, the second image block, and the third image block respectively to obtain corresponding first image description information, second image description information, and third image description information; and generating repair prompt words based on the multiple image description information, including: combining the image description information in the order of third image description information, second image description information, and first image description information to obtain repair prompt words. The image restoration module is used to generate an image of the object to be restored based on the restoration prompt words to obtain a reconstructed image. The module includes: weighting each image description information in the restoration prompt words, with the weights of the image description information decreasing in order; generating an image based on the weighted values ​​of the restoration prompt words to obtain a reconstructed image; and replacing the object to be restored in the damaged video frame image with the reconstructed image to obtain a target video frame image.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images