Image inpainting method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose an image repairing method and device, electronic equipment and storage medium. The method comprises: obtaining an image to be repaired and a text description, wherein the text description is used to represent feature information of at least one target in the image to be repaired and instruct to repair at least one target in the image to be repaired; based on the text description, processing the image to be repaired to obtain a mask image of the target; and repairing the image to be repaired by using the mask image of the target to generate a repaired image. Since the mask pattern of the target is generated based on the text description, by implementing the method of the embodiments of the present application, the target is combined with the text description, which is not limited by a predefined category, is suitable for transfer learning, can process various types of images more flexibly, and effectively improves the versatility.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image restoration method, apparatus, electronic device, and storage medium. Background Technology

[0002] Image restoration is a method that uses computer algorithms to automatically restore damaged or missing parts of an image. In the digital age, image restoration is widely used in fields such as video restoration and old photograph restoration, which can improve the visual quality and viewing experience while preserving historical and cultural heritage.

[0003] In the process of researching and practicing existing technologies, the inventors of the embodiments of this application have discovered that currently, it is necessary to train the model based on a large amount of labeled training data and computing resources, and then use the trained model to repair the image.

[0004] However, traditional unimodal pre-trained models require a large amount of labeled data for training, and currently, the data used to train pre-trained models must be labeled for effective training. If some or all objects in an image are unlabeled, the pre-trained model cannot infer and recognize unlabeled objects of the same type in new input images during subsequent image inpainting stages, thus hindering image inpainting. Considering that actual business needs are constantly changing, current image inpainting methods cannot quickly label larger amounts of data or completely label all objects in an image. This results in an inability to quickly adapt to changing business needs and handle unknown or new categories of data. Overall, this type of image inpainting method suffers from poor versatility. Summary of the Invention

[0005] This application provides an image restoration method, apparatus, electronic device, and storage medium, which can flexibly restore various types of images and improve versatility.

[0006] In a first aspect, embodiments of this application provide an image restoration method, including:

[0007] Obtain an image to be repaired and a text description, wherein the text description is used to represent feature information of at least one target in the image to be repaired and to indicate that at least one target in the image to be repaired is to be repaired;

[0008] Based on the text description, the image to be repaired is processed to obtain a mask image of the target;

[0009] The image to be repaired is repaired using the mask image of the target, resulting in a repaired image.

[0010] Secondly, embodiments of this application also provide an image restoration apparatus, comprising:

[0011] An input / output unit is used to acquire an image to be repaired and a text description, wherein the text description is used to represent feature information of at least one target in the image to be repaired and to indicate that at least one target in the image to be repaired is to be repaired;

[0012] The processing unit is configured to process the image to be repaired based on the text description to obtain a mask image of the target; and to repair the image to be repaired using the mask image of the target to generate a repaired image.

[0013] Thirdly, embodiments of this application also provide an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0014] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, can implement the above-described method.

[0015] Fifthly, embodiments of this application provide a chip that includes a transceiver coupled to a terminal device for executing the technical solution provided in the first aspect of embodiments of this application.

[0016] Sixthly, embodiments of this application provide a chip system including a processor for supporting a terminal device in implementing the functions involved in the first aspect, such as generating or processing information involved in the image restoration method provided in the first aspect. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the terminal. The chip system may be composed of chips or may include chips and other discrete devices.

[0017] In a seventh aspect, embodiments of this application provide a computer program product containing instructions. When the computer program product is run on a computer, it causes the computer to execute the image restoration method provided in the first aspect, thereby achieving the beneficial effects of the image restoration method provided in the first aspect.

[0018] Compared to existing technologies, the solution provided in this application generates a mask image of the target corresponding to the text description based on the image to be processed and the text description. Based on the mask image and the original image to be repaired, a repaired image is obtained. Since the text description represents the feature information of at least one target in the image to be repaired and indicates that at least one target in the image to be repaired should be repaired, the process of processing the image to be repaired based on the text description to obtain the target mask image eliminates the dependence on labeled data. Combining the image to be repaired with the text description to obtain the target mask image for repairing the image to be repaired is not limited by predefined categories, is suitable for transfer learning, and can more flexibly handle various types of data, effectively improving versatility. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram illustrating an application scenario of the first image restoration method provided in the embodiments of this application;

[0021] Figure 2 This is a schematic diagram illustrating an application scenario of the second image restoration method provided in the embodiments of this application;

[0022] Figure 3 This is a schematic diagram illustrating an application scenario of the third image restoration method provided in the embodiments of this application;

[0023] Figure 4 A schematic flowchart illustrating the first image restoration method provided in this application embodiment;

[0024] Figure 5 A flowchart illustrating the second image restoration method provided in this application embodiment;

[0025] Figure 6 A flowchart illustrating a method for repairing dynamic images provided in an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of the structure of an image restoration device provided in an embodiment of this application;

[0027] Figure 8 This is a schematic diagram of the structure of a hardware device provided in an embodiment of this application;

[0028] Figure 9 This is a schematic diagram of a server structure provided in an embodiment of this application. Detailed Implementation

[0029] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.

[0030] Current image inpainting methods typically employ a combination of traditional convolutional neural networks (CNNs) and generative adversarial networks (GANs). However, this approach requires training the model on a large amount of labeled training data and consuming significant computational resources before using the trained model for image inpainting. Because the model trained on large amounts of labeled data can only inpaint images corresponding to the scenes it was trained on, it can only infer and recognize labeled data. For unlabeled and untrained scenes, a new model must be obtained through retraining to perform image inpainting. Furthermore, considering that actual business needs are constantly changing, current image inpainting methods cannot quickly process annotations on larger datasets or completely eliminate the need for labeling of certain objects. This results in a failure to adapt rapidly to changing business requirements and to handle unknown or new types of images, leading to a lack of versatility in current image inpainting methods.

[0031] Based on this, embodiments of this application provide an image restoration method, apparatus, electronic device, and storage medium. Based on the image to be restored and a text description, a mask image of the target is generated, so that the image to be restored can be restored using the mask image to obtain a restored image. The text description represents the feature information of at least one target in the image to be restored and indicates that at least one target in the image to be restored should be restored. By combining the target with the text description, it is not limited by predefined categories and does not rely on a large amount of labeled data. The target corresponding to the text description can be determined from the image to be restored, and a mask image of the target can be generated for restoration. This method is suitable for transfer learning, can handle various types of data more flexibly, and can effectively improve versatility.

[0032] Specifically, the image restoration method provided in this application can be applied to the restoration of dynamic or static images, such as video restoration and text restoration in documents. The following is a combination of... Figures 1-3 This paper introduces the application scenarios of the image restoration method provided in this application.

[0033] The image restoration method provided in this application can generate a mask image of the target based on the target in the image to be restored and the corresponding text description of the target, so as to restore the image to be restored based on the mask image. Please refer to [link to relevant documentation]. Figure 1 The figure is a schematic diagram of an application scenario for the first image restoration method provided in the embodiments of this application.

[0034] Electronic devices need to remove targets from an image to repair it. Since a text description represents the feature information of at least one target in the image to be repaired and indicates that at least one target in the image should be repaired, the target corresponding to the text description can be identified from the image to be repaired based on the image to be repaired, and a mask image of the target can be generated. Using the mask image of the target, the image to be repaired can be repaired to obtain the repaired image.

[0035] The image restoration method provided in this application can be used to restore dynamic images, which may include videos, animated pictures, etc. Please refer to [link / reference]. Figure 2 The figure is a schematic diagram of an application scenario for the second image restoration method provided in the embodiments of this application.

[0036] An electronic device needs to delete a little girl crossing the street in a video, so it targets the little girl. The electronic device acquires the video, a text description of the little girl, and a region of interest (ROI). The ROI can be the area where the little girl is located, and the text description can be "a girl wearing a skirt on a pedestrian crossing." The device crops an image of the ROI from the video. Based on the text description, it detects the ROI image to obtain a bounding box corresponding to the little girl. It then crops an image of the bounding box from the ROI image. Based on the text description, it processes the bounding box image to obtain a mask image of the little girl within the ROI. After obtaining the mask image, it generates a repaired ROI image based on the mask image and the ROI image. Finally, it stitches the repaired ROI image with the original image to obtain the repaired image.

[0037] The image restoration method provided in this application can be used to restore static images, which may include still images, etc. Please refer to [link / reference]. Figure 3 The figure is a schematic diagram of an application scenario for the third image restoration method provided in the embodiments of this application.

[0038] The electronic device needs to delete the characters in the lower right corner of the image. Since the image is a static image, the mask image of the target can be obtained by directly processing the image of the region of interest based on the text description of the target.

[0039] Specifically, the electronic device acquires the image to be repaired, and based on the text description of the characters in the lower right corner of the image, crops out the region of interest (ROI) from the image to be repaired; based on the text description of the characters in the lower right corner, the ROI image is processed to obtain a mask image of the text description of the characters in the lower right corner of the ROI; then, based on the mask image and the ROI image, a repaired ROI image is generated; the repaired ROI image is then stitched together with the original image to be repaired to obtain the repaired image.

[0040] The solutions in this application embodiment can be implemented based on artificial intelligence technology, specifically involving the field of computer vision technology within artificial intelligence technology, which will be briefly introduced below.

[0041] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0042] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0043] Computer vision (CV) is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, model robustness testing, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as fingerprint recognition.

[0044] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0045] It should be noted that the service terminal involved in the embodiments of this application can be a device that provides voice and / or data connectivity to the service terminal, a handheld device with wireless connectivity, or other processing devices connected to a wireless modem. Examples include mobile phones (or "cellular" phones) and computers with mobile terminals, such as portable, pocket-sized, handheld, computer-embedded, or vehicle-mounted mobile devices that exchange voice and / or data with a wireless access network. Examples include Personal Communication Service (PCS) phones, cordless phones, Session Initiation Protocol (SIP) phones, Wireless Local Loop (WLL) stations, and Personal Digital Assistants (PDAs).

[0046] The image restoration method provided in the embodiments of this application is described below.

[0047] The image restoration method provided in this application can restore both dynamic and static images. Please refer to [link / reference]. Figure 4The figure is a flowchart of the first image restoration method provided in the embodiment of this application, which includes the following steps S101-S103.

[0048] S101. Obtain the image to be repaired and its text description.

[0049] The image to be repaired can be a dynamic image, such as a video, or a static image, such as a still picture. The text description is used to represent the feature information of at least one target in the image to be repaired and to indicate that at least one target in the image to be repaired should be repaired.

[0050] In one possible implementation, the text description is used to describe the feature information of a target in the image to be repaired. For example, if the target in an image is a girl wearing blue clothes, the text description of the target could be "girl wearing blue clothes". Based on this text description, repair can be performed on that target, i.e., on the "girl wearing blue clothes" in the image.

[0051] In one possible implementation, the text description is used to describe the feature information of multiple targets in the image to be repaired. For example, if there are multiple targets in an image that need to be repaired, and the text description describes the feature information of multiple targets, then based on the text description, multiple targets in the image can be repaired simultaneously.

[0052] S102. Based on the text description, process the image to be repaired to obtain the mask image of the target.

[0053] In this application, a mask image of the target can be used to repair an image to obtain a repaired image. The mask image is a specific image used for covering; that is, when controlling or processing the image to be repaired, a selected image, graphic, or object can be used to occlude all or part of the image to be repaired.

[0054] Specifically, in the embodiments of this application, the text description and the image to be repaired can be used as inputs to the multimodal segmentation model. The multimodal segmentation model will generate a mask image of the target in the image to be repaired that corresponds to the text description based on the text description, so that the electronic device can obtain the location of the missing area or the area to be repaired in the image to be repaired during the subsequent image repair process.

[0055] For example, in an image, the target to be repaired is a girl wearing blue clothes. The text description describes the feature information of the girl wearing blue clothes, and this text description can instruct the target in the image to be repaired. The multimodal segmentation model will then generate a mask image corresponding to the girl based on this text description, i.e., based on "girl wearing blue clothes", so that subsequent electronic devices can perform image repair on the area where the girl is located.

[0056] The image restoration method provided in this application embodiment can be applied to restore dynamic and static images. When the image to be restored is a dynamic image, in order to obtain the mask image of the target, the image to be restored can be detected to obtain the bounding box of the target; the image of the bounding box is cropped from the image to be restored, and then the image of the bounding box is processed to obtain the mask image of the target, that is, the mask image of the target in the image of the bounding box is generated.

[0057] Specifically, based on the text description of the target, the image to be repaired is detected to obtain the bounding box of the target. For dynamic images to be repaired, the text description of the target and the image to be repaired can be input into the target detection module in the electronic device. The target detection module can detect the target corresponding to the text description from the image to be repaired using a multimodal detection model, based on the text description of the target and the image to be repaired, and generate the bounding box of the target.

[0058] After generating the bounding box of the target, the bounding box image is cropped from the image to be repaired. Based on the text description of the target, the bounding box image is processed to obtain the mask image of the target.

[0059] In one possible implementation, to further improve image restoration efficiency, the detection process can be accelerated using an acceleration module that integrates computationally intensive algorithms.

[0060] Specifically, computationally intensive algorithms are typically applied to systems that consume large amounts of central processing unit (CPU) resources and involve high computational demands, with computational speed being the primary performance indicator. For example, deploying a network management system (NMS) into acceleration modules within electronic devices can accelerate the process, reducing the burden on the CPU and graphics processing unit (GPU), and improving image restoration efficiency through the heterogeneous processing capabilities of the electronic device.

[0061] S103. Use the mask image of the target to repair the image to be repaired, and generate the repaired image.

[0062] The target's mask image can be used to indicate that the target in the image to be repaired needs to be repaired. For example, the target's mask image and the image to be repaired can be used as input to an image inpainting model. The image inpainting model uses the target's mask image to repair the image to be repaired, generating a repaired image.

[0063] In summary, this application, based on a multimodal segmentation model, uses the image to be repaired and text description as input to generate a mask image of the target. This mask image is then used to repair the original image to be repaired, resulting in a repaired image. Due to the zero-shot learning characteristic of multimodal models, combining the image to be repaired with text description is not limited by predefined categories. Based on the text description of the target, the target in the image to be repaired can be repaired to obtain the repaired image. This approach is suitable for transfer learning, allows for more flexible processing of various data types, and effectively improves versatility.

[0064] Meanwhile, the image restoration method based on the multimodal model in this application does not require training of the multimodal model with a large amount of labeled data. Compared with the image restoration method based on the model trained with a large amount of labeled data, the method provided by the embodiments of this application can train and use the model with limited training data, thereby reducing the cost of training data collection and labeling.

[0065] In the embodiments of this application, while improving versatility, the speed of image restoration can also be increased. For example, the image to be restored can be cropped based on a preset region of interest (ROI). The ROI image is then extracted from the image to be restored and processed specifically for the ROI image. By reducing the amount of data processed, the speed of image restoration can be improved. Figure 5 As shown, this figure is a flowchart of the second image restoration method provided in the embodiment of this application, including S201 to S205.

[0066] S201. Obtain the image to be repaired, the text description, and the region of interest in the image to be repaired.

[0067] In machine vision and image processing, the Region of Interest (ROI) is a region in the image to be repaired that is delineated using shapes such as rectangles, circles, ellipses, or irregular polygons. In this embodiment, the ROI is a local image determined from the image to be repaired; it can be at least one region in the image to be repaired specified by the user, containing the target of interest to the user.

[0068] For example, electronic devices can identify the region of interest by obtaining the coordinates corresponding to the region of interest, so that subsequent processing can be performed on the region of interest in the image to be repaired, which can greatly reduce inference time and increase processing accuracy.

[0069] S202. Cropping the region of interest from the image to be repaired.

[0070] Based on the region of interest obtained by the electronic device in S201, the region of interest is cropped from the image to be repaired. Subsequently, only the image of the region of interest needs to be processed, which can greatly reduce the processing time.

[0071] Meanwhile, when the region of interest is not cropped from the image, it is usually necessary to use the resize function to scale the image to a certain size. After stacking, the image outline may appear severely jagged, indicating that the parameter settings of the resize function affect the image quality. In this application, after cropping the region of interest from the image to be repaired, only a portion of the image to be repaired can be processed, eliminating the need to use the resize function to process the entire image. This effectively avoids data distortion caused by using the resize function.

[0072] In one possible implementation, when cropping the region of interest from the image to be repaired, the image to be repaired can be transcoded according to its type to obtain a transcoded image to be repaired; and the region of interest can be cropped from the transcoded image to be repaired.

[0073] Specifically, transcoding each frame of the image to be repaired can effectively reduce transmission bandwidth pressure and improve data processing efficiency. For example, for dynamic images, such as GIFs or videos, smaller transcoding parameters can be set to reduce transmission bandwidth pressure while ensuring the resolution of the image to be repaired and not affecting its display effect; while for static images, such as still images, larger transcoding parameters can be set.

[0074] Therefore, in this embodiment of the application, when transcoding each frame of the image to be repaired, a transcoding parameter corresponding to the type of the image to be repaired can be generated. Based on the transcoding parameter, the image to be repaired is transcoded, so as to effectively reduce the transmission bandwidth pressure and improve the data processing efficiency while ensuring the display effect of the image to be repaired.

[0075] In one possible implementation, when cropping the region of interest from the image to be repaired, the frame rate per second of the image to be repaired can be fixed to a preset frame rate to obtain a fixed image to be repaired; the region of interest is then cropped from the fixed image to be repaired.

[0076] Specifically, by fixing the number of frames per second (FPS) of the image to be repaired to a preset number of transmission frames, the memory usage can be reduced, the number of inferences for high frame rate images to be repaired can be effectively reduced, thereby improving the processing efficiency of the images to be repaired.

[0077] In one possible implementation, data processing efficiency can be improved by reducing the number of inference operations for high-frame-rate images to be repaired through transcoding and fixing the frame rate (FPS). Specifically, the image to be repaired is transcoded according to its type to obtain a transcoded image; the frame rate (FPS) of the transcoded image is then fixed to a preset number to obtain a fixed image.

[0078] S203. Based on the text description, process the image of the region of interest to obtain a mask image of the target.

[0079] The region of interest (ROI) is a local image determined from the image to be repaired, and this local image contains the target. In this embodiment, the text description and the ROI image can be used as input to a multimodal segmentation model. The multimodal segmentation model determines the target corresponding to the text description from the ROI image based on the text description and generates a mask image of the target. Obtaining this mask image facilitates the electronic device in subsequent image repair processes by allowing it to locate the missing or repairable regions within the ROI image.

[0080] In this embodiment, processing can be performed only on a local area of ​​the image to be repaired, that is, on the image of the region of interest, so as to greatly reduce inference time and increase processing accuracy.

[0081] The image inpainting method provided in this application can be applied to inpainting both dynamic and static images. When the image to be inpainted is static, the image of the region of interest (ROI) can be directly processed to obtain a mask image of the target within the ROI. When the image to be inpainted is dynamic, the image of the ROI can be detected to obtain the bounding box of the target; the bounding box image is then cropped from the ROI image, and further processed to obtain a mask image of the target. For example... Figure 6As shown in the figure, this figure is a flowchart of a method for repairing dynamic images provided in an embodiment of this application, wherein S203 can be implemented by S211 to S213.

[0082] S211. Based on the text description, detect the image to be repaired to obtain the bounding box of the target.

[0083] Specifically, for dynamic images to be repaired, such as GIFs or videos, the text description of the target and the image of the region of interest can be input into the target detection module in the electronic device. The target detection module can detect the target corresponding to the text description from the image of the region of interest and generate the bounding box of the target.

[0084] In one possible implementation, to further improve image restoration efficiency, the image to be restored can be detected using a computationally intensive algorithm based on the text description of the target to obtain the target's bounding box. The acceleration module integrating the computationally intensive algorithm is used to accelerate the detection of the image to be restored.

[0085] S212. Cropping the bounding box image from the image to be repaired.

[0086] The bounding box image contains the target. Specifically, the target's bounding box can be detected based on the target's text description. Cropping the bounding box image based on the target's bounding box allows for further processing, improving both image processing speed and accuracy.

[0087] S213. Generate a mask image of the target in the bounding box image.

[0088] For example, the text description of the target and the image of the bounding box can be input into the multimodal segmentation model. The multimodal segmentation model will process the image of the bounding box based on the text description of the target to generate a mask image of the target.

[0089] S204. Use the mask image of the target to repair the image of the region of interest and generate the repaired image of the region of interest.

[0090] Specifically, after obtaining the mask image of the target, the mask image of the target and the image of the region of interest (ROI) can be input into an image inpainting model to generate an inpainted image of the ROI. The image inpainting model is used to inpaint the ROI image using the target's mask image to obtain the inpainted ROI image.

[0091] S205. The image of the repaired region of interest is stitched together with the image to be repaired to obtain the repaired image.

[0092] The region of interest (ROI) is the image that needs to be repaired. In this embodiment, the ROI image is cropped from the image to be repaired so that only the ROI image is processed. Then, after obtaining the repaired ROI image, it is stitched together with the image to be repaired to obtain the repaired image.

[0093] In one possible implementation, the repaired image can be subjected to image fusion processing to obtain a fused repaired image.

[0094] Specifically, image fusion aims to combine parts of different images into a single image. The more natural the resulting image appears, the better the image fusion. It should be noted that this application does not specifically limit the image fusion algorithm used; for example, alpha blending, Laplacian pyramid blending, and Poisson blending algorithms may be employed.

[0095] The core idea of ​​the Poisson fusion algorithm is not to directly superimpose the two images to be fused, but to allow the target image to "grow" a new image in the fusion region based on the guiding field of the source image. Therefore, only the gradient field of the source image is needed, and the target image can generate the fusion region according to its own characteristics and the corresponding gradient field of the source image. Because the target image generates the fusion region based on its own characteristics, the fusion result appears more natural. In other words, in this embodiment, the Poisson fusion algorithm allows the image of the repaired region of interest to "grow" a new image based on the guiding field of the original image to be repaired, thus solving the problem of potential color difference at the stitching edges between the repaired region of interest and the image to be repaired, and improving the effect of the repaired image.

[0096] In summary, the method provided in this application generates a mask image of the target corresponding to the text description based on the image to be repaired and the text description. This mask image can then be used to repair the original image to be repaired, resulting in a repaired image. Since the text description in this application represents the feature information of at least one target in the image to be repaired and indicates the repair of at least one target in the image, combining the image to be repaired with the text description is not limited by predefined categories, making it suitable for transfer learning and enabling more flexible processing of various types of images, effectively improving its versatility.

[0097] Simultaneously, based on the type of the image to be repaired, transcoding can reduce the bitrate of the image to be repaired without compromising its display quality, thus saving bandwidth consumption. By fixing the FPS of the image to be repaired, the number of inference iterations for high-frame-rate images is reduced. Images of regions of interest are extracted and processed for inference, reducing the amount of data processed. Furthermore, by leveraging the heterogeneous processing capabilities of electronic devices, hardware resources are better utilized, improving image repair efficiency.

[0098] like Figure 7 As shown in the figure, this is a schematic diagram of the structure of an image restoration device provided in an embodiment of this application, which can be applied to image restoration scenarios. The image restoration device 1100 in this embodiment of the application can achieve the functions corresponding to the above-mentioned... Figures 4-6 The steps of the image restoration method executed in any corresponding embodiment are described above. The functions implemented by the image restoration device 1100 can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The modules can be software and / or hardware. The image restoration device 1100 may include an input / output module 1101 and a processing module 1102.

[0099] In some embodiments, the input / output module 1101 is used to acquire an image to be repaired and a text description, wherein the text description represents feature information of at least one target in the image to be repaired and indicates that at least one target in the image to be repaired should be repaired. The processing module 1102 is used to process the image to be repaired based on the text description to obtain a mask image of the target; and to repair the image to be repaired using the mask image of the target to generate a repaired image.

[0100] In one possible implementation, the processing module 1102 is configured to: process the image of the region of interest based on the text description to obtain a mask image of the target, wherein the image of the region of interest is a local image determined from the image to be repaired, and the local image contains the target.

[0101] In one possible implementation, the processing module 1102 is used to: repair the image of the region of interest using the mask image of the target to generate a repaired image of the region of interest; and stitch the repaired image of the region of interest with the image to be repaired to obtain a repaired image.

[0102] In one possible implementation, the image to be repaired is a dynamic image, and the processing module 1102 is used to detect the image to be repaired based on the text description to obtain the bounding box of the target; crop the image of the bounding box from the image to be repaired; and generate a mask image of the target in the image of the bounding box.

[0103] In one possible implementation, the processing module 1102 is used to detect the image to be repaired based on the text description using an acceleration module that integrates computationally intensive algorithms, thereby obtaining the bounding box of the target. The acceleration module that integrates computationally intensive algorithms is used to accelerate the detection of the image to be repaired.

[0104] In one possible implementation, the processing module 1102 is specifically used to: transcode the image to be repaired according to the type of the image to be repaired, to obtain a transcoded image to be repaired; and crop out the region of interest from the transcoded image to be repaired.

[0105] In one possible implementation, the processing module 1102 is specifically used for:

[0106] The number of frames transmitted per second for the transcoded image to be repaired is fixed to a preset number of frames to obtain a fixed image to be repaired; the region of interest is cropped from the fixed image to be repaired.

[0107] In one possible implementation, the processing module 1102 further includes a fusion unit;

[0108] The fusion unit is used to perform image fusion processing on the repaired image to obtain a fused repaired image.

[0109] This application also provides a terminal device, such as... Figure 8 As shown, this figure is a schematic diagram of the structure of a hardware device provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. The terminal device can be any terminal device including mobile phones, tablet computers, personal digital assistants (PDAs), point-of-sale (POS) terminals, in-vehicle computers, etc. Taking a mobile phone as an example:

[0110] Figure 8 This diagram illustrates a partial structural representation of a mobile phone related to the terminal device provided in this embodiment. (Reference) Figure 8The mobile phone includes: a radio frequency (RF) circuit 510, a memory 520, an input unit 530, a display unit 540, a sensor 550, an audio circuit 560, a wireless fidelity (Wi-Fi) module 570, a processor 580, and a power supply 590, among other components. Those skilled in the art will understand that... Figure 8 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0111] The following is combined with Figure 8 A detailed introduction to each component of a mobile phone:

[0112] RF circuit 510 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with processor 580; additionally, it transmits uplink data to the base station. Typically, RF circuit 510 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, RF circuit 510 can also communicate wirelessly with networks and other devices. The aforementioned wireless communications may use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Messaging Service (SMS).

[0113] The memory 520 can be used to store software programs and modules. The processor 580 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 520. The memory 520 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0114] The input unit 530 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 530 may include a touch panel 531 and other input devices 532. The touch panel 531, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 531), and drive the corresponding connection devices according to a pre-set program. Optionally, the touch panel 531 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 580, and can also receive and execute commands sent by the processor 580. In addition, the touch panel 531 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 531, the input unit 530 may also include other input devices 532. Specifically, other input devices 532 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0115] Display unit 540 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. Display unit 540 may include display panel 541, optionally configured as a Liquid Crystal Display (LCD), Organic Light-Emitting Diode (OLED), or similar display panel 541. Further, touch panel 531 may cover display panel 541. When touch panel 531 detects a touch operation on or near it, it transmits the information to processor 580 to determine the type of touch event. Subsequently, processor 580 provides corresponding visual output on display panel 541 based on the type of touch event. Although in Figure 8 In this embodiment, the touch panel 531 and the display panel 541 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 531 and the display panel 541 can be integrated to realize the input and output functions of the mobile phone.

[0116] The mobile phone may also include at least one sensor 550, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 541 according to the ambient light level, and the proximity sensor can turn off the display panel 541 and / or backlight when the phone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, taps), etc. Other sensors that may be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0117] Audio circuit 560, speaker 561, and microphone 562 provide an audio interface between the user and the mobile phone. Audio circuit 560 converts received audio data into electrical signals and transmits them to speaker 561, where speaker 561 converts them into sound signals for output. On the other hand, microphone 562 converts collected sound signals into electrical signals, which are received by audio circuit 560, converted into audio data, and then output to processor 580 for processing. The audio data is then transmitted via RF circuit 510 to, for example, another mobile phone, or output to memory 520 for further processing.

[0118] Wi-Fi is a short-range wireless transmission technology. Through the Wi-Fi module 570, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 8 The Wi-Fi module 570 is shown, but it is understood that it is not a necessary component of the mobile phone and can be omitted as needed without changing the nature of the application.

[0119] The processor 580 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It executes software programs and / or modules stored in the memory 520, and calls data stored in the memory 520 to perform various functions and process data, thereby providing overall monitoring of the phone. Optionally, the processor 580 may include one or more processing units; preferably, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 580.

[0120] For example, in this embodiment of the application, the processor 580 can acquire an image to be repaired and a text description, wherein the text description is used to represent feature information of at least one target in the image to be repaired and to indicate that at least one target in the image to be repaired is repaired; based on the text description, the image to be repaired is processed to obtain a mask image of the target; the image to be repaired is repaired using the mask image of the target to generate a repaired image.

[0121] The mobile phone also includes a power supply 590 (such as a battery) that supplies power to various components. The power supply can be logically connected to the processor 580 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0122] Although not shown, mobile phones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0123] In this embodiment of the application, the processor 580 included in the mobile phone also has the function of controlling the execution of, for example Figure 4-6 The flowchart shown illustrates the image restoration method.

[0124] Figure 9This is a schematic diagram of a server structure provided in an embodiment of this application. The server 620 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 622 (e.g., one or more processors) and memory 632, and one or more storage media 630 (e.g., one or more mass storage devices) for storing application programs 642 or data 644. The memory 632 and storage media 630 can be temporary or persistent storage. The program stored in the storage media 630 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 622 may be configured to communicate with the storage media 630 and execute the series of instruction operations in the storage media 630 on the server 620.

[0125] Server 620 may also include one or more power supplies 626, one or more wired or wireless network interfaces 650, one or more input / output interfaces 658, and / or one or more operating systems 641, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.

[0126] The steps performed by the server in the above embodiments can be based on this Figure 9 The server 620 shown has the following structure. For example, the processor 622 performs the following operations by calling instructions in memory 632:

[0127] Obtain the image to be repaired and a text description, wherein the text description represents the feature information of at least one target in the image to be repaired and indicates that at least one target in the image to be repaired should be repaired; based on the text description, process the image to be repaired to obtain a mask image of the target; use the mask image of the target to repair the image to be repaired to generate a repaired image.

[0128] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not described in detail in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0129] The technical solutions provided in the embodiments of this application have been described in detail above. Specific examples have been used in the embodiments of this application to illustrate the principles and implementation methods of the embodiments of this application. The description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of the embodiments of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the embodiments of this application. Therefore, the content of this specification should not be construed as a limitation on the embodiments of this application.

Claims

1. An image restoration method, characterized in that, include: Obtain an image to be repaired and a text description, wherein the text description is used to represent feature information of at least one target in the image to be repaired and to indicate that at least one target in the image to be repaired is to be repaired; Based on the text description, the image to be repaired is processed to obtain a mask image of the target; The objective is identified from the image to be repaired based on the text description; The image to be repaired is repaired using the mask image of the target, generating a repaired image; The step of processing the image to be repaired based on the text description to obtain a mask image of the target includes: When the image to be repaired is a dynamic image, the text description and the image of the region of interest are input into a multimodal detection model. The multimodal detection model detects the target corresponding to the text description from the image of the region of interest based on the text description, and obtains the bounding box of the target. The image of the region of interest is a local image determined from the image to be repaired, and the local image contains the target. The image of the bounding box is cropped from the image of the region of interest. The text description and the image of the bounding box are input into a multimodal segmentation model. The multimodal segmentation model processes the image of the bounding box based on the text description to generate a mask image of the target in the image of the bounding box. When the image to be repaired is a static image, the text description and the image of the region of interest are input into a multimodal segmentation model. The multimodal segmentation model determines the target corresponding to the text description from the image of the region of interest based on the text description, and generates a mask image of the target. The step of repairing the image to be repaired using the mask image of the target to generate a repaired image includes: The image of the region of interest is repaired using the mask image of the target to generate a repaired image of the region of interest; The repaired region of interest image is stitched together with the image to be repaired to obtain the repaired image.

2. The method according to claim 1, characterized in that, The step of detecting the target corresponding to the text description from the image of the region of interest and obtaining the bounding box of the target includes: An acceleration module integrating computationally intensive algorithms is used to detect targets corresponding to the text description from the image of the region of interest, and to obtain the bounding box of the targets. The acceleration module integrating computationally intensive algorithms is used to accelerate the detection of the image to be repaired.

3. The method according to claim 1, characterized in that, The image of the region of interest is obtained through the following steps: Based on the type of the image to be repaired, the image to be repaired is transcoded to obtain the transcoded image to be repaired; The region of interest is cropped from the transcoded image to be repaired.

4. The method according to claim 3, characterized in that, When the image to be repaired is a dynamic image, the step of cropping the region of interest from the transcoded image to be repaired includes: The number of frames per second transmitted for the transcoded image to be repaired is fixed to a preset number of frames to obtain a fixed image to be repaired; The region of interest is cropped from the fixed image to be repaired.

5. The method according to any one of claims 1-4, characterized in that, After repairing the image to be repaired using the mask image of the target and generating the repaired image, the method further includes: The repaired image is then subjected to image fusion processing to obtain a fused repaired image.

6. An image restoration device, characterized in that, include: An input / output unit is used to acquire an image to be repaired and a text description, wherein the text description is used to represent feature information of at least one target in the image to be repaired and to indicate that at least one target in the image to be repaired is to be repaired; The processing unit is configured to process the image to be repaired based on the text description to obtain a mask image of the target; The target is identified from the image to be repaired based on the text description; the image to be repaired is repaired using the mask image of the target to generate a repaired image; The processing unit is specifically used for: When the image to be repaired is a dynamic image, the text description and the image of the region of interest are input into a multimodal detection model. The multimodal detection model detects the target corresponding to the text description from the image of the region of interest based on the text description, and obtains the bounding box of the target. The image of the region of interest is a local image determined from the image to be repaired, and the local image contains the target. The image of the bounding box is cropped from the image of the region of interest. The text description and the image of the bounding box are input into a multimodal segmentation model. The multimodal segmentation model processes the image of the bounding box based on the text description to generate a mask image of the target in the image of the bounding box. When the image to be repaired is a static image, the text description and the image of the region of interest are input into a multimodal segmentation model. The multimodal segmentation model determines the target corresponding to the text description from the image of the region of interest based on the text description, and generates a mask image of the target. The image of the region of interest is repaired using the mask image of the target to generate a repaired image of the region of interest; The repaired region of interest image is stitched together with the image to be repaired to obtain the repaired image.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions that, when executed by a processor, can implement the method as described in any one of claims 1-5.

9. A computer program product containing instructions, characterized in that, The computer program product includes program instructions that, when executed on a computer or processor, cause the computer or processor to perform the method as described in any one of claims 1 to 5.

10. A chip system, characterized in that, include: A communication interface used for inputting and / or outputting information; A processor for executing a computer-executable program, causing a device having the chip system mounted to perform the method as described in any one of claims 1 to 5.

Images