Image super-resolution processing method and device and computer readable medium

Abstract

The embodiment of the invention provides an image super-resolution processing method and device and a computer readable medium, according to the scheme, after text description is generated in a low-resolution image, information contained in the text description can be utilized to make up for the deficiency of information in the low-resolution image, complementation of text information and image information is achieved, and the image super-resolution processing efficiency is improved. In this way, the image super-division process is assisted through text description, and multi-scale image super-division processing is achieved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image super-resolution processing method, apparatus and computer-readable medium. Background Technology

[0002] Images are one of the most important information carriers in the field of information technology. With the development of information technology, people's demand for image resolution is also increasing. High resolution means a high pixel density in the image, which can provide more details, and these details are indispensable in many practical applications. Therefore, image super-resolution technology has gradually become a research hotspot in the industry.

[0003] Image super-resolution technology refers to the technique of reconstructing a higher-resolution image from a lower-resolution image. Current image super-resolution techniques rely on additional reference images; that is, they improve the resolution of the target image by extracting extra image information from the reference image. While this approach compensates for the lack of information in low-resolution target images to some extent, its reliance on additional reference images limits its implementation and makes it difficult to apply to single-image super-resolution scenarios. Summary of the Invention

[0004] One object of this application is to provide an image super-resolution processing method, apparatus, and computer-readable medium to solve the problems existing in the prior art.

[0005] To achieve the above objectives, embodiments of this application provide an image super-resolution processing method, the method comprising:

[0006] Acquire the low-resolution image to be processed;

[0007] Based on the low-resolution image, generate a text description about the low-resolution image;

[0008] Construct information to be processed, which includes text description marker characters, text description, initial image marker characters, and low-resolution image;

[0009] Text feature extraction is performed on the text modal information in the information to be processed to obtain text feature information. At the same time, image feature extraction is performed on the image modal information in the information to be processed to obtain first image feature information. The text modal information includes the text description marker characters, text description and initial image marker characters, and the image modal information includes the low-resolution image.

[0010] The text feature information and the first image feature information, which are concatenated in a preset order, are used as model input information and input into the super-resolution processing model to obtain the second image feature information output by the super-resolution processing model.

[0011] Image reconstruction is performed based on the second image feature information to obtain a high-resolution image corresponding to the second image feature information.

[0012] Further, after reconstructing the image based on the second image feature information to obtain the high-resolution image corresponding to the second image feature information, the method further includes:

[0013] The high-resolution image is used for iterative processing until the resolution of the obtained high-resolution image meets a preset requirement. The iterative processing includes:

[0014] The acquired high-resolution image and its corresponding iterative marker characters are added to the information to be processed to construct new information to be processed;

[0015] Text feature extraction is performed on the text modal information in the information to be processed to obtain text feature information. At the same time, image feature extraction is performed on the image modal information in the information to be processed to obtain first image feature information. The text modal information includes the text description marker characters, text description, initial image marker characters and iteration marker characters. The image modal information includes the low-resolution image and the high-resolution image.

[0016] The text feature information and the first image feature information, which are concatenated in a preset order, are used as model input information and input into the super-resolution processing model to obtain the second image feature information output by the super-resolution processing model.

[0017] Image reconstruction is performed based on the second image feature information to obtain a high-resolution image corresponding to the second image feature information.

[0018] Furthermore, the resolution of the first high-resolution image is twice that of the low-resolution image to be processed, and the resolution of the high-resolution image obtained in each iteration is twice that of the high-resolution image added to the information to be processed in that iteration.

[0019] Further, based on the low-resolution image, a text description about the low-resolution image is generated, including:

[0020] The low-resolution image is processed using the BLIP2 model to obtain a text description of the low-resolution image output by the BLIP2 model.

[0021] Further, text feature extraction is performed on the text modal information in the information to be processed to obtain text feature information, including:

[0022] The tokenizer is used to vectorize the text modal information in the information to be processed, and the vector information corresponding to each text modal information is obtained.

[0023] The vector information is feature-encoded using an MLP model to obtain the text feature information corresponding to each text modality.

[0024] Further, image feature extraction is performed on the image modal information in the information to be processed to obtain first image feature information, including:

[0025] The encoder using the VAE model extracts image features from the image modal information in the information to be processed, and obtains the first image feature information.

[0026] Furthermore, the super-resolution processing model employs a large language model.

[0027] Further, image reconstruction is performed based on the second image feature information to obtain a high-resolution image corresponding to the second image feature information, including:

[0028] The decoder of the VAE model is used to reconstruct the image based on the second image feature information, thereby obtaining a high-resolution image corresponding to the second image feature information.

[0029] Some embodiments of this application also provide an image super-resolution processing apparatus, wherein the apparatus includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the apparatus is triggered to execute the aforementioned image super-resolution processing method.

[0030] Other embodiments of this application also provide a computer-readable medium having computer program instructions stored thereon, which can be executed by a processor to implement the image super-resolution processing method.

[0031] Compared to existing technologies, this application provides an image super-resolution processing scheme. After acquiring a low-resolution image to be processed, the scheme generates a text description of the low-resolution image. Then, it constructs information to be processed, including text description marker characters, the text description itself, initial image marker characters, and the low-resolution image. Text feature extraction is performed on the text modal information in the information to be processed to obtain text feature information. Simultaneously, image feature extraction is performed on the image modal information in the information to be processed to obtain first image feature information. The text feature information and the first image feature information, concatenated in a preset order, are used as input to a super-resolution processing model to obtain second image feature information output by the super-resolution processing model. Image reconstruction is then performed based on the second image feature information to obtain a high-resolution image corresponding to the second image feature information. This scheme, after generating a text description from the low-resolution image, can utilize the information contained in the text description to compensate for the deficiencies in the low-resolution image, achieving complementarity between text and image information. Thus, the text description assists the image super-resolution process, realizing multi-scale image super-resolution processing. Attached Figure Description

[0032] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0033] Figure 1 A flowchart of image super-resolution processing provided in this application embodiment;

[0034] Figure 2 This is a flowchart illustrating the iterative processing in the embodiments of this application;

[0035] Figure 3 A flowchart illustrating the method for improving target image resolution using embodiments of this application;

[0036] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0037] The present application will now be described in further detail with reference to the accompanying drawings.

[0038] In a typical configuration of this application, the terminal and the service network devices each include one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0039] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0040] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer program instructions, data structures, program devices, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only optical disc (CD-ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0041] This application provides an image super-resolution processing method. After acquiring a low-resolution image to be processed, the method generates a text description of the low-resolution image. Then, it constructs processing information including text description marker characters, the text description itself, initial image marker characters, and the low-resolution image. Text feature extraction is performed on the text modal information in the processing information to obtain text feature information. Simultaneously, image feature extraction is performed on the image modal information in the processing information to obtain first image feature information. The text feature information and the first image feature information, concatenated in a preset order, are used as model input information and input to a super-resolution processing model to obtain second image feature information output by the super-resolution processing model. Image reconstruction can then be performed based on the second image feature information to obtain a high-resolution image corresponding to the second image feature information. This scheme, after generating a text description from the low-resolution image, can utilize the information contained in the text description to compensate for the lack of information in the low-resolution image, achieving complementarity between text and image information. Thus, the text description assists the image super-resolution process, realizing multi-scale image super-resolution processing.

[0042] In practical scenarios, the execution subject of this method can be a user device, a network device, or a device composed of user devices and network devices integrated through a network, or it can be an application running on the aforementioned devices. The user device includes, but is not limited to, various terminal devices such as computers, mobile phones, and tablets; the network device includes, but is not limited to, network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets. Here, the cloud consists of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, consisting of a virtual computer composed of a group of loosely coupled computer sets.

[0043] Figure 1 The present application illustrates a processing flow of an image super-resolution processing method according to an embodiment of the present application. The method includes at least the following processing steps:

[0044] Step S101: Obtain the low-resolution image to be processed. Here, the low-resolution image referred to in this solution refers to any image whose resolution needs to be improved using the solution provided in this application embodiment, and is not a limitation on the specific numerical value of the image resolution. For example, in this embodiment, the low-resolution image to be processed can be an image with a resolution of a×b, where a and b can be any positive integers.

[0045] Step S102: Generate a text description of the low-resolution image based on the low-resolution image.

[0046] The text description refers to descriptive text about the content contained in the image. In this embodiment, when generating the text description about the low-resolution image, BLIP2 (Bootstrapping Language-ImagePre-training 2, a second-generation model for pre-training a bootstrapping language and image) can be used. That is, the BLIP2 model is used to process the low-resolution image to obtain the text description about the low-resolution image output by the BLIP2 model.

[0047] Because BLIP2 uses large-scale image and text data for pre-training, it learns rich visual and linguistic feature representations, enabling it to capture the semantic relationships between images and text, thereby generating more accurate and meaningful image and text descriptions. Furthermore, BLIP2 can learn and infer with little or no labeled data, and by utilizing the knowledge learned from pre-training for transfer and generalization, it can generate high-quality text descriptions, thus effectively improving the performance of this solution.

[0048] Step S103: Construct the information to be processed. The information to be processed includes text description marker characters, text description, initial image marker characters, and a low-resolution image.

[0049] Both the text description marker character and the initial image marker character can be pre-defined specific characters. For example, in this embodiment, the text description marker character can be set to a character. <cap>This is used to mark the location of the text description in the information to be processed. The initial image marker character can be set to a character. <low>It is used to mark the location of low-resolution images in the information to be processed.

[0050] Step S104: Extract text features from the text modal information in the information to be processed to obtain text feature information; simultaneously extract image features from the image modal information in the information to be processed to obtain first image feature information.

[0051] The text modal information in the information to be processed includes at least the marker characters and the text description. Since the marker characters include the text description marker characters and the initial image marker characters, text feature extraction can be performed on the text modal information including the text description marker characters, the text description, and the initial image marker characters in this step. Specifically, when performing text feature extraction, a tokenizer can be used to vectorize the text modal information in the information to be processed to obtain the vector information corresponding to each text modal information. Then, the vector information is feature-encoded using an MLP (Multi-Layer Perceptron) model to obtain the text feature information corresponding to each text modal information.

[0052] The image modal information in the information to be processed includes at least the image information. Since the image information at this time only includes the low-resolution image, image feature extraction can be performed on it to obtain the corresponding first image feature information. Specifically, when performing image feature extraction, the encoder of the VAE (Variational Auto Encoder) model can be used to extract image features from the image modal information in the information to be processed to obtain the first image feature information. The VAE model is a generative model that combines the ideas of autoencoders and probabilistic graphical models, aiming to generate new data samples by learning the latent distribution of data. The VAE model encodes the input data into parameters of the latent distribution through the encoder, and samples from the distribution to generate new data. This generation method can not only generate new data, but also perform interpolation and operations in the latent space, providing powerful feature learning and data generation capabilities. Therefore, it can effectively improve the performance of this scheme and achieve faster and more efficient processing.

[0053] Step S105: The text feature information and the first image feature information, which are spliced ​​together in a preset order, are used as model input information and input into the super-resolution processing model to obtain the second image feature information output by the super-resolution processing model.

[0054] The preset order can be pre-set according to the needs of the actual scenario, and the same order used when training the super-resolution processing model can be adopted. For example, the following preset order is used in this embodiment: <cap>Text description <cap> <low>low resolution images <low>That is, the feature information in the model input information is as follows: text description, marker characters. <cap>Text feature information, text description text feature information, text description marker characters <cap>Text feature information, initial image marker characters <low>Textual feature information, image feature information of low-resolution images, and initial image marker characters <low>Textual feature information.

[0055] After concatenating the aforementioned feature information in the preset order, the model input information can be obtained. This model input information can then be input into a super-resolution processing model to obtain the second image feature information output by the super-resolution processing model. The super-resolution processing model utilizes the features described in the text as prior knowledge to supplement and enhance the image features in the low-resolution image. This allows for the combination of text modal information, enabling the feature information of the high-resolution image to be better used to reconstruct the high-resolution image. In some embodiments of this application, the super-resolution processing model can employ an LLM (Large Language Model), thereby enabling fast and efficient image feature enhancement to obtain second image feature information containing more information.

[0056] Step S106: Perform image reconstruction based on the second image feature information to obtain a high-resolution image corresponding to the second image feature information.

[0057] The high-resolution image refers to an output image with a higher resolution than the aforementioned low-resolution image to be processed. For example, the resolution of the high-resolution image can be N times that of the low-resolution image. Taking the aforementioned scenario as an example, if the low-resolution image to be processed in this embodiment has a resolution of a×b, then the high-resolution image can be an image with a resolution of N·a×N·b.

[0058] Since the second image feature information is generated by fusing the information contained in the text description of the low-resolution image with the image information of the low-resolution image, the information contained in the text description is used to make up for the lack of information in the low-resolution image, so as to achieve complementarity between text information and image information. Thus, the text description assists the image super-resolution process, realizing multi-scale image super-resolution processing, thereby enabling the construction of high-resolution images more accurately and improving the resolution of low-resolution images.

[0059] In some embodiments of this application, the present solution can use the decoder of a VAE model to reconstruct the image from the second image feature information, thereby obtaining a high-resolution image corresponding to the second image feature information. The VAE model used in this step can be the same VAE model used in the preceding steps for image feature extraction, thus enabling better image reconstruction based on the second image feature information and obtaining a high-resolution image that meets the requirements.

[0060] In real-world scenarios, when the required resolution multiplier is high, such as increasing the resolution of a low-resolution image by 8 or 16 times, directly generating a high-resolution image of that multiplier may result in severe loss of detail due to a lack of transitional image detail information, leading to poor image quality that does not meet image quality requirements.

[0061] To avoid the aforementioned problems, the solution in this application's embodiments employs an iterative approach when performing high-magnification image super-resolution processing, rather than directly generating a high-resolution image of the target magnification based on a low-resolution image. Specifically, through... Figure 1 After the scheme completes the initial resolution improvement and obtains the high-resolution image corresponding to the second image feature information, it can also use the high-resolution image for iterative processing until the resolution of the obtained high-resolution image meets the preset requirements.

[0062] The specific iterative processing procedure is as follows: Figure 2 The processing steps shown are as follows:

[0063] Step S201: The acquired high-resolution image and its corresponding iteration marker character are added to the information to be processed to construct new information to be processed. The iteration marker character can also be a pre-defined specific character; for example, in this embodiment, it can be set to a character... <m>Where M is the resolution multiple by which the high-resolution image obtained in this iteration is improved compared to the initial low-resolution image. Taking the scenario in this embodiment as an example, if the resolution of the first high-resolution image is set to be twice that of the low-resolution image to be processed, and the resolution of the high-resolution image obtained in each iteration is twice that of the high-resolution image added to the information to be processed in that iteration, then the iteration marker character in the first iteration is: <2> The iteration marker characters in subsequent iterations are as follows: <4> , <8> , <16> ...

[0064] After the acquired high-resolution image and its corresponding iteration marker characters are added to the information to be processed, the newly constructed information to be processed will include the following: text description marker characters, text description, initial image marker characters, low-resolution image, iteration marker characters, and high-resolution image.

[0065] In subsequent iterative processing, new iteration marker characters and newly acquired high-resolution images are continuously added to the information to be processed. Therefore, the newly constructed information to be processed will include multiple sets of iteration marker characters and high-resolution images. That is, in addition to the iteration marker characters and high-resolution images corresponding to this iteration, it will also include the iteration marker characters and high-resolution images from previous iterations.

[0066] Step S202: Extract text features from the text modal information in the information to be processed to obtain text feature information; simultaneously extract image features from the image modal information in the information to be processed to obtain first image feature information.

[0067] Because new content is added to the information to be processed during the iterative processing, the text modal information will include more content than during the first processing. Specifically, it may include the text description marker characters, text description, initial image marker characters, and iteration marker characters, while the image modal information will also include the low-resolution image and the high-resolution image.

[0068] When extracting textual and image feature information, a similar approach to the aforementioned scheme can be adopted. A tokenizer is used to vectorize the textual modal information in the information to be processed, obtaining the vector information corresponding to each textual modality. Then, the vector information is feature-encoded using an MLP model to obtain the textual feature information corresponding to each textual modality. Simultaneously, the encoder of a VAE model is used to extract image features from the image modality information in the information to be processed, obtaining the first image feature information.

[0069] Step S203: The text feature information and the first image feature information, which are spliced ​​together in a preset order, are used as model input information and input into the super-resolution processing model to obtain the second image feature information output by the super-resolution processing model.

[0070] Since feature information about the iteration marker characters is added to the text feature information during the iterative processing, and feature information about the high-resolution image is added to the first image feature information, the preset order can be set as follows: <cap>Text description <cap> <low>low resolution images <low> <m>High-resolution image... That is, the feature information in the model input information is, in order: text description, marker characters. <cap>Text feature information, text description text feature information, text description marker characters <cap>Text feature information, initial image marker characters <low>Textual feature information, image feature information of low-resolution images, and initial image marker characters <low>Text feature information, iterative marker characters <m>Textual feature information, image feature information of high-resolution images, etc.

[0071] During each iteration, a new iteration marker character is appended to the end of the previous second feature information. <m>The text feature information and the image feature information of the high-resolution image are sufficient. For example, the preset order of the model input information in a subsequent iteration could be: <cap>Text description <cap> <low>low resolution images <low><2> 2x higher resolution images <4> 4 times higher resolution images <8> 8 times higher resolution image.

[0072] Step S204: Reconstruct the image based on the second image feature information to obtain a high-resolution image corresponding to the second image feature information. During iterative processing, the multiplier of the high-resolution image obtained in each processing step can be set to a lower multiplier than the previous processing result, such as 2 times. This allows for high-resolution reconstruction based on the feature information from all previously obtained relatively low-resolution images, thereby improving the texture details of the image, avoiding the problem of significant loss of image details due to lack of transition information, and ensuring that the quality of the processed image meets the requirements.

[0073] Figure 3 The present invention illustrates the process of performing super-resolution processing on a target image to obtain an image with 8 times the resolution using the scheme provided in the embodiments of this application. The entire process is implemented iteratively, increasing the resolution by a factor of 2 each time. The specific process is as follows:

[0074] 1. Obtain an image at 2x resolution.

[0075] (1) Input the target image and obtain the text description of the target image through the BLIP2 model.

[0076] (2) Define special characters and combine them with text descriptions and target images to construct the information to be processed. These special characters include text description marker characters. <cap>and initial image marker characters <low>The text description marker character is used to mark the text description, and the initial image marker character is used to mark the input target image.

[0077] (3) Extract features from the information to be processed. The text modal information is obtained by using tokenizer+MLP to obtain text feature information, and the image modal information is obtained by using VAE encoder to obtain the first image feature information.

[0078] (4) All the feature information obtained in (3) are spliced ​​together in the order shown in the figure and used as input to LLM. The feature vectors at the corresponding positions in the output result obtained after LLM processing are used as the second image feature information.

[0079] (5) Use VAE decoder to reconstruct the image based on the feature information of the second image to generate a high-resolution image. The high-resolution image at this time is a 2x resolution image.

[0080] 2. Obtain an image with 4x resolution.

[0081] (1) Combine the 2x resolution image with the corresponding iterative marker characters. <2> This information is appended to the original information to be processed, constructing new information to be processed. Feature extraction is performed on the new content in the information to be processed; that is, the 2x resolution image is used as new image modality information, and the corresponding image feature information is obtained using a VAE encoder, while iteratively marking characters... <2> Text feature information is obtained by using tokenizer+MLP as new text modal information.

[0082] (2) The corresponding feature information extracted from all text modal information and image modal information is concatenated in a preset order and used as the model input information for LLM. At this time, the model input information includes text description marker characters. <cap>Text feature information, text feature information of text description, initial image marker characters <low>Text feature information, image feature information of the target image, iterative marker characters <2> The textual feature information and the image feature information of the 2x resolution image are obtained. After LLM processing, the feature vectors at the corresponding positions in the output result are obtained, which can be used as the second image feature information for this iteration.

[0083] (3) Use VAE decoder to reconstruct the image from the feature information of the second image and generate a high-resolution image. The high-resolution image at this time is a 4x resolution image.

[0084] 3. Obtain an image with 8x resolution.

[0085] (1) Combine the 4x resolution image with the corresponding iterative marker characters. <4> This information is appended to the original information to be processed, constructing new information to be processed. Feature extraction is performed on the new content in the information to be processed; that is, the 4x resolution image is used as new image modality information, and the corresponding image feature information is obtained using a VAE encoder, while iteratively marking characters... <4> Text feature information is obtained by using tokenizer+MLP as new text modal information.

[0086] (2) The corresponding feature information extracted from all text modal information and image modal information is concatenated in a preset order and used as the input of LLM. The model input information at this time includes text description marker characters. <cap>Text feature information, text feature information of text description, initial image marker characters <low>Text feature information, image feature information of the target image, iterative marker characters <2> Textual feature information, image feature information of 2x resolution images, iterative marker characters <4> The textual feature information and the image feature information of the 4x resolution image are processed by LLM. The feature vectors at the corresponding positions in the output result are obtained, which can be used as the second image feature information for this iteration.

[0087] (3) Use VAE decoder to reconstruct the image based on the feature information of the second image to generate a high-resolution image. The high-resolution image at this time is an 8x resolution image.

[0088] Therefore, an 8x resolution image can be obtained through the above three rounds of processing. In practical scenarios, if a higher resolution image is required, more rounds of iterative processing can be used. Compared to directly obtaining a high-resolution image, the iterative approach can improve the texture details of the high-resolution image, avoid the problem of significant loss of image details, and ensure that the processed image quality meets the requirements.

[0089] Based on another aspect of this application, embodiments of this application also provide an image super-resolution processing apparatus, the apparatus including a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the apparatus is triggered to execute the aforementioned image super-resolution processing method.

[0090] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by a processing unit, it performs the functions defined in the methods of this application.

[0091] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0092] In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0093] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0094] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0095] In another aspect, this application also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The aforementioned computer-readable medium carries one or more computer program instructions, which may be executed by a processor to implement the methods and / or technical solutions of the various embodiments of this application.

[0096] It should be noted that this application can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0097] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device in software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any particular order. Numbers corresponding to steps are used to label and distinguish different steps, and the magnitude of these numbers does not limit any particular order of execution.< / low> < / cap> < / low> < / cap> < / low> < / cap> < / low> < / low> < / cap> < / cap> < / m> < / m> < / low> < / low> < / cap> < / cap> < / m> < / low> < / low> < / cap> < / cap> < / m> < / low> < / low> < / cap> < / cap> < / low> < / low> < / cap> < / cap> < / low> < / cap>

Claims

1. An image super-resolution processing method, characterized in that, The method includes: Acquire the low-resolution image to be processed; Based on the low-resolution image, generate a text description about the low-resolution image; Construct information to be processed, which includes text description marker characters, text description, initial image marker characters, and low-resolution image; Text feature extraction is performed on the text modal information in the information to be processed to obtain text feature information. At the same time, image feature extraction is performed on the image modal information in the information to be processed to obtain first image feature information. The text modal information includes the text description marker characters, text description and initial image marker characters, and the image modal information includes the low-resolution image. The text feature information and the first image feature information, which are concatenated in a preset order, are used as model input information and input into the super-resolution processing model to obtain the second image feature information output by the super-resolution processing model. Image reconstruction is performed based on the second image feature information to obtain a high-resolution image corresponding to the second image feature information.

2. The method according to claim 1, characterized in that, After reconstructing the image based on the second image feature information and obtaining the high-resolution image corresponding to the second image feature information, the method further includes: The high-resolution image is used for iterative processing until the resolution of the obtained high-resolution image meets a preset requirement. The iterative processing includes: The acquired high-resolution image and its corresponding iterative marker characters are added to the information to be processed to construct new information to be processed; Text feature extraction is performed on the text modal information in the information to be processed to obtain text feature information. At the same time, image feature extraction is performed on the image modal information in the information to be processed to obtain first image feature information. The text modal information includes the text description marker characters, text description, initial image marker characters and iteration marker characters. The image modal information includes the low-resolution image and the high-resolution image. The text feature information and the first image feature information, which are concatenated in a preset order, are used as model input information and input into the super-resolution processing model to obtain the second image feature information output by the super-resolution processing model. Image reconstruction is performed based on the second image feature information to obtain a high-resolution image corresponding to the second image feature information.

3. The method according to claim 2, characterized in that, The resolution of the first high-resolution image is twice that of the low-resolution image to be processed, and the resolution of the high-resolution image obtained in each iteration is twice that of the high-resolution image added to the information to be processed in that iteration.

4. The method according to claim 1 or 2, characterized in that, Based on the low-resolution image, generate a text description about the low-resolution image, including: The low-resolution image is processed using the BLIP2 model to obtain a text description of the low-resolution image output by the BLIP2 model.

5. The method according to claim 1 or 2, characterized in that, Text feature extraction is performed on the text modal information in the information to be processed to obtain text feature information, including: The tokenizer is used to vectorize the text modal information in the information to be processed, and the vector information corresponding to each text modal information is obtained. The vector information is feature-encoded using an MLP model to obtain the text feature information corresponding to each text modality.

6. The method according to claim 1 or 2, characterized in that, Image feature extraction is performed on the image modal information in the information to be processed to obtain first image feature information, including: The encoder using the VAE model extracts image features from the image modal information in the information to be processed, and obtains the first image feature information.

7. The method according to claim 1 or 2, characterized in that, The super-resolution processing model adopts a large language model.

8. The method according to claim 1 or 2, characterized in that, Image reconstruction is performed based on the second image feature information to obtain a high-resolution image corresponding to the second image feature information, including: The decoder of the VAE model is used to reconstruct the image based on the second image feature information, thereby obtaining a high-resolution image corresponding to the second image feature information.

9. An image super-resolution processing device, wherein, The device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to perform the method of any one of claims 1 to 8.

10. A computer-readable medium having stored thereon computer program instructions that can be executed by a processor to implement the method as described in any one of claims 1 to 8.

Images