Image super-resolution method and image data processing method
By using subimage features of the same sample image to train the image super-scoring model, the problem of poor model effectiveness caused by insufficient sample size is solved, and the performance and detail generation ability of image super-scoring are improved.
Patent Information
- Application Number
- PCT/SG2024/050337
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-22
- Filing Date
- 2024-05-21
- Publication Date
- 2024-11-28
AI Technical Summary
In the prior art, when training an image super-score model based on text information corresponding to the sample image, insufficient sample number leads to poor model training effect, thereby reducing the super-score effect of image super-score.
By using sub-images of the same sample image, training of the image super-segment model is performed based on the image semantic features of the first sample sub-image and the image features of the second sample sub-image, avoiding the guidance of text information of the sample image and improving model performance and detail generation capability.
The performance and super-scoring effect of the image super-scoring model are improved, the poor training effect caused by insufficient sample size is avoided, and the ability to generate image details is improved.
Smart Images

Figure SG2024050337_28112024_PF_FP_ABST
Abstract
Description
[0001] Image Super-Resolution Method and Image Data Processing Method Cross-Reference This disclosure claims priority to Chinese patent application No. 202310591727.0, filed with the China Patent Office on May 22, 2023, entitled "Image Super-Resolution Method and Image Data Processing Method," the entire contents of which are incorporated herein by reference. Technical Field The present disclosure relates to the field of computer technology, and more particularly to an image super-resolution method. Background Art With the advancement of computer technology, trained image super-resolution models, configured to improve image resolution, have become increasingly widely used in the field of image processing. Currently, training image super-resolution models based on text information corresponding to sample images effectively improves their performance, enhances their ability to generate image details, and enhances the image super-resolution effect, significantly enhancing the image super-resolution's ability to generate details. However, the number of sample images with corresponding text information is insufficient. When training image super-resolution models on a large scale, this insufficient number of samples leads to insufficient model training effectiveness, which in turn reduces the image super-resolution effect. Therefore, there is an urgent need for an image super-resolution method that can improve super-resolution effects. In view of this, embodiments of the present disclosure provide an image super-resolution method. One or more embodiments of the present disclosure also relate to another image super-resolution method, an image data processing method, an image super-resolution apparatus, another image super-resolution apparatus, an image data apparatus, an augmented reality (AR) device or a virtual reality (VR) device, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in the prior art. According to a first aspect of an embodiment of the present disclosure, an image super-resolution method is provided, comprising: obtaining a reference image and an initial image of an initial resolution; inputting the reference image and the initial image into an image super-resolution model to obtain a target image of a target resolution, wherein the target resolution is higher than the initial resolution; the image super-resolution model is trained based on a first sample sub-image and a second sample sub-image, wherein the first sample sub-image and the second sample sub-image are sub-images of the same sample image; and image semantic features of the first sample sub-image and image features of the second sample sub-image are used to train the image super-resolution model.According to a second aspect of an embodiment of the present disclosure, an image super-resolution method is provided, applied to a cloud-side device, and includes: receiving an initial image of an initial resolution sent by a client-side device; obtaining a reference image corresponding to the initial image; inputting the reference image and the initial image into an image super-resolution model to obtain a target image of a target resolution, wherein the target resolution is higher than the initial resolution, the image super-resolution model is trained based on a first sample sub-image and a second sample sub-image, the first sample sub-image and the second sample sub-image are sub-images of the same sample image, and semantic features of the first sample sub-image and image features of the second sample sub-image are used to train the image super-resolution model; and sending the target image to the client-side device. According to a third aspect of an embodiment of the present disclosure, an image data processing method is provided, which is applied to a Yunjia device, including: obtaining a sample image, splitting the sample image to obtain a first sample sub-image and a second sample sub-image; reducing the image resolution of the second sample sub-image to obtain a third sample sub-image; inputting the first sample sub-image and the third sample sub-image into an image super-resolution model to obtain a predicted sub-image; calculating a loss value based on the predicted sub-image and the second sample sub-image; adjusting parameters of the image super-resolution model based on the loss value, and obtaining a trained image super-resolution model when a preset training end condition is met; and sending the model parameters of the image super-resolution model to an end-side device. According to a fourth aspect of an embodiment of the present disclosure, an image super-resolution apparatus is provided, comprising: a first acquisition module configured to acquire a reference image and an initial image of an initial resolution; and a first super-resolution module configured to input the reference image and the initial image into an image super-resolution model to obtain a target image of a target resolution, wherein the target resolution is higher than the initial resolution, the image super-resolution model is trained based on a first sample sub-image and a second sample sub-image, the first sample sub-image and the second sample sub-image being sub-images of the same sample image, and image semantic features of the first sample sub-image and image features of the second sample sub-image being used for training the image super-resolution model.According to a fifth aspect of an embodiment of the present disclosure, an image super-resolution apparatus is provided, applied to a cloud-side device, and includes: a receiving module, configured to receive an initial image of an initial resolution sent by an end-side device; a second acquisition module, configured to acquire a reference image corresponding to the initial image; a second super-resolution module, configured to input the reference image and the initial image into an image super-resolution model to obtain a target image of a target resolution, wherein the target resolution is higher than the initial resolution, the image super-resolution model is trained based on a first sample sub-image and a second sample sub-image, the first sample sub-image and the second sample sub-image are sub-images of the same sample image, and semantic features of the first sample sub-image and image features of the second sample sub-image are used to train the image super-resolution model; and a first sending module, configured to send the target image to the end-side device. According to a sixth aspect of an embodiment of the present disclosure, there is provided an image data processing apparatus, which is applied to a cloud-side device and includes: a third acquisition module, configured to acquire a sample image, split the sample image, and obtain a first sample sub-image and a second sample sub-image; a reduction module, configured to lower the image resolution of the second sample sub-image to obtain a third sample sub-image; a prediction module, configured to input the first sample sub-image and the third sample sub-image into an image super-resolution model to obtain a predicted sub-image; a calculation module, configured to calculate a loss value based on the predicted sub-image and the second sample sub-image; an adjustment module, configured to adjust parameters of the image super-resolution model based on the loss value, and obtain a trained image super-resolution model when a preset training end condition is met; and a second sending module, configured to send model parameters of the image super-resolution model to the end-side device. According to a seventh aspect of the embodiments of the present disclosure, an augmented reality (AR) device or a virtual reality (VR) device is provided, comprising: a memory, a processor, and a display; the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement the following steps: obtaining a reference image and an initial image of an initial resolution; inputting the reference image and the initial image into an image super-resolution model to obtain a target image of a target resolution, wherein the target resolution is higher than the initial resolution, the image super-resolution model is trained based on a first sample sub-image and a second sample sub-image, the first sample sub-image and the second sample sub-image are sub-images of the same sample image, and image semantic features of the first sample sub-image and image features of the second sample sub-image are used for training the image super-resolution model; and displaying the target image through the display of the augmented reality (AR) device or the virtual reality (VR) device.According to an eighth aspect of the embodiments of the present disclosure, a computing device is provided, comprising: a memory and a processor; the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement the steps of the above-mentioned image super-resolution method or image data processing method. According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, the medium storing computer-executable instructions, wherein the instructions, when executed by the processor, implement the steps of the above-mentioned image super-resolution method or image data processing method. According to a fifth aspect of the embodiments of the present disclosure, a computer program is provided, wherein, when executed in a computer, the computer is caused to execute the steps of the above-mentioned image super-resolution method or image data processing method. In one or more embodiments of the present disclosure, a reference image and an initial image of an initial resolution are obtained, and the reference image and the initial image are input into an image super-resolution model to obtain a target image of a target resolution, wherein the target resolution is higher than the initial resolution. The image super-resolution model is trained based on a first sample sub-image and a second sample sub-image, wherein the first sample sub-image and the second sample sub-image are sub-images of the same sample image, and image semantic features of the first sample sub-image and image features of the second sample sub-image are used to train the image super-resolution model. By training the image super-resolution model based on sub-images of the same sample image, that is, based on the image semantic features of the first sample sub-image and the image features of the second sample sub-image, the image super-resolution model is trained, thereby avoiding the need to guide the training of the image super-resolution model based on text information corresponding to the sample images, avoiding insufficient model training results caused by insufficient sample number, improving the model performance of the image super-resolution model, improving the ability to generate image details, and improving the super-resolution effect of the image super-resolution.BRIEF DESCRIPTION OF THE DRAWINGS FIG1 is a flowchart of an image super-resolution method provided by one embodiment of the present disclosure; FIG2 is a flowchart of another image super-resolution method provided by one embodiment of the present disclosure; FIG3 is a flowchart of an image data processing method provided by one embodiment of the present disclosure; FIG4 is a schematic diagram of model training of an image super-resolution model in an image super-resolution method provided by one embodiment of the present disclosure; FIG5 is a schematic diagram of the front-end of an image super-resolution method provided by one embodiment of the present disclosure; FIG6 is a flowchart of the processing process of an image super-resolution method configured for portrait super-resolution provided by one embodiment of the present disclosure; FIG7 is a schematic diagram of the structure of an image super-resolution device provided by one embodiment of the present disclosure; FIG8 is a schematic diagram of the structure of another image super-resolution device provided by one embodiment of the present disclosure; FIG9 is a schematic diagram of the structure of an image data processing device provided by one embodiment of the present disclosure; FIG10 is a block diagram of the structure of an augmented reality (AR) device or virtual reality (VR) device provided by one embodiment of the present disclosure; and FIG11 is a block diagram of the structure of a computing device provided by one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS Exemplary embodiments will be described in detail herein, with examples thereof illustrated in the accompanying drawings. In the following description, when referring to the drawings, unless otherwise indicated, the same numerals in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims. Numerous specific details are set forth in the following description to provide a thorough understanding of the present disclosure. However, the present disclosure can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the specific embodiments disclosed below. The terms used in one or more embodiments of the present disclosure are for the purpose of describing specific embodiments and are not intended to be limiting of the present disclosure. As used in one or more embodiments of the present disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the present disclosure refers to and encompasses any and all possible combinations of one or more of the associated listed items. It should be understood that although the terms "first," "second," and so on may be used in one or more embodiments of the present disclosure to describe various information, such information should not be limited to these terms. These terms are used to distinguish information of the same type from one another. For example, without departing from the scope of one or more embodiments of the present disclosure, the first may be referred to as the second, and similarly, the second may be referred to as the first.Depending on the context, the term "if" as used herein can be interpreted as "at the time," "when," or "in response to a determination." Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data, etc.) involved in one or more embodiments of this disclosure are all authorized by the user or fully authorized by all parties. The collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or deny. First, the terms involved in one or more embodiments of this disclosure are explained. Image resolution: This refers to the amount of information stored in an image, specifically the number of pixels per inch. The unit of resolution is PPI (Pixels Per Inch), commonly referred to as pixels per inch. It can also be expressed as the number of pixels in a single direction multiplied by (x). For example, 1960 x 1080 represents 1960 pixels horizontally and 1080 pixels vertically. Super-resolution: A super-resolution algorithm is designed to increase the resolution of a visual image. Essentially, it treats the input image as a noise-added image and denoises it to produce a denoised image, thereby increasing image resolution. Semantic information: The correlation between pixels in an image is used to represent the image's category. It is abstract information contained in the image content. For example, the correlation between pixels in an animal image: stripes, yellow, and black. This can represent the animal's category: tiger. Feature encoding of semantic information involves encoding abstract information into semantic features, resulting in a high-dimensional (abstract) feature encoding vector. Image information refers to pixel information within an image, including specific information such as pixel color and position. Image feature encoding of this information results in a low-dimensional (specific) feature encoding vector. Feature inference refers to the process of performing fine-grained feature transformations on input features under the guidance of guiding features (prior features) to obtain output features. For example, if the input features are 720 x 480 image features, then feature transformations of each dimension are performed on the input features under the guidance of guiding features to obtain 720 x 480 image features as output features.
[0002] CLIP model (Contrastive Language-Image Pre-training): A neural network model pre-trained using matched sample images and text to form sample pairs. This model extracts semantic features based on the correlation between image and text features, and has excellent semantic feature representation capabilities. It includes an image semantic extraction layer and a text semantic extraction layer.
[0003] CNN (Convolutional Neural Networks) model: A multi-layer neural network model with forward propagation and backpropagation.
[0004] FCN model (Fully Convolution Network, fully convolutional neural network model): a neural network model with a single forward propagation mechanism and a fully connected convolutional layer.
[0005] VGG model (Visual Geometry Group Network): A neural network model with small convolutional layers, small pooling layers, deeper layers and wider feature maps.
[0006] VIT model (Vision Transformer, visual translation model): A translation model for visual image processing with a visual attention mechanism.
[0007] ResNet model: A neural network model with a multi-layer network structure and a residual processing module. Examples include ResNet-50 and ResNet-101.
[0008] Transformer (translation) model: A neural network model based on the attention mechanism that extracts and analyzes data features. It can be used for natural language data as well as image data. One or more embodiments of the present disclosure are used for feature extraction and analysis of image data.
[0009] BERT (Bidirectional Encoder Representation from Transformers) model: A neural network model that uses bidirectional attention to encode representations. U-Net model (U-shaped neural network model): A U-shaped neural network model (with decreasing encoding layer size and increasing decoding layer size) that uses a splicing approach to fuse shallow and deep features.
[0010] VAE (Variational Auto-Encoder) model: A probabilistic model based on variational inference (Variational Inference, Variational Bayesian Methods), a neural network model with image generation capabilities.
[0011] GAN (Generative Adversarial Network) model: A neural network model configured for image generation, consisting of a generator and a discriminator. By training the generator and discriminator alternately, a highly accurate generator is obtained to generate images.
[0012] Diffusion model: An image generation model that utilizes a forward process (noise addition) and a backward process (noise removal). Augmented Reality (AR) technology: An image processing technology that integrates virtual visual images with the physical world. It extensively utilizes multiple technologies, including multimedia, 3D modeling, real-time tracking and registration, intelligent interaction, and sensing. The generated virtual visual images are simulated and then applied to the physical world, allowing the virtual visual images and the physical world to complement each other, thereby enhancing the sensory experience of the physical world. This is specifically implemented through AR devices, such as AR environment devices and AR wearable devices. Virtual Reality (VR): Also known as virtual reality or virtual reality technology, it is an image processing technology that replaces the physical world with virtual information. It extensively utilizes multiple technologies, including multimedia, 3D modeling, real-time tracking and registration, intelligent interaction, and sensing. The generated virtual visual images are simulated and then replaced with the physical world, giving users the feeling of being in a virtual world. This is specifically implemented through VR devices, such as VR environment devices and VR wearable devices. This disclosure provides an image super-resolution method. This disclosure also relates to another image super-resolution method, an image data processing method, an image super-resolution device, another image super-resolution device, an image data device, an augmented reality (AR) device or a virtual reality (VR) device, a computing device, a computer-readable storage medium, and a computer program, each of which is described in detail in the following embodiments. Referring to Figure 1, a flowchart of an image super-resolution method provided by one embodiment of the present disclosure is shown, comprising the following specific steps: Step 102: Obtain a reference image and an initial image at an initial resolution. This embodiment of the present disclosure is applicable to applications, web pages, or mini-programs with image super-resolution capabilities. The method can be deployed on a cloud-side device or on a device (client or server). The reference image is a visual image used to guide image super-resolution. It contains semantic information corresponding to the initial image. The reference image can be high-resolution or low-resolution; it only needs to contain the corresponding semantic information. Therefore, for a trained image super-resolution model, the reference image can be a solid color image. The initial image is the visual image to be super-resolved. The image resolution of the initial image is the initial resolution. The initial resolution is the resolution of the image to be improved, which is lower than the target resolution after image super-resolution. The semantic information contained in the initial image and the reference image corresponds to each other.For example, the initial image is a first photo of person A, the reference image is a second photo of person A, the second photo contains semantic information corresponding to the first photo, and the image resolution of the first photo is 720 x 480. Acquiring the reference image and the initial image at the initial resolution can be done by directly acquiring the reference image and the initial image, for example, by receiving the reference image and the initial image uploaded by a user, or by acquiring the initial image and the reference image corresponding to the initial image from an image database. Alternatively, acquiring the initial image and then acquiring the reference image corresponding to the initial image can be done by, for example, receiving the initial image uploaded by a user and acquiring the reference image corresponding to the initial image from an image database. Acquiring the reference image can be done by identifying the image of the initial image or by performing a feature similarity query based on the image features of the initial image. The image database can be a local image database, a remote image database, or an open source image database, without limitation. For example, a first photo of person A at the initial resolution (720 x 480) uploaded by a user is received, and a second photo of person A corresponding to the first photo is acquired from the local image database. A reference image and an initial image of initial resolution are obtained, providing a reference image basis for guiding image super-resolution and an initial image basis for image super-resolution. Step 104: Input the reference image and initial image into an image super-resolution model to obtain a target image of target resolution, where the target resolution is higher than the initial resolution. The image super-resolution model is trained based on a first sample sub-image and a second sample sub-image, which are sub-images of the same sample image. The image semantic features of the first sample sub-image and the image features of the second sample sub-image are used to train the image super-resolution model. The image super-resolution model is a neural network model with image super-resolution capabilities. It performs feature inference on image features of a low-resolution visual image to obtain a high-resolution target image. The image super-resolution model achieves image super-resolution through image feature encoding, feature inference, and image feature decoding. The image super-resolution model can be a single neural network model with image feature encoding, feature inference, and image feature decoding capabilities, or it can be a combination of multiple neural network models with image feature encoding, feature inference, and image feature decoding capabilities, without limitation herein. The image super-resolution model is trained based on a first sample sub-image and a second sample sub-image, wherein the first sample sub-image is a visual image used to guide image super-resolution during training, and the second sample sub-image is a sample image during training.Training of the image super-resolution model is a supervised training process. The first sample sub-image guides the image super-resolution during training, while the second sample sub-image serves as the input for the training process and as a label for comparison with the predicted results of subsequent super-resolution images. The first and second sample sub-images are sub-images of the same sample image. Therefore, the first sample sub-image contains semantic information corresponding to the second sample sub-image. For example, if a sample image of a person is cropped from top to bottom, the first sample sub-image is the sub-image of the upper half, and the second sample sub-image is the sub-image of the lower half. The image semantic features of the first sample sub-image serve as guiding features in a training iteration, while the image features of the second sample sub-image serve as input features in the same training iteration. The image features of the second sample sub-image can be used directly as input features or encoded after reducing the image resolution of the second sample sub-image, without limitation. The target image is the visual image of the initial image after super-resolution. The target resolution is higher than the initial resolution of the initial image, but the image content of the initial and target images must remain consistent before and after super-resolution. For example, the initial image is a first photograph of person A at a resolution of 720 x 480, and the target image is a first photograph at a resolution of 1960 x 1080. The reference image and the initial image are input into the image super-resolution model to obtain a target image at a target resolution. Specifically, the reference image and the initial image are input into the image super-resolution model, and based on the guidance of the semantic information in the reference image, the initial image is super-resolved to obtain a target image at a target resolution. Furthermore, based on the guidance of the semantic information in the reference image, the initial image is super-resolved to obtain a target image at a target resolution. Specifically, based on the image semantic features of the reference image, the initial image is super-resolved to obtain a target image at a target resolution. Furthermore, based on the image semantic features of the reference image, the initial image is super-resolved to obtain a target image at a target resolution. Specifically, based on the image semantic features of the reference image, feature inference is performed on the image features of the initial image based on the image semantic features of the reference image to obtain a target image at a target resolution. It should be noted that, guided by the semantic information contained in the reference image, high-resolution generation of detailed image content in the initial image is achieved. This achieves more accurate image super-resolution. For example, if the initial image is a landscape containing a man and a woman, and the reference image is a close-up photo of the man and woman, guided by the semantic information ("a man and a woman") contained in the reference image, the detailed image content of the man and woman in the initial image is super-resolved at high resolution, resulting in a target image at the target resolution.This method is an image super-resolution method implemented in the form of a guidance graph. Exemplarily, a first photo of person A at an initial resolution (720 x 480) and a second photo of person A are input into an image super-resolution model. Based on the semantic information "person A (male, 40 years old, dark skin...)" in person A's second photo, the first photo of person A is super-resolved to obtain the first photo of person A at a target resolution (1960 x 1080). In this disclosed embodiment, a reference image and an initial image at an initial resolution are obtained, and the reference image and initial image are input into an image super-resolution model to obtain a target image at a target resolution, where the target resolution is higher than the initial resolution. The image super-resolution model is trained based on a first sample sub-image and a second sample sub-image, which are sub-images of the same sample image. The image semantic features of the first sample sub-image and the image features of the second sample sub-image are used to train the image super-resolution model. By training the image super-resolution model based on sub-images of the same sample image, namely, the image semantic features of the first sample sub-image and the image features of the second sample sub-image, the training of the image super-resolution model is avoided by not relying on text information corresponding to the sample images to guide the training of the image super-resolution model. This also avoids insufficient model training due to insufficient sample size, improves the model performance of the image super-resolution model, enhances the ability to generate image details, and improves the image super-resolution effect. In an optional embodiment of the present disclosure, the image super-resolution model includes an image semantic extraction layer, an image encoding layer, a feature inference layer, and a decoding layer. Accordingly, step 104 includes the following specific steps: inputting a reference image into the image semantic extraction layer to obtain image semantic features, inputting an initial image into the image encoding layer to obtain initial image features; inputting the image semantic features and the initial image features into the feature inference layer, performing feature inference on the initial image features based on the image semantic features to obtain target image features; and inputting the target image features into the decoding layer to obtain a target image at a target resolution. The image super-resolution model achieves image super-resolution through image feature encoding, feature inference, and image feature decoding. The model includes an image encoding layer for image feature encoding, a feature inference layer for feature inference, and a decoding layer for image feature decoding. Furthermore, it includes an image semantic extraction layer for extracting image semantic features. The image encoding layer encodes low-dimensional image features of the input image to obtain low-dimensional image features. The image semantic extraction layer encodes high-dimensional image semantic features of the input image to obtain high-dimensional image semantic features.Guided by the input high-dimensional image semantic features, the feature inference layer performs feature inference on the input low-dimensional image features, i.e., transforms them using the feature inference layer, to obtain the corresponding low-dimensional target image features. The decoding layer decodes the low-dimensional target image features to obtain the target image at the target resolution. The image encoding and decoding layers can be those of models such as CNN, FCN, VGG, VIT, and ResNet. The image semantic extraction layer can be the image feature extraction layer of models such as CLIP, FCN, VGG, VIT, and ResNet. The feature inference layer can be the feature transformation layer of models such as U-Net, VAE, GAN, and Diffusion. Image semantic features are high-dimensional feature encoding vectors that contain the semantic information of the reference image. Initial image features are low-dimensional feature encoding vectors of the image information in the initial image. Target image features are low-dimensional feature encoding vectors derived from the semantic information inferred from the reference image. The reference image is input into the image semantic extraction layer to obtain image semantic features. Specifically, the reference image is input into the image semantic extraction layer, and semantic feature encoding is performed on the semantic information in the reference image to obtain image semantic features. The initial image is input into the image encoding layer, and image feature encoding is performed on the image information in the initial image to obtain initial image features. The target image features are input into the decoding layer to obtain a target image at a target resolution. Specifically, the target image features are input into the decoding layer, and image feature decoding is performed on the target image features to obtain a target image at a target resolution. It should be noted that the image feature inference layer performs image feature reconstruction on the initial image features using image semantic features as global feature constraints. This feature reconstruction is achieved by performing feature transformation on the initial image features using image semantic features as prior features. For example, the second photo of person A is input into the image semantic extraction layer, and semantic feature encoding is performed on the semantic information "person A (male, 40 years old, dark skin...)" in the second fat slice to obtain the image semantic feature Embedding_Image. The initial resolution is (720. xA first photo of person A (1960 x 1080) is input into an image encoding layer, image feature encoding is performed on the image information in the first photo to obtain initial image features (Feature image). The image semantic features and the initial image features are input into a feature inference layer, feature inference is performed on the initial image features based on the image semantic features to obtain target image features (Feature TargetImage). The target image features are input into a decoding layer, image feature decoding is performed on the target image features, and the first photo of person A (1960 x 1080) is obtained. In the disclosed embodiment, a reference image is input into an image semantic extraction layer to obtain image semantic features. An initial image is input into an image coding layer to obtain initial image features. The image semantic features and the initial image features are input into a feature inference layer. Feature inference is performed on the initial image features based on the image semantic features to obtain target image features. The target image features are input into a decoding layer to obtain a target image of a target resolution. Through layer-by-layer feature processing in the image semantic extraction layer, the image coding layer, the feature inference layer, and the decoding layer, highly accurate image semantic extraction, image coding, feature inference, and image decoding are achieved, further improving the ability to generate image details and the super-resolution effect of image super-resolution. In an optional embodiment of the present disclosure, the image super-resolution model further includes a text semantic extraction layer and a semantic feature fusion layer. Accordingly, inputting a reference image into the image semantic extraction layer to obtain image semantic features includes the following specific steps: obtaining reference text corresponding to the reference image; inputting the reference image into the image semantic encoding layer to obtain initial image semantic features; inputting the reference text into the text semantic extraction layer to obtain text semantic features; inputting the initial image semantic features and text semantic features into the semantic feature fusion layer to fuse the initial image semantic features and text semantic features to obtain image semantic features. The reference text is text content with semantic information corresponding to the reference image. For example, if the reference image is a close-up of a man and a woman, and the semantic information contained in the reference image is "a man and a woman," the reference text is "a man and a woman." The initial image semantic features are high-dimensional feature encoding vectors of the semantic information in the reference image. The text semantic extraction layer performs text feature encoding on the input text to obtain text semantic features. The text semantic features are feature encoding vectors of the semantic information in the text.The feature fusion layer fuses multiple input feature encoding vectors to obtain a fused feature encoding vector. This fusion process includes, but is not limited to, aligning feature dimensions (feature spaces) and then concatenating them, weighting them after alignment, and mapping multiple feature encoding vectors using a mapping matrix to obtain a fused feature encoding vector. (This is not limited here.) Image semantic features are multimodal feature encoding vectors that contain semantic information from both the reference image and the reference text. They are high-dimensional feature encoding vectors. The text semantic extraction layer can be the text feature extraction layer of models such as the CLIP model, the Transformer model, the BERT model, or the BERT model. The feature fusion layer is a model functional layer with feature fusion capabilities. It fuses multimodal feature vectors to obtain more comprehensive image semantic features for subsequent feature inference. The feature fusion layer is pre-trained. Obtaining reference text corresponding to a reference image can be done by receiving user-uploaded reference text, querying a text database to obtain reference text corresponding to the reference image, or generating reference text corresponding to the reference image using a text generation algorithm. (This is not limited here.) The reference image is input into the image semantic encoding layer to obtain initial image semantic features. Specifically, the reference image is input into the image semantic extraction layer, and image semantic features are encoded on the semantic information in the reference image to obtain initial image semantic features. Reference text is input into the text semantic extraction layer to obtain text semantic features. Specifically, the reference text is input into the text semantic extraction layer, and text semantic features are encoded on the semantic information in the reference text to obtain text semantic features. For example, the description text corresponding to the second photo is obtained from the local database: "Person A is a dark-skinned man, approximately 40 years old..." The second photo is input into the image semantic extraction layer, and image semantic features are encoded on the semantic information "Person A (male, 40 years old, dark-skinned...)" in the second photo to obtain initial image semantic features. , The description text is input into the text semantic extraction layer, and the semantic information in the description text "person A, dark skin, man, around 40 years old" is encoded into text semantic features to obtain the text semantic feature Embedding_Txto. The initial image semantic features and text semantic features are input into the semantic feature fusion layer, and the initial image semantic features and text semantic features are fused to obtain the image semantic feature Embedding_Tmage. oIn this embodiment, reference text corresponding to a reference image is obtained. The reference image is input into the image semantic encoding layer to obtain initial image semantic features. The reference text is input into the text semantic extraction layer to obtain text semantic features. The initial image semantic features and text semantic features are input into the semantic feature fusion layer, where they are fused to obtain image semantic features. This improves the versatility of image super-resolution. Furthermore, more comprehensive image semantic features are used to guide image super-resolution, further enhancing the ability to capture image details and the image super-resolution effect. In an optional embodiment of the present disclosure, an image super-resolution model is trained through the following steps: obtaining a sample image, splitting the sample image to obtain a first sample sub-image and a second sample sub-image; reducing the image resolution of the second sample sub-image to obtain a third sample sub-image; inputting the first and third sample sub-images into the image super-resolution model to obtain a predicted sub-image; calculating a loss value based on the predicted sub-image and the second sample sub-image; and adjusting parameters of the image super-resolution model based on the loss value. When a preset training termination condition is met, a trained image super-resolution model is obtained. The sample images are sample visual images used for training the image super-resolution model, and include multiple sample images. The sample images can be generated, for example, using an image generation model that generates images from text to generate multiple sample images, or they can be obtained from a sample database. The sample database can be a historical database or an open-source database, without limitation. During the training process of the image super-resolution model, to ensure that the trained image super-resolution model can super-resolve low-resolution visual images to obtain high-resolution visual images, the sample image is set to high resolution, and the second sample sub-image obtained by splitting is also set to high resolution. This second sample sub-image is used as the labeled sample image and compared with the predicted sub-image to determine the loss value. The first sample sub-image and the second sample sub-image are sub-images obtained by splitting the sample image. For example, a sample image with a resolution of 1960 x 1080 is divided equally left and right to obtain two 980 x 1080 sub-images. Since the first sample sub-image and the second sample sub-image are sub-images obtained by splitting the same sample image, they naturally contain corresponding semantic information. It should be noted that the first sample sub-image and the second sample sub-image are used for convenience of description; the sample image can be split into multiple parts to obtain multiple sub-images.During the training process of an image super-resolution model, the first sample sub-image is used to guide image super-resolution; the third sample sub-image is a low-resolution sample sub-image for image super-resolution and serves as a training sample image; the second sample sub-image is a high-resolution sample sub-image used to determine the loss value of the image super-resolution result and serves as a labeled sample image. The predicted sub-image is the predicted result obtained by super-resolution of the third sample sub-image and is a high-resolution visual image. The loss value is the image feature loss between the predicted sub-image and the second sample sub-image, representing the image loss between the predicted result and the labeled sample image, reflecting the image super-resolution performance of the image super-resolution model. The loss value can be a cross-beam loss, distance loss, cosine loss, L1 loss, or L2 loss. The preset training termination condition is a pre-set condition for determining the end of iterative training. The preset training termination condition includes, but is not limited to, a preset loss threshold, a preset number of iterations, a preset training duration, and a preset number of training samples. The first sample sub-image and the third sample sub-image are input into an image super-resolution model to obtain a predicted sub-image. Specifically, the first sample sub-image and the second sample sub-image are input into the image super-resolution model, and based on the semantic information in the first sample sub-image, image super-resolution is performed on the second sample sub-image to obtain a predicted sub-image. Furthermore, based on the semantic information in the first sample sub-image, image super-resolution is performed on the second sample sub-image to obtain a predicted sub-image. Specifically, based on the sample image semantic features of the first sample sub-image, image super-resolution is performed on the second sample sub-image to obtain a predicted sub-image. Furthermore, based on the sample image semantic features of the first sample sub-image, image super-resolution is performed on the second sample sub-image to obtain a predicted sub-image. Specifically, based on the sample image semantic features of the first sample sub-image, feature inference is performed on the sample image features of the second sample sub-image based on the sample image semantic features of the first sample sub-image to obtain a predicted sub-image. Reducing the image resolution of the second sample sub-image can be achieved using a related algorithm. For example, using an image sliding window of a predetermined size, the chrominance value of any pixel in the sliding window is determined to be an image of the chrominance values of all pixels in the sliding window. Alternatively, image processing models can be used to reduce the image resolution, which is not limited here. Based on the loss value, parameters of the image super-resolution model are adjusted. Specifically, the parameters of the image super-resolution model are adjusted based on the loss value using a gradient update method. Adjusting the image super-resolution model involves adjusting parameters of one or more of the image semantic extraction layer, the image encoding layer, the feature inference layer, and the decoding layer.It should be noted that training an image super-resolution model is a large-scale model training method, requiring a very large number of sample images. In the disclosed embodiment, sample images are directly split to obtain first sample sub-images, which serve as guidance information for image super-resolution. Text is not required as guidance information. Therefore, to ensure training efficiency, multiple sample images are divided into multiple batches of sample images for multi-batch training. For example, 100,000 sample images are obtained from an open-source database. These 100,000 sample images, SampleImage, are equally split into upper and lower parts to obtain 100,000 first sample sub-images, SampleImagei (i E).
[0013] [1,100000]) and 100,000 second sample sub-images Samplelmage_k (j 6 [1,100000]), reduce the image resolution of the second sample sub-image, and obtain 100,000 third sample sub-images Samplelmage_k (k 6
[0014] [1.100000] ) , input the first sample sub-image and the second sample sub-image into the image super-resolution model, and perform image super-resolution on the second sample sub-image based on the guidance of the semantic information in the first sample sub-image to obtain 100,000 predicted sub-images Predictlmage l ( 1 6 [1, 100000] ) , based on the predicted sub-image and the second sample sub-image: Loss = ( SamplelmageJ, PredictlmageJ, when j = 1), a loss value Loss is calculated, and based on the loss value, the parameters of the image super-resolution model are adjusted using a gradient update method. When a preset loss value threshold is reached, a trained image super-resolution model is obtained. In the disclosed embodiment, a sample image is obtained, the sample image is split to obtain a first sample sub-image and a second sample sub-image. The image resolution of the second sample sub-image is reduced to obtain a third sample sub-image. The first and third sample sub-images are input into the image super-resolution model to obtain a predicted sub-image. A loss value is calculated based on the predicted sub-image and the second sample sub-image. Based on the loss value, the parameters of the image super-resolution model are adjusted. When a preset training end condition is met, a trained image super-resolution model is obtained. By splitting the same sample image to obtain a first sample sub-image and a second sample sub-image, and reducing the image resolution of the second sample sub-image to obtain a third sample sub-image, the image super-resolution model is trained based on the first and third sample sub-images. This avoids guiding the training of the image super-resolution model based on text information corresponding to the sample images, avoids insufficient model training results caused by insufficient sample number, and improves the model performance of the image super-resolution model. In an optional embodiment of the present disclosure, an image super-resolution model includes an image semantic extraction layer, an image coding layer, a feature inference layer, and a decoding layer. Accordingly, inputting a first sample sub-image and a third sample sub-image into the image super-resolution model to obtain a predicted sub-image includes the following specific steps: inputting the first sample sub-image into the image semantic extraction layer to obtain sample image semantic features; inputting the third sample sub-image into the image coding layer to obtain sample initial image features; inputting the sample image semantic features and sample initial image features into the feature inference layer, performing feature inference on the sample initial image features based on the sample image semantic features to obtain predicted image features; and inputting the predicted image features into the decoding layer to obtain a predicted sub-image. The sample image semantic features are high-dimensional feature coding vectors containing semantic information in the first sample sub-image; the sample initial image features are low-dimensional feature coding vectors containing image information in the second sample sub-image; and the predicted image features are low-dimensional feature coding vectors containing image information inferred based on the semantic information in the first sample sub-image. The first sample sub-image is input into the image semantic extraction layer to obtain the semantic features of the sample image. Specifically, the first sample sub-image is input into the image semantic extraction layer, and semantic features are encoded on the semantic information in the first sample sub-image to obtain the semantic features of the sample image.The third sample sub-image is input into the image coding layer to obtain sample initial image features. Specifically, the third sample sub-image is input into the image coding layer, and image feature encoding is performed on the image information in the third sample sub-image to obtain sample initial image features. It should be noted that the image feature inference performed in the feature inference layer uses the semantic features of the sample image as global feature constraints to reconstruct the sample initial image features. This feature reconstruction is achieved by performing feature transformation on the sample initial image features using the semantic features of the sample image as prior features. Exemplarily, 100,000 first sample sub-images Samplelmage_i (i C [1, 100000]) are input into the image semantic extraction layer to obtain corresponding sample image semantic features Embedding_SampleImage_i (i C [1, 100000]), 100,000 third sample sub-images Samplelmage_k (k C [1, 100000]) are input into the image coding layer to obtain corresponding sample initial image features Feature Samplelmage k (k C [1, 100000]), the sample image semantic features and the sample initial image features are input into the feature inference layer, feature inference is performed on the sample initial image features based on the sample image semantic features, and 100,000 predicted image features Feature Predictlmage 1 (1 C [1, 100000]), the predicted image features are input into the decoding layer to obtain 100,000 predicted sub-images Predictlmage 1 ( 1 6 [1,100000] ). In the disclosed embodiment, the first sample sub-image is input into the image semantic extraction layer to obtain sample image semantic features. The third sample sub-image is input into the image encoding layer to obtain sample initial image features. The sample image semantic features and the sample initial image features are input into the feature inference layer. Based on the sample image semantic features, feature inference is performed on the sample initial image features to obtain predicted image features. The predicted image features are input into the decoding layer to obtain a predicted sub-image. Through layer-by-layer feature processing in the image semantic extraction layer, image encoding layer, feature inference layer, and decoding layer, highly accurate image semantic extraction, image encoding, feature inference, and image decoding are achieved, resulting in more accurate predicted sub-images and improving model training efficiency and effectiveness.In an optional embodiment of the present disclosure, sample image semantic features and sample initial image features are input into a feature inference layer, and feature inference is performed on the sample initial image features based on the sample image semantic features to obtain predicted image features. The process includes the following specific steps: inputting the sample image semantic features and sample initial image features into the feature inference layer, performing feature inference on the sample initial image features based on the sample image semantic features to obtain reference predicted image features; determining the reference predicted image features as the sample initial image features; and returning to the step of inputting the sample image semantic features and sample initial image features into the feature inference layer. When a preset inference termination condition is met, the reference predicted image features are determined as the predicted image features. Since image super-resolution is essentially a process of denoising visual images, the output of the feature inference layer can be re-determined as input features, re-input into the feature inference layer for feature inference (denoising), and obtaining denoised output features, thereby improving the accuracy of the predicted image features ultimately derived by inference. The preset inference termination condition is a pre-set termination judgment condition for iterative feature inference, which is a hyperparameter for image super-resolution model training. The preset inference termination conditions include, but are not limited to: preset image feature scores, such as confidence scores, feature distance scores, etc., and preset number of inference iterations.Exemplarily, the sample image semantic feature Embedding Samplelmage i (i € [1, 100000]) and the sample initial image feature Feature Samplelmage k (k 6 [1, 100000]) are input into the feature inference layer, and feature inference is performed on the sample initial image feature according to the sample image semantic feature to obtain a reference predicted image feature Feature_RefImage_ 1 (1 £ [1, 100000]), and the reference predicted image feature is determined as the sample initial image feature. Return to the step of inputting the sample image semantic feature and the sample initial image feature into the feature inference layer, and when a preset inference end condition is met, determine the reference predicted image feature Feature_Reflmage_ 1 (1 6 [1, 100000]) as the predicted image feature Feature Predictlmage 1 (1 6 [1, 100000]). In the embodiment of the present disclosure, the sample image semantic feature and the sample initial image feature are input into the feature inference layer, Based on the semantic features of the sample image, feature inference is performed on the sample initial image features to obtain reference predicted image features. The reference predicted image features are determined as the sample initial image features. The process then returns to the step of inputting the sample image semantic features and the sample initial image features into the feature inference layer. If a preset inference termination condition is met, the reference predicted image features are determined as the predicted image features. This improves the feature accuracy of the predicted image features obtained by feature inference, resulting in more accurate predicted sub-images, further improving model training efficiency and effectiveness. In an optional embodiment of the present disclosure, the preset inference termination condition is a preset image feature score or a preset number of inference iterations. The preset image feature score is a preset score for the feature credibility of the image feature, such as a feature confidence score or a feature distance score (the number of changes in the feature value). The preset number of inference iterations is a preset number of feature inference iterations.Exemplarily, the sample image semantic feature Embedding_SampleImage_i (i C [1,100000]) and the sample initial image feature Feature_SampleImage_k (k 6 [1,100000]) are input into the feature inference layer, and feature inference is performed on the sample initial image feature according to the sample image semantic feature to obtain a reference predicted image feature Feature Reflmage 1 (1 6 [1, 100000]), and the reference predicted image feature is determined as the sample initial image feature. The step of inputting the sample image semantic feature and the sample initial image feature into the feature inference layer is returned to the memory, and when the preset number of inference iterations t=10 is reached, the reference predicted image feature Feature Reflmage 1 (1 C [1, 100000]) is determined to be the predicted image feature Feature_PredictImage_ 1 (1 6 [1,100000]). In the embodiment of the present disclosure, Based on a preset image feature score or a preset number of inference iterations, feature inference iteration termination judgment is performed, thereby improving judgment accuracy. In an optional embodiment of the present disclosure, the image super-resolution model further includes a text semantic extraction layer and a semantic feature fusion layer. Accordingly, inputting the first sample sub-image into the image semantic extraction layer to obtain sample image semantic features includes the following specific steps: obtaining sample text corresponding to the first sample sub-image; inputting the first sample sub-image into the image semantic extraction layer to obtain sample initial image semantic features; inputting the sample text into the text semantic extraction layer to obtain sample text semantic features; inputting the sample initial image semantic features and the sample text semantic features into the semantic feature fusion layer, and fusing the sample initial image semantic features and the sample text semantic features to obtain sample image semantic features. The sample text is text content with semantic information corresponding to the first sample sub-image. Since the first sample sub-image is obtained by segmenting the sample image, the sample image also corresponds to the sample image. For example, the sample image is a close-up photo of a man and a woman. Segmenting it to obtain the first sample sub-image, For the semantic information "a man and a woman" contained in the first sample image, the sample text is: "a man and a woman." The sample text can correspond one-to-one with each first sample sub-image, or multiple first sample sub-images can correspond to one sample text, reducing the difficulty of constructing training samples. The sample text semantic feature is the feature encoding vector of the semantic information in the sample text.The sample image semantic feature is a multimodal feature encoding vector that includes semantic information from the first sample sub-image and semantic information from the sample text. It is a high-dimensional feature encoding vector. The sample text corresponding to the first sample sub-image is obtained together with the sample image. The first sample sub-image is input into the image semantic extraction layer to obtain the sample initial image semantic feature. Specifically, the first sample sub-image is input into the image semantic extraction layer, and the semantic information in the first sample sub-image is encoded using image semantic features to obtain the sample initial image semantic feature. The sample text is input into the text semantic extraction layer to obtain the sample text semantic feature. Specifically, the sample text is input into the text semantic extraction layer, and the semantic information in the sample text is encoded using text semantic features to obtain the sample text semantic feature. Exemplarily, 100,000 sample texts SampleTxt i ( i C [1,100000] ) corresponding to 100,000 first sample sub-images Samplelmage_i ( i € [1, 100000] ) are obtained, the first sample sub-images are input into the image semantic extraction layer, image semantic feature encoding is performed on the semantic information in the first sample sub-images, and sample initial image semantic features Embedding_SampleTmage_i are obtained. ,( i 6 [1, 100000] ) , inputting the sample text into the text semantic extraction layer, performing text semantic feature encoding on the semantic information of the sample text, and obtaining the sample text semantic feature Embedding_SampleTxt_i ( i C: [1, 100000] ) , inputting the sample initial image semantic feature and the sample text semantic feature into the semantic feature fusion layer, performing feature fusion on the sample initial image semantic feature and the sample text semantic feature, and obtaining the sample image semantic feature Embedding SampleImage i ( i 6 [1, 100000] ) . In the embodiment of the present disclosure, a sample text corresponding to a first sample sub-image is obtained, the first sample sub-image is input into the image semantic extraction layer, and the sample initial image semantic feature is obtained, the sample text is input into the text semantic extraction layer, and the sample text semantic feature is obtained, the sample initial image semantic feature and the sample text semantic feature are input into the semantic feature fusion layer, and the sample initial image semantic feature and the sample text semantic feature are fused to obtain the sample image semantic feature. This improves the versatility of image super-resolution. Furthermore, more comprehensive semantic features of sample images are used to guide image super-resolution, further enhancing the ability to generate image details and improving the training effect and efficiency of model training. In an optional embodiment of the present disclosure, adjusting the parameters of the image super-resolution model based on the loss value includes the following specific steps: Adjusting the parameters of the feature inference layer based on the loss value. Generally, the image coding layer, image semantic extraction layer, text semantic extraction layer, and decoding layer in the image super-resolution model can be obtained through pre-training. Training is required for the feature inference layer, the core module of image super-resolution, to specifically improve the image super-resolution effect of the image super-resolution model. The parameters of the image coding layer, image semantic extraction layer, text semantic extraction layer, and decoding layer can be fixed, and only the parameters of the feature inference layer can be adjusted. The parameters of the feature inference layer are adjusted based on the loss value. Specifically, the parameters of the feature inference layer are adjusted based on the loss value using a gradient update method. Exemplarily, the parameters of the feature inference layer are adjusted based on the loss value using a gradient update method. In this embodiment of the present disclosure, Based on the loss value, the parameters of the feature inference layer are adjusted to specifically improve the model performance of the image super-resolution model and the training efficiency of the model.Referring to Figure 2, a flowchart of another image super-resolution method provided by an embodiment of the present disclosure is shown. This method, applied to a cloud-based device, includes the following specific steps: Step 202: Receive an initial image of an initial resolution sent by a client-side device; Step 204: Obtain a reference image corresponding to the initial image; Step 206: Input the reference image and the initial image into an image super-resolution model to obtain a target image of a target resolution, wherein the target resolution is higher than the initial resolution. The image super-resolution model is trained based on a first sample sub-image and a second sample sub-image, which are sub-images of the same sample image. The semantic features of the first sample sub-image and the image features of the second sample sub-image are used to train the image super-resolution model; Step 208: Send the target image to the client-side device. This embodiment of the present disclosure can be configured as a cloud-based device, such as an application, webpage, or mini-program, with image super-resolution functionality. The cloud-based device is deployed with an image super-resolution model and is configured to implement the image super-resolution functionality. The cloud-based device is a virtual device with a network cloud, while the client-side device is a physical device. The client-side device and the cloud-side device are connected via a network, establishing a data link for data transmission. The cloud-side device is equipped with larger storage space and higher computing power. In step 204, a reference image corresponding to the initial image is obtained from an image database. This acquisition can be performed based on image identification of the initial image or by performing a feature similarity query based on image features of the initial image. The image database can be a local image database, a remote image database, or an open-source image database, without limitation. Step 206 is consistent with step 104 in the embodiment of FIG. 1 , which has been described in detail in the embodiment of FIG. 1 and will not be repeated here. In an embodiment of the present disclosure, an initial image of an initial resolution sent by an end-side device is received, a reference image corresponding to the initial image is obtained, and the reference image and the initial image are input into an image super-resolution model to obtain a target image of a target resolution, wherein the target resolution is higher than the initial resolution. The image super-resolution model is trained based on a first sample sub-image and a second sample sub-image, which are sub-images of the same sample image. Semantic features of the first sample sub-image and image features of the second sample sub-image are used to train the image super-resolution model, and the target image is sent to the end-side device.By training the image super-resolution model based on sub-images of the same sample image, that is, based on the image semantic features of the first sample sub-image and the image features of the second sample sub-image, the training of the image super-resolution model is avoided based on the text information corresponding to the sample images. This prevents insufficient model training results caused by insufficient sample numbers, improves the model performance of the image super-resolution model, enhances the ability to generate image details, and improves the image super-resolution effect. Furthermore, by implementing image super-resolution through cloud-based devices, the computing power and storage costs of the end-side devices are reduced, improving the user experience. Referring to Figure 3, an image data processing method provided by an embodiment of the present disclosure is shown. This method, applied to a cloud device, includes the following specific steps: Step 302: Obtain a sample image and split the sample image to obtain a first sample sub-image and a second sample sub-image; Step 304: Reduce the image resolution of the second sample sub-image to obtain a third sample sub-image; Step 306: Input the first and third sample sub-images into an image super-resolution model to obtain a predicted sub-image; Step 308: Calculate a loss value based on the predicted sub-image and the second sample sub-image; Step 310: Adjust the parameters of the image super-resolution model based on the loss value. When a preset training end condition is met, a trained image super-resolution model is obtained; Step 312: Send the model parameters of the image super-resolution model to an end device. This embodiment of the present disclosure is applied to a cloud device with model training capabilities. The cloud device is a network cloud device that provides model training capabilities and is a virtual device. The end device is a terminal device that provides image super-resolution capabilities and is a physical device. The end-side device and the cloud-based device are connected via a network, establishing a data link for data transmission. The cloud-based device has higher computing and storage performance than the end-side device. In the embodiment of the present disclosure, steps 302 to 310 have been described in detail in the embodiment of FIG. 1 above and will not be repeated here. In the embodiment of the present disclosure, a sample image is obtained, the sample image is split to obtain a first sample sub-image and a second sample sub-image. The image resolution of the second sample sub-image is reduced to obtain a third sample sub-image. The first and third sample sub-images are input into an image super-resolution model to obtain a predicted sub-image. A loss value is calculated based on the predicted sub-image and the second sample sub-image. Based on the loss value, the parameters of the image super-resolution model are adjusted. When preset training end conditions are met, a trained image super-resolution model is obtained, and the model parameters of the image super-resolution model are sent to the end-side device.By splitting the same sample image to obtain a first sample sub-image and a second sample sub-image, and reducing the image resolution of the second sample sub-image to obtain a third sample sub-image, the image super-resolution model is trained based on the first sample sub-image and the third sample sub-image. This avoids relying on text information corresponding to the sample image to guide the training of the image super-resolution model, avoids insufficient model training results due to insufficient sample quantity, and improves the performance of the image super-resolution model. Furthermore, training of the image super-resolution model is implemented on cloud-side devices, reducing the computing power cost of end-devices, improving model training efficiency and results, and enhancing the user experience. Figure 4 shows a schematic diagram of model training for an image super-resolution model in an image super-resolution method provided by one embodiment of the present disclosure. As shown in Figure 4: a sample image and sample text corresponding to the sample image are obtained, the sample image is split to obtain a first sample sub-image and a second sample sub-image, and the resolution of the second sample sub-image is reduced to obtain a third sample sub-image. The sample text is input into the text semantic extraction layer of the image super-resolution model to obtain sample text semantic features. The first sample sub-image is input into the image semantic extraction layer of the image super-resolution model to obtain sample initial image semantic features. The third sample sub-image is input into the image encoding layer of the image super-resolution model to obtain sample image features. The sample text semantic features and the sample initial image semantic features are input into the feature fusion layer of the image super-resolution model to obtain sample image semantic features. The sample image semantic features and the sample initial image features are input into the feature inference layer. Feature inference is performed on the sample initial image features based on the sample image semantic features to obtain reference predicted image features. The reference predicted image features are determined as the sample initial image features. The step of inputting the sample image semantic features and the sample initial image features into the feature inference layer is returned to. When a preset number of iterations is reached, the reference predicted image features are determined as the predicted image features. The predicted image features are input into the decoding layer of the image super-resolution model to obtain a predicted sub-image. A loss value is calculated based on the predicted sub-image and the second sample sub-image. Based on the loss value, the parameters of the feature inference layer are adjusted. When a preset training end condition is met, a trained image super-resolution model is obtained.FIG5 shows a front-end schematic diagram of an image super-resolution method provided by one embodiment of the present disclosure. As shown in FIG5 , on a mobile terminal loaded with an image super-resolution application corresponding to an image super-resolution model, the first interface of the image super-resolution application includes a preset initial image preview area, a "Load Initial Image" control, and an "Image Super-Resolution" control. The user first clicks the "Load Initial Image" control to load the initial image at the initial resolution to the client of the image super-resolution application and display it in the initial image preview area. The user then clicks the "Image Super-Resolution" control to super-resolve the initial image. After the image super-resolution is completed, the current interface of the image super-resolution application jumps to a second interface. The second interface includes a preset target image preview area, a "Download" control, a "Copy" control, and a "Back" control. The target image at the target resolution is rendered in the target image preview area. The user can access corresponding functions by clicking corresponding controls. The image super-resolution method provided by the present disclosure will be further described below, using the application of the image super-resolution method provided by the present disclosure in the super-resolution of portrait photos as an example, in conjunction with FIG6 .6 shows a flowchart of a processing process of an image super-resolution method for portrait and eyeglass super-resolution provided by an embodiment of the present disclosure, including the following specific steps: Step 602: Obtain a sample image and sample text corresponding to the sample image; Step 604: Split the sample image to obtain a first sample sub-image and a second sample sub-image; Step 606: Reduce the image resolution of the second sample sub-image to obtain a third sample sub-image; Step 608: Input the first sample sub-image into the image semantic extraction layer to obtain a sample initial image semantic feature, and input the sample text into the text semantic extraction layer to obtain a sample text semantic feature; Step 610: Input the sample initial image semantic feature and the sample text semantic feature into the semantic feature fusion layer, perform feature fusion on the sample initial image semantic feature and the sample text semantic feature to obtain a sample image semantic feature; Step 612: Input the third sample sub-image into the image coding layer to obtain a sample initial image feature; Step 614: Input the sample image semantic feature and the sample initial image feature into the feature inference layer, and perform feature inference on the sample initial image feature based on the sample image semantic feature. Obtain predicted image features, determine the reference predicted image features as sample initial image features, return to the step of inputting the sample image semantic features and the sample initial image features into the feature inference layer, and determine the reference predicted image features as the predicted image features when a preset inference end condition is met; Step 616: Calculate a loss value based on the predicted sub-image and the second sample sub-image; Step 618: Adjust the parameters of the image super-resolution model based on the loss value, and obtain a trained image super-resolution model when a preset training end condition is met; Step 620: Receive an initial face photo of an initial resolution sent by a user; Step 622: Obtain a reference portrait corresponding to the initial face photo from an open source image database; Step 624: Input the reference face photo into the image semantic extraction layer to obtain image semantic features, and input the initial face photo into the image encoding layer to obtain initial face photo features; Step 626: Input the image semantic features and the initial face photo features into the feature inference layer, perform feature inference on the initial face photo features based on the image semantic features, and obtain target image features; Step 628: The target image features are input into the decoding layer to obtain the target face with the target resolution, and the target face photo is fed back to the user.In embodiments of the present disclosure, a multimodal semantic extraction layer is used to extract image semantic features that guide image super-resolution, enabling training of an image super-resolution model. This ensures that the trained image super-resolution model has greater versatility and more accurate image super-resolution performance, while also improving training efficiency. By training the image super-resolution model based on sub-images of the same sample image, namely, the image semantic features of a first sample sub-image and the image features of a second sample sub-image, the training of the image super-resolution model is avoided by guiding the training of the image super-resolution model based on text information corresponding to the sample image. This avoids insufficient model training results due to insufficient sample size, improves the performance of the image super-resolution model, enhances the ability to capture photographic details in portraits and photographs, improves the image super-resolution effect, and enhances the user experience. Corresponding to the aforementioned method embodiments, the present disclosure also provides an embodiment of an image super-resolution device. FIG7 shows a schematic structural diagram of an image super-resolution device provided by one embodiment of the present disclosure. As shown in FIG3 , the apparatus includes: a first acquisition module 702 configured to acquire a reference image and an initial image of an initial resolution; a first super-resolution module 704 configured to input the reference image and the initial image into an image super-resolution model to obtain a target image of a target resolution, wherein the target resolution is higher than the initial resolution; the image super-resolution model is trained based on a first sample sub-image and a second sample sub-image, wherein the first sample sub-image and the second sample sub-image are sub-images of the same sample image; and image semantic features of the first sample sub-image and image features of the second sample sub-image are used for training the image super-resolution model. Optionally, the image super-resolution model includes an image semantic extraction layer, an image encoding layer, a feature inference layer, and a decoding layer; accordingly, the first super-resolution module 704 is further configured to: input a reference image into the image semantic extraction layer to obtain image semantic features; input an initial image into the image encoding layer to obtain initial image features; input the image semantic features and the initial image features into the feature inference layer, perform feature inference on the initial image features based on the image semantic features to obtain target image features; and input the target image features into the decoding layer to obtain a target image of target resolution. Optionally, the image super-resolution model also includes a text semantic extraction layer and a semantic feature fusion layer; accordingly, the first super-resolution module 704 is further configured to: obtain reference text corresponding to the reference image; input the reference image into the image semantic encoding layer to obtain initial image semantic features; input the reference text into the text semantic extraction layer to obtain text semantic features; input the initial image semantic features and the text semantic features into the semantic feature fusion layer, perform feature fusion on the initial image semantic features and the text semantic features to obtain image semantic features.Optionally, the apparatus further includes: a training module configured to obtain a sample image, split the sample image to obtain a first sample sub-image and a second sample sub-image; reduce the image resolution of the second sample sub-image to obtain a third sample sub-image; input the first sample sub-image and the third sample sub-image into an image super-resolution model to obtain a predicted sub-image; calculate a loss value based on the predicted sub-image and the second sample sub-image; and adjust parameters of the image super-resolution model based on the loss value, to obtain a trained image super-resolution model when a preset training end condition is met. Optionally, the image super-resolution model includes an image semantic extraction layer, an image encoding layer, a feature inference layer, and a decoding layer; accordingly, the training module is further configured to: input the first sample sub-image into the image semantic extraction layer to obtain sample image semantic features; input the third sample sub-image into the image encoding layer to obtain sample initial image features; input the sample image semantic features and the sample initial image features into the feature inference layer, and perform feature inference on the sample initial image features based on the sample image semantic features to obtain predicted image features; and input the predicted image features into the decoding layer to obtain a predicted sub-image. Optionally, the training module is further configured to: input the sample image semantic features and the sample initial image features into the feature inference layer, and perform feature inference on the sample initial image features based on the sample image semantic features to obtain reference predicted image features; determine the reference predicted image features as the sample initial image features, return to the step of inputting the sample image semantic features and the sample initial image features into the feature inference layer, and if a preset inference termination condition is met, determine the reference predicted image features as the predicted image features. Optionally, the image super-resolution model further includes a text semantic extraction layer and a semantic feature fusion layer; accordingly, the training module is further configured to: obtain sample text corresponding to the first sample sub-image; input the first sample sub-image into the image semantic extraction layer to obtain sample initial image semantic features; input the sample text into the text semantic extraction layer to obtain sample text semantic features; input the sample initial image semantic features and the sample text semantic features into the semantic feature fusion layer to perform feature fusion on the sample initial image semantic features and the sample text semantic features to obtain sample image semantic features. Optionally, the training module is further configured to: adjust parameters of the feature inference layer based on the loss value.In the embodiments of the present disclosure, the image super-resolution model is trained based on sub-images of the same sample image, namely, based on the image semantic features of the first sample sub-image and the image features of the second sample sub-image. This avoids the need to guide the training of the image super-resolution model based on text information corresponding to the sample images, avoids insufficient model training due to insufficient sample size, improves the model performance of the image super-resolution model, enhances the ability to capture image details, and improves the image super-resolution effect. The above is a schematic diagram of an image super-resolution device according to this embodiment. It should be noted that the technical solution of this image super-resolution device and the technical solution of the aforementioned image super-resolution method are based on the same concept. For details not described in detail in the technical solution of the image super-resolution device, please refer to the description of the technical solution of the aforementioned image super-resolution method. Corresponding to the aforementioned method embodiment, the present disclosure also provides an embodiment of an image super-resolution device. Figure 8 shows a schematic structural diagram of another image super-resolution device according to one embodiment of the present disclosure. As shown in FIG8 , the apparatus is applied to a cloud-side device and includes: a receiving module 802 configured to receive an initial image of an initial resolution sent by a terminal-side device; a second acquisition module 804 configured to acquire a reference image corresponding to the initial image; a second super-resolution module 806 configured to input the reference image and the initial image into an image super-resolution model to obtain a target image of a target resolution, wherein the target resolution is higher than the initial resolution, and the image super-resolution model is trained based on a first sample sub-image and a second sample sub-image, wherein the first sample sub-image and the second sample sub-image are sub-images of the same sample image, and semantic features of the first sample sub-image and image features of the second sample sub-image are used to train the image super-resolution model; and a first sending module 808 configured to send the target image to the terminal-side device. In the disclosed embodiments, the image super-resolution model is trained using sub-images of the same sample image, namely, based on the image semantic features of the first sample sub-image and the image features of the second sample sub-image. This avoids relying on textual information corresponding to the sample images to guide the training of the image super-resolution model, and avoids insufficient model training due to insufficient sample size. This improves the performance of the image super-resolution model, enhances the ability to capture image details, and enhances the image super-resolution effect. Furthermore, image super-resolution is achieved through cloud-based planing devices, reducing the computing power and storage costs of client-side devices and improving the user experience. The above is a schematic diagram of an image super-resolution device in this embodiment. It should be noted that the technical solutions of this image super-resolution device and the technical solutions of the aforementioned image super-resolution method share the same concept. For details not described in detail in the technical solution of the image super-resolution device, please refer to the description of the technical solution of the aforementioned image super-resolution method.Corresponding to the above-mentioned method embodiments, the present disclosure also provides an embodiment of an image data processing device. FIG9 shows a schematic structural diagram of an image data processing device provided by one embodiment of the present disclosure. As shown in FIG9 , the device is applied to a cloud-side E-device and includes: a third acquisition module 902 configured to acquire a sample image and split the sample image to obtain a first sample sub-image and a second sample sub-image; a resolution reduction module 904 configured to reduce the image resolution of the second sample sub-image to obtain a third sample sub-image; a prediction module 906 configured to input the first sample sub-image and the third sample sub-image into an image super-resolution model to obtain a predicted sub-image; a calculation module 908 configured to calculate a loss value based on the predicted sub-image and the second sample sub-image; an adjustment module 910 configured to adjust parameters of the image super-resolution model based on the loss value, and obtain a trained image super-resolution model when a preset training end condition is met; and a second sending module 912 configured to send the model parameters of the image super-resolution model to a device on the end. In the disclosed embodiment, a sample image is split to obtain a first sample sub-image and a second sample sub-image. The resolution of the second sample sub-image is then reduced to obtain a third sample sub-image. The image super-resolution model is then trained based on the first and third sample sub-images. This avoids relying on text information corresponding to the sample image to guide the training of the image super-resolution model. This prevents insufficient training effectiveness due to insufficient sample size, thereby improving the performance of the image super-resolution model. Furthermore, training of the image super-resolution model is implemented using cloud-based devices, reducing the computing power cost of client-side devices, improving model training efficiency and effectiveness, and enhancing the user experience. The above is a schematic diagram of an image data processing device according to this embodiment. It should be noted that the technical solutions of this image data processing device and the technical solutions of the aforementioned image data processing method share the same concept. For details not described in detail in the technical solution of the image data processing device, reference can be made to the description of the technical solution of the aforementioned image data processing method.FIG10 shows a block diagram of an augmented reality (AR) device or a virtual reality (VR) device according to an embodiment of the present disclosure. The device 1000 includes, but is not limited to, a memory 1010, a processor 1020, and a display 1030. The processor 1020, the memory 1010, and the display 1030 are connected via a bus 1040. The memory 1010 is configured to store computer-executable instructions, and the processor 1020 is configured to execute the computer-executable instructions. When executed by the processor 1020, the computer-executable instructions implement the following steps: obtaining a reference image and an initial image at an initial resolution; inputting the reference image and the initial image into an image super-resolution model to obtain a target image at a target resolution, where the target resolution is higher than the initial resolution; the image super-resolution model is trained based on a first sample sub-image and a second sample sub-image, the first sample sub-image and the second sample sub-image being sub-images of the same sample image; and image semantic features of the first sample sub-image and image features of the second sample sub-image being used to train the image super-resolution model; and displaying the target image on the display 1030 of the augmented reality (AR) device 1000 or the virtual reality (VR) device 1000. In the embodiments of the present disclosure, the image super-resolution model is trained using sub-images of the same sample image, namely, based on the image semantic features of the first sample sub-image and the image features of the second sample sub-image. This avoids the need to guide the training of the image super-resolution model based on text information corresponding to the sample images, avoids insufficient model training due to insufficient sample size, improves the performance of the image super-resolution model, enhances the ability to capture image details, and improves the image super-resolution effect. Furthermore, the target image is displayed on a display of an augmented reality (AR) device or a virtual reality (VR) device, improving the user experience. The above is a schematic diagram of an augmented reality (AR) device or a virtual reality (VR) device according to this embodiment. It should be noted that the technical solution of this augmented reality (AR) device or virtual reality (VR) device shares the same concept as the technical solution of the aforementioned image super-resolution method. For details not described in detail in the technical solution of the augmented reality (AR) device or virtual reality (VR) device, refer to the description of the technical solution of the aforementioned image super-resolution method. Figure 11 shows a block diagram of a computing device provided in one embodiment of the present disclosure. Components of the computing device 1100 include, but are not limited to, a memory 1110 and a processor 1120. The processor 1120 is connected to the memory 1110 via a bus 1130. The database 1150 is configured to store data. The computing device 1100 also includes an access device 1140 that enables the computing device 1100 to communicate via one or more networks 1160.Examples of these networks include the Public Switched Telephone Network (PSTN) and Local Area Network (LAN).
Claims
Claims 1. An image super-resolution method, comprising: Obtain a reference image and an initial image of an initial resolution; The reference image and the initial image are input into an image super-resolution model to obtain a target image of a target resolution, wherein the target resolution is higher than the initial resolution, the image super-resolution model is trained based on a first sample sub-image and a second sample sub-image, the first sample sub-image and the second sample sub-image are sub-images of the same sample image, and image semantic features of the first sample sub-image and image features of the second sample sub-image are used for training the image super-resolution model.
2. The method according to claim 1, wherein: The image super-resolution model includes an image semantic extraction layer, an image coding layer, a feature inference layer and a decoding layer; the step of inputting the reference image and the initial image into the image super-resolution model to obtain a target image of a target resolution includes: inputting the reference image into the image semantic extraction layer to obtain image semantic features, inputting the initial image into the image coding layer to obtain initial image features; inputting the image semantic features and the initial image features into the feature inference layer, and performing feature inference on the initial image features according to the image semantic features to obtain target image features; and inputting the target image features into the decoding layer to obtain a target image of a target resolution.
3. The method according to claim 2, wherein: The image super-resolution model also includes a text semantic extraction layer and a semantic feature fusion layer; The step of inputting the reference image into the image semantic extraction layer to obtain image semantic features includes: acquiring a reference text corresponding to the reference image; inputting the reference image into the image semantic encoding layer to obtain initial image semantic features, inputting the reference text into the text semantic extraction layer to obtain text semantic features; inputting the initial image semantic features and the text semantic features into the semantic feature fusion layer, performing feature fusion on the initial image semantic features and the text semantic features to obtain image semantic features.
4. The method according to any one of claims 1 to 3, wherein the image super-resolution model is trained by the following steps: obtaining a sample image, splitting the sample image to obtain a first sample sub-image and a second sample sub-image; ' Reducing the image resolution of the second sample sub-image to obtain a third sample sub-image; Inputting the first sample sub-image and the third sample sub-image into an image super-resolution model to obtain a predicted sub-image; Based on the predicted sub-image and the second sample sub-image, a loss value is calculated; Based on the loss value, the parameters of the image super-resolution model are adjusted, and when a preset training end condition is reached, a trained image super-resolution model is obtained.
5. The method according to claim 4, wherein: The image super-resolution model includes an image semantic extraction layer, an image coding layer, a feature inference layer and a decoding layer; the step of inputting the first sample sub-image and the third sample sub-image into the image super-resolution model to obtain a predicted sub-image includes: inputting the first sample sub-image into the image semantic extraction layer to obtain a sample image semantic feature, inputting the third sample sub-image into the image coding layer to obtain a sample initial image feature; inputting the sample image semantic feature and the sample initial image feature into the feature inference layer, performing feature inference on the sample initial image feature according to the sample image semantic feature to obtain a predicted image feature; and inputting the predicted image feature into the decoding layer to obtain a predicted sub-image.
6. The method according to claim 5, wherein the step of inputting the sample image semantic features and the sample initial image features into the feature inference layer, and performing feature inference on the sample initial image features according to the sample image semantic features to obtain predicted image features comprises: After the sample image semantic features and the sample initial image features are input into the feature inference layer, feature inference is performed on the sample initial image features according to the sample image semantic features to obtain reference predicted image features; the reference predicted image features are determined as the sample initial image features, and the step of inputting the sample image semantic features and the sample initial image features into the feature inference layer is returned to execute, and when a preset inference end condition is met, the reference predicted image features are determined as the predicted image features.
7. The method according to claim 6, wherein: The preset reasoning end condition is a preset image feature score or a preset number of reasoning iterations.
8. The method according to claim 5, wherein: The image super-resolution model also includes a text semantic extraction layer and a semantic feature fusion layer; the step of inputting the first sample sub-image into the image semantic extraction layer to obtain the semantic features of the sample image includes: acquiring a sample text corresponding to the first sample sub-image; inputting the first sample sub-image into the image semantic extraction layer to obtain the semantic features of the sample initial image, and inputting the sample text into the text semantic extraction layer to obtain the semantic features of the sample text; inputting the semantic features of the sample initial image and the semantic features of the sample text into the semantic feature fusion layer, and performing feature fusion on the semantic features of the sample initial image and the semantic features of the sample text to obtain the semantic features of the sample image.
9. The method according to claim 5, wherein adjusting the parameters of the image super-resolution model based on the loss value comprises: Based on the loss value, parameters of the feature inference layer are adjusted.
10. An image super-resolution method, applied to a cloud-side device, comprising: An initial image of an initial resolution sent by a device on the receiving end; Acquire a reference image corresponding to the initial image; Inputting the reference image and the initial image into an image super-resolution model to obtain a target image of a target resolution, wherein the target resolution is higher than the initial resolution, the image super-resolution model is trained based on a first sample sub-image and a second sample sub-image, the first sample sub-image and the second sample sub-image are sub-images of the same sample image, and semantic features of the first sample sub-image and image features of the second sample sub-image are used for training the image super-resolution model; and sending the target image to the terminal device.
11. An image data processing method, applied to a cloud-side device, comprising: A sample image is obtained, and the sample image is split to obtain a first sample sub-image and a second sample sub-image. ' Reducing the image resolution of the second sample sub-image to obtain a third sample sub-image; inputting the first sample sub-image and the third sample sub-image into an image super-resolution model to obtain a predicted sub-image; and calculating a loss value based on the predicted sub-image and the second sample sub-image; Based on the loss value, adjust the parameters of the image super-resolution model, and obtain a trained image super-resolution model when a preset training end condition is met; and send the model parameters of the image super-resolution model to the terminal side device.
12. An augmented reality (AR) device or a virtual reality (VR) device, comprising: Memory, processor and display; The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the following steps are implemented: obtaining a reference image and an initial image of an initial resolution; inputting the reference image and the initial image into an image super-resolution model to obtain a target image of a target resolution, wherein the target resolution is higher than the initial resolution, and the image super-resolution model is based on a first sample sub-image. The target image is obtained by training the first sample sub-image and the second sample sub-image, the first sample sub-image and the second sample sub-image are sub-images of the same sample image, and the image semantic features of the first sample sub-image and the image features of the second sample sub-image are used for training the image super-resolution model; and the target image is displayed through the display of the augmented reality AR device or the virtual reality VR device.
13. A computing device, comprising: Memory and processor; The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the method according to any one of claims 1 to 11 are implemented.
14. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.
Citation Information
Patent Citations
Image processing method and device, electronic device and storage medium
CN110084775A
Model training method, image reconstruction method, storage medium and related equipment
CN110533594A
Image super-resolution reconstruction model training method, ship tracking method and device
CN111445388A
Image processing methods, apparatus and equipment
CN112088393B
Image processing model training method and device and storage medium
CN112508780A
Cited By
Picture reconstruction method, system and device and storage medium
CN120807284A
