Image processing method, terminal equipment and readable storage medium

Abstract

The invention provides an image processing method, terminal equipment and a readable storage medium, and is applied to the technical field of terminals.The method comprises the steps that an obtained low-resolution first image, an obtained noise image and obtained text description information are input into a multi-modal generative amplification model; according to the embodiment of the invention, the high-resolution second image conforming to the text description information can be output, the detail reduction degree of the obtained second image is relatively high, and the image is relatively clear; the text description information is generated based on the first image. According to the embodiment of the invention, the first image is subjected to generative super-magnification processing through the multi-modal generative magnification model, the high-quality second image can be obtained, the detail reduction degree in the second image is high, and the visual experience of a user can be improved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular, to an image processing method, a terminal device and a readable storage medium. Background Art

[0002] With the development of terminal devices, more and more photographing modes are set in the terminal devices. The terminal devices can meet different photographing needs of different users by setting multiple photographing modes.

[0003] Currently, when a user uses a terminal device to shoot an object that is far away from the terminal device, the terminal device can take a photo of the object at an actual magnification ratio expected by the user, thereby obtaining an enlarged image of the object. However, when the actual magnification ratio expected by the user is too large, the details of the enlarged image of the object obtained by the terminal device will be seriously lost, and the image clarity will be poor. Summary of the invention

[0004] The present application provides an image processing method, a terminal device and a readable storage medium, which can make the detail information of the super-resolution enlarged image more complete, improve the clarity of the super-resolution enlarged image, and thus improve the user's visual experience.

[0005] In a first aspect, the present application provides an image processing method, the method comprising:

[0006] Acquire a first image, where the resolution of the first image is a first resolution;

[0007] Based on the first image, text description information of the first image is generated; the text description information is used to describe the first image, and the text description information includes at least first-category information and second-category information; the first-category information includes the time corresponding to the first image and / or the location corresponding to the first image; the second-category information is used to characterize the category to which the content included in the first image belongs;

[0008] Acquire a noise image; the noise image is used to provide a noise distribution feature for the first image;

[0009] The first image, text description information and the noise image are input into a multimodal generative upscaling model, and a second image is output; the resolution of the second image is a second resolution, and the second resolution is greater than the first resolution.

[0010] By acquiring the first image, the present application can obtain the image to be magnified, providing a data basis for the subsequent super-resolution magnification process. Then, based on the obtained first image, the text description information of the first image is generated, which can provide a data basis for the subsequent generative super-resolution magnification process. Acquiring a noise image can provide noise distribution characteristics for the first image. Furthermore, after obtaining the first image, the text description information and the noise image, the first image, the text description information and the noise image can be input into a multimodal generative magnification model, and the multimodal generative magnification model can be used to perform generative super-resolution magnification on the first image based on the input data of the three modes of the first image, the text description information and the noise image, so as to better generate the detail information of the enlarged image and obtain a second image with a higher degree of detail restoration. In other words, after the first image is subjected to generative super-resolution magnification processing by the multimodal generative magnification model under the guidance and constraints of the first image and the text description information, an image with a higher degree of detail restoration is obtained. Thus, it is possible to implement a generative super-resolution upscaling process on a low-resolution (e.g., first resolution) image (e.g., first image) through a multimodal generative upscaling model, and output a high-resolution (e.g., second resolution) image (e.g., second image). Moreover, the output high-resolution image has a higher degree of detail restoration, a higher image clarity, a better image quality, and a better visual experience for users.

[0011] In some possible implementations, inputting the first image, the text description information, and the noisy image into a multimodal generative upscaling model, and outputting the second image includes:

[0012] The first image, text description information and noise image are input into a multimodal generative upscaling model and N iterations are performed to obtain a second image, where N is a positive integer greater than or equal to 1 and less than or equal to a preset iteration threshold.

[0013] In the above implementation, by inputting the three modal data sources of the first image, text description information and noise image into the multimodal generative upscaling model and performing N iterations, the detail information in the second image after the generative super-resolution upscaling processing can be made clearer, and the quality of the second image after the generative super-resolution upscaling processing is higher, which can improve the user's visual experience.

[0014] In some possible implementations, the multimodal generative upscaling model may include a denoising network and a super-resolution upscaling network;

[0015] The first image, the text description information, and the noise image are input into the multimodal generative upscaling model and N iterations are performed to obtain the second image, including:

[0016] When performing the first iteration, the first image, the text description information, and the noisy image are input into the denoising network, and the denoising network performs denoising on the noisy image based on the text description information and the first image to obtain a fourth image;

[0017] Inputting the fourth image into the super-resolution upscaling network, and performing super-resolution upscaling processing on the fourth image from the denoising network through the super-resolution upscaling network to obtain a fifth image;

[0018] When performing the Nth iteration, the fifth image, the text description information, and the first image are input into the denoising network to obtain a fifth image after denoising, and the fifth image after denoising is input into the super-resolution upscaling network to obtain a fifth image after super-resolution upscaling;

[0019] When N is equal to a preset iteration threshold, the fifth image after the super-resolution and enlargement processing is output as the second image.

[0020] In the above implementation, by using a denoising network to denoise the noisy image based on the text description information and the first image when executing the first iteration, a fourth image can be obtained, and the detail information of the fourth image is clearer than the detail information of the first image. Based on this, the fourth image is super-enlarged by the super-resolution enlargement network, and the detail information loss of the fifth image is less. Further, the obtained fifth image, the text description information and the first image are iterated for the next time. After N iterations, the detail information of the obtained second image is clearer and the quality is better.

[0021] In some possible implementations, the denoising network includes a first encoder, a first decoder, and an attention network;

[0022] When performing the first iteration, the first image, the text description information, and the noisy image are input into the denoising network, and the denoising network performs denoising on the noisy image based on the text description information and the first image to obtain a fourth image, including:

[0023] When executing the first iteration, the first feature information, the second feature information, the third feature information and the fourth feature information are input into the attention network to obtain the fifth feature information; the fifth feature information is used to characterize the feature after the attention network fuses multiple features; the first feature information is used to characterize the image feature corresponding to the first image; the second feature information is used to characterize the text feature corresponding to the text description information; the third feature information is used to characterize the noise feature corresponding to the noise image; the fourth feature information is used to characterize the number feature corresponding to the first iteration;

[0024] Inputting the noise image and the fifth feature information into a first encoder, and outputting a first encoding feature map; the first encoder is used to perform feature encoding processing on the noise image and the fifth feature information;

[0025] The first coding feature map is input to the first decoder, and a fourth image is output; the first decoder is used to perform feature decoding processing on the first coding feature map.

[0026] In the above implementation, multiple information is fused through the attention network, which can make the detail information generated in the first encoding feature map more accurate, and thus make the detail information in the fourth image clearer.

[0027] In some possible implementations, the multimodal generative upscaling model further includes a second encoder, a third encoder, and a fourth encoder;

[0028] Before inputting the first feature information, the second feature information, the third feature information, and the fourth feature information into the attention network, the method further includes:

[0029] Inputting the first image into the second encoder to obtain first feature information; the second encoder is used to perform feature encoding processing on the first image;

[0030] Inputting the text description information into a third encoder to obtain second feature information; the third encoder is used to perform feature encoding processing on the text description information;

[0031] Inputting the noise image into the first encoder to obtain third feature information;

[0032] The current number of iterations is input into a fourth encoder to obtain fourth feature information; the fourth encoder is used to perform feature encoding processing on the current number of iterations.

[0033] In the above implementation, different input information is respectively encoded with different encoders to obtain corresponding feature information, which can prepare for the subsequent feature fusion processing of the attention network.

[0034] In some possible implementations, the super-resolution upscaling network includes at least one convolutional layer; the fourth image is input into the super-resolution upscaling network, and the fourth image from the denoising network is super-upscaled by the super-resolution upscaling network to obtain a fifth image, including:

[0035] The fourth image is input into at least one convolution layer, and the fourth image is super-enlarged based on a preset magnification by the at least one convolution layer to obtain a fifth image.

[0036] In the above implementation, by setting at least one convolutional layer to implement super-resolution upscaling of the fourth image, it is possible to avoid the loss of detail information of the fourth image during the super-resolution upscaling process, thereby ensuring that the details of the obtained second image are clearer and the quality of the second image is higher.

[0037] In some possible implementations, generating text description information of the first image based on the first image includes:

[0038] Get the first type of information;

[0039] Input the first image into the classification model and output the second category information;

[0040] The first type of information and the second type of information are fused to obtain text description information.

[0041] In the above implementation, by obtaining system information related to the first image, that is, the first type of information. And, by obtaining high-order semantic feature information of the first image, that is, the second type of information, through the classification model. Based on this, by fusing the first type of information and the second type of information, text information can be obtained. This can provide guidance conditions for the multimodal generative upscaling model to perform generative super-resolution upscaling on the first image, and improve the accuracy of detail information in the first image generated by the multimodal generative upscaling model.

[0042] In some possible implementations, the classification model includes at least an encoder layer and a fully connected layer;

[0043] The first image is input into the classification model, and the second type of information is output, including:

[0044] Inputting the first image into an encoder layer to obtain sixth feature information; the encoder layer is used to perform feature encoding processing on the first image;

[0045] The sixth feature information is input into the fully connected layer to obtain the second category information; the fully connected layer is used to classify the sixth feature information.

[0046] In the above implementation, by first performing feature encoding processing on the first image and then classifying the sixth feature information obtained by the feature encoding processing, the category to which the content included in the first image belongs can be accurately obtained, and the accuracy of the detail information in the first image generated by the multimodal generative magnification model can be further ensured.

[0047] In some possible implementations, before acquiring the first image, the method further includes:

[0048] In response to a user's operation of turning on the camera, displaying a first preview image at a first zoom factor;

[0049] When the first zoom ratio is adjusted to a second zoom ratio, a second preview image is displayed, and the second zoom ratio is greater than the first zoom ratio;

[0050] Acquire the first image, including:

[0051] When receiving a photo-taking instruction, preprocessing the second preview image to obtain the first image; the preprocessing includes one or more of the following: denoising, automatic white balance, demosaicing, histogram equalization, and image enhancement;

[0052] Wherein, outputting the second image comprises:

[0053] The second image is displayed.

[0054] In the above implementation, by preprocessing the second preview image after adjusting the magnification (i.e., the first zoom ratio is adjusted to the second zoom ratio), a first image with enhanced image brightness and contrast can be obtained, thereby providing a good data basis for the subsequent processing of the multimodal generative magnification model.

[0055] In some possible implementations, before displaying the second image, the method further includes:

[0056] The second image is post-processed; the post-processing includes one or more of the following processes: sharpening processing, compression processing and format conversion processing.

[0057] In the above implementation, by post-processing the second image, the clarity of the image after the generative super-resolution upscaling processing by the multimodal generative upscaling model can be further enhanced, thereby improving the user's visual experience.

[0058] In some possible implementations, the multimodal generative upscaling model is obtained using the following training method:

[0059] A first sample image, a second sample image, text description sample information and a third sample image are obtained; the first sample image and the third sample image are images with the same content but different resolutions, and the resolution of the first sample image is smaller than that of the third sample image; the second sample image is a noise image used to increase the magnification diversity of the first sample image; the text description sample information is generated based on the first sample image;

[0060] The first sample image, the second sample image, the text description sample information and the third sample image are input into the original multimodal generative upscaling model for training until the original multimodal generative upscaling model meets the training iteration conditions, and the original multimodal generative upscaling model that meets the training iteration conditions is determined as the multimodal generative upscaling model.

[0061] In the above implementation, after multiple iterative training using the original multimodal generative upscaling model with the same network structure as the multimodal generative upscaling model, the obtained multimodal generative upscaling model has a better effect of generative super-resolution upscaling processing on the first image.

[0062] In a second aspect, the present application provides a terminal device, comprising: one or more processors, and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the terminal device to execute the image processing method in the first aspect and any possible design of the first aspect.

[0063] In a third aspect, the present application provides a chip system, which is applied to a terminal device; the chip system includes one or more processors; the one or more processors are used to call computer instructions so that the terminal device executes the image processing method in the first aspect and any possible design of the first aspect.

[0064] Among them, the chip system may include one chip or multiple chips; when the chip system includes multiple chips, this application does not limit parameters such as the type and quantity of the chips.

[0065] In a fourth aspect, the present application provides a computer-readable storage medium, which includes instructions. When the instructions are executed on a terminal device, the terminal device executes the image processing method in the first aspect and any possible design of the first aspect.

[0066] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the image processing method in the first aspect and any possible design of the first aspect.

[0067] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0069] Figure 1 A schematic diagram of an application scenario of the current image processing method provided in one embodiment of the present application;

[0070] Figure 2 A schematic block diagram of the principle of the current image processing method provided by an embodiment of the present application;

[0071] Figure 3 A schematic diagram of an application scenario of an image processing method provided in one embodiment of the present application;

[0072] Figure 4 A schematic diagram of an image magnification effect provided by an embodiment of the present application;

[0073] Figure 5 A schematic block diagram of the principle of an image processing method provided in one embodiment of the present application;

[0074] Figure 6 A flowchart of an image processing method provided in one embodiment of the present application;

[0075] Figure 7 A flowchart of an image processing method provided in one embodiment of the present application;

[0076] Figure 8 A schematic diagram of a principle for outputting first-category information provided in an embodiment of the present application;

[0077] Fig. 9 A schematic diagram of a network architecture of a classification model provided in one embodiment of the present application;

[0078] Fig.10 A schematic block diagram of the principle of a language conversion module provided in one embodiment of the present application;

[0079] Fig.11 A schematic diagram of a network architecture of a multimodal generative amplification model provided in one embodiment of the present application;

[0080] Fig.12 A schematic diagram of the principle of a diffusion model provided in one embodiment of the present application;

[0081] Fig.13 A schematic block diagram of an application of a multi-modal generative amplification model provided in one embodiment of the present application;

[0082] Fig.14 A schematic diagram of a network architecture of a multimodal generative amplification model provided in one embodiment of the present application;

[0083] Fig.15 A schematic block diagram of a training process of a multimodal generative upscaling model provided in one embodiment of the present application;

[0084] Fig.16 A schematic diagram of a hardware system of a terminal device provided in one embodiment of the present application;

[0085] Fig.17 A software system diagram of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0086] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0087] In the following, the terms "first", "second", and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", and "third" may explicitly or implicitly include one or more of the features.

[0088] The image processing method provided in the embodiment of the present application can be applied to the scene where the image is super-resolution magnified. In some embodiments, the image processing method provided in the embodiment of the present application can be applied to the scene where the image is super-resolution magnified, such as shooting, recording, and screenshot.

[0089] The image processing method provided in the embodiment of the present application can be applied to a terminal device. The terminal device can be a terminal device with display hardware and corresponding software support. For example, the terminal device can be a mobile phone, a folding screen, a smart screen, a tablet computer, a wearable terminal device, a vehicle-mounted terminal device, an augmented reality (AR) device, a virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a home appliance, a projector, etc. The embodiment of the present application does not impose any restrictions on the specific type of the terminal device.

[0090] Combine the following Figure 1 and Figure 2 , which introduces in detail the specific implementation process of the current image processing methods.

[0091] See also Figure 1 , Figure 1 A schematic diagram of an application scenario of an image processing method is shown.

[0092] like Figure 1 As shown, when a user uses the terminal device 100 to take an image of a pine tree, in response to the user adjusting the magnification (for example, the magnification is 60 times), the terminal device 100 can display a preview image after enlarging some details of the pine tree. Further, in response to the user's photo operation, the terminal device 100 can obtain an enlarged image of the pine tree (for example, Figure 1 The arrow in the middle points to the image of a pine tree at 60x magnification).

[0093] It should be noted that Figure 1 The structure of the pine tree shown is a schematic structure, not the structure of the pine tree actually photographed. The pine tree with the schematic structure and the terminal device 100 can embody a complete application scenario.

[0094] It should also be noted that Figure 1 The pine tree image with a magnification of 60 times pointed by the middle arrow is an enlarged effect picture obtained by actual shooting. The enlarged effect picture obtained by actual shooting can reflect the effect of the terminal device 100 magnifying some details of the pine tree.

[0095] The terminal device 100 can implement the above image processing process through multiple software modules. Figure 2 The schematic diagram of the principle of the current image processing method is shown in FIG. Figure 2 As shown, the multiple software modules may include an acquisition module 10 , a pre-processing module 11 , an amplification module 12 and a post-processing module 13 .

[0096] The terminal device 100 can collect the image to be captured through the capture module 10. The image to be captured is an original image that has not been processed in any way.

[0097] Exemplarily, the acquisition module 10 may be a camera (including a telephoto lens and a short focal lens, etc.) installed on the terminal device 100. The camera can transmit the content to be photographed in the form of a digital image to the subsequent preprocessing module 11 by converting the optical signal into an electrical signal, thereby providing a data basis for subsequent preprocessing.

[0098] After obtaining the image to be captured, the acquisition module 10 may input the image to be captured into the preprocessing module 11 for preprocessing (or pre-processing) to obtain the preprocessed image to be captured. The preprocessed image to be captured is a preview image before the user adjusts the magnification (for example, including Figure 1 image of pine tree shown).

[0099] When the preprocessed image to be captured is obtained, in response to the user's operation of adjusting the magnification (for example, the user selects a magnification greater than the current magnification in the terminal device 100), the terminal device 100 can enlarge some details of the preprocessed image to be captured through the enlargement module 12 to obtain the enlarged image to be captured. The enlarged image to be captured is a preview image after the user adjusts the magnification.

[0100] Optionally, in some embodiments, the magnification module 12 may use an image interpolation algorithm (such as nearest neighbor interpolation or bilinear interpolation, etc.) to perform super-resolution magnification on the pre-processed image to be captured.

[0101] It should be noted that the magnification module 12 may also use a generative adversarial algorithm to perform super-resolution magnification on the pre-processed image to be captured, which is not limited in the embodiment of the present application.

[0102] When the image to be captured after the enlargement processing is obtained, in response to the user's photo-taking operation, the terminal device 100 can input the image to be captured after the enlargement processing to the post-processing module 13 through the enlargement module 12 for post-processing to obtain a post-processed image. The post-processed image is the image displayed to the user by the terminal device 100 (for example, Figure 1 The arrow in the middle points to the image of a pine tree at 60x magnification).

[0103] However, the image obtained by the above image processing method has serious detail loss and poor image clarity (such as Figure 1 The image shown is a pine tree image with a magnification of 60 times). Moreover, as the magnification of the image increases, the detail loss of the image obtained by the above image processing method will be more serious, and the image will be more blurred.

[0104] In view of this, an embodiment of the present application proposes an image processing method, by acquiring a low-resolution first image, and acquiring a noise image. Then, based on the first image, text description information of the first image can be generated. Furthermore, the acquired first image, text description information and noise image are input into a multimodal generative upscaling model, and the multimodal generative upscaling model can complete the super-resolution upscaling processing of the first image based on the information of the three modes of the first image, the text description information and the noise image, and output a high-resolution second image. Thus, by inputting information of multiple modes, the multimodal generative upscaling model can accurately generate the detail information of the first image under the guidance of the text description information, thereby making the detail information in the second image after the super-resolution upscaling processing clearer, and further improving the quality of the second image after the super-resolution upscaling processing.

[0105] Combine the following Figures 3 to 5 , briefly introduces the specific implementation process of an image processing method provided in an embodiment of the present application.

[0106] See also Figure 3 , Figure 3 A schematic diagram of an application scenario of an image processing method provided in an embodiment of the present application is shown.

[0107] When a user uses the terminal device 100, the terminal device 100 may display the following Figure 3 The interface 11 shown in FIG. (1) is shown in FIG. Figure 3 As shown in FIG. 1 (1), multiple applications are installed in the terminal device 100. The multiple applications may include a camera 101, an address book, a phone, and a photo album.

[0108] like Figure 3 As shown in Figure (1), the terminal device 100 can display the internal interface of the camera 101 application in response to the user's triggering operation on the camera 101 in the interface 11 (for example, a single click, multiple clicks or a long press, etc.).

[0109] like Figure 3 As shown in (2) of FIG. 1 , the interface 12 displayed by the terminal device 100 is an internal interface of the camera 101. The interface 12 may include multiple shooting modes of the camera 101. The multiple shooting modes may include an aperture mode, a night scene mode, a portrait mode, a photo mode, a video mode, and a professional mode. Among them, the professional mode is set by default to a mode without adding any algorithm.

[0110] It should be noted that different shooting modes produce different image effects. In actual use, the user can obtain different image effects by manually switching different shooting modes, and the embodiments of the present application are not limited to this.

[0111] Optionally, the image processing method of the embodiment of the present application can be integrated in the camera 101. Alternatively, the image processing method of the embodiment of the present application can be integrated in any one of a plurality of shooting modes. For example, the image processing method of the embodiment of the present application can be integrated in the photo mode. Considering that the interface when the camera 101 in the terminal device 100 is turned on usually defaults to the interface in the photo mode (for example, interface 12), therefore, the image processing algorithm of the embodiment of the present application is integrated in the photo mode, and the user only needs to click the control 106 without switching the mode to obtain the image processed by the image processing algorithm of the embodiment of the present application, which is more convenient for user operation and can increase the user experience.

[0112] It should be noted that, in practical applications, those skilled in the art may also integrate the image processing algorithm of the embodiment of the present application into other shooting modes (e.g., video recording mode) in addition to the photo mode according to actual use requirements, and the embodiment of the present application does not limit this. For example, when the terminal device 100 integrates the image processing method of the embodiment of the present application into the video recording mode, during the video recording process, the image captured during the video recording process can be subjected to generative super-resolution magnification processing to obtain an image with higher resolution and clearer detail information.

[0113] like Figure 3 As shown in FIG. (2), the interface 12 may further include a viewfinder 102, a control 103, a control 104, a control 105, and a control 106. In response to the user triggering operations on different controls in the interface 12, the terminal device 100 may perform different operations.

[0114] Optionally, the viewfinder 102 is used to display the image to be captured (eg Figure 3 For example, in the photo mode of the camera 101, the viewfinder 102 is used to display a preview image of the pine tree to be photographed. In the video mode of the camera 101, the viewfinder 102 is used to display a video of the pine tree to be recorded.

[0115] It should be noted that Figure 3 The pine tree included in the preview image of the pine tree to be photographed shown in FIG. (2) is a schematic structure, and the purpose is to fully reflect the preview image currently displayed in the viewfinder 102. Figure 3 As shown in (2) of FIG. 1 , the interface 12 may also display the optical magnification of the lens in the terminal device 100. The optical magnification of the lens refers to the ability of the lens to magnify the observed object on the imaging sensor or film. The optical magnification of the lens is a fixed value, which is determined by the physical properties of the lens and cannot be changed by the user.

[0116] Optionally, the optical magnification of the lens may include the optical magnification of the short-focus lens and the optical magnification of the long-focus lens. Exemplarily, the optical magnification of the short-focus lens is 1, and the magnification of the long-focus lens is 3.

[0117] Optionally, the control 103 is used to display the optical magnification of the short-focus lens in the terminal device 100. For example, Figure 3 As shown in FIG. 2 (2), the value 1 can be displayed in the control 103.

[0118] Optionally, the control 104 is used to display the optical magnification of the telephoto lens in the terminal device 100. For example, Figure 3As shown in (2) of FIG. 1 , control 104 can display a value of 3.

[0119] Considering that the optical magnification of the lens in the terminal device is a fixed value, in actual use, it is necessary to combine the digital zoom magnification of the terminal device 100 to achieve the actual magnification expected by different users. The value of the digital zoom magnification of the terminal device 100 is a value that the user can change through corresponding operations.

[0120] Optionally, the actual magnification ratio expected by the user may be the product of the optical magnification ratio of the lens and the digital zoom ratio of the terminal device 100. When the terminal device 100 detects the actual magnification ratio expected by the user, it can perform corresponding magnification processing on the image to be captured displayed in the viewfinder 102.

[0121] It should be noted that, in actual applications, in order to ensure the quality of the enlarged image, the digital zoom ratio of the terminal device 100 is usually set with an upper limit value (for example, the upper limit value is 100 or the upper limit value is 200) and stored in the memory of the terminal device 100. In the case where the quality of the image is not required, the digital zoom ratio of the terminal device 100 may not be set with an upper limit value, and the embodiments of the present application are not limited to this.

[0122] It should also be noted that, since the actual magnification expected by the user may be the product of the optical magnification of the lens and the digital zoom ratio of the terminal device 100, the actual magnification expected by the user is usually displayed in the terminal device 100 so that the user can intuitively see the current magnification value, and the upper limit value of the digital zoom ratio of the terminal device 100 is usually not displayed in the interface.

[0123] When the actual magnification desired by the user is greater than the optical magnification of the short focus lens and less than the optical magnification of the long focus lens, the actual magnification desired by the user is the product of the optical magnification of the short focus lens and the digital zoom magnification of the terminal device 100 .

[0124] Exemplarily, the actual magnification expected by the user is 1.5 times, and the optical magnification of the short-focus lens is 1 times, that is, the digital zoom ratio of the terminal device 100 is 1.5 times. At this time, the actual magnification expected by the user is the product of 1 and 1.5.

[0125] Optionally, in some embodiments, when the actual magnification desired by the user is greater than the optical magnification of the short-focus lens and less than the optical magnification of the long-focus lens, in response to the user's operation of adjusting the magnification in the interface 12 (for example, Figure 3The terminal device 100 may display the change process of the actual magnification ratio expected by the user in the control 103, for example, the value displayed in the control 103 gradually increases from 1 to 1.5, and then gradually increases from 1.5 to 2.9 ( Figure 3 (2) in the figure is not shown).

[0126] It should be noted that the pinch operation may refer to sliding two fingers of the user on the screen so that the relative distance between the two fingers gradually increases or decreases, ultimately achieving zooming in and out of the operation object.

[0127] It should also be noted that, in response to the user's operation of adjusting the magnification in the interface 12, the control 103 may not display the actual magnification change process expected by the user, and the embodiment of the present application is not limited to this.

[0128] In response to the user's operation of adjusting the magnification in the interface 12, when the actual magnification desired by the user is equal to the optical magnification of the telephoto lens, the terminal device 100 can switch from the short-focus lens to the telephoto lens for shooting. At this time, the actual magnification desired by the user displayed in the terminal device 100 can be switched from the optical magnification of the short-focus lens to the optical magnification of the telephoto lens. For example, Figure 3 The current actual magnification shown in FIG. 2 is switched from the value 1 displayed by the control 103 to the value 3 displayed by the control 104 .

[0129] It should be noted that, in addition to the above-mentioned adjustment of the magnification by the user to enable the terminal device 100 to switch from the short-focus lens to the telephoto lens for shooting, in response to the user swiping left or right on the display screen of the terminal device 100, the terminal device 100 can also switch from the short-focus lens to the telephoto lens for shooting, and the embodiments of the present application are not limited to this.

[0130] It should also be noted that as the user adjusts the magnification, the image to be captured displayed in the viewfinder 102 will also be magnified accordingly. For example, when the user adjusts the magnification to 1.5 times, the image to be captured displayed in the viewfinder 102 will be magnified 1.5 times.

[0131] Optionally, the control 105 is used to display the captured image (e.g., Figure 3 In response to the user's triggering operation on the control 105, the terminal device 100 can display the captured image or the recorded video.

[0132] Optionally, the control 106 is used to trigger the start of shooting or recording. In response to the user's triggering operation on the control 106, the terminal device 100 can start shooting images or start recording videos.

[0133] In the case where the actual magnification expected by the user is greater than the optical magnification of the telephoto lens, such as Figure 3 As shown in Figure (2), in response to the user's operation of adjusting the magnification in the interface 12 (for example, the user's two fingers slide on the screen to perform a pinching operation so that the relative distance between the two fingers gradually increases), the terminal device 100 can perform corresponding magnification processing on the image to be captured displayed in the viewfinder 102.

[0134] In the case where the actual magnification desired by the user is greater than the optical magnification of the telephoto lens, the actual magnification desired by the user is the product of the optical magnification of the telephoto lens and the digital zoom magnification of the terminal device 100 .

[0135] For example, Figure 3 As shown in Figure (3), the actual magnification ratio expected by the user is 60 times, and the optical magnification ratio of the telephoto lens is 3 times, that is, the digital zoom ratio of the terminal device 100 is 20 times. At this time, the actual magnification ratio expected by the user is the product of 3 and 20.

[0136] like Figure 3 As shown in FIG. 3 , the interface 13 includes a control 107. The control 107 is used to display the current actual magnification greater than the optical magnification of the telephoto lens. For example, the control 107 displays a value of 60.

[0137] It should be noted that the embodiment of the present application uses control 107 to display the current actual magnification that is greater than the optical magnification of the telephoto lens as an example, but the embodiment of the present application is not limited to this. For example, the terminal device 100 can also directly display the current actual magnification that is greater than the optical magnification of the telephoto lens through control 104.

[0138] like Figure 3 As shown in FIG. (3) of FIG. 1 , the viewfinder 102 of the interface 13 displays an enlarged preview image corresponding to the current actual magnification in the control 107. For example, the viewfinder 102 of the interface 13 displays a preview image of a pine tree enlarged 60 times.

[0139] It should be noted that Figure 3 The pine tree included in the preview image of the pine tree magnified 60 times as shown in FIG. (3) is a schematic structure, and the purpose is to fully reflect the preview image currently displayed in the viewfinder 102.

[0140] like Figure 3As shown in FIG. (3), in response to the user's triggering operation on the control 106, the terminal device 100 may display the following Figure 3 The interface 14 shown in FIG. (4) in FIG. 14 . At this time, the control 105 of the interface 14 can display the image of the pine tree after being magnified 60 times.

[0141] It should be noted that Figure 3 The pine tree included in the image of the pine tree magnified 60 times displayed in the control 105 shown in FIG. (4) is a schematic structure, the purpose of which is to fully reflect the image currently displayed by the control 105.

[0142] like Figure 3 As shown in FIG. (4) of FIG. 1 , in response to the user's triggering operation on the control 105, the terminal device 100 can display the image of the pine tree after being magnified 60 times. For example, the terminal device 100 can display the following Figure 4 The interface 15 shown is an image of a pine tree magnified 60 times.

[0143] like Figure 4 As shown, the interface 15 may include controls 108 and 109. Control 108 is used to return the image stored in the image of the pine tree after being magnified 60 times. Control 109 is used to edit the image of the pine tree after being magnified 60 times currently displayed in the interface 15 (for example, crop the image size, etc.).

[0144] It should be noted that, in addition to controls 108 and 109, Figure 4 The interface 15 shown may also include a share control, a delete control, etc. In response to the user clicking on different controls, the terminal device 100 may perform corresponding processing or display a corresponding interface. For example, in response to the user clicking on the delete control, the terminal device 100 may delete the image of the pine tree magnified 60 times from the memory. In addition, Figure 4 The interface 15 shown may also display information such as the time when the current image was taken, which is not limited in this embodiment of the present application.

[0145] Figure 4 The image of the pine tree shown in the interface 15 is magnified 60 times compared to Figure 1 For the pine tree image with a magnification of 60 times shown in the video, the detail information is more complete, the image is clearer and of better quality, which can enhance the user's visual experience.

[0146] It should be noted that Figure 4 The image of the pine tree magnified 60 times shown in the figure is an enlarged effect diagram obtained by actual shooting. The enlarged effect diagram obtained by actual shooting can reflect the effect of the terminal device 100 magnifying some details of the pine tree.

[0147] It should also be noted that Figure 4 In the example, the image of the pine tree after being magnified 60 times is displayed in the middle of the interface 15, but the embodiments of the present application are not limited to this. For example, the image of the pine tree after being magnified 60 times fills the interface 15 display (or occupies the entire interface display).

[0148] After the terminal device 100 completes the shooting, the control 105 of the interface 14 may display the image of the scenery 1 that has been shot and magnified 60 times. After the terminal device 100 stores the image of the scenery 1 that has been magnified 60 times, the viewfinder 102 of the interface 14 may display the image captured in real time by the camera 101 ( Figure 3 (4) in the figure is not shown).

[0149] It should be understood that the above-mentioned interfaces and controls in each interface are only examples and do not constitute a limitation on the interfaces in the embodiments of the present application. In another embodiment of the present application, based on different application scenarios, each interface may include more or fewer controls than those in the above examples, and the position of each control may also be adjusted accordingly, and the present application does not impose specific limitations on them. For example, Figure 3 The control 103 and the control 104 shown in FIG. (2) can be displayed on the right side of the viewfinder 102, or on the bottom of the viewfinder 102. For example, Figure 3 As shown in Figure (4), after the shooting is completed, the control 107 will not be displayed in the interface.

[0150] It should also be understood that the above descriptions related to directions (such as right, left, and bottom, etc.) are based on Figure 3 The direction shown is described. When the direction of the terminal device 100 changes, the direction and position of each control can be automatically adjusted accordingly, which is not limited in the embodiments of the present application.

[0151] Based on the above description, when the terminal device includes multiple software modules, the image processing method of the embodiment of the present application can be implemented by multiple software modules.

[0152] See also Figure 5 , Figure 5 A schematic block diagram of the principles of an image processing method provided in an embodiment of the present application is shown.

[0153] like Figure 5 As shown, the terminal device 100 may include a collection module 301 , a pre-processing module 302 , a multimodal generative amplification model 303 , a post-processing module 304 and a language-to-text conversion module 305 .

[0154] When the user adjusts the magnification (for example, Figure 3Before the pinch operation shown in FIG. (2) in FIG. 1 ), the terminal device 100 can collect the preview image whose magnification is not adjusted through the collection module 301 (for example, Figure 3 (a preview image displayed in the viewfinder 102 shown in FIG. 2 ).

[0155] In response to the user's operation of adjusting the magnification, the terminal device 100 can collect the preview image after the magnification is adjusted through the collection module 301 (for example, Figure 3 (a preview image is displayed in the viewfinder 102 shown in FIG. (3)).

[0156] When the preview image after the magnification is adjusted is obtained, in response to the user's photographing operation (for example, the user Figure 3 (3) in the figure, the trigger operation of the control 106 is shown), the acquisition module 301 can input the preview image after the magnification is adjusted into the preprocessing module 302 for preprocessing to obtain the preprocessed image.

[0157] Optionally, in some embodiments, preprocessing is used to eliminate noise of the preview image after the magnification is adjusted, improve the contrast of the preview image, and correct the distortion of the preview image. The preprocessing may include one or more of the following: denoising processing, automatic white balance processing, demosaicing processing, image contrast processing, histogram equalization processing, and image enhancement processing.

[0158] By preprocessing the preview image after adjusting the magnification, the terminal device 100 can improve the quality of the preview image after adjusting the magnification (for example, improving the brightness of the preview image after adjusting the magnification and improving the contrast of the preview image after adjusting the magnification, etc.), thereby providing a good data basis for subsequent generative super-resolution upscaling processing and post-processing.

[0159] Optionally, in some embodiments, when a preprocessed image is obtained, the terminal device 100 may send the preprocessed image to the multimodal generative magnification model 303 and the language-to-text conversion module 305 respectively through the preprocessing module 302 .

[0160] Furthermore, the terminal device 100 may generate text description information corresponding to the preprocessed image based on the preprocessed image through the language conversion module 305 .

[0161] Optionally, in some embodiments, after obtaining the text description information corresponding to the preprocessed image, the terminal device 100 may send the text description information corresponding to the preprocessed image to the multimodal generative magnification model 303 through the language conversion module 305 .

[0162] It should be noted that, in the embodiment of the present application, the multimodal generative magnification model 303 and the language-to-text module 305 are set independently of each other for exemplary description, and the terminal device 100 can also integrate the multimodal generative magnification model 303 and the language-to-text module 305 into one module to implement the image processing method of the embodiment of the present application.

[0163] In addition, the terminal device 100 may also acquire a noise image pre-stored in the terminal device 100 through the acquisition module 306 , and send the acquired noise image to the multimodal generative upscaling model 303 .

[0164] Optionally, in some embodiments, after obtaining the preprocessed image, the text description information corresponding to the preprocessed image, and the noise image, the terminal device 100 can perform generative super-resolution upscaling on the preprocessed image based on the input preprocessed image, the text description information corresponding to the preprocessed image, and the noise image through the multimodal generative upscaling model 303 to obtain the generative super-resolution upscaling processed image. That is, under the guidance constraints of the text description information and the preprocessed image, the detail information in the preprocessed image is generated, so that the detail information of the generative super-resolution upscaling processed image is restored to a higher degree, the image is clearer, and the user's visual experience can be improved.

[0165] Optionally, in some embodiments, when the image after generative super-resolution upscaling is obtained, the terminal device 100 may send the image after generative super-resolution upscaling to the post-processing module 304 through the multimodal generative upscaling model 303. Thus, the terminal device 100 performs post-processing on the image after generative super-resolution upscaling through the post-processing module 304 to obtain the image after post-processing (for example, Figure 3 The image displayed in the control 105 shown in (4) in FIG. 1 , or Figure 4 The image shown). Post-processing the image after generative super-resolution upscaling can further improve the clarity of the image, thereby further improving the user's visual experience.

[0166] Optionally, in some embodiments, post-processing is used to further optimize the quality of the image after the generative super-resolution upscaling process. The terminal device 100 can achieve a specific visual effect for the image after the generative super-resolution upscaling process by post-processing the image after the generative super-resolution upscaling process and display it to the user.

[0167] Optionally, post-processing may include one or more of the following processes: sharpening, compression, and format conversion. Sharpening is used to increase the edges and details of an image. Compression is used to reduce the storage space occupied by an image without changing the clarity of the image. Format conversion is used to convert an image into a format such as jpg and store it in the memory of the terminal device 100, so that the user can view the captured image.

[0168] It should be noted that in response to the user's photo-taking operation, the terminal device 100 can directly display the image after generative super-resolution upscaling processing, that is, the image after post-processing, without displaying the image output by the preprocessing module 302 or the image output by the multimodal generative upscaling model 303, thereby providing a better user experience.

[0169] In summary, the image processing method in the embodiment of the present application inputs the three modal data sources of the preprocessed image, the text description information corresponding to the preprocessed image, and the noise image into the multimodal generative upscaling model 303, so that the multimodal generative upscaling model 303 can accurately generate the detail information in the preprocessed image under the guidance and constraints of the preprocessed image and the text description information corresponding to the preprocessed image, and obtain the generative super-resolution upscaling processed image, which can make the detail information of the generative super-resolution upscaling processed image clearer and improve the quality of the generative super-resolution upscaling processed image. Compared with Figure 2 The amplification module 12 performs a simple amplification process, and the amplification effect is better, the image obtained by the terminal device 100 is clearer, and the user's visual experience can be improved.

[0170] based on Figures 3 to 6 The following description is combined with Figure 6 , describes in detail the specific implementation process of an image processing method provided in an embodiment of the present application.

[0171] See also Figure 6 , Figure 6 A flowchart of an image processing method provided in an embodiment of the present application is shown.

[0172] like Figure 6 As shown, the image processing method in the embodiment of the present application includes the following steps:

[0173] S101, acquiring a first image.

[0174] Optionally, the first image is a low-resolution image to be super-resolution magnified. Figure 5 As shown, the first image is equivalent to Figure 5The image output by the preprocessing module 302. For example, in a photo shooting scene, in response to a user's photo shooting operation, a first image after preprocessing can be obtained.

[0175] Optionally, the resolution of the first image is a first resolution. Exemplarily, the first resolution may be 512 pixels*512 pixels.

[0176] Optionally, in some embodiments, the terminal device responds to the user's operation of turning on the camera (for example, the user Figure 3 The trigger operation of the camera 101 shown in (1) of FIG. 1 can display a first preview image at a first zoom factor. By displaying the first preview image, a data basis can be provided for subsequent acquisition of the first image.

[0177] Optionally, the first preview image is an original low-resolution image captured by the terminal device. Figure 5 As shown in FIG. 1 , the first preview image is equivalent to the preview image without adjusting the magnification captured by the capture module 301. For example, the first preview image can be as follows: Figure 3 The preview image is displayed in the viewfinder 102 shown in FIG.

[0178] It should be noted that the terminal device may also store the first preview image and perform subsequent processing based on the stored first preview image, which is not limited in the embodiment of the present application.

[0179] Optionally, the first zoom ratio is an actual magnification ratio of the terminal device that has not been adjusted. Exemplarily, the first zoom ratio can be Figure 3 The control 103 shown in FIG. 2 (2) shows the optical magnification of the short-focus lens.

[0180] It should be noted that the terminal device can also set the first zoom ratio to the optical magnification of the telephoto lens or other magnifications (for example, a digital zoom magnification of a specific value, etc.) according to actual usage requirements, and the embodiments of the present application are not limited to this.

[0181] When the first zoom ratio is adjusted to the second zoom ratio, the terminal device may display the second preview image. By displaying the second preview image, a data basis can be further provided for subsequent acquisition of the first image.

[0182] Optionally, the second zoom ratio is the actual magnification ratio adjusted by the terminal device. The second zoom ratio is greater than the first zoom ratio. Exemplarily, the second zoom ratio can be Figure 3 The control 107 shown in FIG. 3 ( 3 ) shows the current actual magnification.

[0183] Optionally, the second preview image is an image after the terminal device adjusts the magnification. Figure 5 As shown, the second preview image is equivalent to Figure 5 The second preview image can be a preview image after the magnification is adjusted and captured by the capture module 301 in FIG. Figure 3 The preview image is displayed in the viewfinder 102 shown in FIG.

[0184] In some embodiments, the terminal device responds to the user's operation of adjusting the magnification (eg, Figure 3 By performing a pinch operation as shown in (2) in FIG. 1 , the first zoom ratio can be adjusted to a second zoom ratio.

[0185] In another embodiment, the terminal device may also adjust the first zoom ratio to the second zoom ratio in response to the user's operation of switching the shooting mode.

[0186] It should be noted that in addition to the above two methods, the terminal device can also detect the distance between the content to be photographed (such as scenery, buildings or people, etc.) and the terminal device. When it is detected that the distance between the content to be photographed and the terminal device exceeds the processing distance of the short-focus lens, the terminal device automatically adjusts the first zoom ratio to a larger second zoom ratio. The embodiments of the present application are not limited to this.

[0187] Optionally, in some embodiments, when the terminal device receives the photo taking instruction, it can pre-process the second preview image (equivalent to Figure 5 By preprocessing the second preview image, a first image with better image brightness and image contrast can be obtained, thereby providing a good data basis for subsequent generative super-resolution upscaling processing.

[0188] Exemplarily, the terminal device responds to the user's photo-taking operation (for example, the user Figure 3 The trigger operation of the control 106 shown in (3) in the figure can receive a photo taking instruction.

[0189] It should also be noted that the terminal device can obtain the first image through a variety of possible implementation methods, which are not limited in the embodiments of the present application. For example, the terminal device can obtain the first image pre-stored from the memory of the terminal device. For another example, the terminal device can obtain the first image by intercepting an image during the video recording process.

[0190] In summary, the terminal device can obtain the first image, thereby providing a good data basis for subsequent processing of the multimodal generative upscaling model.

[0191] S102: Generate text description information of the first image based on the first image.

[0192] Optionally, in some embodiments, text description information is used to describe the first image. The text description information includes at least first-category information and second-category information. The first-category information may be system information related to the first image acquired by the terminal device. The system information related to the first image refers to other attribute information (e.g., time attributes and location attributes, etc.) other than the content contained in the first image. The second-category information may be high-order semantic feature information of the first image (e.g., color information and texture information of the content contained in the first image, etc.).

[0193] It should be noted that the embodiment of the present application does not limit the language type of the text description information. The voice type may depend on the user's settings. The text description information may be described in languages ​​including but not limited to Chinese, English, Korean, Russian, and Japanese. For example, the text description information may be the pine trees in Huangshan, Anhui on May 3.

[0194] Optionally, the first type of information may include the time corresponding to the first image and / or the location corresponding to the first image. The time corresponding to the first image may be expressed in one or more of the following forms: date (e.g., 20230503), week (e.g., Thursday), season (e.g., spring), and time (e.g., 8:00), etc.

[0195] It should be noted that the time corresponding to the first image may be the shooting time of the first image, the time when the user takes a screenshot of the first image, the time when the user edits and saves the first image, or the time currently displayed by the terminal device 100, etc. The embodiment of the present application is not limited to this.

[0196] Optionally, the place corresponding to the first image may be expressed in one or more of the following forms: geographic coordinates (e.g., longitude 122°18′, latitude 36°57′), administrative regions (e.g., Beijing), landmarks and buildings (e.g., the Forbidden City), electronic fences, and locations marked on maps, etc. Exemplarily, in a shooting scenario, the place corresponding to the first image may be the shooting location.

[0197] It should be noted that the location corresponding to the first image can be determined by the Beidou navigation system or the global positioning system (GPS), and the embodiments of the present application are not limited thereto. For example, a variety of sensors (including electronic compasses, accelerometers, and gyroscopes, etc.) provided in the terminal device can determine the location corresponding to the first image in combination with wireless network signals.

[0198] Optionally, the second type of information is used to characterize the category to which the content included in the first image belongs. Optionally, the category to which the content included in the first image belongs may include flowers, plants, and trees (e.g., pine trees, peonies, camphor trees, and azaleas), buildings (e.g., the Forbidden City and Big Wild Goose Pagoda), animals (e.g., cats and dogs), people (e.g., children and the elderly), and vehicles (e.g., cars, trucks, vans, and bicycles).

[0199] After executing S101, the terminal device can obtain the first image. After obtaining the first image, the terminal device can generate text description information of the first image (equivalent to Figure 5 By generating text description information of the first image, a data basis can be provided for subsequent processing of the multimodal generative magnification model.

[0200] S103, obtaining a noise image.

[0201] It should be noted that there is no sequence or order between S102 and S103, that is, S102 and S103 can be executed simultaneously or in sequence, depending on the actual application scenario. For example, S102 can be executed first and S103 can be executed later; for another example, S102 can be executed later and S103 can be executed first, which is not limited in the embodiments of the present application.

[0202] Optionally, in some embodiments, the noise image is used to provide a noise distribution feature for the first image. The noise image may be an image pre-stored in the terminal device. Exemplarily, the noise image may be a random Gaussian noise image or a Gaussian white noise image. The resolution of the noise image may be 512 pixels*512 pixels.

[0203] Optionally, in another embodiment, the noise image may also be an image generated in the terminal device according to a pre-set software code, which is not limited in this embodiment of the present application.

[0204] Optionally, in some embodiments, the terminal device may Figure 5 The acquisition module 306 shown acquires the pre-stored noise image from the memory of the terminal device. By acquiring the noise image, a data basis can be provided for the subsequent processing of the multimodal generative upscaling model, which can ensure the normal operation of the multimodal generative upscaling model, and can increase the diversity of the generative super-resolution upscaling processing of the first image by the multimodal generative upscaling model, so that the detail information (such as the length and direction of the branches of the pine tree) in the first image generated by the multimodal generative upscaling model is more diverse.

[0205] S104, input the first image, text description information and the noise image into a multimodal generative upscaling model, and output a second image.

[0206] Optionally, a multimodal generative upscaling model (e.g. Figure 5 ) is a pre-trained model stored in the terminal device. Multimodality refers to two or more different types of imaging technologies or data sources (for example, the first image, text description information and noise image).

[0207] Optionally, in some embodiments, the resolution of the second image is a second resolution, and the second resolution is greater than the first resolution. Exemplarily, the second resolution may be 4096 pixels*4096 pixels.

[0208] After executing S101 to S103, the terminal device can input the obtained first image, text description information and noise image into a pre-trained multimodal generative upscaling model for processing, and output the processed second image (equivalent to Figure 5 The multimodal generative magnification model 303 shown in FIG. 3 performs the processing). Thus, the terminal device can obtain the second image at the second zoom factor.

[0209] It should be noted that the multimodal generative upscaling model in the terminal device can use the same noise image to perform generative super-resolution upscaling on the first image containing different contents, or can use different noise images to perform generative super-resolution upscaling on the first image containing different contents. The embodiments of the present application are not limited to this.

[0210] Optionally, in some embodiments, in response to the user's photo-taking operation, the terminal device may display the second image. Before displaying the second image, the terminal device may also post-process the second image (equivalent to Figure 5 The post-processing module 304 shown in FIG. 3 performs the processing to obtain a second image after post-processing. The second image after post-processing has a higher degree of detail restoration and better quality, which can improve the user's visual experience.

[0211] Optionally, in some embodiments, the terminal device may store the second image after post-processing. Exemplarily, the second image after post-processing may be as follows: Figure 3 The image of the pine tree in the control 105 shown in FIG. (4) is magnified 60 times. Figure 3 As shown in FIG. (4), in response to the user's triggering operation on the control 105 in the interface 14, the terminal device can display the following Figure 4 The image shown is for user convenience.

[0212] It should be noted that the image processing method of the embodiment of the present application is a process performed between the time when the camera of the terminal device starts to capture the image and the time when the image capture is completed. In addition, there is a certain delay between the time when the camera of the terminal device starts to capture the image and the time when the image capture is completed. Therefore, the first image acquired by the terminal device is only used as the data basis for the terminal device to perform super-resolution magnification processing, and is not the final image displayed to the user. The image that the terminal device finally displays to the user is the second image after post-processing, so as to enhance the user's experience.

[0213] It should also be noted that the image processing method of the embodiment of the present application can be used in the photo mode of the camera 101 to obtain a clear second image. When the user needs to compare the effects, the shooting mode can be switched to the professional mode to take photos. Since the image processing method of the embodiment of the present application is not used in the professional mode, the obtained image is relatively blurred.

[0214] The image processing method in the embodiment of the present application can obtain an image to be magnified by acquiring a first image, providing a data basis for subsequent super-resolution magnification processing. Then, based on the obtained first image, text description information of the first image is generated, which can provide a data basis for subsequent generative super-resolution magnification processing. Obtaining a noise image can provide noise distribution characteristics for the first image. Furthermore, after obtaining the first image, the text description information and the noise image, the first image, the text description information and the noise image can be input into a multimodal generative magnification model, and the first image can be generatively super-resolution magnification processed based on the input data of the three modes of the first image, the text description information and the noise image through the multimodal generative magnification model, so that the detail information of the enlarged image can be better generated, and a second image with a higher degree of detail restoration can be obtained. In other words, after the first image is generatively super-resolution magnified by the multimodal generative magnification model under the guidance and constraints of the first image and the text description information, an image with a higher degree of detail restoration is obtained. Thus, it is possible to implement a generative super-resolution upscaling process on a low-resolution (e.g., first resolution) image (e.g., first image) through a multimodal generative upscaling model, and output a high-resolution (e.g., second resolution) image (e.g., second image). Moreover, the output high-resolution image has a higher degree of detail restoration, a higher image clarity, a better image quality, and a better visual experience for users.

[0215] It should be noted that the multimodal generative upscaling model in the image processing method of the embodiment of the present application can rely on the terminal device to process the image in real time, or the image processing method of the embodiment of the present application can be independently implemented in the form of a model to achieve offline processing of the image. For example, the first image, text description information and a noise image are input into an independent multimodal generative model, and the second image is directly output.

[0216] Based on the description in S102, the terminal device in the embodiment of the present application may adopt a variety of possible implementation methods to generate text description information. Figures 7 to 10 , a possible implementation method for generating text description information in an embodiment of the present application is introduced in detail. Figure 7 The method can be applied to Figure 5 In the language conversion module 305 shown.

[0217] See also Figure 7 , Figure 7 A flowchart of an image processing method provided in an embodiment of the present application is shown.

[0218] like Figure 7 As shown, the image processing method in the embodiment of the present application includes the following steps:

[0219] S201, obtaining first type of information.

[0220] Optionally, in some embodiments, after executing S101, the terminal device 100 may Figure 5 The language conversion module 305 shown is used to obtain system information related to the first image to obtain the first type of information.

[0221] Considering that the first category information has various forms of expression, after obtaining the first category information, the terminal device can convert the first category information through the language conversion module 305, and the first category information after conversion is expressed in text. By converting the first category information in other forms (such as numbers or symbols, etc.) into the first category information in text form, a data basis can be provided for the generation of text description information, so that the generated text description information is more accurate.

[0222] Optionally, when the first category of information includes the time corresponding to the first image, and the expression form of the time corresponding to the first image is not in textual form, the terminal device can convert the time corresponding to the first image in other forms of expression into time in textual form through the index algorithm in the language-to-text conversion module 305 (for example, a hash algorithm or a bitmap index, etc.).

[0223] Exemplarily, the time corresponding to the first image may be a shooting date of the first image, and the shooting date of the first image is 20230503 in digital form. Figure 8 The text conversion process of the time corresponding to the first image is shown. Figure 8As shown, the terminal device matches the shooting date of the first image 20230503 in digital form obtained with the date text description in the index list pre-stored in the terminal device through the language conversion module 305. The date text description in the index list includes at least the month and the day. That is, the terminal device can match 05 in 20230503 with May in the index list, and match 03 in 20230503 with the 3rd in the index list through the language conversion module 305, and can obtain that the shooting date of the first image is May 3rd.

[0224] Optionally, when the first category of information includes a location corresponding to the first image, and the expression form of the location corresponding to the first image is not in textual form, the terminal device can convert the location corresponding to the first image in other expressions into a location in textual form by calling the coordinate-to-text call function in the language-to-text conversion module 305.

[0225] Exemplarily, the location corresponding to the first image may be a geographical coordinate determined by a global positioning system in the terminal device, and the geographical coordinate may be a longitude of 122°18′ and a latitude of 36°57′.

[0226] Optionally, the coordinate-to-text calling function can be:

[0227] String location Address1=Location And Geocoder Util.get Instance(this).get Address By Location(location1);

[0228] Among them, location Address1 is used to represent the location in text form corresponding to the first image; Location And Geocoder Util is a class used to handle tasks related to geographic location and address coding; getInstance(this) means returning an instance of the Location And Geocoder Util class; get Address ByLocation(location1) is to query and return the address string corresponding to the geographic coordinates of location1.

[0229] For example, when the longitude of location1 is 122°18′ and the latitude is 36°57′, the terminal device converts the longitude and latitude by calling the coordinate-to-text call function in the language-to-text conversion module, and can obtain that the location corresponding to the first image is Huangshan, Anhui.

[0230] It should be noted that the above description is based on the example that the index algorithm and the coordinate-to-text call function are integrated in the language-to-text module 305, and the embodiments of the present application are not limited thereto. For example, the index algorithm and the coordinate-to-text call function can also be independently set in the terminal device, and the language-to-text module 305 implements the corresponding processing by calling the index algorithm and the coordinate-to-text call function set in the terminal device.

[0231] In summary, the terminal device can obtain the first type of information. Obtaining the first type of information can provide a data basis for the generation of subsequent text description information.

[0232] S202, input the first image into the classification model, and output the second category information.

[0233] It should be noted that there is no time sequence or order between S201 and S202. That is, S201 and S202 can be executed simultaneously, or in the order of S201 first and S202 later, or in the order of S201 later and S202 first, and the present embodiment of the application does not limit this.

[0234] Optionally, in some embodiments, the language conversion module 305 includes a pre-trained classification model.

[0235] The terminal device can input the obtained first image into a pre-trained classification model in the language-to-text conversion module, and output the second type of information, so as to obtain the category to which the content included in the first image belongs.

[0236] It should be noted that the classification model in the embodiment of the present application can be obtained by training the original classification model on the unprocessed low-resolution sample images and the corresponding category labeling results, or by training the original classification model on the preprocessed low-resolution sample images and the corresponding category labeling results. The embodiment of the present application is not limited to this.

[0237] It should also be noted that the difference between the above two training methods is that the preprocessed low-resolution sample images have better brightness and contrast than the unpreprocessed low-resolution sample images. Therefore, the classification effect of the classification model obtained by training the preprocessed low-resolution sample images and the corresponding category labeling results with the original classification model is more accurate.

[0238] Optionally, in some embodiments, Fig. 9As shown, the classification model includes at least an encoder layer and a fully connected layer. The terminal device can input the first image into the encoder layer of the classification model to obtain the sixth feature information. In other words, the terminal device can perform feature encoding processing on the first image through the encoder layer to obtain the sixth feature information. After obtaining the sixth feature information, the terminal device can input the sixth feature information into the fully connected layer to obtain the second category of information. In other words, the terminal device can classify the sixth feature information through the fully connected layer to obtain the second category of information.

[0239] It should be noted that the network structure of the original classification model adopts the same network structure as the classification model. That is, the original classification model also includes an encoder layer and a fully connected layer.

[0240] S203, fusing the first type of information and the second type of information to obtain text description information.

[0241] Optionally, in some embodiments, after executing S201 and S202, the terminal device may Figure 5 The language conversion module 305 shown performs a fusion process on the first type of information and the second type of information, thereby obtaining text description information, which can prepare for the subsequent processing of the multimodal generative amplification model 303.

[0242] Optionally, in some embodiments, Fig.10 As shown, the terminal device can also input the first image and the first type of information into the language conversion module 305, and output the text description information of the first image. Exemplarily, when the input first image is a low-resolution pine tree image, the time corresponding to the input first image is 20230503, and the location corresponding to the input first image is 122°18′ longitude and 36°57′ latitude, the terminal device outputs the text description information of the pine tree in Huangshan, Anhui on May 3 through the language conversion module 305.

[0243] Based on the description of S104, the multimodal generative amplification model in the embodiment of the present application can adopt a variety of network structures. Fig.11 , a network structure of a multimodal generative amplification model in an embodiment of the present application is introduced in detail.

[0244] like Fig.11 As shown, the multimodal generative upscaling model in the embodiment of the present application may include a denoising network and a super-resolution upscaling network. The terminal device may input the first image, the text description information, and the noisy image into the denoising network in the multimodal generative upscaling model for denoising, and then input the denoised intermediate image into the super-resolution upscaling network in the multimodal generative upscaling model for super-resolution upscaling to obtain a second image.

[0245] Optionally, in some embodiments, the denoising network can adopt any of the following diffusion models: a basic diffusion model (stable diffusion, SD), a deep learning-based diffusion model (differentiable predictive coding, DDPM), a stochastic differential equations-based diffusion model (stochastic differential equations, SDE), a score-based generative model (score based generative model, SGM) and a diffusion model based on a convolutional neural network architecture (u-shaped convolutional networks, U-NET).

[0246] It should be noted that the above diffusion model is only for illustrative purposes. In practical applications, those skilled in the art may also adopt other diffusion models based on actual needs, and the embodiments of the present application are not limited to this.

[0247] like Fig.12 As shown in the figure, the process of diffusion model from XT→X0 is the denoising process, and the process of diffusion model from X0→XT is the denoising process. Among them, XT represents the original noise image in the denoising process; XT-1 represents the intermediate denoised image when the time step is T-1; X1 represents the intermediate denoised image when the time step is 2; X0 represents the denoised image when the time step is 1; T-1 represents the total number of time steps, that is, the number of denoising steps that the diffusion model needs to go through from XT→X0.

[0248] It should be noted that the above total number of time steps determines the level of detail of the denoising process. The degree of denoising performed in each step is different, and during the denoising process, as the number of time steps increases, the degree of denoising gradually decreases, which can ensure that the quality of the denoised image after denoising is higher.

[0249] It should also be noted that the specific network structure of the diffusion model can be found in the description of the relevant technology and will not be described here.

[0250] Considering that the denoising process of the diffusion model requires a total time step of T-1 to obtain a denoised image with a higher quality, a preset iteration threshold is set in the multimodal generative upscaling model in the embodiment of the present application, and the multimodal generative upscaling model is used to perform multiple iterations to ensure that a second image with a higher quality is obtained.

[0251] It should be noted that the preset iteration threshold can be set by those skilled in the art according to actual application requirements, and the embodiments of the present application do not limit this. For example, the preset iteration threshold can be set to 1000.

[0252] based on Fig.11 and Fig.12 The following description is combined with Fig.13 , describes in detail the specific process of the terminal device outputting the second image through the multimodal generative upscaling model when the denoising network adopts the diffusion model. Optionally, in some embodiments, Fig.13 A schematic diagram of the application of the multimodal generative amplification model is shown, Fig.13 As shown, the terminal device can input the first image, text description information and noise image into the multimodal generative amplification model to perform N iterations to obtain a second image, where N is a positive integer greater than or equal to 1 and less than or equal to a preset iteration threshold.

[0253] The terminal device can determine the current number of iterations (or the current number of iteration steps) through the multimodal generative amplification model.

[0254] It should be noted that different numbers of iterations in the embodiment of the present application correspond to different degrees of denoising processing. Therefore, the terminal device can determine the degree of denoising processing performed in the current number of iterations by determining the current number of iterations.

[0255] After determining the current number of iterations N, the terminal device may determine whether the current number of iterations N is equal to 1. When it is determined that the current number of iterations N is equal to 1, the terminal device may perform the first iteration.

[0256] When executing the first iteration, if Fig.13 As shown, the terminal device can input the first image, the noisy image and the text description information into the denoising network, and the denoising network can perform denoising on the input noisy image based on the input text description information and the input first image to obtain a fourth image.

[0257] After obtaining the fourth image, the terminal device can input the fourth image into the super-resolution amplification network, output the fifth image, and perform super-resolution amplification processing on the fourth image from the denoising network through the super-resolution amplification network to obtain the fifth image. Exemplarily, the resolution of the fifth image can be 1024 pixels * 1024 pixels.

[0258] When the current number of iterations N is equal to 1, Fig.13 As shown, the fifth image is an intermediate image after the multimodal generative upscaling model performs a denoising process and a super-resolution upscaling process on the noisy image under the guidance of the first image and text description information.

[0259] When the current number of iterations N is less than the preset iteration threshold, the Nth iteration is performed, and the terminal device can input the fifth image, the text description information and the first image into the denoising network to obtain the fifth image after denoising, and input the fifth image after denoising into the super-resolution enlargement network to obtain the fifth image after super-resolution enlargement, and use the fifth image after super-resolution enlargement, the text description information and the first image together as the input of the next iteration until the current number of iterations N is equal to the preset iteration threshold.

[0260] Exemplarily, when the current number of iterations N is 2, the terminal device may perform the second iteration. When performing the second iteration, the terminal device may input the first image, the fifth image, and the text description information into the denoising network to obtain the fifth image after denoising. Further, the terminal device may input the fifth image after denoising into the super-resolution up-scaling network to obtain the fifth image after super-resolution up-scaling. The fifth image after super-resolution up-scaling, the first image, and the text description information are used as inputs for the third iteration.

[0261] It should be noted that, when the current number of iterations N is less than the preset iteration threshold, each iteration process is similar. Please refer to the relevant description of the second iteration above, which will not be repeated here.

[0262] like Fig.13 As shown, when the current number of iterations N is equal to the preset iteration threshold, the terminal device can output the fifth image after super-resolution upscaling as a second image through a multimodal generative upscaling model, and can obtain a second image with a higher degree of detail restoration, thereby improving the clarity of the image after super-resolution upscaling.

[0263] Combine the following Fig.14 , which introduces in detail the specific implementation process of an image processing method in an embodiment of the present application.

[0264] like Fig.14 As shown, the multimodal generative amplification model in the embodiment of the present application may include a second encoder, a third encoder, a fourth encoder, a denoising network and a super-resolution amplification network. The super-resolution amplification network includes at least one convolutional layer. The denoising network includes a first encoder, a first decoder and an attention network. Among them, the output of the second encoder, the output of the third encoder and the output of the fourth encoder are used as inputs to the denoising network, and the output of the denoising network is used as the output of the super-resolution amplification network. In the denoising network, data can be exchanged between the first encoder and the attention network, and the output of the first encoder is used as the input of the first decoder. The output of the first decoder is used as the input of the super-resolution amplification network.

[0265] It should be noted that Fig.14The attention network shown in is only for illustrative purposes, and only the attention network corresponding to the first encoder is shown in the embodiment of the present application. In practical applications, those skilled in the art can also set the attention network corresponding to the first decoder according to usage requirements, and the embodiment of the present application does not limit this. In addition, the embodiment of the present application does not limit the number and specific network structure of the attention networks. For example, the number of attention networks can be set according to the number of layers of the first encoder. Alternatively, the attention network can also be set according to the number of layers of the first decoder. The specific network structure of the attention network can be found in the description of the relevant technology, which will not be repeated here.

[0266] In the case where an attention network corresponding to the first decoder is also provided in the denoising network, Fig.14 The first feature information, the second feature information and the fourth feature information also need to be input into the attention network corresponding to the first decoder for feature fusion processing, and the fused feature information is input into the first decoder.

[0267] It should also be noted that Fig.14 The attention network shown is used to fuse multiple feature information. In practical applications, according to the use requirements, the terminal device can also fuse two feature information through the attention network. For example, the terminal device fuses the first feature information and the third feature information through the attention network 1, and fuses the second feature information and the third feature information through the attention network 2, and inputs the feature information fused by the attention network 1 and the feature information fused by the attention network 2 into the first encoder for subsequent processing.

[0268] like Fig.14 As shown, an image processing method according to an embodiment of the present application specifically includes the following steps:

[0269] S301, input a first image into a second encoder to obtain first feature information.

[0270] Optionally, the second encoder is used to perform feature encoding processing on the first image. Exemplarily, the second encoder may be an image encoder. The first feature information is used to characterize image features corresponding to the first image.

[0271] S302: Input the text description information into a third encoder to obtain second feature information.

[0272] Optionally, the third encoder is used to perform feature encoding processing on the text description information. Exemplarily, the third encoder can be a text encoder. The second feature information is used to characterize text features corresponding to the text description information.

[0273] It should be noted that the specific network architecture of the third encoder mentioned above can refer to the specific network architecture of the text encoder described in the relevant technology, which will not be repeated here.

[0274] Optionally, in some embodiments, the third encoder may also be a feature encoder. In this case, any form of the first type of information may be directly input into the feature encoder, and the corresponding feature information may be output. That is, when the third encoder is a feature encoder, Figure 5 The language-to-text conversion module 305 in the terminal device shown is an optional module.

[0275] It should be noted that the specific network architecture of the above-mentioned feature encoder can be found in the description of the relevant technology and will not be repeated here.

[0276] S303: Input the noise image to the first encoder to obtain third feature information.

[0277] It should be noted that there is no time sequence or order between S301, S302 and S303. That is to say, S301, S302 and S303 can be executed simultaneously, or in other order, for example, in the order of S301 in the middle, S302 in the front, and S303 in the back, or in the order of S301 in the front, S302 in the back, and S303 in the middle.

[0278] Optionally, the first encoder is used to perform feature encoding processing on the noise image. Exemplarily, the first encoder may be an image encoder. The third feature information is used to characterize noise features corresponding to the noise image.

[0279] It should be noted that the first encoder and the second encoder may use the same image encoder or different image encoders, which is not limited in the embodiment of the present application.

[0280] Optionally, the image encoder may include a first encoding layer (encoder block), a second encoding layer, a third encoding layer, and a fourth encoding layer from top to bottom. Exemplarily, the first encoding layer may output 4 feature maps of size 64*64. The second encoding layer may output 4 feature maps of size 32*32. The third encoding layer may output 4 feature maps of size 16*16. The fourth encoding layer may output 4 feature maps of size 8*8.

[0281] S304, input the current iteration number to the fourth encoder to obtain fourth feature information.

[0282] Optionally, the fourth encoder is used to perform feature encoding processing on the current iteration number. Exemplarily, the fourth encoder may be a step number encoder. The fourth feature information is used to characterize the number of features corresponding to the current (eg, first) iteration.

[0283] It should be noted that the specific network architecture of the fourth encoder mentioned above can refer to the specific network architecture of the step encoder described in the related art, which will not be described in detail here.

[0284] S305, when executing the first iteration, the first feature information, the second feature information, the third feature information and the fourth feature information are input into the attention network to obtain the fifth feature information.

[0285] Optionally, the attention network is used to perform feature fusion processing. The fifth feature information is used to characterize the features after the attention network fuses multiple features.

[0286] It should be noted that S303 is an optional step. In some embodiments, the terminal device can directly fuse the first feature information, the second feature information and the fourth feature information through the attention network, and continue to execute S306 to S309, which can also implement the image processing method of the embodiment of the present application.

[0287] S306, input the noise image and the fifth feature information into a first encoder, and output a first encoded feature map.

[0288] Optionally, the first encoder is used to perform feature encoding processing on the noise image and the fifth feature information.

[0289] By executing S305 and S306, the terminal device can fuse the third feature information corresponding to the noise image together, so that the fusion effect of the attention network is better.

[0290] S307, input the first encoding feature map to the first decoder, and output a fourth image.

[0291] Optionally, the first decoder is used to perform feature decoding processing on the first encoded feature map. Exemplarily, the first decoder may be an image decoder.

[0292] Optionally, the image decoder may include a first decoding layer (decoder block), a second decoding layer, a third decoding layer, and a fourth decoding layer from top to bottom. Exemplarily, the first decoding layer may output 4 feature maps of size 8*8. The second decoding layer may output 4 feature maps of size 16*16. The third decoding layer may output 4 feature maps of size 32*32. The fourth decoding layer may output 4 feature maps of size 64*64.

[0293] It should be noted that the number of decoding layers in the image decoder and the number of encoding layers in the image encoder may be the same or different, and this is not limited in the embodiments of the present application.

[0294] S308: Input the fourth image into at least one convolution layer, and perform super-resolution upscaling processing on the fourth image based on a preset upscaling factor through the at least one convolution layer to obtain a fifth image.

[0295] It should be noted that, in practical applications, in order to improve the clarity of the image, the super-resolution amplification network may also include an upsampling layer, which is not limited in the embodiments of the present application. The upsampling layer can restore the spatial resolution of the image from the feature map output by the convolution layer through the upsampling operation of pixel shuffle, thereby further improving the clarity of the image.

[0296] It should also be noted that the preset magnification ratio may be a preset magnification ratio of a certain iteration, or the preset magnification ratio may be the product of magnification ratios of at least two iterations. When the actual magnification ratio of the image to be captured exceeds the preset magnification ratio, the terminal device may combine digital zoom to achieve further magnification of the image to be captured.

[0297] S309: When the current number of iterations N is equal to a preset iteration threshold, the fifth image after the super-resolution upscaling process is output as the second image.

[0298] It should be noted that the above steps are described using the first iteration as an example, and S305 to S308 are specific implementation steps for the first iteration. When the current number of iterations N is less than the preset iteration threshold, the specific implementation process of the Nth iteration is similar to S305 to S308. Please refer to the description of S305 to S308, which will not be repeated here.

[0299] In summary, the embodiment of the present application inputs the data sources of three modes, namely, the first image, the noisy image and the text description information, into the multimodal generative upscaling model. Under the guidance and constraints of the first image and the text description information, the multimodal generative upscaling model can perform generative super-resolution upscaling on the first image, output a second image with more complete detail expansion and higher clarity, thereby improving the user's visual experience.

[0300] Combination of the above Figures 3 to 14 The specific implementation process of the image processing method of the embodiment of the present application is described in detail. Fig.15 , introduces a possible implementation method of obtaining a multimodal generative amplification model in an embodiment of the present application.

[0301] The multimodal generative amplification model in the embodiment of the present application is obtained by the following training method:

[0302] Optionally, in some embodiments, the terminal device may acquire a first sample image, a second sample image, text description sample information, and a third sample image.

[0303] Optionally, the first sample image and the third sample image are images with the same content but different resolutions, and the resolution of the first sample image is smaller than that of the third sample image. The second sample image is a noise image used to increase the magnification diversity of the first sample image. The text description sample information is generated based on the first sample image.

[0304] It should be noted that the first sample image and the third sample image may be acquired by a terminal device or other acquisition devices (eg, a SLR camera), and this embodiment of the present application does not limit this.

[0305] like Fig.15 As shown, the terminal device can input the first sample image, the second sample image, the text description sample information and the third sample image into the original multimodal generative upscaling model for training until the original multimodal generative upscaling model meets the training iteration conditions, and determine the original multimodal generative upscaling model that meets the training iteration conditions as the multimodal generative upscaling model.

[0306] Optionally, in some embodiments, the training iteration condition may be a training iteration number threshold or a loss threshold, etc.

[0307] It should be noted that, in the actual training process, technical personnel in this field can set corresponding training iteration conditions according to actual conditions to ensure that a multimodal generative amplification model with better effects is obtained.

[0308] It should also be noted that Fig.15 The network structure of the original multimodal generative upscaling model shown can be adopted Fig.14 The network structure of the multimodal generative amplification model shown is used for training.

[0309] The above describes in detail the specific implementation process of the image processing method in the embodiment of the present application. In practical applications, the image processing method in the embodiment of the present application can be implemented by relying on the hardware system and software system of the terminal device. Fig.16 and Fig.17 , respectively introduce the hardware system and software system of a terminal device in an embodiment of the present application.

[0310] See also Fig.16 , Fig.16 A hardware system schematic diagram of a terminal device provided in an embodiment of the present application is shown.

[0311] like Fig.16As shown, the terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L and a bone conduction sensor 180M, etc.

[0312] It should be noted that Fig.16 The structure shown does not constitute a specific limitation on the hardware system of the terminal device 100. In other embodiments of the present application, the hardware system of the terminal device 100 may include: Fig.16 The components shown are more or less than the components shown, or the hardware system of the terminal device 100 may include Fig.16 A combination of some of the components shown, or a hardware system of the terminal device 100 may include Fig.16 Subcomponents of some of the components shown. For example, Fig.16 The illustrated proximity light sensor 180G may be optional. Fig.16 The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0313] The processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and a neural-network processing unit (NPU). Different processing units may be independent devices or integrated devices.

[0314] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0315] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0316] Fig.16 The connection relationship between the modules shown is only a schematic illustration and does not constitute a limitation on the connection relationship between the modules of the terminal device 100. Optionally, the modules of the terminal device 100 may also adopt a combination of multiple connection modes in the above embodiments.

[0317] The terminal device 100 can realize the display function through the GPU, the display screen 194 and the application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0318] The display screen 194 can be used to display images or videos. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini light-emitting diode (Mini LED), a micro light-emitting diode (Micro LED), a micro OLED (Micro OLED) or a quantum dot light emitting diode (QLED). In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1. In some embodiments, the unfolded state of the display screen 194 can be as follows: Figure 3 Any of the 4 states shown in .

[0319] The terminal device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0320] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can perform algorithm optimization on the noise, brightness and color of the image. The ISP can also optimize the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0321] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB or YUV. In some embodiments, the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0322] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0323] The terminal device 100 can implement audio functions, such as music playback and recording, through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor.

[0324] The touch sensor 180K is also called a touch control device. The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, which is also called a touch control screen. The touch sensor 180K is used to detect a touch operation acting on or near it. The touch sensor 180K can pass the detected touch operation to the application processor to determine the type of touch event. In other embodiments, the touch sensor 180K can also be set on the surface of the terminal device 100 and set at a different position from the display screen 194.

[0325] Optionally, in some embodiments, the processor 110 is used to input the acquired low-resolution first image, the noise image, and the text description information into the multimodal generative upscaling model, and the multimodal generative upscaling model performs generative super-resolution upscaling processing on the low-resolution first image under the guidance of the low-resolution first image and the text description information, so as to generate image detail information, obtain a high-resolution second image, and convert the second image (such as Figure 4 The image is displayed to the user (as shown), which can enhance the user's visual experience.

[0326] The above describes in detail the hardware system of the terminal device 100, and the following introduces the software system of the terminal device 100. The software system can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the layered architecture as an example to exemplarily describe the software system of the terminal device 100.

[0327] See also Fig.17 , Fig.17 A software system schematic diagram of a terminal device provided in an embodiment of the present application is shown.

[0328] The software system of the terminal device 100 can be divided into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, Fig.17 As shown, the software system of the terminal device 100 can be an Android system architecture, and the Android system architecture can be divided into four layers, from top to bottom, namely, an application layer (application, APP), an application framework layer (application framework), a system runtime layer and a kernel layer (linux kernel).

[0329] The application layer may include multiple applications. For example, the multiple applications may be an audio application, a settings application, a camera application, a calendar application, and an album application, etc. It is understood that the multiple applications in the application layer may be third-party applications installed by the user or system applications.

[0330] like Fig.17 As shown in FIG, the camera application may also include a multimodal generative zoom model and a language conversion module. The multimodal generative zoom model and the language conversion module can be found in Figure 5 The camera application can also include Figure 5 Other modules shown, such as post-processing modules, etc. ( Fig.17 not shown).

[0331] The application framework layer is a framework layer for supporting the operation of multiple applications in the application layer. In some embodiments, the application framework layer may include a binary archive file (android archive, AAR) of an Android library project.

[0332] The system runtime layer may include graphics hardware, which is used to abstract hardware. Graphics hardware may include an external library and a hardware abstraction layer library (HAL library).

[0333] The kernel layer is an abstract layer between hardware and software. Exemplarily, the kernel layer may include driver modules such as audio driver, display driver, hardware interface driver (eg, headphone interface driver) and sensor driver.

[0334] The image processing method of the embodiment of the present application can be executed in the application layer. Specifically, the image processing method of the embodiment of the present application can be executed by a camera application in the application layer.

[0335] In some embodiments, in the shooting mode, the camera application in the terminal device 100 can perform generative super-resolution magnification processing on the first image under the guidance of the first image and text description information through the multimodal generative magnification model in the camera application, so as to output a second image with a higher degree of detail information restoration, so that the user can use the controls in the album application or the camera application (such as Figure 3 (3) in the figure to view a second image with higher definition (such as Figure 4 shown).

[0336] It should be understood that Fig.17 The hierarchical structure shown does not constitute a specific limitation on the software system of the terminal device 100. In other embodiments of the present application, the software system of the terminal device 100 may include: Fig.17 The layered architecture shown may be more or less than the architecture shown, or each layer of the software system of the terminal device 100 may include more than Fig.17 The component structures shown may be more or less structures, and the embodiments of the present application are not limited thereto.

[0337] Exemplarily, the present application provides a computer-readable storage medium, which includes instructions. When the instructions are executed on the terminal device 100, the terminal device 100 implements the method in the above embodiment when executing.

[0338] Exemplarily, the present application provides a chip system, which is applied to a terminal device 100; the chip system includes one or more processors; the one or more processors are used to call computer instructions so that the terminal device 100 executes the method in the foregoing embodiment.

[0339] Exemplarily, the present application provides a computer program product, which enables the terminal device 100 to implement the method in the foregoing embodiment when the computer program product is executed on a computer.

[0340] In the above embodiments, all or part of the functions can be implemented by software, hardware, or a combination of software and hardware. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer codes or instructions. When the computer program code or instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer code or instructions can be stored in a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integration. Available media can be magnetic media, (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid state disk (SSD)), etc.

[0341] A person skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: a read-only memory (ROM) or a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

Claims

1. An image processing method, characterized in that: Applied to a terminal device, the method comprises: Acquire a first image, where the resolution of the first image is a first resolution; Based on the first image, text description information of the first image is generated; the text description information is used to describe the first image, and the text description information includes at least first-category information and second-category information; the first-category information includes a time corresponding to the first image and / or a location corresponding to the first image; the second-category information is used to characterize a category to which content included in the first image belongs; Acquire a noise image; the noise image is used to provide a noise distribution feature for the first image; The first image, the text description information and the noise image are input into a multimodal generative upscaling model, and a second image is output; the resolution of the second image is a second resolution, and the second resolution is greater than the first resolution.

2. The method according to claim 1, characterized in that: The step of inputting the first image, the text description information, and the noise image into a multimodal generative upscaling model and outputting a second image comprises: The first image, the text description information and the noise image are input into a multimodal generative upscaling model and N iterations are performed to obtain a second image, where N is a positive integer greater than or equal to 1 and less than or equal to a preset iteration threshold.

3. The method according to claim 2, characterized in that The multimodal generative amplification model includes a denoising network and a super-resolution amplification network; The step of inputting the first image, the text description information, and the noise image into a multimodal generative upscaling model and performing N iterations to obtain a second image includes: When performing the first iteration, the first image, the text description information and the noisy image are input into the denoising network, and the denoising network performs denoising processing on the noisy image based on the text description information and the first image to obtain a fourth image; Inputting the fourth image into the super-resolution upscaling network, and performing super-resolution upscaling processing on the fourth image from the denoising network through the super-resolution upscaling network to obtain a fifth image; When performing the Nth iteration, the fifth image, the text description information and the first image are input into the denoising network to obtain a fifth image after denoising, and the fifth image after denoising is input into the super-resolution upscaling network to obtain a fifth image after super-resolution upscaling; When N is equal to the preset iteration threshold, the fifth image after the super-resolution enlargement process is output as the second image.

4. The method according to claim 3, characterized in that The denoising network includes a first encoder, a first decoder and an attention network; When performing the first iteration, the first image, the text description information and the noisy image are input into the denoising network, and the denoising network performs denoising on the noisy image based on the text description information and the first image to obtain a fourth image, including: When performing the first iteration, the first feature information, the second feature information, the third feature information and the fourth feature information are input into the attention network to obtain the fifth feature information; the fifth feature information is used to characterize the feature after the attention network fuses multiple features; the first feature information is used to characterize the image feature corresponding to the first image; the second feature information is used to characterize the text feature corresponding to the text description information; the third feature information is used to characterize the noise feature corresponding to the noise image; the fourth feature information is used to characterize the number feature corresponding to the first iteration; Inputting the noise image and the fifth feature information into the first encoder, and outputting a first encoding feature map; the first encoder is used to perform feature encoding processing on the noise image and the fifth feature information; The first coding feature map is input to the first decoder, and the fourth image is output; the first decoder is used to perform feature decoding processing on the first coding feature map.

5. The method according to claim 4, characterized in that The multimodal generative upscaling model also includes a second encoder, a third encoder, and a fourth encoder; Before inputting the first feature information, the second feature information, the third feature information, and the fourth feature information into the attention network, the method further includes: Inputting the first image into the second encoder to obtain the first feature information; the second encoder is used to perform feature encoding processing on the first image; Inputting the text description information into the third encoder to obtain the second feature information; the third encoder is used to perform feature encoding processing on the text description information; Inputting the noise image into the first encoder to obtain the third feature information; The current number of iterations is input into the fourth encoder to obtain the fourth feature information; the fourth encoder is used to perform feature encoding processing on the current number of iterations.

6. The method according to any one of claims 3 to 5, characterized in that The super-resolution amplification network includes at least one convolutional layer; The step of inputting the fourth image into the super-resolution amplification network, and performing super-resolution amplification processing on the fourth image from the denoising network through the super-resolution amplification network to obtain a fifth image comprises: The fourth image is input into the at least one convolution layer, and the at least one convolution layer performs super-resolution upscaling processing on the fourth image based on a preset magnification to obtain the fifth image.

7. The method according to any one of claims 1 to 6, characterized in that The step of generating text description information of the first image based on the first image includes: obtaining the first type of information; Inputting the first image into a classification model and outputting the second category information; The first category of information and the second category of information are fused to obtain the text description information.

8. The method according to claim 7, characterized in that The classification model includes at least an encoder layer and a fully connected layer; The step of inputting the first image into a classification model and outputting the second category of information comprises: Inputting the first image into the encoder layer to obtain sixth feature information; the encoder layer is used to perform feature encoding processing on the first image; The sixth feature information is input into the fully connected layer to obtain the second category information; the fully connected layer is used to classify the sixth feature information.

9. The method according to any one of claims 1 to 8, characterized in that Before acquiring the first image, the method further includes: In response to a user's operation of turning on the camera, displaying a first preview image at a first zoom factor; When the first zoom ratio is adjusted to a second zoom ratio, displaying a second preview image, the second zoom ratio being greater than the first zoom ratio; The acquiring of the first image comprises: When receiving a photo-taking instruction, preprocessing the second preview image to obtain the first image; the preprocessing includes one or more of the following: denoising, automatic white balance, demosaicing, histogram equalization, and image enhancement; Wherein, outputting the second image includes: The second image is displayed.

10. The method according to claim 9, characterized in that Before displaying the second image, the method further includes: The second image is post-processed; the post-processing includes one or more of the following processes: sharpening processing, compression processing and format conversion processing.

11. The method according to any one of claims 1 to 10, characterized in that The multimodal generative upscaling model is obtained by the following training method: Acquire a first sample image, a second sample image, text description sample information and a third sample image; the first sample image and the third sample image are images with the same content but different resolutions, and the resolution of the first sample image is smaller than that of the third sample image; the second sample image is a noise image used to increase the magnification diversity of the first sample image; the text description sample information is generated based on the first sample image; The first sample image, the second sample image, the text description sample information and the third sample image are input into an original multimodal generative upscaling model for training until the original multimodal generative upscaling model meets a training iteration condition, and the original multimodal generative upscaling model that meets the training iteration condition is determined as the multimodal generative upscaling model.

12. A terminal device, characterized in that: The terminal device includes: one or more processors, and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the terminal device to execute the method as described in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises instructions, and when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 11.

14. A chip system, characterized in that: The chip system is applied to a terminal device; the chip system comprises one or more processors; the one or more processors are used to call computer instructions so that the terminal device executes the method as described in any one of claims 1 to 11.

Images