Image processing method and apparatus, device, and readable storage medium

Abstract

The present application provides an image processing method and apparatus, a device, and a readable storage medium. The method comprises: separately extracting an encoding feature representation of an image to be processed and an encoding feature representation of visual feature information in different dimensions of the image, by using the two encoding feature representations as inputs of a feature conversion network, using feature fusion and feature mapping to enhance the texture performance of the image, and obtaining a target image in which the texture of an object included in the image is closer to the texture of an image captured by using an object entity, thereby implementing adaptive image texture enhancement on the basis of features and requirements of different objects.

Description

Image processing method, device, equipment and readable storage medium TECHNICAL FIELD

[0001] The present application relates to the technical field of image, and particularly relates to an image processing method, device, equipment and readable storage medium. BACKGROUND

[0002] Image texture refers to the tactile sensation or surface characteristics presented by an image. In many scenarios, it is required to continuously improve the image texture to accurately reflect the real visual perception of an object in the image, so as to obtain better visual effects and user experience. For example, in the design, marketing and e-commerce sales process of clothing, luggage and home goods, the product image accurately reflecting the real design and texture of the product can improve the competitiveness of the product and bring good shopping experience to users.

[0003] In the related art, software is used for digital customization and three-dimensional simulation of products, and product images are generated through three-dimensional rendering technologies such as rasterization rendering, ray tracing rendering and volume rendering. However, the image texture of the three-dimensional rendered product image is poor, and the product image formed thereby has problems such as rigid shape, distorted fabric material, stiff texture and large light and shadow effect difference, which is quite different from the product image texture captured by the product real object, and it is difficult to accurately reflect the real product design and texture. SUMMARY

[0004] Therefore, in order to solve the above technical problems, the present application provides an image processing method, device, equipment and readable storage medium.

[0005] Specifically, the present application is realized by the following technical solutions.

[0006] According to a first aspect of an embodiment of the present application, an image processing method is provided, which comprises: performing feature encoding processing on a to-be-processed image to obtain a corresponding first feature representation; respectively extracting visual feature information of different dimensions of the to-be-processed image, and performing feature encoding processing on each kind of dimension of visual feature information to obtain a corresponding second feature representation; the dimensions at least include at least one of structure information, shape information, color information, texture pattern and material information; inputting the first feature representation and the second feature representation into a feature conversion network to obtain a third feature representation after feature fusion and feature mapping to a target feature space by the feature conversion network; performing feature decoding processing on the third feature representation to obtain a target image corresponding to the to-be-processed image.

[0007] Optionally, the method further comprises: after obtaining a target image corresponding to the image to be processed, extracting a pixel-level physical form feature from the target image; the physical form feature is used to represent enhanced material texture details and form wrinkle information; and performing pixel-level fusion of the physical form feature and the image to be processed to obtain an enhanced image corresponding to the image to be processed.

[0008] Optionally, the extraction of the pixel-level physical form feature from the target image comprises: removing high-frequency features at the pixel level in the target image through a filter to obtain a corresponding grayscale image; and the grayscale image represents the physical form feature.

[0009] Optionally, the method further comprises: constructing a mask for the image to be processed; the mask is used to identify a pixel region in the image to be processed that remains unchanged; and performing pixel-level fusion of the physical form feature and the image to be processed according to the mask.

[0010] Optionally, the pixel-level fusion of the physical form feature and the image to be processed comprises: performing fusion operations on pixel attributes in the physical form feature and pixel attributes at the same position in the image to be processed.

[0011] Optionally, the method further comprises: receiving a control input for indicating an image processing effect, converting the control input into control information features, and inputting the control information features together with the second feature representation into the feature conversion network.

[0012] Optionally, the control input includes at least one type of information of a text type and an image type, and the conversion of the control input into control information features comprises extracting corresponding control information features from the control input according to a pre-trained multi-modal understanding network.

[0013] Optionally, the feature conversion network further comprises a control input, and the method further comprises a training step of the feature conversion network, the training step comprising: obtaining training samples; the training samples comprising an image pair and labeled control inputs; the image pair comprising an object modeling and rendering image and a corresponding object entity photographed image; performing feature encoding processing on the object modeling and rendering image to obtain a fourth feature representation, and performing feature encoding processing on different dimension visual feature information of the object modeling and rendering image to obtain corresponding fifth feature representations; obtaining corresponding first control information features from the training samples according to the labels of the control inputs; inputting the first control information features, the fourth feature representation, and the fifth feature representation into the feature conversion network, and adjusting network parameters of the feature conversion network according to a network output image and the corresponding object entity photographed image.

[0014] Optionally, the extracting the visual feature information of different dimensions of the image to be processed respectively comprises: inputting the image to be processed into an image understanding network to obtain the visual feature information of each dimension; the image understanding network comprises an extraction submodule for extracting visual features of different dimensions.

[0015] Optionally, the feature encoding processing of the image to be processed comprises: encoding the image to be processed by using a variational autoencoder to obtain the first feature representation.

[0016] Optionally, the feature encoding processing of the visual feature information of each dimension comprises: encoding the visual feature information of each dimension by using a U-Net encoder respectively to obtain the second feature representation corresponding to the dimension.

[0017] According to a second aspect of the embodiment of the present application, an image processing device is provided, which comprises: a first encoding processing module, configured to perform feature encoding processing on an image to be processed to obtain a corresponding first feature representation; a second encoding processing module, configured to extract visual feature information of different dimensions of the image to be processed respectively, and perform feature encoding processing on the visual feature information of each dimension to obtain a corresponding second feature representation; the dimensions at least include at least one of structure information, shape information, color information, texture pattern and material information; a feature fusion and mapping module, configured to input the first feature representation and the second feature representation into a feature conversion network to obtain a third feature representation after feature fusion and feature mapping to a target feature space by the feature conversion network; and an image output module, configured to perform feature decoding processing on the third feature representation to obtain a target image corresponding to the image to be processed.

[0018] Optionally, the device further comprises: a physical form feature extraction module, configured to extract pixel-level physical form features from the target image after obtaining the target image corresponding to the image to be processed; the physical form features are used to represent enhanced material texture details and form wrinkle information; and a fusion enhancement module, configured to perform pixel-level fusion on the physical form features and the image to be processed to obtain an enhanced image corresponding to the image to be processed.

[0019] Optionally, the physical form feature extraction module is specifically configured to: remove pixel-level high-frequency features in the target image by a filter to obtain a corresponding grayscale image; and the grayscale image represents the physical form features.

[0020] Optionally, the device further comprises: constructing a mask of the to-be-processed image; the mask is used to identify a pixel region in the to-be-processed image that remains unchanged; and performing pixel-level fusion of the physical form feature and the to-be-processed image according to the mask.

[0021] Optionally, the fusion enhancement module is specifically configured to: perform fusion operation on a pixel attribute in the physical form feature and a pixel attribute at a same position in the to-be-processed image.

[0022] Optionally, the device further comprises: receiving a control input used to indicate an image processing effect, converting the control input into control information features; and inputting the control information features and the second feature representation into the feature conversion network.

[0023] Optionally, the control input comprises at least one type of information of a text type or an image type, and the conversion of the control input into control information features comprises: extracting corresponding control information features from the control input according to a pre-trained multi-modal understanding network.

[0024] Optionally, the feature conversion network further comprises a control input, and the device further comprises a training step of the feature conversion network, the training step comprising: obtaining training samples; the training samples comprising an image pair and a labeled control input; the image pair comprising an object modeling rendered image and a corresponding object entity photographed image; performing feature encoding processing on the object modeling rendered image to obtain a fourth feature representation, and performing feature encoding processing on different dimensional visual feature information of the object modeling rendered image to obtain corresponding fifth feature representations; obtaining corresponding first control information features from the training samples according to the label of the control input; inputting the first control information features, the fourth feature representation and the fifth feature representation into the feature conversion network, and adjusting network parameters of the feature conversion network according to an output image and the corresponding object entity photographed image.

[0025] Optionally, the second encoding processing module is specifically configured to: input the to-be-processed image into an image understanding network to obtain the visual feature information of each dimension; and the image understanding network comprises an extraction submodule used to extract visual features of different dimensions.

[0026] Optionally, the first encoding processing module is specifically configured to: perform encoding processing on the to-be-processed image by using a variational autoencoder to obtain the first feature representation.

[0027] Optionally, the second encoding processing module is specifically configured to: perform encoding processing on the visual feature information of each dimension by using a U-shaped network U-Net encoder respectively to obtain the second feature representation corresponding to the dimension.

[0028] According to a third aspect of the embodiments of the present application, an electronic device is provided, which comprises a memory and a processor; the memory is configured to store a computer program; the processor is configured to execute the image processing method by invoking the computer program.

[0029] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores a computer program, and the program is executed by a processor to implement the image processing method.

[0030] The technical solutions provided by the embodiments of the present application can include the following beneficial effects.

[0031] In the technical solutions provided by the embodiments of the present application, the pre-trained feature conversion network is used to apply the prior knowledge related to the texture of the commodity to the image, the encoding feature representation of the to-be-processed image and the encoding feature representation of the visual feature information of the image in different dimensions are extracted respectively, the details of the commodity in the image can be maintained, the design details of the commodity in the input image can be avoided to be modified by the network calculation, and the encoding feature representations corresponding to the to-be-processed image and the visual feature information thereof are mapped to the feature space after the texture is improved and are decoded, the image texture effect is enhanced, and the output image is closer to the high-quality imaging effect of the real shooting of the commodity.

[0032] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. In addition, any embodiment in the present application does not need to achieve all the effects described above. BRIEF DESCRIPTION OF DRAWINGS

[0033] The accompanying drawings, which are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the present application.

[0034] FIG. 1 is a flow diagram of an image processing method according to an example embodiment of the present application.

[0035] FIG. 2A is a block diagram of an image processing process including pixel-level fusion processing according to an example embodiment of the present application.

[0036] FIG. 2B is an example diagram of an image processing process when the to-be-processed image is a clothing image according to an example embodiment of the present application.

[0037] FIG. 3A is a block diagram of an image processing process with control input according to an example embodiment of the present application.

[0038] FIG. 3B is a flow diagram of an image processing method with control input according to an example embodiment of the present application.

[0039] FIG3C is a diagram illustrating an example of image processing in which a control input is introduced, according to an exemplary embodiment of the present application.

[0040] FIG4 is a flow chart of a method for training a feature conversion network in image processing, shown in an exemplary embodiment of the present application.

[0041] FIG5 is a schematic structural diagram of an image processing device according to an exemplary embodiment of the present application.

[0042] FIG6 is a hardware diagram of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0043] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0044] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0045] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, the first classification threshold may also be referred to as the second classification threshold, and similarly, the second classification threshold may also be referred to as the first classification threshold. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0046] Three-dimensional rendering technology is a technique in the field of computer graphics that simulates how light interacts with objects in a virtual scene, such as reflection, refraction, scattering, and shadow effects, and simulates the texture, color, and transparency of materials, and converts three-dimensional models into two-dimensional images for imaging display. Common three-dimensional rendering techniques include rasterization rendering, ray tracing rendering, and volume rendering, among others. Rasterization rendering is a real-time rendering technique that projects three-dimensional models onto a two-dimensional screen and then fills in pixels to calculate their corresponding colors, commonly used in real-time imaging and interactive applications, with the characteristics of simplicity and efficiency, but the effect is not realistic enough. Ray tracing rendering is a physics-based rendering technique that simulates the propagation of light in a three-dimensional world by tracking the path of light, which can calculate refraction, reflection, shadow, and other effects to produce high-quality images, but the computational cost is high. Volume rendering is commonly used for visualization of three-dimensional volume data, which samples volume data and calculates light propagation to obtain images, and can be applied to analysis and understanding of internal structures of volume data such as medical images and fluid simulation.

[0047] Image texture is crucial for conveying the surface characteristics and tactile sensations of objects, and in scenarios such as architectural design, digital art creation, and other scenarios that require conveying realism, performance quality, or enhancing user experience through images, improving image texture can more accurately reflect the true perception of objects in images, making it easier to better showcase objects in images, allowing users to better perceive and understand objects in images, and enhancing the immersion and experience of viewers.

[0048] For example, in the design, selection, marketing, and e-commerce sales of clothing, luggage, home goods, and other commodities, commodity images that can accurately reflect the design and texture of real objects have important value for merchants and users. There are two ways to obtain such commodity images.

[0049] The first is the traditional scheme, which requires making patterns from the designer's manuscript to obtain a commodity sample, then hiring photographers, retouchers, and even models and makeup artists to take photos and select pictures, thereby obtaining usable commodity images. This scheme has a cumbersome process, takes a long time, and is costly.

[0050] The second is three-dimensional rendering imaging. Existing three-dimensional rendering imaging digitally customizes a commodity through software, simulates a real commodity through three-dimensional simulation, and obtains a corresponding commodity image through three-dimensional rendering imaging technology to select, distribute, and market the style of the commodity. However, the three-dimensional rendering commodity image generated by using the existing rendering technology has poor image quality, and the commodity quality presented by the image is quite different from the commodity quality of the image taken by the real commodity. Based on three-dimensional modeling and simulation effects, it is difficult to reflect the normal shape of the fabric, such as the natural sag of different fabrics, and it is also difficult to simulate the small wrinkles generated when the commodity is placed, hung, or worn by a model. The rendering imaging has problems such as stiff and heavy commodity shape, distorted commodity fabric material quality, stiff texture, and too simple light and shadow effects that are quite different from natural light. It is difficult to reflect the real design and physical quality of the commodity, and the image quality needs to be improved.

[0051] To solve the above technical problems, the present application provides an image processing method. The method processes a to-be-processed image, such as a three-dimensional modeling rendering image, that needs to enhance the image quality, so that the object quality presented by the processed image is closer to the quality effect of the object entity. The method can be executed by any electronic device or apparatus, which is not limited in the present application. As shown in FIG. 1, the method can include the following steps S101 to S104.

[0052] S101, performing feature encoding processing on the to-be-processed image to obtain a corresponding first feature representation.

[0053] The to-be-processed image refers to a picture that needs to improve the object quality presented by the image. It can include an image with poor quality obtained by modeling and simulating an article or object through three-dimensional rendering technology and then rendering through graphics, or an image with poor object quality presented by using other image acquisition methods, such as an image taken by a mobile phone or an image taken in an environment with poor light conditions. The image is of a commodity real object or a landscape or a real object. The poor quality is relative to the object quality presented by an image that accurately reflects the object entity, which is taken under normal shooting conditions. The object quality presented by the to-be-processed image is lower than that of the image that accurately reflects the object entity.

[0054] Feature encoding processing refers to converting input data into a feature representation in a latent space through an encoder network to capture the context information of the input data to obtain a high-level abstract feature representation. The feature representation includes key information of the input data and can be used for image reconstruction. In this embodiment, the feature encoding processing refers to taking the to-be-processed image as input, extracting the features of the input image layer by layer, mapping the related information describing the object quality in the image, such as the edges and textures of the objects or objects in the image, to a latent, continuous, and small-dimension feature space, and the mapping process is reversible.

[0055] In some embodiments, the feature encoding of the to-be-processed image can be achieved by encoding the to-be-processed image using a VAE (Variational Autoencoder) encoder to obtain the first feature representation. Other encoders that can perform feature encoding on images can also be used for feature encoding, which is not limited in the present application.

[0056] S102, respectively extracting visual feature information of different dimensions of the to-be-processed image, and performing feature encoding on the visual feature information of each dimension to obtain a corresponding second feature representation; the dimensions at least include at least one of structure information, shape information, color information, texture pattern, and material information.

[0057] The visual feature information is numerical or vector form data representing image content and features extracted from the to-be-processed image, which is used to describe the texture features of the objects in the image presented by the image, including but not limited to structure information, shape information, color information, texture pattern, and material information, and other dimensional description of related feature information of objects in the image.

[0058] The structure information refers to the spatial structure information of the objects in the image, and the spatial relationship and layout information between the objects, which can include key point information representing the structure of the objects, such as shoulder points, body points, and waist line positions of clothes as key point information. The shape information is used to describe the shape and contour of the objects in the image, which can be obtained by edge detection and contour extraction methods. The color information is used to represent the color and brightness of the image, which can include color histogram, color distribution, and dominant color, and the brightness can include brightness, saturation, and contrast information. The texture information is used to describe the complexity and fineness of the texture in the image, which can be extracted by a gray level co-occurrence matrix and a HOG (Histogram of Oriented Gradient). The material information is used to describe the characteristics and texture of the material of the objects in the image, which can include diffuse reflection, specular reflection, and transparency, such as a leather sofa, and the material information is used to describe the leather material characteristics of the sofa.

[0059] The visual feature information of the to-be-processed image can be extracted in any of the following manners: a pre-trained convolutional neural network model such as VGG, ResNet, Inception, etc. can be used to extract the visual feature information, the to-be-processed image is input into the model, and the feature representation of the intermediate layer is extracted as the visual feature information; a tool or library specially designed for image feature extraction such as OpenCV, Scikit-learn, etc. can also be used to run the feature extractor on the image to obtain visual feature information including color histograms, texture features, shape features, etc.; or traditional feature extraction algorithms can also be used in combination with other feature extraction methods to obtain the visual feature information, for example, traditional computer vision feature extraction algorithms such as SIFT (Scale-Invariant Feature Transform), HOG (Histogram of Oriented Gradients) can be used to extract local feature information of the image.

[0060] The feature encoding processing in step S102 is described above in the meaning indicated by the feature encoding processing in step S101. In this embodiment, each dimension of visual feature information is encoded in units of dimensions to obtain a feature representation of the visual feature information corresponding to the dimension in the latent space. For example, five dimensions of visual feature information including structure information, shape information, color information, texture patterns, and material information are obtained, and then five corresponding second feature representations are obtained after the feature encoding processing.

[0061] Based on the U-Net encoder, large-scale features in the image can be captured through multi-level downsampling operations, and the skip connection design can obtain more detailed information from the shallow features. In some embodiments, the feature encoding of the visual feature information can use a U-Net encoder or other encoders that can perform feature encoding processing on the visual feature information, and each dimension of visual feature information is encoded to obtain a second feature representation corresponding to the dimension.

[0062] S103, input the first feature representation and the second feature representation into the feature conversion network to obtain a third feature representation after the feature conversion network performs feature fusion and feature mapping to the target feature space.

[0063] The feature conversion network is used for feature fusion and mapping of the first feature representation and the second feature representation of the image to be processed, and maps them to a specified feature space, i.e., a feature space with improved texture, to obtain a mapped feature representation. The image generated by the mapped feature representation has higher texture than the image to be processed. The feature conversion network can be an open source network model pre-trained on a large number of open source data sets, which can map the input image features to a feature distribution space similar to a photograph of a real object, thereby generating a new image. The open source network model can include any one of a generative network such as a GAN, a VAE network, or a diffusion network. Alternatively, the open source network model can be trained based on the above-mentioned open source network model, using a three-dimensional rendered image of the same product and a corresponding product photograph as two images with different textures, and fine-tuning the network parameters to make the feature conversion network more suitable for the present application, so as to map the feature representation of the image with lower texture to the feature space with improved texture and improve the texture processing effect.

[0064] In the present embodiment, the encoded features of the image to be processed, i.e., the first feature representation, and the encoded features of the visual feature information in the dimensions of structure, color, and material, i.e., the second feature representation, are respectively input into the feature conversion network. The feature conversion network fuses the first feature representation and the second feature representation by simple feature splicing or other fusion strategies such as feature weighted fusion, or more complex attention mechanisms, etc., to obtain a comprehensive feature representation. According to the feature mapping processing learned by the feature conversion network during pre-training, the comprehensive feature representation is mapped to a feature space with improved texture, and the mapped feature representation is determined as a third feature representation.

[0065] During inputting the first feature representation and the second feature representation into the feature conversion network, the first feature representation contains richer image feature information, while the second feature representation in each dimension is more abstract. Therefore, the first feature representation can be input into the input layer of the feature conversion network, and the second feature representation can be input into the middle layer of the feature conversion network. The first feature representation and the second feature representation are fused in the middle layer, which can reduce information loss during information transmission and processing, help to retain more original feature information, and improve the expression ability of the network and the texture enhancement effect.

[0066] In some embodiments, the feature conversion network can include a U-Net model, and the first feature representation can be input into an input layer of the U-Net model, and the second feature representation of each dimension can be added to an intermediate layer of the U-Net model. After the first feature representation is calculated step by step to each intermediate layer through the input layer, the feature fusion is performed on the intermediate layer after the feature fusion, and the feature mapping processing is performed on the fused feature representation. In the feature fusion, the first feature representation of the image to be processed and the second feature representation of each dimension can be adjusted to the same dimension by dimension expansion or projection, etc. to realize dimension alignment.

[0067] S104, performing feature decoding processing on the third feature representation to obtain a target image corresponding to the image to be processed.

[0068] The feature decoding processing refers to converting the high-level feature mapping extracted by the encoder into the same size or the same dimension output as the input data through the decoder part in the neural network. In the present embodiment, the feature decoding processing is used to decode the third feature representation obtained after the feature conversion network processing into the corresponding output image, so as to convert the abstract feature representation back to the image data.

[0069] The feature decoding processing can be realized by a feature decoding network, which is symmetrical in structure with the feature encoding network. The abstract feature representation is gradually converted into a feature map of the same size as the input image through the deconvolution layer, the up-sampling operation, etc. In each layer of the feature decoding, the feature fusion operation corresponding to the encoding part can be considered to retain the details and semantic information transmitted from the encoder. The feature decoding network gradually restores the abstract feature representation to the image data, and the restoration process involves gradually expanding the spatial dimension of the feature map, and decoding the details and structure information of the image by using the learned parameters.

[0070] Through the feature decoding processing, the third feature representation in the feature space with improved texture after the feature mapping is converted into a target image, and the object texture in the image presented by the target image is higher than that of the image to be processed.

[0071] In the present disclosure, for the problem that the object texture in the image presented by the image is poor and the real object entity is different, the encoding feature representation of the image to be processed and the encoding feature representation of the visual feature information of different dimensions of the image are extracted respectively, and the two kinds of encoding feature representations are taken as the input of the feature conversion network, the related features of the object in the image can be kept to a higher degree, the texture performance of the image is enhanced by feature fusion and feature mapping, the target image with object texture closer to the object entity texture is obtained by feature decoding, the adaptive image texture enhancement according to the characteristics and needs of different objects is realized, the effect presented by the image is closer to the real object texture, and the image processing efficiency is improved.

[0072] According to the image processing method described in the above embodiment, assuming that the image to be processed is a flat and smooth clothing image, the object in the image is the clothing, and the surface presented by the image is completely horizontal, the flat image can present the real wrinkles of the clothing fabric by the image processing method. For example, the input image is a dressing image of a 3D simulated skirt worn on a model, the texture effect presented by the image has problems such as straight and stiff fabric drape board, unclear material information, and unclear human body curve wrinkles, and the target output image obtained by the above method can present the natural drape of the fabric, the material details and the clear human body curve wrinkles. Based on the image processing method, various application scenarios requiring image texture enhancement are applicable, and the above is an example for illustrative purposes, which does not limit the specific application of the present application.

[0073] In the foregoing embodiment, after the image to be processed is processed by the image processing method described in the foregoing steps S101-S104, the object texture in the target image obtained is higher than that in the image to be processed, and the image material, texture, details, shape wrinkles and other forms are relatively enhanced. However, in some cases, part of the design and pattern features in the image to be processed will be modified to different degrees in the image processing process, thereby deviating from the original design and production target of the real object. To solve this problem, the present embodiment proposes a further optimization method, which further processes the target image corresponding to the image to be processed to obtain an enhanced image that enhances the material shape and other forms and the sense of reality while retaining the original graphic design. As shown in FIG. 2A, the image to be processed is processed by feature encoding, feature conversion network and feature decoding to obtain a target image corresponding to the image to be processed. After obtaining the target image, pixel-level feature extraction S105 and fusion processing S106 processes can be further included.

[0074] S105, after obtaining the target image corresponding to the image to be processed, extracting a pixel-level physical form feature from the target image; the physical form feature is used to represent enhanced material texture details and form wrinkle information.

[0075] In this embodiment, the pixel-level physical form feature represents an image feature extracted from the pixel level of the target image, which can reflect the physical characteristics of the object surface. It can include but is not limited to the related features of structure, shape, color, wrinkle, texture, and material, etc. At the pixel level, the physical form feature can be represented as the distribution of pixel values, the spatial relationship between pixels, and the change of pixel intensity in the image.

[0076] It should be understood that the pixel-level physical form feature and the feature representing the physical characteristics of the object surface in the vector space are information in two different concept spaces, which respectively represent the description of the image at different abstraction levels. Among them, the pixel-level physical form feature is described based on the pixel data of the image, which reflects the details and physical characteristics of the image at the pixel level; and the feature representing the physical characteristics of the object surface in the vector space is a set of high-dimensional vectors obtained by mapping the image from the pixel space to the vector space. Each vector represents the information of the image in a certain feature dimension. These feature vectors have specific geometric meaning and statistical properties in the vector space, and can be used to describe the global or local characteristics of the image, such as shape, texture, color, etc.

[0077] The extraction of the physical form feature is to obtain the morphological information of the medium and low frequency material, shape wrinkle change, etc., while avoiding the introduction of modified pattern design and other detail features. Regarding the extraction of the physical form feature, the physical form feature representing the object in the target image can be extracted from the target image by combining and applying methods such as texture analysis, edge detection, morphological processing, and feature description. The high-frequency edge information is usually strongly related to the pattern design, and the physical form feature can also be extracted by filtering the high-frequency features in the target image. Alternatively, a feature extraction network dedicated to extracting the pixel-level physical form feature can be trained through samples by machine learning or neural network, so that the target image is input into the feature extraction network to obtain the physical form feature output by the feature extraction network.

[0078] S106, pixel-level fusion of the physical form feature and the image to be processed is performed to obtain an enhanced image corresponding to the image to be processed.

[0079] Pixel-level fusion refers to combining pixel information of multiple images to generate a new image. The fusion operation occurs at the pixel points of the image, and pixel values of different images at the same position are combined through pixel replacement or weighted averaging. In the pixel-level fusion process, the color, brightness, contrast and other attributes of the pixel points are recalculated to reflect the comprehensive characteristics of the fused image.

[0080] In this embodiment, pixel-level fusion is used to fuse the image containing physical form features with the to-be-processed image. That is, the pixel attributes in the physical form features can be fused with the pixel attributes at the same position in the to-be-processed image. For the pixel points on the to-be-processed image, the original pixel point information of the to-be-processed image is fused with the pixel point information of the corresponding physical form features to generate an enhanced image. The enhanced image further enhances the material, shape and other forms and the sense of reality of the image on the basis of maintaining the original graphic design of the to-be-processed image, that is, the enhanced image not only retains the key design elements in the to-be-processed image, but also makes the image more realistic and delicate in vision through the fusion of physical form features, thereby improving the image quality while ensuring that the design elements in the image meet the original design and production target.

[0081] As exemplarily shown in FIG. 2B, one example of the image processing process of a to-be-processed image is a garment, and the to-be-processed image is processed through feature encoding, feature conversion network and feature decoding to obtain a corresponding target image, and a better enhancement effect of the clothing material and wrinkle shape is obtained. However, as can be seen from the marked area and the enlarged view of the marked area in the figure, the clothing collar logo and hem icon design are modified to different degrees. In this embodiment, the physical form features representing the enhanced material texture details, wrinkle shape and the like are extracted from the target image, and are fused with the to-be-processed image, and finally the enhanced image on the right is output, which not only retains the enhancement effect of the material texture details, wrinkle shape and other surface physical forms in the target image, but also maintains the original pattern design details of the clothing.

[0082] In the embodiments of the present disclosure, based on the fact that the object texture, material, texture, details and shape wrinkles and other physical forms presented in the target image are enhanced relative to the to-be-processed image, by extracting the pixel-level physical form features from the target image and performing pixel-level fusion with the to-be-processed image, it can be ensured that the original design and pattern features in the to-be-processed image are retained while the image texture, material, texture, details and shape wrinkles are enhanced, thereby solving the problem of feature distortion or deviation from the original design and production target caused by excessive enhancement in the image processing process.

[0083] In some embodiments, regarding the physical form features described in FIG. 2A, the physical form features can be extracted by removing high-frequency features at the pixel level in the target image through a filter such as a Gaussian blur filter. In order to highlight the reflection characteristics of physical materials under light and shape wrinkles and other form details in the target image, the obtained physical form features can be represented in the form of a grayscale image, reducing unnecessary interference information, that is, the corresponding grayscale image can be obtained by removing high-frequency features at the pixel level in the target image. The grayscale image represents the physical form features and contains the reflection characteristics of physical materials under light and shape wrinkles and other form details in the preliminary enhanced target image. The image size of the grayscale image is the same as the image size of the image to be processed, that is, the pixel points of the two correspond one by one. When the image size of the grayscale image is inconsistent with the image size of the image to be processed, image interpolation techniques such as bilinear interpolation or nearest neighbor interpolation can be used to adjust the grayscale image to the same size as the image to be processed, ensuring accurate correspondence of each pixel point during pixel-level fusion.

[0084] Based on the grayscale image, when the physical form features are fused with the image to be processed at the pixel level, the pixel attributes representing the physical form features in the grayscale image can be fused with the pixel attributes at the same position in the image to be processed.

[0085] In some embodiments, direct pixel-level fusion during image fusion may result in some areas that do not need to be fused being incorrectly fused, thereby affecting the final image enhancement effect. In order to improve the accuracy and controllability of fusion, the present embodiment proposes a pixel-level fusion control method that more accurately controls pixel area fusion. This method realizes precise adjustment of specific areas of an image by precisely controlling pixel area fusion through a mask, ensuring that the fused image retains the pattern design of the image to be processed while naturally incorporating the enhanced physical form features. This method can be implemented as follows: a mask of the image to be processed is constructed; the mask is used to identify pixel areas in the image to be processed that remain unchanged; and the physical form features are fused with the image to be processed at the pixel level according to the mask.

[0086] In image processing, a mask is a two-dimensional array or matrix or binary image that is the same size as an image and is used to identify specific areas in an image to be processed. Each element in the mask is used to indicate whether the corresponding image pixel participates in pixel-level fusion operations. For example, the elements in the mask can be set to 0 or 1, which are used to identify pixel areas in the image to be processed that remain unchanged and pixel areas that need to be fused, respectively.

[0087] The mask can be constructed according to image content analysis of the to-be-processed image. For example, the to-be-processed image is a clothing image, and the pixel region related to the pattern design on the clothes does not need to be fused with the physical form feature. The pixel region related to the pattern design can be identified through semantic analysis of the image, and the corresponding pixel region in the mask is marked as a region to be kept unchanged. Alternatively, the mask can also be constructed according to user specification, that is, the user can mark which region in the to-be-processed image is to be kept unchanged according to his own needs, and then create a mask according to the user's specification. For example, the user can use a brush tool to draw the region that does not need to be fused on the image in the image editing software, and the software constructs the mask according to the region drawn by the user.

[0088] According to the mask, when the physical form feature is fused with the to-be-processed image at the pixel level, for the pixel region marked as unchanged in the mask, the pixel value in the to-be-processed image is directly retained in the fused image. For the region marked as fusable in the mask, the physical form feature and the pixel attribute of the same pixel point on the to-be-processed image are fused according to the pixel-level fusion. The weight α can be set to control the fusion proportion of the pixel attributes of the physical form feature and the to-be-processed image.

[0089] The elements in the mask can also be set as numerical values representing fusion weights, so that when the physical form feature is fused with the to-be-processed image according to the mask, the pixel attributes of the same pixel point of the physical form feature and the to-be-processed image can be weighted and fused according to the element value at the same position in the mask corresponding to the pixel point on the to-be-processed image. For example, when the fusion weight represents the proportion of the physical form feature in the enhanced image, and (1-fusion weight) represents the proportion of the pixel attribute of the to-be-processed image in the enhanced image, the fusion weight of the corresponding pixel point can be set to 0 for the pixel region marked as unchanged in the to-be-processed image.

[0090] In the embodiments of the present disclosure, by using the mask as a control tool in the fusion process, the fusion and retention of the pixel region can be accurately controlled, unnecessary regions are avoided from being incorrectly fused, the controllability of the pixel region fusion is enhanced, and greater freedom is provided for image processing and enhancement, thereby improving the accuracy of image fusion, ensuring that the fused image is more accurate in details, and the enhanced image obtained by fusion can well retain the pattern design and key features of the to-be-processed image, so that the enhanced image is more realistic and natural in visual effect, and the image quality is improved.

[0091] In some embodiments, in order to further improve the flexibility and accuracy of image processing, enhance the quality and richness of the target image, as shown in FIG. 3A, in the process of encoding the image to be processed and inputting it into the feature conversion network, an additional control input is introduced, which is converted into control information features and input into the feature conversion network together with the encoding processing result corresponding to the image to be processed. The control input is used to guide the generation of image quality enhancement processing effect. The control input can be a control text or a reference image, which provides more detailed and specific guidance for image generation, can enrich and optimize image generation, and thus can generate a high-quality image that meets the expectations.

[0092] Based on the above-mentioned additional control input, the above-mentioned method further includes a processing step of controllable image quality enhancement, so that the enhanced image is more in line with user requirements. As shown in FIG. 3B, the processing step can include steps S301 and S302.

[0093] S301, receiving a control input for indicating image processing effect, and converting the control input into control information features.

[0094] S302, inputting the control information features and the second feature representation into the feature conversion network.

[0095] The control input is an input signal for indicating image quality enhancement processing effect. By processing the input, the quality performance of the output image can be controlled. The control input can be various forms of parameters or indication information. For example, the control input is a parameter indicating the brightness of the adjusted image; or the control input can be a symbol representing different material types. By selecting different material types, the quality characteristics of the objects in the output image can be controlled. For example, in furniture display, by selecting different material types such as wood, metal or leather, the appearance quality of the furniture in the image can be changed.

[0096] The control information features are feature representations of specific control information extracted from the control input. The control information features are used to guide the feature conversion network to perform quality enhancement processing on the image. Keywords, labels or features can be extracted from the control input to convert them into corresponding vector representation forms. For example, in the example of brightness control, the control input can be a brightness value in the range of 0-1, and the control information features can be a feature representation vector obtained by feature extraction network from the corresponding brightness value. The feature representation vector guides the image brightness processing process in the feature conversion network, so that the brightness of the final output target image meets the brightness requirement of the control input.

[0097] In some embodiments, in the case of inputting the aforementioned second feature representation into the intermediate layer of the feature conversion network, after obtaining the control information feature, the control information feature is input into the intermediate layer of the feature conversion network together with the second feature representation.

[0098] Based on the control information feature as the input of the feature conversion network, the format of the obtained control information feature meets the input format requirements of the feature conversion network, and processing steps such as normalization and dimension adjustment of the feature can be performed to ensure that the control information feature can be effectively utilized by the feature conversion network.

[0099] In the present disclosure, by introducing control input and control information feature to guide the texture enhancement processing effect, based on the control information feature, more detailed control of the texture performance of the image to be processed can be provided, and the user can adjust the texture features of different dimensions according to specific needs, thereby realizing personalized customization of the image texture enhancement processing effect of different types and different styles, obtaining a target image that meets the user's own needs and preferences, better meeting the user's needs, and having higher controllability and flexibility; and the user can instantly view the image effect after texture enhancement processing, and adjust the control input according to real-time feedback to achieve the best texture performance, thereby improving the processing efficiency and accuracy of the effect.

[0100] In some embodiments, the aforementioned control input can include at least one type of information of text type and image type, and based on this, the control information feature is extracted from the control input, which involves parsing the text description, processing the parameters of the user input, etc., to obtain a specific indication vector of the texture improvement direction.

[0101] Among them, the control input of the text type can be used to describe the related features of the material and shape, and the features described by the control input can be features that the image to be processed contains but the reality or accuracy is not enough, such as the three-dimensional sense of knitting and the luster of silk, etc., or features that the image to be processed does not express, such as the details of the hair and the wrinkles, etc. The control input of the text type can be processed through a text feature extraction or a multi-modal understanding network to obtain a corresponding feature vector, which is used to guide the subsequent image generation.

[0102] The control input of the image type can supplement the features that are difficult to describe by the control input of the text type, for example, the image to be processed lacks some significant features of certain materials, and the significant features are difficult to accurately describe by text, so the reference image of the corresponding target material can be used as the control input. The control input of the image type can be processed through an image feature extraction or a multi-modal understanding network to extract visual information such as color, texture, and texture related to the material as control information features, which are used to guide the subsequent image generation.

[0103] As exemplarily shown in FIG. 3C, an image processing example diagram of introducing control input, for the image to be processed, both text type control input and image type control input are introduced at the same time, which are used to represent that the material of the clothes is silk material, and based on the control input, the generation of the target image is guided, and the target image obtained by introducing the control input presents better visual enhancement effect of silk in terms of image texture, wrinkle morphology, etc. compared with the target image obtained by not introducing the control input.

[0104] Based on this, for the conversion of the control input into control information features described in the foregoing step S301, the following methods can be used:

[0105] According to the pre-trained multi-modal understanding network, the corresponding control information features are extracted from the control input.

[0106] The multi-modal understanding network is a deep learning model for processing and fusing different types of data, and can integrate information of multiple data types together to improve the understanding and processing ability of multi-modal data. The multi-modal understanding network can accept multi-modal input from different data sources, such as text description, image content, audio clip, etc., and use different processing branches for each data type to extract effective feature representation in the respective field; for example, methods suitable for natural language processing can be used for text data, and convolutional neural networks can be used for image data. Among them, cross-modal information interaction and fusion can be carried out inside the multi-modal understanding network, so that different data types can influence and complement each other. In this embodiment, the multi-modal understanding network can be realized based on the CLIP (Contrastive Language-Image Pre-training) model after visual and text alignment, or other multi-modal network models can also be used.

[0107] In this disclosure, by using the pre-trained multi-modal understanding network to extract control information features related to the quality effect of the commodity from the control input, the quality enhancement processing process of the feature conversion network can be better guided, the semantic information of multi-modal data is combined, and the quality features of the commodity can be more comprehensively understood and expressed, so as to realize more accurate and expected quality enhancement processing effect, and improve the accuracy and efficiency of the processing effect.

[0108] In some embodiments, as shown in FIG. 4, for the feature conversion network used in the image processing method embodiment described above in FIG. 3A, the method can further include a training step of the feature conversion network, and a suitable generation network or diffusion network structure is pre-selected, and for the image quality enhancement task, the selected network can be a network that can process high-dimensional data and generate high-quality images, such as GAN or VAE. Based on the selected network, the training step can include steps S401 to S404.

[0109] S401, obtaining a training sample; the training sample includes an image pair and a labeled control input; the image pair includes an object modeling rendered image and a corresponding object entity photographed image.

[0110] The image pair refers to two images of different quality levels, and in this embodiment, one of the images is obtained by three-dimensional simulation and graphics rendering, and the other is obtained by photographing a real object. The object modeling rendered image refers to simulating the entity of the object by three-dimensional simulation, and obtaining the corresponding image by three-dimensional graphics rendering imaging technology, while the corresponding object entity photographed image is a higher quality image obtained by photographing the object with the same shooting angle as the object modeling rendered image. That is, the image pair includes a low-quality rendered image and its corresponding high-quality version, and the labeled control input refers to control information describing the desired quality processing effect, and the label indicates the corresponding control information features.

[0111] In the case where the image to be processed includes an image obtained by graphics rendering after modeling and simulation of a commodity, the object modeling rendered image is an image obtained by rendering after modeling of the commodity, and the corresponding object entity photographed image is an image obtained by photographing a commodity sample or a commodity real object. It can be understood that the image pair can be adaptively adjusted according to the type of the object in the image in the application scenario, for example, if the object is a landscape, two images of different quality levels taken at the same shooting angle and scene can be used as the image pair, or an image accurately reflecting the quality of the object can be blurred or noise-added to obtain another image with poor quality, and the two images can be used as the image pair. The above are exemplary examples, and the specific content selection of the training sample is not limited in the present application.

[0112] S402, performing feature encoding processing on the object modeling rendered image to obtain a fourth feature representation, and performing feature encoding processing on different dimensional visual feature information of the object modeling rendered image to obtain a corresponding fifth feature representation.

[0113] S403, obtaining corresponding first control information features from the training sample according to the label of the control input.

[0114] S404, input the first control information feature, the fourth feature representation and the fifth feature representation into the feature conversion network, and adjust the network parameters of the feature conversion network according to the output image and the corresponding object entity shooting image.

[0115] The way of inputting parameters into the feature conversion network is the same as the foregoing embodiment. The fourth feature representation corresponding to the object modeling rendering image can also be input into the input layer of the feature conversion network, and the first control information feature and the fifth feature representation of different dimensional features can be input into the middle layer of the feature conversion network, and the image output by the feature network is obtained.

[0116] After obtaining the input of the feature conversion network to be trained according to the training sample, the feature conversion network is trained according to the input and the corresponding real commodity real shooting image, so that the network learns how to generate high-quality images and how to adjust the quality of the generated images according to the given control information feature.

[0117] In the training process, a suitable loss function is set in advance to quantify the difference between the quality of the network output image and the real commodity real shooting image. The loss function can include a pixel-level loss such as L1 or L2 loss, a perception loss such as the output of the middle layer based on the feature extraction network, and a loss term specially for the control information feature. In each iteration of the training process, the value of the loss function is calculated and the gradient of each network parameter with respect to the loss function is calculated using the back propagation algorithm, so that the parameters of the network are adjusted according to the gradient and a suitable optimization algorithm, until the value of the loss function converges or a preset stopping condition is reached.

[0118] In the present disclosure, by utilizing a large number of training samples, continuously minimizing the loss function and adjusting the network parameters, the feature conversion network will gradually learn the correlation between the control information feature and the feature representation of different dimensions of the image, thereby realizing controllable enhancement processing of the quality of the image.

[0119] In some embodiments, the foregoing step S102 of extracting different dimensional visual feature information of the image to be processed can also be realized by the following method: inputting the image to be processed into an image understanding network to obtain the visual feature information of each dimension; the image understanding network comprises an extraction submodule for extracting visual features of different dimensions.

[0120] The image understanding network is provided with extraction submodules of visual feature information of different dimensions, which are respectively used to extract visual feature information of different dimensions. The extraction submodule can be a convolutional neural network, and the best extraction network corresponding to the visual feature information of each dimension in related technologies can be used as the extraction submodule, so as to obtain the visual feature information of each dimension independently output.

[0121] In the present disclosure, the feature extraction is performed on the to-be-processed image by using the extraction sub-modules for extracting visual features of different dimensions, each of which can be designed and trained specifically for a certain type of visual feature, improving the distinguishability and accuracy of the features, and can simultaneously capture visual feature information of different dimensions in the image, which helps to improve the comprehensive understanding of the image content, thereby better supporting the subsequent processing and application.

[0122] In some embodiments, for the feature fusion described in the foregoing step S103, before the feature fusion, a pre-processing operation can be performed on the second feature representation of the visual feature information of different dimensions and the first feature representation of the to-be-processed image, which is used to ensure that the visual feature information is spatially matched with the to-be-processed image, so that the corresponding positions are consistent in space.

[0123] The pre-processing operation can be implemented in various ways, for example, a spatial transformation network (STN) can be used to automatically learn the alignment transformation between the features and the to-be-processed image; or the size of the feature map can be adjusted to be consistent with the original image size through feature mapping and up-sampling operation.

[0124] In the present disclosure, by performing the pre-processing on the second feature representation of the extracted visual feature information and the first feature representation of the original image before the feature fusion, the spatial consistency is ensured during the feature fusion, so that the feature information can be accurately corresponded to the corresponding area of the to-be-processed image, and the feature conversion network can more comprehensively utilize the information in the image, improving the understanding and representation ability of the image content, thereby improving the accuracy and efficiency of image processing.

[0125] Corresponding to the foregoing embodiments of the image processing method, referring to FIG. 5, the present application also provides an embodiment of an image processing device, which comprises a first encoding processing module 501, a second encoding processing module 502, a feature fusion and mapping module 503 and an image output module 504.

[0126] The first encoding processing module 501 is configured to perform feature encoding processing on the to-be-processed image to obtain a corresponding first feature representation.

[0127] The second encoding processing module 502 is configured to extract visual feature information of different dimensions of the to-be-processed image respectively, and perform feature encoding processing on the visual feature information of each dimension to obtain a corresponding second feature representation; the dimensions at least include at least one of structure information, shape information, color information, texture pattern and material information.

[0128] The feature fusion and mapping module 503 is configured to input the first feature representation and the second feature representation into the feature conversion network to obtain a third feature representation after feature fusion and feature mapping to a target feature space by the feature conversion network.

[0129] The image output module 504 is configured to perform feature decoding processing on the third feature representation to obtain a target image corresponding to the image to be processed, and the target image presents a higher product texture than the image to be processed.

[0130] In some embodiments, the apparatus further includes: receiving a control input for indicating an image processing effect, converting the control input into control information features; inputting the control information features and the second feature representation into the feature conversion network.

[0131] In some embodiments, the control input includes at least one type of information of a text type or an image type, and the conversion of the control input into control information features includes: extracting corresponding control information features from the control input according to a pre-trained multi-modal understanding network.

[0132] In some embodiments, the feature conversion network further includes a control input, and the apparatus further includes a training step of the feature conversion network, the training step including: obtaining training samples; the training samples including an image pair and labeled control inputs; the image pair including an object modeling rendered image and a corresponding object entity photographed image; performing feature encoding processing on the object modeling rendered image to obtain a fourth feature representation, and performing feature encoding processing on different dimensional visual feature information of the object modeling rendered image to obtain corresponding fifth feature representations; obtaining corresponding first control information features from the training samples according to the labels of the control inputs; inputting the first control information features, the fourth feature representation, and the fifth feature representation into the feature conversion network, and adjusting network parameters of the feature conversion network according to a network output image and the corresponding object entity photographed image.

[0133] In some embodiments, the second encoding processing module is specifically configured to: input the image to be processed into an image understanding network to obtain the visual feature information of each dimension; and the image understanding network includes an extraction submodule for extracting visual features of different dimensions.

[0134] In some embodiments, the first encoding processing module is specifically configured to: perform encoding processing on the image to be processed by using a variational autoencoder to obtain the first feature representation.

[0135] In some embodiments, the second encoding processing module is specifically configured to: encode each dimension of visual feature information respectively by using a U-shaped network U-Net encoder to obtain a second feature representation corresponding to the dimension.

[0136] In some embodiments, the device further comprises: a physical form feature extraction module configured to extract a pixel-level physical form feature from the target image after obtaining the target image corresponding to the to-be-processed image; the physical form feature is used to represent enhanced material texture details and form wrinkle information; and a fusion enhancement module configured to perform pixel-level fusion between the physical form feature and the to-be-processed image to obtain an enhanced image corresponding to the to-be-processed image.

[0137] In some embodiments, the physical form feature extraction module is specifically configured to: remove pixel-level high-frequency features in the target image by using a filter to obtain a corresponding grayscale image; and the grayscale image represents the physical form feature.

[0138] In some embodiments, the device further comprises: a mask of the to-be-processed image; the mask is used to identify a pixel region in the to-be-processed image that remains unchanged; and the physical form feature is fused with the to-be-processed image at a pixel level according to the mask.

[0139] In some embodiments, the fusion enhancement module is specifically configured to: perform fusion operation on pixel attributes in the physical form feature and pixel attributes at the same position in the to-be-processed image.

[0140] The implementation process of the functions and roles of each unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0141] For the device embodiment, since it basically corresponds to the method embodiment, the related parts can be referred to the part of the method embodiment. The device embodiments described above are only illustrative, and the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, some or all of the modules can be selected to achieve the purpose of the present application. Those skilled in the art can understand and implement without creative labor.

[0142] The electronic device provided in the embodiments of the present application includes at least one processor, a memory and a bus. The at least one processor is electrically connected with the memory. The memory is configured to store at least one computer-executable instruction. The processor is configured to execute the at least one computer-executable instruction, so as to perform the steps of any image processing method provided in any one of the embodiments or any one of the optional implementation manners of the present application.

[0143] Further, the processor 601 can be a FPGA (Field-Programmable Gate Array, field programmable gate array) or other device with logic processing capability, such as an MCU (Microcontroller Unit, microcontroller unit) or a CPU (Central Process Unit, central processing unit).

[0144] The embodiments of the present application further provide another readable storage medium storing a computer program, which is used to perform the steps of any image processing method provided in any one of the embodiments or any one of the optional implementation manners of the present application when the processor executes the computer program.

[0145] The readable storage medium provided in the embodiments of the present application includes but is not limited to any type of disk (including a floppy disk, a hard disk, an optical disk, a CD-ROM, and a magneto-optical disk), a ROM (Read-Only Memory, read-only memory), a RAM (Random Access Memory, random access memory), an EPROM (Erasable Programmable Read-Only Memory, erasable programmable read-only memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory, electrically erasable programmable read-only memory), a flash memory, a magnetic card or an optical card. That is, the readable storage medium includes any medium that can store or transmit information in a form readable by a device (for example, a computer).

[0146] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0147] While this specification contains many specifics, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptive of particular embodiments of the application. Some features described in the specification can be combined with features of other embodiments in order to produce new embodiments which fall within the scope of the application. Other combinations and sub-combinations can be made and are intended to fall within the scope of the application. Still other embodiments can be apparent to those of ordinary skill in the art from the above description which is intended to be illustrative only and not limiting. For example, while the application is described in terms of specific embodiments, it is contemplated that the application also applies to any data processing systems that fall within the scope of the claims. Furthermore, it is not intended that the application be limited by any particular preferred embodiment. The application as described and claimed herein is thus applicable regardless of whether it is described using specific and preferred language to describe the particular embodiments.

[0148] The foregoing is considered as illustrative only of the principles of the application. Numerous modifications and changes can be devised by those skilled in the art which fall within the scope of the application. Accordingly, the scope of the application should be determined by the scope of the appended claims and their equivalents.

Claims

1. An image processing method, characterized in that: The method comprises: Performing feature encoding processing on the image to be processed to obtain a corresponding first feature representation; Extracting visual feature information of different dimensions of the image to be processed respectively, and performing feature coding processing on the visual feature information of each dimension to obtain a corresponding second feature representation; the dimension includes at least one of structural information, shape information, color information, texture pattern, and material information; Inputting the first feature representation and the second feature representation into the feature conversion network to obtain a third feature representation after the feature conversion network performs feature fusion and feature mapping to a target feature space; Perform feature decoding processing on the third feature representation to obtain a target image corresponding to the image to be processed.

2. The method according to claim 1, characterized in that The method further comprises: After obtaining a target image corresponding to the image to be processed, extracting pixel-level physical morphological features from the target image; the physical morphological features are used to represent enhanced material texture details and morphological wrinkle information; The physical morphological features are fused with the image to be processed at the pixel level to obtain an enhanced image corresponding to the image to be processed.

3. The method according to claim 2, characterized in that The extracting pixel-level physical morphological features from the target image includes: The high-frequency features at the pixel level in the target image are removed by a filter to obtain a corresponding grayscale image; the grayscale image represents the physical morphological features.

4. The method according to claim 2, characterized in that The method further comprises: Constructing a mask of the image to be processed; the mask is used to identify pixel areas that remain unchanged in the image to be processed; The physical morphological features are fused with the image to be processed at pixel level according to the mask.

5. The method according to claim 2 or 4, characterized in that The pixel-level fusion of the physical morphological features and the image to be processed includes: The pixel attributes in the physical morphological features are fused with the pixel attributes at the same position in the image to be processed.

6. The method according to claim 1, characterized in that The method further comprises: receiving a control input for indicating an image processing effect, and converting the control input into a control information feature; The control information feature and the second feature representation are input into the feature conversion network.

7. The method according to claim 6, characterized in that The control input includes at least one type of information of text type and image type, and converting the control input into a control information feature includes: According to the pre-trained multimodal understanding network, corresponding control information features are extracted from the control input.

8. The method according to claim 6, characterized in that The method further includes a step of training the feature conversion network, the training step comprising: Acquire training samples; the training samples include image pairs and labeled control inputs; the image pairs include object modeling rendering images and corresponding object entity photographed images; Performing feature coding processing on the object modeling rendering image to obtain a fourth feature representation, and performing feature coding processing on visual feature information of different dimensions of the object modeling rendering image to obtain corresponding fifth feature representations; Acquiring corresponding first control information features from the training samples according to the labeling of the control input; The first control information feature, the fourth feature representation, and the fifth feature representation are input into the feature conversion network, and network parameters of the feature conversion network are adjusted according to the network output image and the corresponding object entity captured image.

9. The method according to claim 1, characterized in that The extracting visual feature information of different dimensions of the image to be processed respectively includes: The image to be processed is input into an image understanding network to obtain the visual feature information of each dimension; the image understanding network includes an extraction submodule for extracting visual features of different dimensions.

10. The method according to claim 1, characterized in that The step of performing feature encoding processing on the image to be processed to obtain a corresponding first feature representation includes: encoding the image to be processed using a variational autoencoder to obtain the first feature representation; Alternatively, performing feature encoding processing on the visual feature information of each dimension to obtain a corresponding second feature representation includes: The U-net encoder is used to encode the visual feature information of each dimension to obtain the second feature representation corresponding to the dimension.

11. An image texture processing device, characterized in that: The device comprises: A first encoding processing module is used to perform feature encoding processing on the image to be processed to obtain a corresponding first feature representation; a second encoding processing module, configured to extract visual feature information of different dimensions of the image to be processed, and perform feature encoding processing on the visual feature information of each dimension to obtain a corresponding second feature representation; the dimension includes at least one of structural information, shape information, color information, texture pattern, and material information; a feature fusion and mapping module, configured to input the first feature representation and the second feature representation into the feature conversion network to obtain a third feature representation after feature fusion and feature mapping to a target feature space by the feature conversion network; An image output module is used to perform feature decoding processing on the third feature representation to obtain a target image corresponding to the image to be processed.

12. An electronic device, characterized in that: include: Processor, memory; The memory is used to store computer programs; The processor is configured to execute the method according to any one of claims 1 to 10 by calling the computer program.

13. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images