Image style conversion method and device
By combining feature extraction and fusion of original images and point cloud data, and utilizing cross-attention mechanisms and large language models, the problem of low conversion efficiency in image stylization reconstruction is solved, achieving efficient image style conversion results.
Patent Information
- Application Number
- CN202511112105.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-08
- Publication Date
- 2025-11-21
AI Technical Summary
Existing technologies struggle to effectively improve the efficiency and quality of image transformation between different domains in image stylization reconstruction, especially the transformation between sketch images and photographs.
By acquiring the original image and its point cloud data, features are extracted using the image-text contrast learning model CLIP, visual transformer, multilayer perceptron MLP and transformer encoder, and features are fused through cross-attention mechanism and large language model LLM. Finally, 2D transposed convolution is used to reconstruct the target style image.
It enhances the effect of image style transfer, improving the efficiency and quality of image conversion between different domains.
Smart Images
Figure CN120997034A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to an image style transfer method and apparatus. Background Technology
[0002] With the development of technology, the devices available for image acquisition are becoming increasingly diverse, and the demands for the representation of images themselves are also becoming more varied. For example, in the process of forensic evidence collection, it is necessary to compare photos of multiple suspects with sketches drawn by painters based on eyewitness descriptions to identify the perpetrator. Generating suspect photos directly from the sketches would greatly improve the efficiency of suspect identification. Images taken in daily life also often need to be converted into oil painting-style images to improve visual effects. All of these applications aim to achieve the conversion of images between different representational forms, that is, the stylized reconstruction of images.
[0003] Image stylization reconstruction aims to transform an input image into one consistent with the target domain, given an image in that domain. For example, given a sketch image, how can a photograph be converted into a corresponding sketch? Because images in different domains differ significantly, even those depicting the same scene, the conversion results are often poor. Therefore, improving the efficiency of style transfer on the original image is a crucial issue in image reconstruction. Summary of the Invention
[0004] One objective of this application is to provide an image style conversion method and apparatus that, by utilizing the combination of the original image and its point cloud data, achieves image style conversion of the original image to a target style, thereby enhancing the effect of image style conversion.
[0005] According to one aspect of this application, an image style transfer method is provided, wherein the method includes:
[0006] Acquire the original image to be converted and its point cloud data, and determine the target style to be converted from the original image;
[0007] Feature extraction is performed on the original image to obtain semantic and visual features; feature extraction is performed on the target style to obtain text features; feature extraction is performed on the point cloud data to obtain point cloud features;
[0008] Information fusion is performed on the semantic features and the text features to obtain image-style features; information fusion is performed on the visual features and the point cloud features to obtain image-3D features;
[0009] The image-style features and the image-3D features are converged and fused to obtain a converged and fused feature vector.
[0010] A 2D transposed convolution is used to reconstruct the converged and fused feature vector into a style image corresponding to the target style, wherein the style image is the image corresponding to the original image when it is converted to the target style.
[0011] Furthermore, in the above method, determining the target style to be converted from the original image includes:
[0012] In response to user input, the target style to be converted from the original image and its descriptive information are obtained.
[0013] Furthermore, in the above method, feature extraction is performed on the original image to obtain semantic features and visual features, including:
[0014] The image encoder in the image-text contrast learning model CLIP is used to extract features from the original image to obtain semantic features;
[0015] The original image is used to extract visual features using a visual transformer.
[0016] Furthermore, in the above method, feature extraction is performed on the target style to obtain text features, including:
[0017] The text encoder in the image-text contrast learning model CLIP is used to extract features from the descriptive information of the target style to obtain text features.
[0018] Furthermore, in the above method, feature extraction is performed on the point cloud data to obtain point cloud features, including:
[0019] The point cloud data is processed using a multilayer perceptron (MLP) and a transformer encoder to extract features and obtain point cloud features.
[0020] Furthermore, in the above method, information fusion is performed on the semantic features and the text features to obtain image-style features, including:
[0021] The semantic features are used as the query, and the text features are used as the key and value. They are input together into the cross-attention mechanism for information fusion to obtain image-style features.
[0022] Furthermore, in the above method, information fusion is performed on the visual features and the point cloud features to obtain image-3D features, including:
[0023] The visual features are used as the Query, and the point cloud features are used as the Key and Value. They are input together into the cross-attention mechanism for information fusion to obtain image-3D features.
[0024] Furthermore, in the above method, information convergence and fusion are performed on the image-style features and the image-3D features to obtain a converged and fused feature vector, including:
[0025] The image-style features are used as the query, and the image-3D features are used as the key and value. They are input together into the cross-attention mechanism for information fusion to obtain the first fused feature vector.
[0026] The image-3D features are added element-wise to the first fused feature vector, and then input into the Large Language Model (LLM) to output the second fused feature vector.
[0027] The image-3D features are added to the second fused feature vector element by element to obtain the converged fused feature vector.
[0028] According to another aspect of this application, a non-volatile storage medium is also provided, on which computer-readable instructions are stored, which, when executed by a processor, cause the processor to implement the image style transfer method described above.
[0029] According to another aspect of this application, an image style conversion apparatus is also provided, wherein the apparatus includes:
[0030] One or more processors;
[0031] Computer-readable medium for storing one or more computer-readable instructions.
[0032] When the one or more computer-readable instructions are executed by the one or more processors, the one or more processors implement the image style transfer method as described above.
[0033] Compared with existing technologies, this application first acquires the original image to be converted and its point cloud data, and determines the target style to be converted from the original image; then, it extracts features from the original image to obtain semantic and visual features; it extracts features from the target style to obtain text features; it extracts features from the point cloud data to obtain point cloud features; next, it fuses the semantic and text features to obtain image-style features; it fuses the visual and point cloud features to obtain image-3D features; then, it converges and fuses the image-style features and the image-3D features to obtain a converged and fused feature vector; finally, it uses 2D transposed convolution to reconstruct the converged and fused feature vector into a style image corresponding to the target style, wherein the style image is the image corresponding to the original image when converted to the target style. By utilizing the combination of the original image and its point cloud data, it achieves image conversion of the original image to the target style, thus enhancing the effect of image style conversion. Attached Figure Description
[0034] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0035] Figure 1 A flowchart illustrating an image style transfer method according to one aspect of this application is shown;
[0036] Figure 2 This diagram illustrates the processing flow of an image style transfer method according to one aspect of this application in a practical application scenario.
[0037] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation
[0038] The present application will now be described in further detail with reference to the accompanying drawings.
[0039] In a typical configuration of this application, the terminal, the device of the service network, and the trusted party all include one or more processors (CPUs), input / output interfaces, network interfaces, and memory.
[0040] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0041] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.
[0042] like Figure 1 As shown, Figure 1 This is a flowchart illustrating an image style transfer method proposed in one aspect of this application. The method is applied in scenarios such as image DIY, AI face swapping, or image style transfer generation. The method includes steps S11, S12, S13, S14, and S15, specifically comprising the following steps:
[0043] Step S11: Obtain the original image to be converted and its point cloud data, and determine the target style to be converted from the original image; here, the target style may include, but is not limited to, abstract art, impressionism, realism, expressionism, futurism, metaphysics, cubism, Fauvism, Art Nouveau, surrealism, Dadaism, Suprematism, action art, pop art, minimalism, etc.
[0044] Step S12: Extract features from the original image to obtain semantic and visual features; extract features from the target style to obtain text features; extract features from the point cloud data to obtain point cloud features; here, the extraction of features carried in the three dimensions of the original image, target style, and point cloud data is realized.
[0045] Step S13: Perform information fusion on the semantic features and the text features to obtain image-style features; perform information fusion on the visual features and the point cloud features to obtain image-3D features;
[0046] Step S14: Perform information convergence and fusion on the image-style features and the image-3D features to obtain a converged and fused feature vector;
[0047] Step S15: Using 2D transposed convolution, the converged and fused feature vector is reconstructed into a style image corresponding to the target style, wherein the style image is the image corresponding to the original image when it is converted to the target style.
[0048] Through steps S11 to S15 above, by combining the original image and its point cloud data, the target style image conversion of the original image is achieved, thereby enhancing the effect of image style conversion.
[0049] Following the above embodiments of this application, the step S11 of determining the target style to be converted in the original image includes:
[0050] In response to user input, the target style to be converted from the original image and its descriptive information are obtained.
[0051] In a preferred embodiment of this application, such as Figure 2 As shown, when it is determined that a style conversion needs to be performed on the original image, the user needs to input the descriptive information of the target style required for the style conversion of the original image, such as "anime", while obtaining the original image. In response to the user's input of "anime", the descriptive information of the target style to be converted of the original image is obtained: "anime", thus realizing the acquisition and determination of the target style and its descriptive information.
[0052] Following the above embodiments of this application, step S12, which involves extracting features from the original image to obtain semantic and visual features, includes:
[0053] The image encoder in the image-text contrast learning model CLIP is used to extract features from the original image to obtain semantic features;
[0054] The original image is used to extract visual features using a visual transformer.
[0055] Here, CLIP stands for Contrastive Language-Image Pre-training, which is an image-text contrastive learning model. It is a pre-training method or model based on contrastive text-image pairs. CLIP is a multimodal model based on contrastive learning. CLIP's training data is text-image pairs: an image and its corresponding text description. The goal is to enable the model to learn the matching relationship between text-image pairs through contrastive learning.
[0056] VIT stands for Visual Transformer. It's a model based on the Transformer architecture used for visual tasks such as image classification and object detection. In Python, the Transformers library makes it easy to extract features from VIT models.
[0057] In a preferred embodiment of this application, such as Figure 2 As shown, in obtaining the original image (corresponding to...) Figure 2 After processing the original image, the image encoder in the image-text contrast learning model CLIP is used to extract features to obtain semantic features; at the same time, the VIT (visual transformer) is used to extract features to obtain visual features, thereby realizing the extraction of semantic and visual features from the original image.
[0058] Following the above embodiments of this application, step S12, which involves extracting features from the target style to obtain text features, includes:
[0059] The text encoder in the image-text contrast learning model CLIP is used to extract features from the descriptive information of the target style to obtain text features.
[0060] In a preferred embodiment of this application, such as Figure 2 As shown, after determining the target style and its descriptive information of the original image to be converted, the text encoder in the image-text contrast learning model CLIP is used to extract features from the descriptive information of the target style, such as "anime", to obtain text features. This realizes the acquisition of text features by extracting features from the descriptive information of the target style.
[0061] Following the above embodiments of this application, step S12, which involves extracting features from the point cloud data to obtain point cloud features, includes:
[0062] The point cloud data is processed using a multilayer perceptron (MLP) and a transformer encoder to extract features and obtain point cloud features.
[0063] Here, the Multilayer Perceptron (MLP) is a feedforward artificial neural network model that maps multiple input datasets to a single output dataset.
[0064] In a preferred embodiment of this application, such as Figure 2As shown, after acquiring the point cloud data of the original image, a multilayer perceptron (MLP) and a transformer encoder are used to extract features from the point cloud data to obtain point cloud features, thereby realizing the extraction of point cloud features from the point cloud data corresponding to the original image.
[0065] Following the above embodiments of this application, step S13, which involves fusing the semantic features and the text features to obtain image-style features, includes:
[0066] The semantic features are used as the query, and the text features are used as the key and value. They are input together into the cross-attention mechanism for information fusion to obtain image-style features.
[0067] In a preferred embodiment of this application, such as Figure 2 As shown, after extracting the semantic features of the original image and the text features of the target style, the semantic features are used as the Query, and the text features are used as the Key and Value, and are input into the cross-attention mechanism. The cross-attention mechanism fuses the information of the target style and the original image to obtain image-style features, thereby realizing the feature fusion of the information of the original image and the target style.
[0068] Following the above embodiments of this application, step S13, which involves fusing the visual features and the point cloud features to obtain image-3D features, includes:
[0069] The visual features are used as the Query, and the point cloud features are used as the Key and Value. They are input together into the cross-attention mechanism for information fusion to obtain image-3D features.
[0070] In a preferred embodiment of this application, such as Figure 2 As shown, after extracting the visual features of the original image and the point cloud features of the point cloud data, the visual features are used as the Query, and the point cloud features are used as the Key and Value, and are input into the cross-attention mechanism. The cross-attention mechanism fuses the 3D information carried by the point cloud data with the information of the original image to obtain the image-3D features, thereby realizing the feature fusion of the 3D information of the original image and the point cloud data.
[0071] Following the above embodiments of this application, step S14, which involves fusing information from the image-style features and the image-3D features to obtain a fused feature vector, includes:
[0072] The image-style features are used as the query, and the image-3D features are used as the key and value. They are input together into the cross-attention mechanism for information fusion to obtain the first fused feature vector.
[0073] The image-3D features are added element-wise to the first fused feature vector, and then input into the Large Language Model (LLM) to output the second fused feature vector.
[0074] The image-3D features are added to the second fused feature vector element by element to obtain the converged fused feature vector.
[0075] In a preferred embodiment of this application, such as Figure 2 As shown, after obtaining image-style features and image-3D features through information fusion, the image-style features are used as the Query, and the image-3D features are used as the Key and Value, respectively, and are input into the cross-attention mechanism. The cross-attention mechanism fuses the information of the original image, point cloud data, and target style to obtain the first fused feature vector. In order to better utilize the original image and point cloud data, the image-3D features are added to the first fused feature vector element by element, and then passed through the Large Language Model (LLM) to obtain the second fused feature vector output by the LLM. The second fused feature vector output by the LLM is then added to the image-3D features element by element to obtain the converged fused feature vector, thereby realizing the convergence and fusion of information from the original image, point cloud data, and target style. Finally, in step S15, the converged fused feature vector is reconstructed into the style image of the original image after target style conversion by 2D transpose convolution, thus realizing the conversion of the original image into the style image corresponding to the target style.
[0076] According to another aspect of this application, a non-volatile storage medium is also provided, on which computer-readable instructions are stored, which, when executed by a processor, cause the processor to implement the image style transfer method as described above.
[0077] According to another aspect of this application, an image style conversion apparatus is also provided, wherein the apparatus includes:
[0078] One or more processors;
[0079] Computer-readable medium for storing one or more computer-readable instructions.
[0080] When the one or more computer-readable instructions are executed by the one or more processors, the one or more processors implement the image style transfer method as described above.
[0081] For details of the various embodiments of the image style conversion device, please refer to the corresponding parts of the above-described image style conversion method embodiments, which will not be repeated here.
[0082] In summary, this application first acquires the original image to be converted and its point cloud data, and determines the target style of the original image to be converted; then, it extracts features from the original image to obtain semantic and visual features; it extracts features from the target style to obtain text features; it extracts features from the point cloud data to obtain point cloud features; next, it fuses the semantic and text features to obtain image-style features; it fuses the visual and point cloud features to obtain image-3D features; then, it converges and fuses the image-style features and the image-3D features to obtain a converged and fused feature vector; finally, it uses 2D transposed convolution to reconstruct the converged and fused feature vector into a style image corresponding to the target style, wherein the style image is the image corresponding to the original image when converted to the target style. By utilizing the combination of the original image and its point cloud data, it achieves image conversion of the original image to the target style, thus enhancing the effect of image style conversion.
[0083] It should be noted that this application can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.
[0084] Furthermore, a portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. The program instructions invoking the methods of this application may be stored in a fixed or removable recording medium, and / or transmitted via data streams in broadcast or other signal carrying media, and / or stored in the working memory of a computer device operating according to the program instructions. Here, one embodiment of this application includes an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein, when the computer program instructions are executed by the processor, the apparatus is triggered to operate the methods and / or technical solutions based on the foregoing embodiments of this application.
[0085] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device in software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.
Claims
1. An image style transfer method, wherein, The method includes: Acquire the original image to be converted and its point cloud data, and determine the target style to be converted from the original image; Feature extraction is performed on the original image to obtain semantic and visual features; feature extraction is performed on the target style to obtain text features; feature extraction is performed on the point cloud data to obtain point cloud features; Information fusion is performed on the semantic features and the text features to obtain image-style features; information fusion is performed on the visual features and the point cloud features to obtain image-3D features; The image-style features and the image-3D features are converged and fused to obtain a converged and fused feature vector. A 2D transposed convolution is used to reconstruct the converged and fused feature vector into a style image corresponding to the target style, wherein the style image is the image corresponding to the original image when it is converted to the target style.
2. The method according to claim 1, wherein, Determining the target style to be converted from the original image includes: In response to user input, the target style to be converted from the original image and its descriptive information are obtained.
3. The method according to claim 1, wherein, Feature extraction is performed on the original image to obtain semantic and visual features, including: The image encoder in the image-text contrast learning model CLIP is used to extract features from the original image to obtain semantic features; The original image is used to extract visual features using a visual transformer.
4. The method according to claim 2, wherein, Feature extraction is performed on the target style to obtain text features, including: The text encoder in the image-text contrast learning model CLIP is used to extract features from the descriptive information of the target style to obtain text features.
5. The method according to claim 1, wherein, Feature extraction is performed on the point cloud data to obtain point cloud features, including: The point cloud data is processed using a multilayer perceptron (MLP) and a transformer encoder to extract features and obtain point cloud features.
6. The method according to claim 1, wherein, The semantic features and the text features are fused to obtain image-style features, including: The semantic features are used as the query, and the text features are used as the key and value. They are input together into the cross-attention mechanism for information fusion to obtain image-style features.
7. The method according to claim 1, wherein, The visual features and the point cloud features are fused to obtain image-3D features, including: The visual features are used as the Query, and the point cloud features are used as the Key and Value. They are input together into the cross-attention mechanism for information fusion to obtain image-3D features.
8. The method according to claim 1, wherein, The image-style features and the image-3D features are converged and fused to obtain a converged and fused feature vector, including: The image-style features are used as the query, and the image-3D features are used as the key and value. They are input together into the cross-attention mechanism for information fusion to obtain the first fused feature vector. The image-3D features are added element-wise to the first fused feature vector, and then input into the Large Language Model (LLM) to output the second fused feature vector. The image-3D features are added to the second fused feature vector element by element to obtain the converged fused feature vector.
9. A non-volatile storage medium having stored computer-readable instructions thereon, which, when executed by a processor, cause the processor to perform the method as described in any one of claims 1 to 8.
10. An image style conversion device, wherein, The device includes: One or more processors; Computer-readable medium for storing one or more computer-readable instructions. When the one or more computer-readable instructions are executed by the one or more processors, the one or more processors perform the method as described in any one of claims 1 to 8.