Image processing methods, apparatus, computer equipment and storage media

Abstract

This application discloses an image processing method, apparatus, computer device, and storage medium, belonging to the field of image processing technology. Applied to a computer device, the method includes: obtaining a first feature map and a second feature map based on an image to be processed and a mask image of the image to be processed; obtaining a first fusion feature map based on the first feature map and the second feature map; obtaining a second fusion feature map based on the first fusion feature map and a style feature map; and obtaining a target fusion image based on the image to be processed, the mask image, and the second fusion feature map, wherein the background information in the target fusion image is the background information of the style feature map contained in the second fusion feature map. The computer device of this application can combine style feature maps, using the background information of any style feature map as the background information in the target fusion image, improving the flexibility and efficiency of background transfer in image processing.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the development of science and technology, all kinds of terminal devices have appeared in people's daily lives. People can use terminal devices to take pictures, play videos and record their lives.

[0003] After taking photos, users often edit them. For example, for selfies, users can smooth skin, add stickers, or blur areas to obtain a more suitable image. Currently, background replacement applications typically require traditional image processing algorithms to replace the background. This not only necessitates testing and feature extraction on the entire image, but also results in relatively simple background modifications and low efficiency when transferring background data. Summary of the Invention

[0004] To address the problems of existing technologies and improve the efficiency of computer devices in background migration of images, this application provides an image processing method, apparatus, computer device, and storage medium. The technical solution is as follows:

[0005] In one aspect, this application provides an image processing method applied to a computer device, the method comprising:

[0006] Based on the image to be processed and its mask image, a first feature map and a second feature map are obtained. The first feature map is a feature map obtained by extracting and encoding semantic features from the image to be processed, and the second feature map is a feature map obtained by extracting and encoding semantic features from the mask image.

[0007] A first fused feature map is obtained based on the first feature map and the second feature map, wherein the first fused feature map contains background information of the image to be processed;

[0008] Based on the first fused feature map and the style feature map, a second fused feature map is obtained, wherein the second fused feature map contains the background information of the style feature map;

[0009] Based on the image to be processed, the mask image, and the second fusion feature map, a target fusion image is obtained. The foreground information in the target fusion image is the foreground information of the image to be processed, and the background information in the target fusion image is the background information of the style feature map contained in the second fusion feature map.

[0010] In one aspect, this application provides an image processing apparatus for use in a computer device, the apparatus comprising:

[0011] The first acquisition module is used to acquire a first feature map and a second feature map based on the image to be processed and a mask image of the image to be processed. The first feature map is a feature map obtained by extracting semantic features from the image to be processed and encoding them, and the second feature map is a feature map obtained by extracting semantic features from the mask image and encoding them.

[0012] The second acquisition module is used to acquire a first fused feature map based on the first feature map and the second feature map, wherein the first fused feature map contains background information of the image to be processed;

[0013] The third acquisition module is used to acquire a second fusion feature map based on the first fusion feature map and the style feature map, wherein the second fusion feature map contains background information of the style feature map;

[0014] The fourth acquisition module is used to acquire a target fused image based on the image to be processed, the mask image, and the second fusion feature map, wherein the foreground information in the target fused image is the foreground information of the image to be processed, and the background information in the target fused image is the background information of the style feature map contained in the second fusion feature map.

[0015] In another aspect, this application provides a computer device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the image processing method as described in one aspect.

[0016] In another aspect, this application provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the image processing method as described in one aspect.

[0017] On the other hand, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to perform the image processing method as described in one aspect above.

[0018] On the other hand, embodiments of this application provide an application publishing platform for publishing computer program products, wherein when the computer program product is run on a computer, the computer performs the image processing method as described in one aspect above.

[0019] The beneficial effects of the technical solutions provided in this application include at least the following:

[0020] Based on the image to be processed and its mask image, a first feature map and a second feature map are obtained. The first feature map is obtained by extracting and encoding semantic features from the image to be processed, and the second feature map is obtained by extracting and encoding semantic features from the mask image. Based on the first and second feature maps, a first fusion feature map is obtained, which contains background information of the image to be processed. Based on the first and style feature maps, a second fusion feature map is obtained, which contains background information of the style feature map. Based on the image to be processed, the mask image, and the second fusion feature map, a target fusion image is obtained. The foreground information in the target fusion image is the foreground information of the image to be processed, and the background information in the target fusion image is the background information of the style feature map contained in the second fusion feature map. The computer device of this application can extract features based on the image to be processed and the mask image, fuse the extracted feature maps, and combine them with style feature maps to generate a second fused feature map containing background information of the style feature map. The background information of any style feature map is used as the background information in the target fused image, thereby more flexibly transferring the background information in the image to be processed to the background information of the style feature map, improving the flexibility and efficiency of the computer device in transferring the background of the image. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a structural schematic diagram of a usage scenario for modifying the background of an image, provided by an exemplary embodiment of this application;

[0023] Figure 2 This is a flowchart of an image processing method provided in an exemplary embodiment of this application;

[0024] Figure 3 This is a flowchart of an image processing method provided in an exemplary embodiment of this application;

[0025] Figure 4 This is a schematic diagram of an image selection interface according to an exemplary embodiment of this application;

[0026] Figure 5This is a schematic diagram of a first editing interface according to an exemplary embodiment of this application;

[0027] Figure 6 This is a schematic diagram of a mask image according to an exemplary embodiment of this application;

[0028] Figure 7 This is a schematic diagram of the structure of an encoder according to an exemplary embodiment of this application;

[0029] Figure 8 This is a flowchart illustrating a feature fusion process according to an exemplary embodiment of this application;

[0030] Figure 9 This is a schematic diagram of the structure of a network bottleneck resetting procedure block according to an exemplary embodiment of this application;

[0031] Figure 10 This is a schematic diagram of the structure of a second feature fusion network according to an exemplary embodiment of this application;

[0032] Figure 11 This application relates to an exemplary embodiment. Figure 10 A schematic diagram of the structure of an AMA module;

[0033] Figure 12 This is a structural block diagram of an image processing apparatus provided in an exemplary embodiment of this application;

[0034] Figure 13 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application;

[0035] Figure 14 This is a schematic diagram of the structure of a terminal device provided in an exemplary embodiment of this application. Detailed Implementation

[0036] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0037] The solution provided in this application can be used in everyday scenarios where computer devices are used for image editing, image processing, etc. To facilitate understanding, the scenario architecture of the application scenarios involved in the embodiments of this application will be briefly introduced below.

[0038] With the development of science and technology, the intelligence of terminal devices is becoming increasingly common. Various terminal devices can establish communication connections and transmit data, enabling the interaction of various video and audio data. Among these, people can use terminal devices with cameras to take photos, record videos, and document their daily lives.

[0039] After obtaining the recorded photos and videos, users can crop and edit them using installed editing applications. For example, users with mobile phones can take photos and edit them using image editing applications to modify effects such as background, color, resolution, and saturation.

[0040] Please refer to Figure 1 This illustrates a structural diagram of a use case for modifying the background of an image, provided by an exemplary embodiment of this application. Figure 1 As shown, it includes terminal device 110, server 120, and network providing device 130.

[0041] Among them, terminal device 110 can be an electronic device with image capture and image editing functions. For example, terminal device can include, but is not limited to, wearable devices (such as wristbands, smartwatches, smart glasses, etc.), mobile phones, tablets, laptops, smart glasses, smartwatches, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), desktop computers, laptop computers, smart home devices, and other terminal devices with database storage functions.

[0042] Server 120 may be at least one of the following: a single server, multiple servers, a cloud computing platform, and a virtualization center. Server 120 is used to provide background services for applications that support the virtual environment. Optionally, server 120 may undertake the primary computing task, and terminal device 110 may undertake the secondary computing task; or, server 120 may undertake the secondary computing task, and terminal device 110 may undertake the primary computing task; or, server 120 and terminal device 110 may collaborate on computing using a distributed computing architecture.

[0043] Network providing device 130 can be a WiFi device, wireless access point (AP) device, etc. in a home environment.

[0044] Terminal device 110 can establish a wireless communication connection with network providing device 130 to interact with each other or with server 120. Optionally, the terminal device establishes a communication connection with the server and then transmits data such as images and videos through the communication connection. This communication connection can be referred to as a communication network or network connection, and it uses standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to any combination of Local Area Network (LAN), Metropolitan Area Network (MAN), Wide Area Network (WAN), mobile, wired or wireless networks, private networks, or virtual private networks. In some embodiments, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can be used to encrypt all or some of the links. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0045] In the above scenario, when a user captures an image using a terminal device and needs to reshape the background, they can either use their own image processing software or send the image to a server via a communication connection for background reshaping. Currently, background replacement applications typically require traditional image processing algorithms. These algorithms not only involve testing and feature extraction on the entire image but also often result in limited background modification, leading to low efficiency in background reshaping.

[0046] To address the problems existing in the aforementioned related technologies and improve the efficiency of computer devices in transferring the background of images, this application provides an image processing method that can utilize style feature maps and fused feature maps for background transfer, flexibly changing the background information of the image to be processed into the background information of any style feature map.

[0047] Please refer to Figure 2This document illustrates a flowchart of an image processing method provided in an exemplary embodiment of this application. This image processing method can be executed by a computer device, which may be the aforementioned... Figure 1 The terminal device in the process can also be one of the above. Figure 1 The server in the middle. For example... Figure 2 As shown, the image processing method may include the following steps:

[0048] Step 201: Based on the image to be processed and its mask image, obtain a first feature map and a second feature map. The first feature map is a feature map obtained by extracting and encoding semantic features from the image to be processed, and the second feature map is a feature map obtained by extracting and encoding semantic features from the mask image.

[0049] Optionally, the image to be processed can be any image from which the user needs to perform background migration. For example, the image to be processed could be an image containing portraits and landscapes captured by the user through a terminal device. The mask image is obtained by the computer device through binarization processing of the image to be processed. For example, the computer device can divide the information contained in the image to be processed into background information and foreground information, representing the pixel values ​​of the foreground information with 0 and the pixel values ​​of the background information with 1, thereby obtaining a mask image composed of black background information and white foreground information after binarization processing.

[0050] Optionally, the computer device extracts semantic features from the image to be processed and encodes them into a first feature map. The computer device also extracts semantic features from the mask image of the image to be processed and encodes them into a second feature map. Optionally, the first and second feature maps can have the same dimension. The first feature map can be considered as a feature map with the same resolution as the image to be processed but fewer channels (e.g., a 3-channel feature map). Similarly, the second feature map can be considered as a feature map with the same resolution as the mask image of the image to be processed but fewer channels (e.g., a 3-channel feature map).

[0051] Step 202: Obtain a first fused feature map based on the first feature map and the second feature map. The first fused feature map contains background information of the image to be processed.

[0052] The computer device fuses the first feature map and the second feature map to obtain the first fused feature map. During the fusion process, the background information of the image to be processed can be preserved, ensuring that the boundary between the foreground information and the background information of the image to be processed is smoother.

[0053] Step 203: Obtain a second fusion feature map based on the first fusion feature map and the style feature map. The second fusion feature map contains background information of the style feature map.

[0054] Optionally, the computer device continues to obtain a second fusion feature map based on the obtained first fusion feature map and style feature map, thereby obtaining a second fusion feature map containing background information of the style feature map. Optionally, the second fusion feature map has the same resolution as the first fusion feature map.

[0055] Optionally, the style feature map can be pre-designed by the developers in the computer device. When a user needs to perform background style transfer on an image to be processed, the user can choose the style feature map they want to replace. The Level 93 device then fuses the style feature map selected by the user with the first fusion feature map obtained above to obtain a second fusion feature map.

[0056] Step 204: Based on the image to be processed, the mask image, and the second fusion feature map, obtain the target fusion image. The foreground information in the target fusion image is the foreground information of the image to be processed, and the background information in the target fusion image is the background information of the style feature map contained in the second fusion feature map.

[0057] Optionally, the computer device obtains a target fused image based on the image to be processed, the mask image, and the obtained second fused feature map. In the target fused image, the foreground information is the foreground information of the image to be processed, and the background information is the background information of the style feature map contained in the second fused feature map. That is, through the above processing, the computer device transfers the background information of the image to be processed to the background information of the style feature map, resulting in a target fused image where the foreground information is the foreground information of the image to be processed, and the background information is the background information of the style feature map contained in the second fused feature map.

[0058] In summary, based on the image to be processed and its mask image, a first feature map and a second feature map are obtained. The first feature map is obtained by extracting and encoding semantic features from the image to be processed, and the second feature map is obtained by extracting and encoding semantic features from the mask image. Based on the first and second feature maps, a first fusion feature map is obtained, which contains background information of the image to be processed. Based on the first and style feature maps, a second fusion feature map is obtained, which contains background information of the style feature map. Based on the image to be processed, the mask image, and the second fusion feature map, a target fusion image is obtained. The foreground information in the target fusion image is the foreground information of the image to be processed, and the background information in the target fusion image is the background information of the style feature map contained in the second fusion feature map. The computer device of this application can extract features based on the image to be processed and the mask image, fuse the extracted feature maps, and combine them with style feature maps to generate a second fused feature map containing background information of the style feature map. The background information of any style feature map is used as the background information in the target fused image, thereby more flexibly transferring the background information in the image to be processed to the background information of the style feature map, improving the flexibility and efficiency of the computer device in transferring the background of the image.

[0059] In one possible implementation, taking feature map extraction as an example, the computer device uses pre-configured encoders composed of convolutional neural networks. Each encoder has a different function, and the computer device achieves the above-mentioned functionality through these encoders. Figure 2 The image processing method shown.

[0060] Please refer to Figure 3 The diagram illustrates a flowchart of an image processing method provided in an exemplary embodiment of this application. This image processing method can be executed by a first terminal device acting as the video initiator during a video call. The first terminal device can be the one described above. Figure 1 Terminal devices in the process. For example... Figure 3 As shown, the image processing method may include the following steps:

[0061] Step 301: Determine the image to be processed.

[0062] Optionally, taking a computer device as a terminal device as an example, after a user takes any image through the terminal device, they can perform background migration processing on any image stored in the terminal device. When the user selects an image for background migration processing, the terminal device determines that the image that needs to be background migrated is the image to be processed.

[0063] For example, please refer to Figure 4This illustrates a schematic diagram of an image selection interface according to an exemplary embodiment of this application. Figure 4 As shown, the image selection interface 400 includes an image to be selected 401 and a background transition control 402. When the user selects the image to be selected 401 and triggers the background transition control 402, the terminal device can determine the image to be selected as the image to be processed. Alternatively, the computer device can also be the aforementioned server. After the user selects an image as the image to be processed on the terminal device, the user sends the image to be processed to the server, so that the server receives the image to be processed determined by the user. This application is not limited in its comparison.

[0064] In one possible implementation, when the computer device is a terminal device, the terminal device can also display a first editing interface containing a first image; determine a background migration area of ​​the first image; and determine the image to be processed based on the background migration area. For example, please refer to... Figure 5 This illustrates a schematic diagram of a first editing interface according to an exemplary embodiment of this application. Figure 5 As shown, the first editing interface 500 includes a first image 501 and a migration control 502. The user can select a portion of the first image 501 as a background migration area by triggering the migration control 502, and the terminal device will use the user-selected background migration area as the image to be processed. Similarly, the computer device can also be the aforementioned server. After the user selects an image as the image to be processed on the terminal device, the user sends the image to be processed to the server, allowing the server to receive the image specified by the user.

[0065] It should be noted that in practical applications, the above... Figure 4 and Figure 5 The display method can be through the terminal device, such as when the user triggers an editing operation in the photo album application.

[0066] Step 302: Based on the image to be processed and its mask image, obtain a first feature map and a second feature map. The first feature map is a feature map obtained by extracting and encoding semantic features from the image to be processed, and the second feature map is a feature map obtained by extracting and encoding semantic features from the mask image.

[0067] Optionally, after determining the image to be processed, the computer device can obtain a corresponding mask image for that image. The method for obtaining the mask image can be to identify foreground and background information of the image to be processed using a pre-set image recognition algorithm, and then binarize the foreground and background information to obtain the mask image of the image to be processed. Please refer to [reference needed]. Figure 6 This illustrates a schematic image of a mask image according to an exemplary embodiment of this application. Figure 6As shown, the human figure contains foreground information, as shown in the white area 601, and background information, as shown in the black area 602.

[0068] Optionally, the recognition algorithm for the image to be processed can be pre-trained using machine learning algorithms. In practical applications, users can also actively annotate the foreground and background information in the image to be processed, making it easier for computer devices to obtain the corresponding mask image.

[0069] Optionally, after obtaining the image to be processed and its mask image, the computer device inputs the image to be processed and its mask image into respective encoders composed of convolutional neural networks to generate a first feature map and a second feature map. For example, encoder A, composed of convolutional neural networks, is used to extract and encode semantic features from the image to be processed. The computer device inputs the image to be processed into encoder A, extracts semantic features from the image to be processed, and encodes them into a first feature map. Encoder B, composed of convolutional neural networks, is used to extract and encode semantic features from the mask image of the image to be processed. The computer device inputs the mask image into encoder B, extracts semantic features from the mask image, and encodes it into a second feature map.

[0070] Optionally, both encoder A and encoder B can be VGG9 networks (a 9-layer 3x3 convolutional encoding network). The following explanation uses encoder A encoding an RGB image as an example. Please refer to [link / reference]. Figure 7 This illustration shows a schematic diagram of an encoder according to an exemplary embodiment of this application. The entire encoding process is as follows: Figure 7 As shown, the input RGB image is first resized to a fixed size (320 pixels * 320 pixels), then encoded into a 16-channel feature map via a first 3x3 convolution. This is followed by the remaining 3x3 convolutions to obtain a 128-dimensional (number of channels) first feature map. Encoder B is another VGG9 network with the same structure as encoder A, and its encoding process is similar to... Figure 7 The details will not be elaborated upon here.

[0071] Step 303: Concatenate the first feature map and the second feature map to obtain a concatenated feature map.

[0072] Step 304: Input the cascaded feature map into the first feature fusion network to obtain the first fused feature map.

[0073] That is, the computer device obtains a first fused feature map based on the first feature map and the second feature map, and the first fused feature map contains background information of the image to be processed.

[0074] For example, during steps 303 to 304, the computer device concatenates the first feature map and the second feature map obtained by encoder A and encoder B to obtain a concatenated feature map, and inputs the concatenated feature map into the first feature fusion network to obtain a first fused feature map. In other words, the computer device concatenates the first and second feature maps together, transforming the original two 128-dimensional feature maps into a 256-dimensional concatenated feature map, and inputs the 256-dimensional concatenated feature map into the first feature fusion network to output the first fused feature map.

[0075] Please refer to Figure 8 This illustrates a flowchart of a feature fusion process according to an exemplary embodiment of this application. Figure 8 As shown, the diagram includes an image to be processed 801, a mask image 802, a cascaded feature step 803, a first feature fusion network 804, and a result image 805. The computer device cascades the obtained first and second feature maps and inputs them into the first feature fusion network 804 to obtain the result image 805, which is the first fused feature map. Optionally, the first feature fusion network 804 can be a feature fusion network composed of eight resnet bottleneck blocks connected in series. Please refer to [reference needed]. Figure 9 The diagram illustrates a structural schematic of a network bottleneck resetting procedure block according to an exemplary embodiment of this application.

[0076] Optionally, in this scheme, when the first feature fusion network outputs the first fused feature map, it can also calculate the MSE loss function of the first fused feature map, that is, for the above... Figure 8 The first feature fusion network 804 can not only output the resulting image 805, but also the MSE loss function of the resulting image 805. Here, f(x) represents the output image, where x is each pixel, and y represents the feature map obtained by the developer through annotation of the image to be processed. Optionally, y can be obtained by annotation using the Inpainting algorithm, or other annotation methods can be used; there are no limitations here.

[0077] Step 305: Obtain a second fusion feature map based on the first fusion feature map and the style feature map. The second fusion feature map contains background information of the style feature map.

[0078] Optionally, the computer device obtains a second fused feature map based on the obtained first fused feature map and a pre-selected style feature map. In one possible implementation, the computer device can input the first fused feature map and the style feature map into a second feature fusion network to obtain a second fused feature map. The second fused feature map has the same dimension as the first fused feature map, and the second feature fusion network contains at least three attentional manifold alignment (AMA) modules. For example, please refer to... Figure 10 This illustrates a flowchart of a second feature fusion network according to an exemplary embodiment of this application. Figure 10 As shown, it includes a first AMA module 1001, a second AMA module 1002, and a third AMA module 1003. The computer device inputs the first fused feature map and the style feature map into the second feature fusion network, according to... Figure 10 The sequence indicated by the middle arrow flows through the network and finally outputs the second fused feature map after passing through a decoder composed of eight resnet bottleneck blocks. The structure of the resnet bottleneck block can be referenced above. Figure 8 This will not be elaborated upon here. Please refer to [the relevant documentation]. Figure 11 It illustrates an exemplary embodiment of this application relating to Figure 10 A schematic diagram of the structure of an AMA module.

[0079] In one possible implementation, a style image can be determined before performing this step; based on the style image, a style feature map is obtained, which is obtained by extracting image features from the style image. The style image can be pre-stored in a computer device, which can extract features from any style image to obtain the corresponding style feature map. For example, a user can choose a specific style image, and the computer device, in response to the user's selection, inputs the style image into an encoder C composed of a convolutional neural network, where the encoder C extracts features to obtain the style feature map.

[0080] Optionally, the computer device can obtain the style feature map based on the style image in the following ways: receiving the image features to be extracted set for the style image; and obtaining the style feature map based on the image features to be extracted and the style image. In this application, the user can also independently set the image features to be extracted when performing feature extraction on the style image, such as extracting the main color tone, saturation, color distribution, etc. of the background information of the style image. When extracting image features, the computer device extracts the style image based on the image features to be extracted set by the user to obtain the style feature map.

[0081] Step 306: Based on the image to be processed, the mask image, and the second fusion feature map, obtain the target fusion image. The foreground information in the target fusion image is the foreground information of the image to be processed, and the background information in the target fusion image is the background information of the style feature map contained in the second fusion feature map.

[0082] In one possible implementation, when a computer device obtains a target fused image based on the image to be processed, the mask image, and the second fusion feature map, it can obtain the foreground information of the image to be processed through the image to be processed and the mask image, and obtain the background information of the second fusion feature map through the mask image and the second fusion feature map, and then stitch the two together to obtain the target fused image.

[0083] Please refer to the formula: Target fused image F = A & B + E & (~B);

[0084] In this diagram, A represents the image to be processed, B represents the mask image, and C represents the second fusion feature map. The computer device obtains the foreground information of the image to be processed using image A and mask image B, and obtains the background information of the second fusion feature map using mask image B and the second fusion feature map E. The two images are then stitched together to obtain the target fusion image F.

[0085] In summary, based on the image to be processed and its mask image, a first feature map and a second feature map are obtained. The first feature map is obtained by extracting and encoding semantic features from the image to be processed, and the second feature map is obtained by extracting and encoding semantic features from the mask image. Based on the first and second feature maps, a first fusion feature map is obtained, which contains background information of the image to be processed. Based on the first and style feature maps, a second fusion feature map is obtained, which contains background information of the style feature map. Based on the image to be processed, the mask image, and the second fusion feature map, a target fusion image is obtained. The foreground information in the target fusion image is the foreground information of the image to be processed, and the background information in the target fusion image is the background information of the style feature map contained in the second fusion feature map. The computer device of this application can extract features based on the image to be processed and the mask image, fuse the extracted feature maps, and combine them with style feature maps to generate a second fused feature map containing background information of the style feature map. The background information of any style feature map is used as the background information in the target fused image, thereby more flexibly transferring the background information in the image to be processed to the background information of the style feature map, improving the flexibility and efficiency of the computer device in transferring the background of the image.

[0086] In addition, the multimodal image background stylization transfer method proposed in this scheme can perform stylization transformation on only the background region end-to-end, and can autonomously select the background region and style image to achieve arbitrary style transformation, thereby improving the flexibility of background stylization transfer.

[0087] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0088] Please refer to Figure 12 This diagram illustrates a structural block diagram of an image processing apparatus provided in an exemplary embodiment of this application. The image processing apparatus 700 can be used in a computer device, which may be the one described above. Figure 1 Mid-terminal equipment, or it can be Figure 1 The server in the middle. To execute Figure 2 or Figure 3 The illustrated embodiment provides all or part of the steps performed by a computer device in the method. The image processing apparatus 1200 includes:

[0089] The first acquisition module 1201 is used to acquire a first feature map and a second feature map based on the image to be processed and a mask image of the image to be processed. The first feature map is a feature map obtained by extracting semantic features from the image to be processed and encoding them, and the second feature map is a feature map obtained by extracting semantic features from the mask image and encoding them.

[0090] The second acquisition module 1202 is used to acquire a first fused feature map based on the first feature map and the second feature map, wherein the first fused feature map contains background information of the image to be processed;

[0091] The third acquisition module 1203 is used to acquire a second fusion feature map based on the first fusion feature map and the style feature map, wherein the second fusion feature map contains background information of the style feature map;

[0092] The fourth acquisition module 1204 is used to acquire a target fused image based on the image to be processed, the mask image, and the second fusion feature map, wherein the foreground information in the target fused image is the foreground information of the image to be processed, and the background information in the target fused image is the background information of the style feature map contained in the second fusion feature map.

[0093] In summary, based on the image to be processed and its mask image, a first feature map and a second feature map are obtained. The first feature map is obtained by extracting and encoding semantic features from the image to be processed, and the second feature map is obtained by extracting and encoding semantic features from the mask image. Based on the first and second feature maps, a first fusion feature map is obtained, which contains background information of the image to be processed. Based on the first and style feature maps, a second fusion feature map is obtained, which contains background information of the style feature map. Based on the image to be processed, the mask image, and the second fusion feature map, a target fusion image is obtained. The foreground information in the target fusion image is the foreground information of the image to be processed, and the background information in the target fusion image is the background information of the style feature map contained in the second fusion feature map. The computer device of this application can extract features based on the image to be processed and the mask image, fuse the extracted feature maps, and combine them with style feature maps to generate a second fused feature map containing background information of the style feature map. The background information of any style feature map is used as the background information in the target fused image, thereby more flexibly transferring the background information in the image to be processed to the background information of the style feature map, improving the flexibility and efficiency of the computer device in transferring the background of the image.

[0094] Optionally, the fourth acquisition module 1204 includes: a first acquisition unit, a second acquisition unit, and a third acquisition unit;

[0095] The first acquisition unit is used to acquire foreground information of the image to be processed based on the image to be processed and the mask image;

[0096] The second acquisition unit is used to acquire background information of the second fusion feature map based on the mask image and the second fusion feature map;

[0097] The third acquisition unit is used to stitch together the foreground information of the image to be processed and the background information of the second fusion feature map to obtain the target fusion image.

[0098] Optionally, the device further includes:

[0099] The first determining module is used to determine the style image before obtaining the second fusion feature map based on the first fusion feature map and the style feature map;

[0100] The fifth acquisition module is used to acquire the style feature map based on the style image, wherein the style feature map is obtained by extracting image features based on the style image.

[0101] Optionally, the fifth acquisition module includes: a first receiving unit and a fourth acquisition unit;

[0102] The first receiving unit is configured to receive the image features to be extracted set for the style image;

[0103] The fourth acquisition unit is used to acquire the style feature map based on the image features to be extracted and the style image.

[0104] Optionally, the second acquisition module 1202 includes: a fifth acquisition unit and a sixth acquisition unit;

[0105] The fifth acquisition unit is used to concatenate the first feature map and the second feature map to obtain a concatenated feature map.

[0106] The sixth acquisition unit is used to input the cascaded feature map into the first feature fusion network to acquire the first fused feature map.

[0107] Optionally, the third acquisition module 1203 is further used for,

[0108] The first fused feature map and the style feature map are input into the second feature fusion network to obtain the second fused feature map. The dimension of the second fused feature map is the same as that of the first fused feature map. The second feature fusion network contains at least three AMA modules.

[0109] Optionally, the device further includes:

[0110] The first display module is used to display a first editing interface before obtaining the first feature map and the second feature map based on the image to be processed and the mask image of the image to be processed. The first editing interface contains the first image.

[0111] The second determining module is used to determine the background migration region of the first image;

[0112] The third determining module is used to determine the image to be processed based on the background migration region.

[0113] Optionally, the aforementioned computer equipment may be Figure 1 The server in the middle, Figure 13This is a schematic diagram of a computer device provided in an exemplary embodiment of this application. The computer device may be a server. The computer device 1300 includes a central processing unit (CPU) 1301, a system memory 1304 including random access memory (RAM) 1302 and read-only memory (ROM) 1303, and a system bus 1305 connecting the system memory 1304 and the CPU 1301. The computer device 1300 also includes a basic input / output system (I / O system) 1306 to facilitate information transfer between various devices within the computer, and a mass storage device 1307 for storing an operating system 1317, application programs 1314, and other program modules 1315.

[0114] The basic input / output system 1306 includes a display 1308 for displaying information and an input device 1309 for user input, such as a mouse or keyboard. Both the display 1308 and the input device 1309 are connected to the central processing unit 1301 via an input / output controller 1310 connected to the system bus 1305. The basic input / output system 1306 may also include the input / output controller 1310 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1310 also provides output to a display screen, printer, or other types of output devices.

[0115] The mass storage device 1307 is connected to the central processing unit 1301 via a mass storage controller (not shown) connected to the system bus 1305. The mass storage device 1307 and its associated computer-readable media provide non-volatile storage for the computer device 1300. That is, the mass storage device 1307 may include computer-readable media (not shown) such as a hard disk or a CD-ROM drive.

[0116] The computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technologies, CD-ROM, DVD or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 1304 and the mass storage device 1307 described above can be collectively referred to as memory.

[0117] Computer device 1300 can be connected to the Internet or other network devices via network interface unit 1311 connected to the system bus 1305.

[0118] The memory also includes one or more programs, which are stored in the memory, and the central processing unit 1301 implements the functions by executing the one or more programs. Figure 2 , Figure 3 or Figure 4 All or part of the steps of the method shown.

[0119] It should be understood that, in the embodiments of this application, the processor may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0120] Optionally, the aforementioned computer equipment may be Figure 1 For terminal devices, please refer to Figure 14 This illustrates a schematic diagram of the structure of a terminal device provided in an exemplary embodiment of this application. For example... Figure 14 As shown, the terminal device includes a processor 1410, a transceiver 1420, and a display unit 1470. The display unit 1470 may include a display screen.

[0121] Optionally, the terminal device may also include a memory 1430. The processor 1410, transceiver 1420 and memory 1430 can communicate with each other through an internal connection path to transmit ranging data. The memory 1430 is used to store computer programs, and the processor 1410 is used to call and run the computer programs from the memory 1430.

[0122] The processor 1410 and memory 1430 can be combined into a single processing device, but more commonly they are separate components. The processor 1410 executes the program code stored in memory 1430 to achieve the aforementioned functions. In specific implementations, memory 1430 can be integrated into processor 1410, or it can be independent of processor 1410.

[0123] Understandable, Figure 14The terminal device shown may include one or more processing units. For example, processor 1410 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0124] The processor 1410 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 1410 is a cache memory. This memory can store instructions or data that the processor 1410 has just used or that are used repeatedly. If the processor 1410 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 1410, and thus improves the efficiency of the system.

[0125] In some embodiments, the processor 1410 may include one or more interfaces. Interfaces may include inter-integrated circuit (IC) interfaces, inter-integrated circuit sound (IS) interfaces, pulse code modulation (PCM) interfaces, universal asynchronous receiver / transmitter (UART) interfaces, mobile industry processor interfaces (MIPI), general-purpose input / output (GPIO) interfaces, subscriber identity module (SIM) interfaces, and / or universal serial bus (USB) interfaces, etc.

[0126] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 1410 and the transceiver 1420. For example, the processor 1410 communicates with the Bluetooth module in the transceiver 1420 via the UART interface to implement Bluetooth functionality.

[0127] The MIPI interface can be used to connect the processor 1410 to peripheral devices such as the display unit 1470. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 1410 and the display unit 1470 communicate via the DSI interface to realize the display function of the terminal device.

[0128] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 1410 to the display unit 1470, transceiver 1420, etc. The GPIO interface can also be configured as an IC interface, IS interface, UART interface, MIPI interface, etc.

[0129] Transceiver 1420 can provide solutions for wireless communication applications on terminal devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. Transceiver 1420 can be one or more devices integrating at least one communication processing module; for example, it may include a Bluetooth module.

[0130] The memory 1430 can be used to store computer executable program code, which includes instructions. The memory 1430 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the terminal device (such as location data), etc. Furthermore, the memory 1430 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. The processor 1410 executes various functional applications and data processing of the terminal device by running instructions stored in the memory 1430 and / or instructions stored in memory disposed in the processor.

[0131] In addition, to further enhance the functionality of the terminal device, it may also include one or more of the following: power supply 1450, input unit 1460, audio circuit 1480, and sensor 1402.

[0132] Power supply 1450 is used to provide power to various devices or circuits in the terminal device. Preferably, power supply 1450 can be logically connected to processor 1410 through a power management device, thereby enabling functions such as charging, discharging, and power consumption management through the power management device.

[0133] The input unit 1460 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the terminal device. Specifically, the input unit 1460 may include a touch panel and other input devices. A touch panel, also known as a touch screen, can collect touch operations performed by the user on or near it, such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel, and drive corresponding connected devices according to a pre-set program. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 1410, and can receive and execute commands from the processor 1410. Furthermore, the touch panel can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel, the input unit 1460 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of function keys, trackballs, joysticks, etc.

[0134] Display unit 1470 can be used to display information input by the user or information provided to the user, as well as various menus of the terminal device. Display unit 1470 may include a display panel, optionally configured as a Liquid Crystal Display (LCD), Organic Light-Emitting Diode (OLED), or similar display panel. Furthermore, a touch panel may cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to processor 1410 to determine the type of touch event. Subsequently, processor 1410 provides corresponding visual output on the display panel based on the type of touch event.

[0135] The terminal device may also include at least one sensor 1402, such as a gyroscope sensor, a motion sensor, and other sensors. Specifically, the gyroscope sensor can be used to determine the motion posture of the terminal device. In some embodiments, the angular velocity of the terminal device around three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor. The gyroscope sensor can also be used in navigation and motion-sensing game scenarios. As a type of motion sensor, an accelerometer can detect the magnitude of acceleration in various directions (i.e., the x, y, and z axes), and when stationary, it can detect the magnitude and direction of gravity. It can be used for applications that identify the posture of the terminal device (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometers, taps), etc. Other sensors that may be configured on the terminal device, such as pressure gauges, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0136] The audio circuit 1480 may include a speaker and a microphone, providing an audio interface between the user and the terminal device. The audio circuit 1480 can convert received audio data into electrical signals and transmit them to the speaker, where the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by the audio circuit 1480, converted back into audio data, and processed by the processor 1410 before being transmitted via RF circuitry to, for example, another terminal device, or output to the memory 1430 for further processing.

[0137] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the terminal device. In other embodiments of this application, the terminal device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0138] It should be understood that, in the embodiments of this application, the processor may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0139] This application also provides a computer-readable medium storing at least one instruction, which is loaded and executed by the processor to implement all or part of the steps performed by the computer device in the image processing methods described in the above embodiments.

[0140] This application also provides a computer program product that stores at least one instruction, which is loaded and executed by the processor to implement the image processing method described in the above embodiments, including all or part of the steps performed by the computer device.

[0141] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when controlling computer equipment. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0142] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0143] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0144] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image processing method, characterized by, Applied to a computer device, the method comprises: According to the to-be-processed image and the mask image of the to-be-processed image, a first feature map and a second feature map are obtained, the first feature map being a feature map obtained by extracting semantic features from the to-be-processed image and encoding, and the second feature map being a feature map obtained by extracting semantic features from the mask image and encoding; According to the first feature map and the second feature map, a first fusion feature map is obtained, the first fusion feature map containing background information of the to-be-processed image; According to the first fusion feature map and a style feature map, a second fusion feature map is obtained, the second fusion feature map containing background information of the style feature map; According to the to-be-processed image, the mask image and the second fusion feature map, a target fusion image is obtained, the foreground information in the target fusion image being the foreground information of the to-be-processed image, and the background information in the target fusion image being the background information of the style feature map contained in the second fusion feature map.

2. The method of claim 1, wherein, The method further comprises: According to the to-be-processed image and the mask image, foreground information of the to-be-processed image is obtained; According to the mask image and the second fusion feature map, background information of the second fusion feature map is obtained; The foreground information of the to-be-processed image and the background information of the second fusion feature map are spliced to obtain the target fusion image.

3. The method of claim 1, wherein, Before the step of obtaining the second fusion feature map according to the first fusion feature map and the style feature map, the method further comprises: A style image is determined; According to the style image, the style feature map is obtained, the style feature map being obtained by extracting image features based on the style image.

4. The method of claim 3, wherein, The step of obtaining the style feature map according to the style image comprises: Receiving to-be-extracted image features set for the style image; According to the to-be-extracted image features and the style image, the style feature map is obtained.

5. The method of claim 1, wherein, The step of obtaining the first fusion feature map according to the first feature map and the second feature map comprises: The first feature map and the second feature map are concatenated to obtain a concatenated feature map; The concatenated feature map is input into a first feature fusion network to obtain the first fusion feature map.

6. The method of claim 1, wherein, The step of obtaining the second fusion feature map according to the first fusion feature map and the style feature map comprises: The first fusion feature map and the style feature map are input into a second feature fusion network to obtain the second fusion feature map, the dimension of the second fusion feature map being the same as that of the first fusion feature map, and the second feature fusion network containing at least three AMA modules.

7. The method according to any one of claims 1 to 6, characterized in that, Before the step of obtaining the first feature map and the second feature map according to the to-be-processed image and the mask image of the to-be-processed image, the method further comprises: A first editing interface is displayed, the first editing interface containing a first image; A background migration area of the first image is determined; According to the background migration area, the to-be-processed image is determined.

8. An image processing apparatus characterized by comprising: The device is applied to a computer device and comprises: The first obtaining module is configured to obtain a first feature map and a second feature map according to the to-be-processed image and a mask image of the to-be-processed image, the first feature map being a feature map obtained by extracting and encoding semantic features of the to-be-processed image, and the second feature map being a feature map obtained by extracting and encoding semantic features of the mask image; The second obtaining module is configured to obtain a first fusion feature map according to the first feature map and the second feature map, the first fusion feature map containing background information of the to-be-processed image; The third obtaining module is configured to obtain a second fusion feature map according to the first fusion feature map and a style feature map, the second fusion feature map containing background information of the style feature map; The fourth obtaining module is configured to obtain a target fusion image according to the to-be-processed image, the mask image and the second fusion feature map, the foreground information in the target fusion image being the foreground information of the to-be-processed image, and the background information in the target fusion image being the background information of the style feature map contained in the second fusion feature map.

9. A computer device, comprising: The computer device comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the image processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the image processing method according to any one of claims 1 to 7.

Images