Image processing method and device, electronic equipment and computer readable storage medium

Abstract

Embodiments of the present application provide an image processing method and device, electronic equipment and computer readable storage medium, and belong to the field of image processing. The method comprises: segmenting a to-be-processed image to obtain a target region in the to-be-processed image; and performing style conversion on the target region. Based on the scheme provided in the embodiments of the present application, the processing effect of the image can be improved, and the actual application requirements can be better met.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an image processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Deep learning methods have accelerated the development of computer vision, enabling machines to recognize multimedia information more quickly and accurately. The widespread adoption of smart terminals, especially mobile terminals, has led to a surge in image data, and people's demand for the entertainment value of images has reached unprecedented levels. Image style transfer is an urgent need for users in image editing and processing, and how to improve image processing effects has always been a very important research area in the field of image processing. Summary of the Invention

[0003] The purpose of this application is to provide an image processing method, apparatus, electronic device, and computer-readable storage medium that can effectively improve image processing results.

[0004] In a first aspect, embodiments of this application provide an image processing method, the method comprising:

[0005] The image to be processed is segmented to obtain the target region in the image;

[0006] Perform style conversion on the target area.

[0007] Secondly, embodiments of this application provide an image processing apparatus, the apparatus comprising:

[0008] The image segmentation module is used to segment the image to be processed to obtain the target region in the image;

[0009] The image style conversion module is used to perform style conversion on the target region.

[0010] Thirdly, embodiments of this application provide an electronic device, which includes a processor and a memory:

[0011] Memory, used to store machine-readable instructions;

[0012] A processor, configured to execute the method described in the first aspect when executing the machine-readable instructions described above.

[0013] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method described in the first aspect above.

[0014] The beneficial effects of the technical solution provided in this application are as follows: When performing style conversion on the image to be processed, the solution provided in the embodiments of this application obtains the target region in the image to be processed by segmenting the object to be processed, thereby realizing the style conversion of the target region in the image to be processed. Based on this solution, the effect of image style conversion can be effectively improved, and the needs of practical applications can be better met. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0016] Figure 1 A schematic flowchart of an image processing method provided in an embodiment of this application is shown;

[0017] Figure 2 A schematic diagram of a scheme for rotating an image is shown in one example of this application;

[0018] Figure 3 A flowchart illustrating a preprocessing scheme for an image to be processed, as shown in an example of this application, is presented.

[0019] Figure 4 A schematic diagram of a scheme for rotating an image is shown in one example of this application;

[0020] Figure 5 A schematic diagram of a scheme for predicting the rotation angle of an image to be processed is shown in one example of this application;

[0021] Figure 6 A flowchart illustrating a scheme for predicting the rotation angle of an image to be processed in one example of this application is shown.

[0022] Figure 7a A schematic diagram of the structure of a feature extraction unit in an existing angle prediction model is shown.

[0023] Figure 7b This paper shows a schematic diagram of the structure of a feature extraction unit of an angle prediction model provided in an example of this application;

[0024] Figure 8 A schematic diagram of a human skeleton structure is shown in one example of this application;

[0025] Figure 9 This paper illustrates a flowchart of a method for predicting the rotation angle of a person based on key points of the human skeleton, as shown in one example of this application.

[0026] Figure 10a , Figure 10b , Figure 10c and Figure 10dThe following are schematic diagrams illustrating the principle of predicting the rotation angle of a person based on key points of the human skeleton in one example of this application;

[0027] Figure 11 This illustration shows a schematic diagram of a scheme for rotating an object region in an image to be processed based on a rotation angle, as shown in one example of this application.

[0028] Figure 12 A flowchart illustrating a scheme for rotation correction and image segmentation of the image to be processed in an example of this application is shown.

[0029] Figure 13a , Figure 13b as well as Figure 13c This illustration shows a schematic diagram of a scheme for filtering and merging multiple-person regions in an image, as shown in an example of this application.

[0030] Figure 13d This is a schematic diagram of the image to be processed in one example of this application;

[0031] Figure 13e To adopt existing solutions Figure 13d The diagram shows the object segmentation results of the image to be processed.

[0032] Figure 13f for Figure 13e Enlarged schematic diagram;

[0033] Figure 13g The solution provided based on the application embodiments Figure 13d A schematic diagram of the object detection results in the image to be processed;

[0034] Figure 13h To Figure 13g The image of the object region after cropping the object detection results;

[0035] Figure 13i for Figure 13h A schematic diagram of the object segmentation results of the image shown;

[0036] Figure 14a This diagram illustrates the structure of a pyramid scene analysis network in the prior art.

[0037] Figure 14b A schematic diagram of the structure of an improved pyramid scene analysis network provided in an embodiment of this application is shown;

[0038] Figure 15 The diagram illustrates a process flow of a style transfer method based on Markov random fields for generating ground truth values ​​of style-transferred images, as shown in one example of this application.

[0039] Figure 16A schematic diagram of a generative adversarial network in one example of this application is shown;

[0040] Figure 17a A schematic diagram of the structure of an existing generator is shown;

[0041] Figure 17b A schematic diagram of the structure of a generator provided in one example of this application is shown;

[0042] Figure 18a A schematic diagram of an image probability distribution is shown;

[0043] Figure 18b This illustrates the prior art regarding Figure 18a The diagram shows the probability distribution map after binarization.

[0044] Figure 18c A schematic diagram of the probability distribution graph after linear transformation of the probability distribution graph shown in 18a is shown in one example of this application;

[0045] Figure 18d This diagram illustrates the structure of an object segmentation model in the prior art.

[0046] Figure 18e This paper shows a schematic diagram of the structure of an object segmentation model provided in an embodiment of this application;

[0047] Figure 18f This diagram illustrates the effect of multi-style image fusion based on existing image segmentation techniques.

[0048] Figure 18g A schematic diagram illustrating the effect of multi-style image fusion based on the soft segmentation technique proposed in the embodiments of this application is shown.

[0049] Figure 18h This paper illustrates a flowchart of image style transfer processing in an example of this application.

[0050] Figure 18i This paper illustrates a flowchart of image style transfer processing in another example of this application.

[0051] Figure 19 A schematic diagram illustrating the principle of an image processing method provided in an example of this application is shown;

[0052] Figure 20 This diagram illustrates the structure of an image processing apparatus according to an embodiment of this application.

[0053] Figure 21 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0054] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting the invention.

[0055] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0056] To better understand the solutions provided in the embodiments of this application, the relevant technologies involved in this application will be briefly described below.

[0057] Style transfer is a transformation from one image to another, which can be achieved using popular deep learning-based image generation methods. Deep learning methods enable machines to recognize multimedia information more quickly and accurately, especially human-related data such as faces and body poses. Achieving accurate human image segmentation is a crucial research area, particularly for the lightweight and real-time requirements of mobile devices. This opens up possibilities for further processing of portraits or backgrounds differently, enabling various entertainment functions and better meeting people's practical application needs.

[0058] In addition, the widespread use of smartphones and other mobile devices has led to a surge in image data and an unprecedented increase in people's demand for the entertainment value of images. Style transfer is an urgent need for users in image editing and processing, and it is also an important research direction for neural networks. Due to the complexity of neural networks, the processing speed of neural network-based style transfer processing on mobile devices with limited storage space and computing power is slow, making it difficult to meet users' needs.

[0059] The inventors of this application have discovered through research that existing image processing methods have at least the following technical problems or areas that need improvement:

[0060] 1) When users edit images, they may rotate the image. The objects in the image (such as people) will rotate accordingly. At this time, the angle of the object is not a normal vertical or horizontal angle. That is, the object has a large rotation angle (also called a deflection angle) relative to the central axis of the image. It can also be considered that the posture of the object in the image is abnormal or unconventional. In addition, sometimes when users take pictures, the shooting device may not be placed horizontally. In this case, the angle of the object in the obtained image is not a normal vertical or horizontal angle. That is, the posture of the object is abnormal. If style transfer processing is performed on rotated images or images taken at non-vertical or non-horizontal angles, the accuracy of determining the corresponding regions of objects in the image in existing technologies is low, resulting in poor style transfer effects. The network that determines the corresponding regions of objects in the image can be called a region segmentation network, object segmentation network, segmentation network, object segmentation model, region segmentation model, segmentation model, etc. In other words, because the objects in the image are in abnormal poses, the segmentation accuracy of existing object segmentation models is low. Even when training the object segmentation model using images containing objects in abnormal poses, it is difficult to cover all cases. This training method may also affect the segmentation accuracy of objects in normal poses.

[0061] 2) Existing technologies mostly use edge extraction to determine the region of a target object. However, edge extraction cannot distinguish object categories; for example, it cannot differentiate between a person and an animal. Therefore, users cannot flexibly select the object category when performing image style transfer. Furthermore, if two objects overlap, edge extraction cannot accurately extract the regions of both objects. In summary, existing technologies have low accuracy and flexibility in determining object regions.

[0062] 3) When a user takes a photo, if the person occupies a small proportion of the entire image, the accuracy of determining the target person area for the entire image will be low.

[0063] 4) Due to network complexity, existing object segmentation and style transfer models are slow to implement on terminal devices. Therefore, they are less likely to be implemented on terminal devices or have poor performance, making it difficult to guarantee real-time processing. In particular, style transfer models are processed in multiple iterations, which is very time-consuming and cannot achieve real-time style transfer.

[0064] 5) When performing style transfer processing on images, the existing technology does not achieve ideal fusion of the transition areas (i.e., the edges of different areas) between the foreground and background parts of the image. For example, jagged edges may appear, the edges are not natural enough, and the visualization effect is poor.

[0065] In view of at least one of the above-mentioned technical problems or areas that need improvement in the prior art, embodiments of this application provide an image processing method, apparatus, electronic device and computer-readable storage medium to improve the image processing effect and better meet the needs of practical applications.

[0066] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0067] Figure 1 This document illustrates a flowchart of an image processing method provided in an embodiment of this application. Figure 1 As shown, the method may include the following steps:

[0068] Step S110: Segment the image to be processed to obtain the target region in the image to be processed;

[0069] Step S120: Perform style conversion on the target area.

[0070] The image to be processed can be any image that requires style conversion. For example, it can be an image captured by an image acquisition device (such as a camera on a mobile terminal), an image received from another device, an image obtained from the network, or an image that has undergone other editing processes. The image to be processed can be an image selected by the user for processing, or an image that needs to be processed automatically. For example, in an application scenario, if it is pre-configured to automatically perform style conversion on all or part of the captured, received, or otherwise acquired images, then these images are the images to be processed.

[0071] It is understood that in this embodiment, the target region refers to the image region in the image to be processed that needs to be processed. Specifically, when style conversion is required for an image, the target region refers to the image region in the image to be processed that needs style conversion. The target region can be one or more of the foreground image region, background image region, and image regions containing at least one or more objects in the image to be processed. In practical applications, the target region can be configured to be a specific part of the image according to actual needs. The terminal device can automatically determine the target region based on pre-configured requirements (e.g., if style conversion is pre-configured for people in the foreground, the terminal device can automatically determine that the target region of the image is the region where the people in the foreground are located), or the user can specify the target region according to actual needs. By segmenting the image to be processed, the image can be divided into several regions. Specifically, segmentation can be based on objects in the image to obtain the target region, which is the region requiring style conversion. For example, if the region to be style-transferred is a foreground or background image region, object segmentation can be performed on the image to divide it into foreground and background image regions, thus obtaining the target region. As another example, if the target region is the image region containing a certain type of object in the image to be processed, the image region containing that type of object can be segmented based on the object's category, and the image region containing that type of object, i.e., the target region, can be obtained based on the segmentation results.

[0072] Additionally, it should be noted that in this embodiment, the object in the image to be processed refers to all or part of the main body of the image excluding the background, i.e., all or part of the foreground object in the image, or foreground object that meets the set conditions. It is understood that the main body in the image can include, but is not limited to, people, animals, buildings, etc. For example, for an image of a person, the object in the image can include the person in the image; for an image including both people and animals, the object in the image can include at least one of the people and animals in the image, which can be a designated object in the image. The designated object can be at least one object in the image, which can be specified by the user or pre-configured by the system. For example, in practical applications, the user can specify that the style transfer is being performed on the person in the image to be processed; in this case, the person is the designated object, and the area of ​​the person in the image to be processed is the target area in the image to be processed.

[0073] Furthermore, the definition of foreground and background in an image can be based on image content or on configuration information. This configuration information can be pre-configured by the system or configured by the user according to their needs. For example, the configuration information can specify which attributes or types (categories) are the foreground of the image. In this case, the portion of the image that matches the configuration information is the foreground, and the remaining portion is the background.

[0074] It should also be noted that the target region for style transfer can be a single region or multiple regions. When there are multiple target regions, each target region can correspond to the same image style or multiple image styles. Furthermore, the entire image to be processed can also require style transfer. For example, the image to be processed can be divided into a foreground image region and a background image region, both of which are target regions. The foreground image region needs to be converted to one style, and the background image region needs to be converted to another style.

[0075] For ease of description, the image to be processed will be referred to as the original image in the following text.

[0076] It is understandable that, after completing the style transfer of the target region, if the original image is segmented into target and non-target regions, in order to obtain the complete image after style transfer, the method of this application embodiment may further include:

[0077] The target region after style transfer and the non-target region in the original image are fused to obtain the target image (i.e., the final image after style transfer of the original image).

[0078] If the entire original image needs to undergo style transfer, that is, the original image is segmented into target regions, then after completing the style transfer of each target region, it may also include: fusing the style-transferred target regions to obtain the target image.

[0079] In this embodiment of the application, step S120, when performing style conversion on the target region, can either perform style conversion on the entire image to be processed or only on the target region. The following two examples illustrate these two different conversion methods in detail.

[0080] In one example, when performing style transfer, the entire original image can be style transferred, and the target region in this example can be at least one of the foreground image region and the background image region in the image. Specifically, if the user chooses to convert the foreground image of the original image to the first target style while leaving the background image unchanged, then during style conversion, the entire original image can be converted to the first target style. In post-processing, based on the object segmentation results of the original image, the foreground image region in the style-converted image is merged with the background image region in the original image before conversion to obtain the target image. Alternatively, if the user chooses to convert the background image to the second target style while leaving the foreground image unchanged, then during style conversion, the entire original image can be converted to the second target style. In post-processing, based on the object segmentation results, the background image region in the style-converted image is merged with the foreground image region in the original image before conversion to obtain the target image. Furthermore, if the user chooses to convert the foreground image to the first target style and the background image to the second target style, then during style conversion, the entire original image can be converted to both the first and second target styles respectively. In post-processing, based on the object segmentation results, the foreground image region in the image converted to the first target style is merged with the background image region in the image converted to the second target style to obtain the target image.

[0081] In another example, when performing style transfer, you can perform style transfer only on the area that needs to be transferred, i.e., the target area. Specifically, if the user chooses to convert the foreground image to the first target style while leaving the background image unchanged, the target region is the foreground image region. During style conversion, based on the object segmentation results, only the foreground image region is converted to the first target style. Then, in post-processing, the style-converted foreground image region is fused with the background image in the original image. Alternatively, if the user chooses to convert the background image to the second target style while leaving the foreground image unchanged, the target region is the background image region. During style conversion, based on the object segmentation results, only the background image region is converted to the second target style. Then, in post-processing, the style-converted background image region is fused with the foreground image region in the original image. Or, if the user chooses to convert the foreground image to the first target style and the background image to the second target style, meaning the target region includes both the foreground and background images, during style conversion, based on the object segmentation results, the foreground image region is converted to the first target style, and the background image region is converted to the second target style. Then, in post-processing, the foreground object region converted to the first target style is fused with the background image region converted to the second target style.

[0082] The image processing method provided in this application, when performing style transfer on an image to be processed, segments the image to be processed and obtains the target region that needs to be style transferred based on the image segmentation result, thereby realizing style transfer of the target region in the image to be processed. Based on this method, the effect of image style transfer can be effectively improved.

[0083] It should be noted that, in the embodiments of this application, the segmentation of the image to be processed can be performed directly on the image to be processed, or it can be performed after certain preprocessing of the object to be processed, such as detecting each object region in the image to be processed and performing style on each object region, or merging each object region and then segmenting the merged image, or performing rotation correction on the image to be processed and segmenting the image based on the corrected image.

[0084] In an optional embodiment of this application, step S110 above, segmenting the image to be processed, may specifically include:

[0085] Predict the rotation angle corresponding to the image to be processed;

[0086] Perform rotation correction on the image to be processed based on the rotation angle;

[0087] The corrected image to be processed is segmented.

[0088] When performing style transfer on images that have been rotated or edited, or images taken at non-horizontal or non-vertical angles, existing technologies typically involve directly feeding the image into an object segmentation model for segmentation. However, when objects in the image have a large rotation angle relative to the central axis, the model's segmentation results are less accurate, leading to poor style transfer performance.

[0089] To address this issue, the solution in this application embodiment, before segmenting the image to be processed to obtain the target region, can predict the rotation angle required to straighten the image, i.e., the rotation angle corresponding to the image to be processed, for images that have been rotated by the user during image editing or images taken at non-horizontal or non-vertical angles. By using the predicted rotation angle to straighten the image before determining the segmentation of the image to obtain the target region in the image, the segmentation effect of the image to be processed can be improved, and the accuracy of the obtained target region can be improved.

[0090] It should be noted that in practical applications, after predicting the rotation angle of the image to be processed, if the rotation angle is not 0 degrees, rotation correction can be performed based on that angle. Alternatively, it can be determined first whether the rotation angle meets certain conditions. If the preset conditions are met, rotation correction is performed; otherwise, it is not necessary. For example, if the rotation angle is not within a preset angle range, rotation correction may not be performed.

[0091] In an optional embodiment of this application, predicting the rotation angle corresponding to the image to be processed may specifically include:

[0092] Detect the first object region where the object is located in the image to be processed;

[0093] Predict the rotation angle corresponding to the first object region, and use the rotation angle corresponding to the first object region as the rotation angle corresponding to the image to be processed.

[0094] In other words, besides using the rotation angle of the image to be processed as the corresponding rotation angle of the image to be processed, the rotation angle of the region where the object is located in the image to be processed can also be used as the corresponding rotation angle of the image to be processed. That is, the rotation angle of a part of the image to be processed can be used as the rotation angle of the image to be processed.

[0095] As an example, such as Figure 2 As shown in the figure, the input image is the image to be processed in this example. We can first detect the region where the object is located in the input image, i.e., target object detection as shown in the figure. In this example, the first object region is the region where the person shown in the figure is located in the input image. Figure 2 The image shows the middle part of the image. Then, the rotation angle of the first object region is predicted, and the image to be processed can be rotated and corrected according to the rotation angle. That is, the rotation angle prediction and correction shown in the figure is used to obtain the rotated and corrected image, that is, the corrected image shown in the figure.

[0096] In an optional embodiment of this application, the first object region may be an object region that meets preset conditions, wherein the preset conditions may include at least one of the following:

[0097] The specified object area, the area where the specified object is located, the area where the specified type of object is located, and the object area that meets the preset area size.

[0098] In other words, object regions in the original image can be filtered, and further processing can be performed only on object regions that meet preset conditions, thereby reducing the amount of data to be processed and improving data processing efficiency. These preset conditions can be configured according to actual needs, either through system settings or user-defined settings. The object regions that meet the preset conditions can refer to objects in the image to be processed that meet the preset conditions, or the regions where objects are located that meet the preset conditions (e.g., the first object region is the region where a specified object is located). For example, the preset conditions could be based on the object's type (e.g., the first object region is the region where an object of a specified type is located), or the object's location and / or size (e.g., the first object region is the object region at a specified location or the object region meeting a preset size).

[0099] As an example, Figure 3 The diagram illustrates a preprocessing procedure for objects before segmentation. Before segmenting the image (input image shown), an image detection model can first detect the presence of objects, i.e., object regions, as shown in the diagram. If the image detection model outputs that no objects are present, object segmentation is unnecessary; the image need not be input into the model, thus reducing unnecessary workload. This advantage is particularly pronounced when batch processing images or video image sequences is required. If the image detection model outputs that objects are present, further object filtering can be performed to determine if objects meet preset conditions. For example, the object might be considered compliant based on its position and / or size. Specifically, as an option, these requirements can be pre-defined, such as pre-setting that an object meets a preset size requirement. If an object meets the preset conditions, subsequent processing is performed, such as image rotation correction or object segmentation. If an object does not meet the preset conditions, image segmentation is not performed.

[0100] In an optional embodiment of this application, predicting the rotation angle corresponding to the image to be processed may include:

[0101] Determine the rotation angle of the image to be processed based on the rotation angle of the object in the image.

[0102] As an alternative approach, in practical applications, the rotation angle of the image to be processed can be determined based on the rotation angle of the objects contained in the image. Specifically, for example, the rotation angle of the objects contained in the image to be processed, or the rotation angle of the objects in the first object region of the image to be processed, can be used as the rotation angle of the image to be processed. Figure 2 In the example shown, the rotation angle of the figure in the image can be used as the corresponding rotation angle of the image to be processed in that example. Based on this rotation angle, the image to be processed is rotated and corrected to obtain the corrected image. For example, including... Figure 4 The image of the person shown is detected to have a clockwise rotation angle of α on the image plane. The image can then be rotated counterclockwise by an angle of α to make the person in a vertical posture on the image plane (which can be considered a normal or conventional posture). Further processing can then be performed on the rotated image to improve the image processing effect.

[0103] In an optional embodiment of this application, if the image to be processed includes at least two objects, determining the rotation angle of the image to be processed based on the rotation angle of the objects in the image to be processed may specifically include:

[0104] The rotation angle of the object with the largest area in the image to be processed is determined as the corresponding rotation angle of the image to be processed; or...

[0105] The rotation angles of each object in the image to be processed are merged, and the merged rotation angle is determined as the corresponding rotation angle of the image to be processed; or...

[0106] The rotation angle of each object is determined to be the corresponding rotation angle of the image to be processed.

[0107] As an optional approach, when determining the rotation angle of each object as the corresponding rotation angle of the image to be processed, rotation correction is performed on the image to be processed based on the rotation angle corresponding to the object to be processed. Specifically, this may include:

[0108] For each object's region, rotation correction is performed according to the corresponding rotation angle.

[0109] Accordingly, the corrected image to be processed is segmented, including:

[0110] The regions of each object after rotation correction are segmented separately, and the segmentation results of each object's region are merged to obtain the segmentation result of the image to be processed. Alternatively, the regions of each object after rotation correction are merged, and the merged image is segmented to obtain the segmentation result of the image to be processed.

[0111] For images containing multiple (including two) objects or multiple objects meeting preset conditions, rotation correction can be performed on the image to be processed based on the rotation angle of the single object with the largest area in the image (i.e., the object in the first object region that occupies the largest area in the image to be processed). Alternatively, the rotation angle can be predicted for each object separately, and then the rotation angles of each object can be fused to obtain the fused rotation angle. The image can then be corrected based on the fused rotation angle. The method of rotation angle fusion is not limited in this embodiment, and may include, but is not limited to, averaging or weighted fusion of multiple rotation angles. Alternatively, the rotation angle can be calculated separately for each object in the image. For the region where each object is located, correction processing can be performed separately according to the rotation angle of its corresponding object. Then, object segmentation processing can be performed on the regions corresponding to each corrected object. The segmentation results are then merged together to obtain the segmentation result of the entire image. Or, to avoid the impact of segmenting multiple object regions separately on image processing speed, the regions where each corrected object is located can be merged into one image. The merged image can then be segmented once, and the corresponding regions of the segmentation results can be taken to obtain the segmentation result of each object. These segments can then be merged together to obtain the segmentation result of the entire image.

[0112] In an optional embodiment of this application, predicting the rotation angle corresponding to the image to be processed may include:

[0113] Method 1: Predict the rotation angle of the image to be processed using an angle prediction model. The angle prediction model includes an image feature extraction module, and the number of BN (Batch Normalization) layers in at least one feature extraction unit of the image feature extraction module is less than a set value.

[0114] This method utilizes an angle prediction model to obtain the rotation angle of the image to be processed, such as the rotation angle of an object within the image. Specifically, by training an angle prediction model based on a deep neural network, the rotation angle of an image relative to the central axis of the image can be obtained, encompassing any content including humans, buildings, nature, and scenery.

[0115] In practical applications, the rotation angle can be represented as a clockwise rotation angle or a counter-clockwise rotation angle. For ease of description, unless otherwise specified, the rotation angle will be described using a clockwise example. Furthermore, the rotation angle described in this embodiment can be an actual rotation angle value or an angle range corresponding to the rotation angle. For example, it can be any value between 0 and 360 degrees, or it can be the range of the object's actual rotation angle, such as 0-15 degrees, 16-30 degrees, 31-45 degrees, etc. In other words, in practical applications, besides using a rotation angle value, the rotation angle space (i.e., 0-360 degrees) can be divided into multiple angle ranges, and the range where the object's rotation angle lies can be used as the object's rotation angle. In this case, a corresponding rotation angle value can be configured for each angle range. Once the angle range where the object's rotation angle lies is determined, rotation correction can be performed according to the rotation angle value corresponding to that range.

[0116] As an example, for such Figure 5 The flowchart illustrating the process of predicting the rotation angle of objects in the image to be processed using a pre-trained angle prediction model is shown below. Figure 6 As shown, specifically, the image to be processed can be used as the input image of a deep neural network. The deep neural network extracts features from the input image and predicts the rotation angle based on the extracted feature information, such as... Figure 5 The character's rotation angle (the predicted angle shown in the image) is 15 degrees, that is... Figure 5 The figure in the image is deflected at an angle of 15 degrees relative to the vertical central axis of the image.

[0117] When performing image processing using neural networks, the complexity of these networks results in slow processing speeds on mobile devices with limited storage and computing power, making it difficult to meet user needs. Therefore, how to perform lightweight optimization of neural networks to make them suitable for mobile devices, and not just limited to their deployment, is a crucial issue that needs to be addressed.

[0118] To address this issue, in the first method described in this application embodiment, a lightweight feature extraction module (corresponding to...) can be used when performing feature extraction through a deep neural network. Figure 6A lightweight feature extraction method (in image feature extraction modules) is implemented to be suitable for operation on mobile terminal devices, meaning that the number of BN layers in at least one feature extraction unit of the image feature extraction module is less than a set value. By reducing the number of BN layers in the feature extraction unit, the overall structure of the neural network is simplified, and the computational load of the neural network during image processing and the memory usage on the device are reduced, thereby improving the portability and versatility of the neural network and enabling it to be deployed on resource-constrained mobile terminal devices.

[0119] The statement that the number of BN layers is less than a set value means that the number of BN layers in the feature extraction unit is less than the number of BN layers in commonly used feature extraction units. In other words, the maximum value of the set value can be the number of BN layers in commonly used feature extraction units. Of course, the set value can also be zero, meaning that at least one feature extraction unit of the image feature extraction module in this application example may not include BN layers.

[0120] As an example, Figure 7a The diagram illustrates the structure of a basic feature extraction unit (shown in the dashed box) in the existing MobileNetV2 technology (one or more of which can be used in practice). As shown, this feature extraction unit can consist of depthwise convolution layers, several batch normalization (BN) layers, and convolution layers, outputting image feature information. BN layers are deployed between adjacent convolution layers (the depthwise and convolutional layers shown in the diagram). The convolutional and depthwise convolutional layers are responsible for basic feature extraction, while the BN layers assist the convolutional layers. BN layers can accelerate and facilitate the training process of deep learning neural networks. In some networks, the presence of BN layers is crucial; without them, the entire network may fail to converge or function properly. However, in other networks, training and normal operation can be completed even without BN layers.

[0121] As an example, Figure 7b This figure shows a schematic diagram of a basic unit for lightweight feature extraction that can be deployed in a mobile terminal device, as provided in an embodiment of this application (as shown in the dashed box). As shown in the figure, compared to... Figure 7aThe structure shown illustrates a lightweight feature extraction unit that simplifies the overall structure by removing some batch normalization (BN) layers. Experiments demonstrate that this lightweight feature extraction unit structure enables the network to function correctly during use. Therefore, BN layers are unnecessary in the deployment of this embodiment and do not significantly impact the overall network performance. On the other hand, using BN layers incurs certain costs. BN layers increase computational load and memory usage, and they reduce the network's portability and versatility, as not all mobile platforms support BN layers. Therefore, the benefits of removing BN layers from the lightweight basic unit are obvious: it not only reduces computational load but also improves the network's platform compatibility.

[0122] In an optional embodiment of this application, if the rotation angle of the image to be processed is determined based on the rotation angle of the object in the image to be processed, the rotation angle of the object in the image to be processed can be predicted by the following method:

[0123] Method 2: Detect key points of the object skeleton using a pose estimation model, and predict the rotation angle of the object based on the key points of the object skeleton.

[0124] In this approach, when determining the rotation angle of an image based on the rotation angle of an object in the image, the rotation angle of the object can be obtained through keypoint information detected by the pose estimation model.

[0125] As described above, objects in an image can include, but are not limited to, people, animals, or other objects with skeletal keypoints. In practical applications, the specific skeletal keypoints to be detected, and the specific keypoints used to predict the object's rotation angle, can be configured according to actual needs and / or application scenarios. Of course, the same pose estimation model can be used for detecting the skeletal keypoints of different objects, or different pose estimation models can be used for different objects.

[0126] As an example, the following explanation uses a human figure as an example to illustrate the key points of the human skeleton. For a human figure, a human pose estimation model can be used to detect the key points of the human skeleton. The results of the human pose estimation can include, but are not limited to, those mentioned above. Figure 8 The diagram shows a human skeleton composed of 17 key points (hereinafter referred to as bone nodes), which may also include the hip bone nodes. For ease of description, Figure 8 Each skeletal node has been numbered, and the following description will use the numbers shown in the figure to represent the corresponding skeletal nodes, as detailed below:

[0127] Bone nodes 1 and 2 are bone nodes in the eye area;

[0128] Bone nodes 4 and 5 are bone nodes in the ear area;

[0129] Bone node 3 is a bone node in the nasal region;

[0130] Bone nodes 6 and 9 are bone nodes in the shoulder area;

[0131] Bone nodes 7 and 10 are bone nodes in the elbow area;

[0132] Bone nodes 8 and 11 are bone nodes in the wrist region;

[0133] Bone nodes 12 and 15 are bone nodes in the lumbar region;

[0134] Bone nodes 13 and 16 are bone nodes in the knee area;

[0135] Bone nodes 14 and 17 are bone nodes in the ankle area.

[0136] For ease of description, the corresponding bone nodes can be represented by p1 to p17 in the following text. For example, p6 represents bone node 6.

[0137] Based on Method 2, in this example, the rotation angle of the human body in the image can be predicted based on the above pose estimation results, i.e., the detected human skeleton key points. The specific method for human pose estimation is not limited in this embodiment.

[0138] In an optional embodiment of this application, predicting the rotation angle of a person based on key points of the object's skeleton may include:

[0139] Predict the rotation angle of the object based on the positional relationship of at least two key points in the object's skeleton keypoints.

[0140] Since two keypoints can form a line, the rotation angle of an object can be predicted based on at least two object skeleton keypoints. The positional relationship between these two keypoints refers to the positional relationship that characterizes the deviation angle of the object relative to the corresponding image axis (which can be a horizontal or vertical axis). The specific positional relationship can be configured according to actual needs, and this application does not limit the specific form of the positional relationship. For example, the positional relationship may include, but is not limited to, the line connecting two keypoints; the deviation angle of this line from the image axis can then be used as the rotation angle of the object. When there are at least three object skeleton keypoints, the positional relationship can also refer to the positional relationship between the line connecting two keypoints and other keypoints, etc.

[0141] In an optional embodiment of this application, at least two key points include at least two corresponding key points.

[0142] Specifically, the two corresponding key points can refer to the corresponding bone nodes on the left and right sides of the object. For example, for a human figure, the two corresponding key points of the human skeleton can refer to the corresponding skeletal key points on the left and right sides of the body, such as... Figure 8 The bone nodes 6 and 9 shown are bone nodes in the shoulder area, and bone nodes 12 and 15 are bone nodes in the waist area. These are two corresponding key points.

[0143] Since the key points of the skeletons of two objects are generally symmetrical when the objects are relatively upright, the rotation angle of the objects can be predicted based on the positional relationship between the corresponding key points, thereby improving the accuracy of the prediction.

[0144] As an optional approach, continue with Figure 8 The example shown is used to illustrate this. Figure 9 The image shows a method based on Figure 8 The diagram shows a flowchart illustrating how key points of the human skeleton are used to predict the rotation angle of the human body. Figure 10a , Figure 10b , Figure 10c and Figure 10d The diagram illustrates four methods for calculating human body rotation angles. Specifically, such as... Figure 9 , Figure 10a , Figure 10b , Figure 10c and Figure 10d As shown, the methods for predicting the rotation angle of the human body in this example can mainly include:

[0145] (1) If the number of bone nodes is 0, it can be assumed that there are no people in the image. Therefore, the rotation angle of the image is 0. Otherwise, proceed to (2).

[0146] (2) Since the shoulder and waist bone nodes best reflect the posture of the human body, this solution proposes that the shoulder and waist bone nodes can be detected in sequence first. If bone node p6 and bone node p9 are detected, then proceed to (3); otherwise proceed to (6).

[0147] (3) If bone node p12 and bone node p15 are detected, proceed to (4); otherwise proceed to (5).

[0148] (4) Calculate the human body rotation angle according to Scheme 1, such as Figure 10a As shown, the two endpoints of line segment l1 are the midpoint A of the line connecting bone node p6 and bone node p9, and the midpoint B of the line connecting bone node 12 and bone node 15. Calculate the angle α1 between l1 and the image y-axis (vertical central axis). Then α1 is the rotation angle of the human figure (the rotation angle of the human body).

[0149] (5) Calculate the human body rotation angle according to Scheme 2, such as Figure 10b As shown, the two endpoints of line segment l2 are bone node p6 and bone node p9, respectively. Calculate the angle α2 between l2 and the image x-axis (horizontal central axis, i.e., horizontal central axis). Then α2 is the rotation angle of the portrait.

[0150] (6) If bone node p12 and bone node p15 are detected, proceed to (7); otherwise proceed to (8).

[0151] (7) Calculate the human body rotation angle according to Scheme 3, such as Figure 10c As shown, the two endpoints of line segment l3 are bone node p12 and bone node p15, respectively. Calculate the angle α3 between l3 and the x-axis of the image. Then α3 is the rotation angle of the portrait.

[0152] (8) If bone nodes p1 to p3 are detected, then proceed to (9); otherwise, the rotation angle of the image is considered to be 0.

[0153] (9) Calculate the human body rotation angle according to Scheme 4, such as Figure 10d As shown, the two endpoints of line segment l4 are the midpoint C of the line connecting bone node p1 and bone node p2, and bone node p3, respectively. Calculate the angle α4 between l4 and the y-axis of the image, then α4 is the rotation angle of the portrait.

[0154] It should be noted that the methods for predicting rotation angles provided in the embodiments of this application can be used individually or simultaneously. For example, as an example, method one and / or method two can be used to predict the rotation angle of an object in the image to be processed. The rotation angle of the image to be processed is determined based on the rotation angle of the object predicted by method one and / or method two. For example, the rotation angle of the image can be determined based on the rotation angle predicted by one of the methods. Alternatively, the rotation angles of the object predicted by method one and method two can be fused, and the fused angle can be used as the rotation angle of the image. In addition, different methods can be used to predict the rotation angle of objects based on the object category in the image. For example, method two can be used to predict the rotation angle of a person in the image, and method one can be used to predict the rotation angle of other objects in the image besides people. The rotation angle of the image to be processed is then determined based on the prediction results obtained by at least one method. It is understood that this example only lists a few optional solutions. In practical applications, the specific selection scheme can be set by the user based on the two methods provided in the embodiments of this application, according to actual needs or by providing user setting options.

[0155] In an optional embodiment of this application, rotation correction of the image to be processed based on the rotation angle may include:

[0156] Perform rotational correction on the first object region;

[0157] The corrected image to be processed is segmented, including:

[0158] The corrected first object region is segmented;

[0159] Based on the segmentation results of the first object region, the segmentation results of the object to be processed are obtained.

[0160] When the proportion of objects (such as people) in the image to be processed is small, segmenting the entire image to determine the target region will result in low segmentation accuracy and a large amount of data to process, leading to low processing efficiency. To address this issue, as an alternative approach, when performing rotation correction on the image to be processed, one can either rotate the entire image or rotate only the object region in the original image, then segment the rotated object region, and finally obtain the segmentation result of the original image based on the segmentation result of the rotated object region. This method reduces the amount of data to be processed and improves image segmentation efficiency because it avoids segmenting the entire original image, and it also maintains image segmentation accuracy by segmenting only the rotated object region.

[0161] As an example, such as Figure 2 The image in the middle section shown is the object region, so image rotation correction can be performed only on this region, such as... Figure 11 As shown, the rotated region image is obtained. Then, when performing image segmentation, only the rotated part of the image can be segmented to obtain the segmented human and non-human parts. The segmentation result is then rotated in reverse based on the predicted rotation angle, and the reverse rotation result is merged with the non-object region in the original image to obtain the segmentation result of the original image.

[0162] In an optional embodiment of this application, segmenting the image to be processed may include:

[0163] Detect the second object region where the object in the image to be processed is located;

[0164] Each second object region can be segmented separately; or, the second object regions can be merged, and the merged image can be segmented.

[0165] Based on the segmentation results of the second object region, the segmentation results of the image to be processed are obtained.

[0166] It should be noted that the image to be processed in this scheme can be either an image without rotation correction or an image after rotation correction. The second object region in this scheme may be the same as or different from the first object region described above. The first object region is used to determine the rotation angle corresponding to the object to be processed when performing rotation correction. The rotation angle corresponding to the first object region is determined as the rotation angle corresponding to the object to be processed. During rotation correction, the image to be processed or the first object region can be corrected based on this rotation angle to obtain the corrected image. The second object region is the object region in the image to be segmented when segmenting the image to be processed or the rotated image. Image segmentation can be performed by segmenting each second object region in the image, or by merging the second object regions before style processing.

[0167] In addition, in practical applications, when performing rotation correction on an image based on the rotation angle corresponding to the first object region, the segmentation of the image to be processed can be performed by detecting each second object region and segmenting based on each second object region, or by segmenting the corrected first object region as described above and obtaining the segmentation result of the object to be processed based on the segmentation result of the first object region.

[0168] Optionally, the second object region can be segmented separately, which may specifically include: enlarging or shrinking each second object region, and then segmenting each enlarged or shrunk second object region separately.

[0169] Optionally, the second object regions are merged, and the merged image is segmented. Specifically, this may include: merging the second object regions, enlarging or reducing the merged image, and segmenting the enlarged or reduced merged image.

[0170] To improve segmentation accuracy when segmenting an image or a rotated image, object regions can be detected first. By segmenting the detected object regions, the amount of data to be processed can be reduced while improving segmentation accuracy.

[0171] As an alternative, to further improve the segmentation accuracy of the image, the object region in the corrected image, i.e., the second object region mentioned above, can be detected after the rotation correction of the image is completed. That is, the object region in the corrected image is detected again after the image correction is completed. Since the detection is based on the corrected image, the detected object region is more accurate. The image to be processed is accurately segmented by segmenting the re-detected object region, and the amount of data to be processed can also be reduced.

[0172] When a second object region is detected, each second object region can be segmented separately to obtain segmentation results for each region. Alternatively, when there are multiple second object regions, to improve segmentation efficiency and avoid the impact of separate segmentation on segmentation speed, multiple second object regions can be merged into a single image before segmentation. Furthermore, to improve segmentation accuracy, when segmenting the image to be segmented (each second object region or the merged image), to ensure segmentation accuracy even when the image to be segmented or the object region within it is small, the image to be segmented can be first enlarged before segmentation. Correspondingly, after segmentation, the segmentation result can be downsized to obtain the final segmentation result. Conversely, when the image to be segmented is large or the object region within it is large, to ensure segmentation efficiency, the image to be segmented can be downsized before segmentation. Similarly, after segmentation, the segmentation result can be enlarged to obtain the final segmentation result.

[0173] Furthermore, based on the solution provided in the embodiments of this application, in addition to improving segmentation accuracy and speed, the method of first detecting the location of each object can filter out irrelevant object categories in the image. Therefore, compared with directly using segmentation processing, adding this detection processing can reduce the error probability of the segmentation result containing irrelevant categories and improve segmentation performance. Higher segmentation accuracy will translate into better style transfer quality.

[0174] As an example, such as Figure 12As shown in the figure, before segmenting the object to be processed by the object segmentation model, a detection module (target object detection shown in the figure) and an object rotation angle prediction and correction module can be added before the object segmentation model. The detection module can detect the object region in the input image (i.e., the first object region), which is the middle image at the top of the figure. The object rotation angle prediction and correction module predicts the rotation angle of the object in this region, and performs rotation correction on the input image based on the rotation angle to obtain the corrected image. Then, the object position in the corrected image is detected again to obtain the object region at this time, which is the second object region. In this example, specifically... Figure 12 The rightmost image in the bottom three images shows that the object region detected based on the rotation-corrected image is significantly more accurate than the object region detected based on the original image. Using this second object region as the input image for the object segmentation model yields a more accurate object segmentation result (the middle image in the bottom three images). This result can then be processed to generate the object segmentation result of the original image for subsequent image processing. Specifically, the precisely segmented image shown in the image can be rotated in the opposite direction according to the rotation angle, and then the rotated image can be fused with the image regions in the original image except for the rotated region to obtain the accurate segmentation result of the original image, which is the leftmost image in the bottom three images.

[0175] Specifically, after detecting the first object region and predicting the corresponding rotation angle α, the first object region can be cropped and rotated based on the following transformation matrix to obtain the desired result from the input image. Figure 12 The middle image at the top shown:

[0176] M = RT

[0177]

[0178] Where R is the rotation correction matrix, T is the cropping matrix, dx and dy are the position information of the object region in the image to be processed (usually the x and y coordinates of the top left corner of the object region), and sx and sy represent the scaling ratio of the object region when cropping it. For example, if the width and height of the object region in the image to be processed are w and h respectively, and the width and height of the cropped object region are W and H respectively, then sx = w / W and sy = h / H. It can be understood that if no scaling is performed, then w = W and h = H.

[0179] As described above, another alternative approach is to segment the image based on the unrotated image, such as... Figure 12In the example shown, after the detection module detects the region of the object in the input image (the middle image at the top shown in the figure), the object region can be directly segmented to obtain the object segmentation result. The object segmentation result is then fused with the image regions in the original image other than the object region to obtain the segmentation result of the original image.

[0180] In an optional embodiment of this application, the second object region is an object region in the image to be processed or the corrected image to be processed that meets preset conditions.

[0181] Similar to the principle described above, where the first object region is an object region that meets the preset conditions, the second object region can also be an object region in the image to be processed or in the rotated and corrected image that meets the preset conditions, so as to reduce the amount of data to be processed and improve data processing efficiency.

[0182] It is understandable that the preset conditions corresponding to the first object area and the preset conditions corresponding to the second object area can be the same or different, and can be configured and adjusted according to actual application needs.

[0183] In addition, when an image contains multiple objects that meet the preset conditions, that is, when there are multiple second object regions, as an option, the regions of all objects can be merged before subsequent processing to avoid the speed reduction caused by segmenting multiple object regions separately.

[0184] As an example, such as Figure 13a The image shown is an image to be processed or an image after rotation correction. For this image, the image detection model detects the second object region, and its output is three object regions, namely object regions ①, ②, and ③ shown in the figure. Among them, object regions ① and ② meet the preset conditions, while object region ③ does not meet the preset conditions, so object region ③ is ignored. When segmenting the detected object regions ① and ②, object regions ① and ② can be merged first as the input image for subsequent segmentation processing. The merging method is not limited in this embodiment and can be configured according to actual needs. For example, the merging method can simply use the regions of the image corresponding to the maximum bounding boxes of multiple objects (such as...). Figure 13b (As shown) can be used as input for subsequent segmentation processing, or an image stitched together from multiple object regions can be used (e.g. Figure 13c (As shown in the figure) is used as input for subsequent segmentation processing.

[0185] The specific method for detecting object regions (first object region and / or second object region) in the image is not limited in this embodiment and can be configured according to actual needs. For example, as an optional solution, object detection can be performed using methods based on YOLO (You Only Look Once) or SSD (Single Shot MultiBox Detector). These detection models have small storage requirements and fast running speed, making them suitable for deployment on mobile terminal devices.

[0186] Compared to existing image segmentation schemes that directly segment the original object to be processed, the scheme provided in this application, which first detects object regions in the image to be processed or the rotated corrected image to be processed, and then segments each object region separately or merges the object regions before segmentation, can effectively improve the accuracy of image segmentation. To better illustrate this effect, a specific example is provided below.

[0187] Figure 13d The diagram shows a schematic of an image to be processed provided in this example, with the object in the diagram being the task shown. Figure 13e This demonstrates the use of existing image segmentation methods for... Figure 13d The diagram shows the image after segmentation. Figure 13f for Figure 13e An enlarged illustration of the character / object portion, by Figure 13e and Figure 13f It can be seen that the image segmentation results obtained by using the existing method have low segmentation accuracy and the object edge segmentation effect is blurry.

[0188] for Figure 13d The object to be processed shown can first be detected in the image using image detection models (such as YOLO, SSD detection models), for example... Figure 13g The object region detection result shown is illustrated here. dx and dy represent the coordinates of the top-left corner of the bounding box of the object region in the image to be processed, respectively. The origin of these coordinates is typically the bottom-left corner of the image to be processed. w and h represent the width and height of the bounding box, which are the width and height of the detected object region, respectively. After detecting the object region, the image to be processed can be cropped based on the positional information of the detected object region (including dx, dy, w, and h) to obtain the image of the object region, as shown below. Figure 13h As shown, the object region can then be segmented to obtain... Figure 13i The segmentation results shown are compared with Figure 13i and Figure 13fIt can be seen that the solution based on the embodiments of this application effectively improves the image segmentation accuracy. Specifically, when cropping the image to be processed to obtain the image of the object region, it can be obtained through the following expression:

[0189]

[0190] Where x and y represent the coordinates of the image coordinate points of the object region in the image to be processed, respectively. Figure 13g The coordinates of the object region shown are x′ and y′, which are the coordinates of the corresponding points in the cropped object region image. Figure 13h Zhongyu Figure 13g The coordinates of the points corresponding to the object region in the image are given. sx and sy represent the image scaling ratios; specifically, sx = w / W, sy = h / H, where W and H represent the width and height of the scaled image, respectively. Figure 13h The image shows the width and height of the cropped object region.

[0191] In an optional embodiment of this application, segmenting the image to be processed may include:

[0192] The image to be processed is segmented using an object segmentation model;

[0193] The object segmentation model is obtained by training a neural network.

[0194] In addition, object segmentation models can also be models that segment input images based on the categories of objects in the input image.

[0195] In existing technologies, edge extraction is mostly used to determine the region of an object. However, edge extraction cannot distinguish object categories, such as whether an object in an image is a person or an animal. Therefore, it cannot flexibly select the target object category when performing image style transfer. Furthermore, if there are overlapping areas between two or more objects in an image, edge extraction cannot accurately extract the regions of both objects. Therefore, existing technologies have low accuracy and flexibility in determining object regions, resulting in less than ideal image segmentation results.

[0196] To address this issue, in an optional embodiment of this application, a neural network-based object segmentation method can be used when determining the region of an object. This method can not only achieve segmentation based on different object categories, but also accurately segment the objects of each target even when there are overlapping regions between objects, thereby improving the accuracy or flexibility of determining the object region.

[0197] In an optional embodiment of this application, the target region may include the region where objects of a specified category are located in the image to be processed.

[0198] Since the object segmentation model provided in this application can achieve image segmentation based on object categories, in practical applications, style transfer can be performed on regions containing one or more objects of a specified category in an image according to actual needs. The specified category can be a system-preconfigured category or a category specified by the user according to actual needs.

[0199] In an optional embodiment of this application, the method may further include:

[0200] Receive a style transfer request. The style transfer request includes the specified category of the object to be style transferred. That is, the style transfer request can carry the category of the object. This object category is the specified category. In other words, the user specifies the object of this category as the image to be style transferred. The area where this object is located in the image to be processed is the target area.

[0201] In other words, when performing style transfer on an image, users can flexibly select the object categories to be style transferred according to their actual needs, better meeting user requirements. Of course, it is understandable that in practical applications, if the area in the image that needs style transfer does not include the area containing one or more objects of a specified category, the style transfer request can still include the specified object category. In this case, the specified category is the category of the object that does not need style transfer.

[0202] In an optional embodiment of this application, the object segmentation model includes PSPNet (Pyramid Scene Parsing Network), wherein the number of layers in the pyramid pooling module of PSPNet is less than a set number of layers. This reduces the number of layers in the pyramid pooling module, achieving a lightweight model.

[0203] To enable the object segmentation model to be better adapted to more terminal devices and to run normally on mobile terminal devices, the object segmentation model used in this embodiment is an improvement on the original PSPNet. By reducing the number of layers in the pyramid pooling module in the existing PSPNet, the model is lightweighted while maintaining segmentation accuracy, making it well-suited for mobile terminal devices.

[0204] The original PSPNet provided a global context prior structure, namely the Pyramid Pooling Module. This module gave the network structure a hierarchical global priority, containing information at different scales between different sub-regions, resulting in more accurate segmentation results. For example... Figure 14aAs shown, in the original PSPNet, for the image to be processed (the input image shown in the figure), a feature extraction network (the residual network shown in the figure) is first used to obtain the feature map of the input image. The feature map is used as the input to the pyramid pooling module, which fuses features at four different pyramid scales. As shown in the figure, the feature map is pooled at four different scales to obtain four feature maps, namely the features shown in the figure. Figure 1 To characteristics Figure 4 To control the proportion of global features, the results of each pooling are then passed through their respective convolutional layers to refine the features. Figure 1 To characteristics Figure 4 Dimensionality reduction is performed to obtain the reduced features. Figure 1 'To characteristics Figure 4 ', then the low-dimensional features Figure 1 'To characteristics Figure 4 Upsampling is performed, and the upsampled feature maps are concatenated with the feature maps input to the pyramid pooling module (also known as splicing) to obtain the final feature map. The final feature map is then input into the convolutional layer to obtain the final image segmentation result, which is the output image shown in the image.

[0205] To better suit the model for terminal devices, as an optional solution, the third layer of the pyramid pooling module can be removed from the object segmentation model used in this embodiment, i.e., the layer for features... Figure 3 The operation yielded information at three different pyramid scales. Removing this layer preserved the multi-scale information of PSPNet, thus ensuring segmentation accuracy, while also reducing the network structure, allowing the model to run faster on terminals and thus enabling it to run on more types of terminal devices, including mobile devices. In the example above, the number of layers can be set to 4, meaning that the pyramid pooling module has fewer than 4 layers.

[0206] As an example, Figure 14b The diagram shows a structural schematic of an object segmentation model provided in an embodiment of this application. Figure 14a In contrast, this model removes the third layer of the pyramid model, namely the 3*3 convolutional kernel layer shown in the figure. Based on this model, features at three different pyramid scales can be obtained. While maintaining the multi-scale characteristics of the pyramid, it reduces the parameters of one scale, thus ensuring accuracy and reducing the running time of the model.

[0207] In an optional embodiment of this application, style transfer of the target region includes:

[0208] Image style transfer is performed on the target region using a style transfer model, where the style transfer model includes a Generative Adversarial Network (GAN). The network structure of the generator in a GAN satisfies any of the following:

[0209] It includes cascaded convolutional layers, residual networks (also known as residual connectives), and deconvolutional layers;

[0210] This includes cascaded convolutional and deconvolutional layers;

[0211] It includes convolutional layers, depthwise convolutional layers, and deconvolutional layers that are cascaded in sequence;

[0212] It includes sequentially cascaded convolutional layers, deep convolutional layers, residual networks, and deconvolutional layers;

[0213] In a network structure, the number of channels in at least one convolutional layer is less than a set number.

[0214] The network structure of the GAN generator provided in this application embodiment, compared with existing network structures, can achieve a lightweight network structure by reducing the number of channels in at least one convolutional layer and thus reducing the number of parameters. Furthermore, by adding deconvolutional layers, the consistency of image size between the input and output images can be ensured. Alternatively, at least one of residual networks and deep convolutional layers can be used to replace some convolutional layers to further improve image processing performance and reduce the number of parameters. A detailed description of the effects of the network structure of the generator provided in this application embodiment can be found in the further description in the examples below.

[0215] Generative Adversarial Networks (GANs) are a commonly used training method in image generation, achieving good results in applications such as face generation, facial attribute editing, and image colorization. Therefore, as an alternative approach, style transfer models, including GANs, can be used to achieve image style transfer of target regions.

[0216] For style transfer tasks, GANs need to be trained on paired samples before and after style transfer for each content image. Therefore, other methods are needed to generate the corresponding style transfer image ground truth for each training content image. As an alternative approach, a slow style transfer method based on Markov random fields can be used to generate the style transfer image ground truth.

[0217] As an example, Figure 15 The figure shows a flowchart illustrating a method for generating style-transferred image ground truth using a Markov random field-based style transfer approach. As shown, for a given content image X...C and style image X S Our goal is to generate an image X (i.e., the style transfer target image) such that X has the same characteristics as X. S Similar style and with X C Having the same content, that is, the content image X C Convert to style image X S An image X with the same style. X can be initialized with random noise, and in each subsequent iteration, X, X... C X S Inputting these features into a convolutional neural network with fixed parameters yields several layers of feature maps F (i.e., the feature maps of the style transfer target image). C (i.e., content image feature map), F S (i.e., style image feature map), where the neural network can be selected according to actual needs, and may include, but is not limited to, neural network structures such as VGG (Visual Geometry Group), AlexNet, etc. Calculate F and Fi. S The Markov random field loss between them, and F and F C The L2 loss (mean squared error loss) between the two losses is weighted and backpropagated to X to update X, i.e., the backpropagation gradient is calculated. Markov random field loss focuses more on local information, while L2 loss focuses more on global information. The iteration stops when the weighted loss converges; at this point, X should be equal to X0. C Has consistent content and is consistent with X S Having a similar style, X in this case is X. C The ground truth. All training images can be used to generate style-transferred ground truth images for adversarial generative training of GANs.

[0218] As an example, Figure 16 The figure shows a schematic diagram of a GAN network structure. As shown in the figure, in a GAN, the generator is responsible for image style transfer and confusing the discriminator. The content image X C The input is fed into the generator, which generates a style-transferred image X. G The discriminator is responsible for determining whether the generated image is generated by the current model or is a style-transferred ground truth image; that is, determining whether the image X generated by the generator is true. G Is it a content image X? C The style-transferred image is the ground truth. The generator's loss includes GAN loss, feature loss, and pixel loss. GAN loss refers to the loss of the style-transferred image X output by the current generator. G The cross-entropy loss is the output of the discriminator after inputting the style-transferred image ground truth X into the discriminator. Feature loss refers to X.G X and X are input into a VGG network with fixed parameters (or other types of neural networks), and their L2 loss is calculated in the feature maps of certain layers. Pixel loss refers to the loss of X. G The pixel-wise loss for X includes methods such as Total Variation (TV) loss and L2 loss. When training a GAN network, the parameters of the discriminator can be fixed during generator training, and the generator parameters can be updated by weighting and summing the three aforementioned losses. Conversely, when training the discriminator, the generator parameters can be fixed, and X can be used as the basis for updating the generator parameters. G The discriminator is updated using the classification cross-entropy loss output by X after inputting X into it. The generator and discriminator are trained alternately until both are close to convergence, completing the training of the GAN. At this point, the style-transferred image generated by the generator should be very close to the ground truth style-transferred image and have high quality.

[0219] In general, deep learning models often employ complex network structures to produce better results, resulting in a high number of parameters and slow execution speed. To enable style transfer models to run in real-time on portable devices such as mobile phones and tablets, and to operate on a wider range of devices, this application's embodiments have lightweighted both the model size and its execution speed on electronic devices. Specifically, the network structure of the style transfer model's generator has been optimized, reducing the model size and improving execution speed.

[0220] As an optional structure in the lightweight generator network structure provided in this application embodiment, residual connection blocks can be used. Compared with ordinary convolution operations, residual connection blocks help reduce the number of parameters. In addition, residual connection blocks have many bypass branches that directly connect the input to the subsequent layers, so that the subsequent layers can directly learn the residuals. Since traditional convolutional layers will more or less suffer from information loss and damage during information transmission, residual connection blocks solve this problem to some extent. By directly passing the input information to the output through a bypass, the integrity of the information is protected. The entire network only needs to learn the part that is different between the input and the output, simplifying the learning objective and difficulty.

[0221] Furthermore, the number of channels (number of convolutional kernels) in at least one convolutional layer in this scheme can be reduced accordingly to further reduce the number of parameters, for example... Figure 17b In the network structure of the generator shown, the number of convolutional kernels in the convolutional layers (which can be at least one of the convolutional layers, deep convolutional layers, or deconvolutional layers shown in the figure) can be reduced accordingly as needed.

[0222] As an example, Figure 17aThe figure shows a schematic diagram of an existing generator network structure based on VGG-19. As shown, the generator includes multiple levels of convolutional layers and pooling layers located between the convolutional layers. In the convolutional layer diagram, the first number indicates the number of filters (i.e., the number of convolutional kernels), and the second number indicates the size of the convolutional kernels. Similarly, the number after the pooling layer diagram indicates the size of the pooling kernels. For example, in the convolutional layer (64,3) shown in the figure, 64 represents the number of convolutional kernels, and 3 indicates that the size of the convolutional kernels is 3×3. The pooling layer (2) shown in the figure indicates that the pooling kernel is 2×2. Figure 17a The VGG-19 network structure shown is approximately 115Mb in size and takes about 2500ms to process one image. The model is too large and the image processing speed is too slow, which cannot meet the needs of practical applications, especially its applicability on mobile terminal devices is too poor.

[0223] As an example, Figure 17b The figure shows a schematic diagram of the network structure of a generator for a GAN network provided in an embodiment of this application. As shown in the figure, the network structure of the generator may include a pooling layer, four convolutional layers, one depthwise convolutional layer, four residual connect blocks (the part enclosed by the dashed line in the figure is a residual connect block), and three deconvolutional layers. Based on this network structure, since the number of channels in the network structure is less than the number of channels in commonly used generators, such as... Figure 17b The number of channels in the convolutional kernels in the network structure shown is less than Figure 17a The number of convolutional kernels in the network structure shown reduces the model size, achieving lightweight architecture. Furthermore, this example utilizes a deep convolutional layer structure. Since deep convolutional layers have fewer parameters than regular convolutional layers, using deep convolutional layers can further reduce the amount of data to be processed. Figure 17b The deep convolutional layer (128,4) shown has a kernel size of 4, which means it uses two consecutive kernels of 4×1 and 1×4 for calculation. Compared with ordinary convolutional layers (such as convolutional layer (64,3)), the number of parameters can be effectively reduced.

[0224] As described above, compared to convolutional layers, Figure 17b The residual network, specifically the residual connection block, offers better data processing performance and can further reduce the number of parameters. Furthermore, since convolution operations reduce the size of the original input image, this example uses deconvolution layers to achieve upsampling, thereby restoring the scaled-down feature map to the same size as the input image, improving image processing performance and better meeting practical needs.

[0225] Figure 17bThe lightweight generator shown is approximately 2Mb in size and processes an image in about 250ms. It is evident that, compared to existing style transfer model structures, the GAN network model provided in this embodiment is significantly smaller, and image processing efficiency is greatly improved, better meeting practical needs and enhancing the user experience.

[0226] In an optional embodiment of this application, the method further includes:

[0227] The image to be processed is segmented using an object segmentation model to obtain a probability distribution map of objects corresponding to each image style in the image after style conversion. For each image style, each pixel in the probability distribution map represents the probability that the pixel corresponding to that element in the image to be processed belongs to the segmentation category corresponding to that image style. When the segmentation category is converted to the corresponding image style, each element in the probability distribution map represents the probability that the pixel corresponding to that element in the image to be processed belongs to the image style.

[0228] Based on the object probability distribution map corresponding to each image style, the style images of the image to be processed corresponding to each image style are fused to obtain the target image.

[0229] Specifically, based on the object probability distribution map corresponding to each image style, the style images of the image to be processed corresponding to each image style are fused to obtain the target image, which may include:

[0230] The object probability distribution map corresponding to each image style is used as the weight map of the corresponding style image. The pixel values ​​of each pixel in each style image are weighted and summed to obtain the pixel value of each pixel in the target image.

[0231] Optionally, when using the object probability distribution map corresponding to each image style as the weight map of the corresponding style image, and performing a weighted summation of the pixel values ​​of each pixel in each style image to obtain the pixel value of each pixel in the target image, the following method can be used:

[0232] Preprocess the object probability distribution map corresponding to each image style;

[0233] The object probability distribution map corresponding to each image style after preprocessing is used as the weight map of the corresponding style image. The pixel values ​​of each pixel in each style image are weighted and summed to obtain the pixel value of each pixel in the target image.

[0234] This preprocessing can be either a linear transformation or a nonlinear transformation.

[0235] Based on the object segmentation results, after style conversion is performed on the target region of the image to be processed, if style conversion is performed on only a part of the image to be processed, i.e., the target region is only a part of the image to be processed, then the non-target region (i.e., the image of the region that has not undergone style conversion) and the converted target region image are fused together. Alternatively, if style conversion is performed on all regions of the image to be processed, i.e., the target region is the entire image region, then the images of the target regions corresponding to different styles need to be fused together to obtain an image in which multiple styles coexist, i.e., the style-converted target image.

[0236] For ease of description, the image style of the image to be processed, i.e. the original image, will be referred to as the original style, and the style that the target region needs to be converted to will be referred to as the target style.

[0237] The method provided in this application is applicable to scenarios where one or more style conversions are performed on an image. In scenarios where only one style conversion is performed, the converted image contains both the original style and a target style. In scenarios where multiple style conversions are performed on multiple target regions, the image styles contained in the converted image can differ depending on the specific application requirements. For example, if two different style conversions are performed on the foreground and background regions of an image, the converted image contains two different target styles. Similarly, if one style conversion is performed on one or more target regions of an image, another style conversion is performed on one or more other target regions, and non-target regions also exist, the converted image contains two different target styles and one original style.

[0238] It is understandable that for the same pixel, the sum of the values ​​of the corresponding element points on the object probability distribution map for each image style is 1. The aforementioned probability values ​​can be floating-point data, specifically values ​​ranging from [0.0, 1.0].

[0239] In practical applications, since the object segmentation result is highly correlated with the performance of the segmentation model used, the transition region may be too large or too small. To avoid the transition region being too large or too small and affecting the fusion effect between images of different styles in the transition region, the object probability distribution map corresponding to each style image can be preprocessed before fusing the images of each style. Specifically, if the transition region is too small, the transition is not smooth enough, and a fuzzy algorithm can be used for processing, which will not be elaborated in this invention; if the transition region is too large, the object probability distribution map can be transformed again to reduce the transition region. The transformation methods include, but are not limited to, linear transformation and nonlinear transformation.

[0240] It should be noted that in practical applications, the preprocessing of the object probability distribution map can be performed either within the segmentation model or outside of it. For example, the preprocessing can be incorporated as a layer in the model, connected to the output layer of the object probability distribution map. Alternatively, the object probability distribution map can be output from the model, and then the preprocessing can be performed outside the model.

[0241] The image fusion scheme provided in the embodiments of this application will be described below with reference to specific examples.

[0242] Example 1

[0243] This example illustrates a scenario where the image after style conversion contains two styles. The two styles can be the original style and the target style, or two target styles. In this example, we take the original style and the target style as examples.

[0244] This example assumes that style transfer is performed only on the background image region of the image to be processed, and not on the foreground image region. Specifically, after segmenting the image to be processed to obtain the background image region, style transfer can be performed on the background image region to obtain the corresponding style image. As described above, during style transfer, only the background image region can be transferred, and after the transfer is completed, the transferred background image region and the foreground image region of the image to be processed are merged to obtain the corresponding style image. Alternatively, the entire image to be processed can be transferred, and then the background image region of the transferred image is merged with the foreground image region of the original image to obtain the corresponding style image.

[0245] For the image to be processed, the object probability distribution maps corresponding to the original style and the target style in the converted image can be obtained by the image segmentation model. Since the sum of the probabilities of a pixel in the two probability distribution maps is 1 when there are only two image styles, in practical applications, when the converted image contains only two styles, the object segmentation model can output only one object probability distribution map (which can be denoted as P2), and the other object probability distribution map can be represented as 1-P2. After obtaining the object probability distribution maps corresponding to each style, the style image corresponding to the original style and the style image corresponding to the target style can be fused based on these two probability distribution maps.

[0246] Specifically, it can be implemented based on the following expression:

[0247] Output_style=P2·Foreground_style+(1-P2)·Background_style

[0248] Wherein, Output_style represents the target image, Foreground_style represents the style image of the image to be processed corresponding to the foreground style (i.e., the original style), which is the original image, Background_style represents the style image of the image to be processed corresponding to the background style (i.e., the target style), which can be understood as the image after the image to be processed is transformed according to the target style, P2 represents the object probability distribution map corresponding to the foreground style, and 1-P2 represents the object probability distribution map corresponding to the target style.

[0249] In this embodiment, the object segmentation result is represented as an object probability distribution map, which is the probability that each pixel in the original image belongs to the foreground object. The value range can be represented as [a1, a2]. This embodiment does not specifically limit a1 and a2, and generally [0, 1] or [0, 100] can be used. Here, 0 indicates that the pixel is not a foreground object, that is, the pixel is a pixel in the background area, and 1 or 100 indicates that the pixel is a foreground object, that is, a pixel in the foreground area. The following description uses [0, 1] as an example.

[0250] In existing technologies, object segmentation results are typically achieved by binarizing the object probability map. This involves setting a probability threshold a0; if the object's probability is greater than a0, the probability of that pixel is set to 1; otherwise, it is set to 0. For example... Figure 18a This is a schematic diagram of an object probability distribution map p0. Based on the existing scheme, if the probability threshold a0 is 0.5, then the probability values ​​in the object probability distribution map p0 that are not less than 0.5 are binarized to 1, and the probability values ​​that are less than 0.5 are binarized to 0. Figure 18bThe binarized object probability distribution map p1 shown in the figure is used directly as the object region and background fusion. When the transition part is used directly, there will be a serious visual contrast. The existing technology generally blurs p1 to make the transition region smooth. However, this operation makes the size of the transition region constant and does not keep in line with the original transition region, resulting in poor visual effect. To address the aforementioned problems in the existing technology, this application proposes an embodiment that performs style conversion on the foreground image region of the image to be processed, while the background image region is not style converted. The target image includes the original style and a target style. Based on the image soft segmentation scheme provided by this application embodiment, the object probability distribution map of the object soft segmentation result proposed in this application can be directly used as a template for fusing the foreground and background. Since the result output by the object segmentation model is an object probability distribution map, which can be a distribution map of the probability of the object foreground, i.e., the target style (i.e., the probability that each pixel corresponding to each element point in the distribution map is the foreground image region in the image to be processed, i.e., the probability that the pixel is the target style pixel), or it can be a probability distribution map of the object background, i.e., the original style, then directly using it as a template for fusing the foreground and background will preserve the size and position of the transition region in the original image to the greatest extent possible, making the transition region more natural and obtaining a better visual effect.

[0251] Specifically, the non-target region of the image to be processed and the transformed target region are fused, including any of the following:

[0252] A: For any pixel in the image to be processed, if the probability value of the target region of the pixel is not greater than the first set probability value, then the probability value of the target region of the pixel is set to the minimum probability value.

[0253] B: For any pixel in the image to be processed, if the probability value of the target region of the pixel is not less than the second set probability value, then the probability value of the target region of the pixel is set to the set maximum probability value.

[0254] C: For any pixel in the image to be processed, if the target region probability value of the pixel is between the first set probability value and the second set probability value, then the target region probability value of the pixel is transformed, such as by linear transformation.

[0255] Among them, the second set probability value is greater than the first set probability value, and the maximum probability value is greater than the minimum probability value.

[0256] Furthermore, since the object segmentation result is highly correlated with the performance of the segmentation model used, the transition region may be too large or too small. If the transition region is too small, the transition is not smooth enough and can be processed using a fuzzy algorithm, which will not be elaborated upon in this invention. If the transition region is too large, the original segmentation model result can be transformed again to reduce the transition region. The transformation methods include, but are not limited to, linear transformation and nonlinear transformation.

[0257] As an example, taking linear transformation as an example, two probability thresholds α0 and α1 can be set as the probability cutoff values ​​for the background and the object, respectively. If the object probability value p0 ≤ α0, the object probability value can be set to 0; if the object probability value p0 ≥ α1, the object probability value can be set to 1. The rest can be calculated according to the following linear transformation formula. Process it, Figure 18a The object probability distribution diagram p2 after transformation of the probability distribution diagram in the image is shown below. Figure 18c As shown, comparison Figure 18a and Figure 18c It can be seen that the transition region after the transformation is smaller than the original object probability map p0, which is more in line with the actual situation.

[0258] As an example, Figure 18d The diagram illustrates the structure of an existing object segmentation model. As shown, this model comprises a residual network, a pyramid pooling module, a decoding network, a SoftMax layer, and an Argmax layer, all cascaded sequentially. For details regarding the residual network and the pyramid pooling module, please refer to the previous section on... Figure 14a The descriptions of the residual network and pyramid pooling module are provided, while the decoding network, SoftMax layer, and Argmax layer correspond to... Figure 14aThe specific implementation of the last convolutional layer shown is as follows: the decoding network upsamples the concatenated feature map, projects it onto the pixel space, and then uses the SoftMax and ArgMax layers to label the category of each pixel, resulting in the segmentation result for each pixel. In the existing model, for the output of the SoftMax layer, each output channel represents a category. The value of each element in the feature map (also called the probability distribution map) corresponding to each channel of the SoftMax layer is the probability that the corresponding pixel belongs to the category corresponding to that channel. The ArgMax layer following the SoftMax layer compares the values ​​of each pixel in the probability distribution map of each output channel of the SoftMax layer and selects the category corresponding to the maximum probability value of each pixel as the final category of that pixel, obtaining the binary classification result for each pixel. For the various image styles contained in the target image, one image style can correspond to one category. Based on the existing segmentation model, for each pixel, the corresponding style result is only a binary result of either belonging to that category or not belonging to that category.

[0259] Based on the image soft segmentation scheme provided in the embodiments of this application, Figure 18e The diagram shows a schematic of the object segmentation model based on the principle of this scheme. Figure 18d As can be seen from the comparison, this model can not only adopt the simplified pyramid pooling module provided in this application (such as the existing pyramid pooling module), but also... Figure 14b In addition to the pyramid pooling model shown, the Argmax layer is removed, and the output of the Softmax layer is directly used as the final output. This allows us to obtain a probability distribution map corresponding to each category, that is, to obtain the probability of each pixel belonging to each category, instead of a simple result of either 0 or 1. This achieves a softer image style, so that when fusing images of different styles based on this segmentation result, the fusion result can be more natural.

[0260] As an example, Figure 18f and Figure 18g The images show segmentation results based on existing object segmentation models and the object segmentation model provided in this application embodiment, respectively. They also illustrate the effect of fusing images of different styles. In this example, the background portion of the image underwent style conversion, while the foreground portion (the person wearing a hat in the image; not all people in the image are shown) did not. That is, the foreground corresponds to one style, and the background corresponds to another. For the transition region between the foreground and background portions in the target image (i.e., the edge of different regions), taking the area circled by the black ellipse in both images as an example, it can be seen from the two images... Figure 18f The blending effect in the image is not ideal; the edges are not natural enough. Figure 18f The blending effect is significantly improved compared to 18g, and the visualization effect of the edge parts is more natural.

[0261] By applying the solutions provided in the embodiments of this application, the results of background image stylization or object stylization can be obtained. For images without objects, the style transfer model is directly applied to perform artistic style conversion; for images with objects, the object segmentation module and the style transfer model are applied to achieve style conversion of objects and background using different artistic styles. The object segmentation model can ensure its accuracy through the rotation angle prediction module and the object detection module, thus better meeting practical needs.

[0262] It is understood that the rotation angle prediction and object segmentation schemes provided in the embodiments of this application are not only applicable to style transfer processing of images, but also applicable to other image processing schemes that require image segmentation. For example, conventional processing schemes that edit specified objects or regions in an image can be segmented based on the schemes provided in the embodiments of this application to obtain the target region in the image, i.e. the region to be processed, thereby improving the accuracy of image segmentation and improving the image processing effect and efficiency.

[0263] The solution provided in this application will now be described with reference to a specific optional example.

[0264] Figure 19 The figure shows a schematic diagram of the principle process in this optional example. As shown in the figure, the image processing method in this example can mainly include the following steps:

[0265] Obtain the image to be processed, i.e., the original image to be stylized as shown in the figure;

[0266] Image segmentation is performed on the image to be processed to obtain the target regions in the image, which is the step of obtaining the style image shown in the figure. Specifically, after determining each target region in the image to be processed, the lightweight style transfer model based on GAN (also known as a style transfer network, such as...) provided in the embodiments of this application can be used. Figure 17b The network structure shown in the figure) implements image style transfer of the target region to obtain the converted image (the style image shown in the figure). The specific method for segmenting the image to be processed to obtain the target region for style transfer can be to use existing image segmentation methods.

[0267] To improve the accuracy of image segmentation, the image can first be rotated for the desired image. This involves predicting the rotation angle of the image and then correcting it based on this angle to obtain the corrected image. This corresponds to the steps of predicting and rotating the image shown in the figure. Specifically, this step can be achieved using an angle prediction model (corresponding to the rotation network shown in the figure, such as...). Figure 7b The model structure shown in the figure is implemented by inputting the image to be processed into the model and obtaining the rotation angle α of the image to be processed through the model. In this example, a condition for determining whether to perform rotation correction on the image is pre-configured, that is, the rotation angle range of 45° to 315° shown in the figure. If the rotation angle α predicted by the angle prediction model is within this angle range, the image to be processed is corrected based on the rotation angle (the corrected image shown in the figure) to obtain the corrected image. If the predicted rotation angle α does not belong to this angle range, then rotation correction is not required.

[0268] After completing the above correction process, the target object region (i.e., the object region that meets the preset conditions) can be detected based on the uncorrected image or the corrected image. Specifically, this can be achieved through an image detection model (the detection network shown in the figure), such as using... Figure 3 The scheme shown inputs the image to be detected (i.e., the uncorrected image or the corrected image) into an image detection model. The image detection model detects whether objects exist in the image and obtains the detection results, specifically the position information of the bounding boxes of the target objects in the image. If the number of bounding boxes is greater than 0, that is, the model outputs the position information of at least one bounding box, it indicates that an object region exists. The object region can then be cropped based on the bounding box position information to obtain each object region. These detected object regions are the image regions that need to be segmented. If no object exists, that is, the number of bounding boxes is 0, it means that there is no object region in the image that meets the preset conditions, and deep object segmentation processing is not required.

[0269] After obtaining each object region, the object segmentation model based on the soft segmentation scheme (the simplified soft segmentation network shown in the figure) provided in the embodiments of this application can be used for segmentation to obtain the object probability distribution map corresponding to each image style in the style-transformed image. Then, based on the object probability distribution map corresponding to each image style, the style images of the image to be processed corresponding to each image style are fused to obtain the target image (i.e., the output style-transformed image).

[0270] Compared with existing image processing methods, the image processing method provided in this application embodiment has at least the following advantages:

[0271] 1) This application proposes that an object rotation angle prediction and correction module can be added before determining the object region. This module predicts the rotation angle corresponding to the image. For images rotated by the user during image editing, or images taken at non-horizontal or non-vertical angles, the rotation angle required to straighten the image is predicted. After straightening the image by the predicted rotation angle, the object region in the image is determined, thereby improving the accuracy of determining the object region and improving the accuracy of object segmentation.

[0272] 2) This application proposes that a neural network-based object segmentation method can be used to determine the region of an object. This method can not only achieve segmentation based on different object categories, but also accurately segment objects even when there are overlapping areas, improving the accuracy and flexibility of determining the object region. The object segmentation scheme of this application is applicable not only to the segmentation model for people, but also to the segmentation model for other objects, effectively improving the accuracy of each segmentation model. In practical applications, multiple segmentation models can be used to detect and segment multiple types of objects, or multiple segmentation models can be used to detect and segment different types of objects separately. In addition, when segmenting an image, the rotated image can be segmented, or the object region in the rotated image can be detected first, and then the detected object region can be segmented to further improve the segmentation accuracy. This method is more effective when the proportion of the object in the image is small, as the object occupies a larger proportion of the detected object region, resulting in better segmentation results.

[0273] 3) This application proposes that an object detection module can be added before object segmentation. This module obtains the bounding box of the object (i.e., the first object region). Based on the bounding box, objects that meet the requirements, i.e., the preset conditions, can be detected. This can improve the accuracy of object segmentation and reduce the amount of data that needs further processing. In addition, it reduces the probability that the segmentation result contains pixels of irrelevant categories, thereby improving the segmentation accuracy.

[0274] 4) The solution provided in this application has been greatly optimized for mobile devices with limited computing power, but is not limited to deployment on mobile devices. It enables the solution of the embodiments of this application to be applied to more types of terminal devices and greatly improves the user experience on mobile terminal devices.

[0275] 5) The object segmentation scheme provided in this application can achieve image segmentation through various processing methods such as image rotation correction and object region segmentation, making object segmentation more robust and ensuring that the final segmentation result can maintain good segmentation accuracy even for complex images to be processed.

[0276] This application proposes an image style transfer system based on object segmentation, enabling rapid generation of artistic creations from real images. The system proposes appropriate preprocessing of the input image to improve the accuracy of object segmentation; improvements and optimizations to the original image style transfer algorithm to enhance the visual effect of the output image after style transfer; and significant optimizations to address the limited computing power of mobile devices. However, it is not limited to deployment on mobile devices, greatly improving the user experience on mobile terminals.

[0277] In one example, a system for performing style transfer on images may include:

[0278] 1. Preprocessing module: Detects the object region and rotation angle of the object in the image, and performs rotation correction on the image;

[0279] 2. Object Segmentation Module: Accurately segments the object regions in the calibrated image and outputs an object probability map;

[0280] 3. Style Transfer Module: Apply the target style to objects and / or backgrounds;

[0281] 4. Post-processing module: Merges the object and background based on the object segmentation results.

[0282] In one example, such as Figure 18h As shown, style transfer can be performed on the entire original image (i.e., the input image). Specifically, if the user chooses to convert the target object to the first target style while leaving the background image unchanged, the style transfer module converts the entire original image to the first target style during style transfer. Then, in post-processing, the post-processing module merges the style-transformed target object region with the background image in the original image based on the segmentation results from the object segmentation module. Alternatively, if the user chooses to convert the background image to the second target style while leaving the target object unchanged, the style transfer module converts the entire original image to the second target style during style transfer. Then, in post-processing, the post-processing module merges the style-transformed background image region with the target object region in the original image based on the segmentation results from the object segmentation module. Or, if the user chooses to convert the target object to the first target style and the background image to the second target style, the style transfer module converts the entire original image to both the first and second target styles during style transfer. Then, in post-processing, the post-processing module merges the target object region converted to the first target style with the background image region converted to the second target style based on the segmentation results from the object segmentation module, resulting in the output image. In addition, before the object segmentation module performs segmentation processing, the preprocessing module can predict the rotation angle of the image and perform rotation correction processing on the image based on the rotation angle.

[0283] In another example, such as Figure 18i As shown, during style transfer, only the regions requiring conversion can be style-transformed. Specifically, if the user chooses to convert the target object to the first target style while leaving the background image unchanged, the style transfer module, based on the object segmentation module's results, converts only the target object's region to the first target style. Then, in post-processing, the post-processing module merges the style-transformed target object region with the background image in the original image. Alternatively, if the user chooses to convert the background image to the second target style while leaving the target object unchanged, the style transfer module, based on the object segmentation module's results, converts only the background image region to the second target style. Then, in post-processing, the post-processing module merges the style-transformed background image region with the target object region in the original image. Or, if the user chooses to convert the target object to the first target style and the background image to the second target style, the style transfer module, based on the object segmentation module's results, converts the target object region to the first target style and the background image region to the second target style. Then, in post-processing, the post-processing module merges the target object region converted to the first target style with the background image region converted to the second target style to obtain the output image. In addition, before the object segmentation module performs segmentation processing, the preprocessing module can predict the rotation angle of the image and perform rotation correction processing on the image based on the rotation angle.

[0284] Based on and Figure 1 Using the same principle as the method shown, this application also provides an image processing apparatus, such as... Figure 20 As shown, the image processing device 100 may include an image segmentation module 110 and an image style conversion module 120. Wherein:

[0285] Image segmentation module 110 is used to segment the image to be processed to obtain the target region in the image to be processed;

[0286] Image style conversion module 120 is used to perform style conversion on the target area.

[0287] Optionally, when segmenting the image to be processed, the image segmentation module 110 is specifically used for:

[0288] Predict the rotation angle corresponding to the image to be processed;

[0289] Perform rotation correction on the image to be processed based on the rotation angle;

[0290] The corrected image to be processed is segmented.

[0291] Optionally, when predicting the rotation angle corresponding to the image to be processed, the image segmentation module 110 is specifically used for:

[0292] Detect the first object region where the object is located in the image to be processed;

[0293] Predict the rotation angle corresponding to the first object region, and use the rotation angle corresponding to the first object region as the rotation angle corresponding to the image to be processed.

[0294] Optionally, the first object region is an object region that meets preset conditions, wherein the preset conditions include at least one of the following:

[0295] The specified object area, the area where the specified object is located, the area where the specified type of object is located, and the object area that meets the preset area size.

[0296] Optionally, when predicting the rotation angle corresponding to the image to be processed, the image segmentation module 110 can be specifically used for:

[0297] Determine the rotation angle of the image to be processed based on the rotation angle of the object in the image.

[0298] Optionally, if the image to be processed contains at least two objects, the image segmentation module 110, when determining the rotation angle of the image to be processed based on the rotation angle of the objects in the image, can specifically be used for:

[0299] The rotation angle of the object with the largest area in the image to be processed is determined as the corresponding rotation angle of the image to be processed; or...

[0300] The rotation angles of each object in the image to be processed are merged, and the merged rotation angle is determined as the corresponding rotation angle of the image to be processed; or...

[0301] The rotation angle of each object is determined to be the corresponding rotation angle of the image to be processed.

[0302] Optionally, when determining the rotation angle of each object as the corresponding rotation angle of the image to be processed, the image segmentation module 110 can be used to perform rotation correction on the image to be processed based on the rotation angle:

[0303] For each object's region, rotation correction is performed according to the corresponding rotation angle.

[0304] Accordingly, the image segmentation module 110 segments the corrected image to be processed, including:

[0305] The regions of each object after rotation correction are segmented separately, and then the segmentation results of each object's region are merged to obtain the segmentation result of the image to be processed. Alternatively, the regions of each object after rotation correction are merged, and the merged image is segmented to obtain the segmentation result of the object to be processed.

[0306] Optionally, the image segmentation module 110 can specifically be used to predict the rotation angle corresponding to the image to be processed in the following ways:

[0307] The rotation angle of the image to be processed is predicted by an angle prediction model, wherein the angle prediction model includes an image feature extraction module, and the number of BN layers in at least one feature extraction unit of the image feature extraction module is less than a set value.

[0308] Optionally, the image segmentation module 110 predicts the rotation angle of objects in the image to be processed in the following manner:

[0309] The object's skeleton key points are detected by a pose estimation model, and the object's rotation angle is predicted based on the object's skeleton key points.

[0310] Optionally, when predicting the rotation angle of an object based on its skeleton key points, the image segmentation module 110 can specifically be used for:

[0311] Predict the rotation angle of the object based on the positional relationship of at least two key points in the object's skeleton keypoints.

[0312] Optionally, at least two keypoints include at least two corresponding object skeleton keypoints.

[0313] Optionally, when performing rotation correction on the image to be processed based on the rotation angle, the image segmentation module 110 is specifically used for:

[0314] Perform rotational correction on the first object region;

[0315] When segmenting the corrected image to be processed, the image segmentation module 110 is specifically used for:

[0316] The corrected first object region is segmented;

[0317] Based on the segmentation results of the first object region, the segmentation results of the object to be processed are obtained.

[0318] Optionally, when segmenting the image to be processed, the image segmentation module 110 is specifically used for:

[0319] Detect the second object region where the object in the image to be processed is located;

[0320] Each second object region can be segmented separately, or the second object regions can be merged and the merged image can be segmented.

[0321] Based on the segmentation results of the second object region, the segmentation results of the image to be processed are obtained.

[0322] Optionally, the second object region is an object region in the image to be processed or the corrected image to be processed that meets preset conditions.

[0323] Optionally, when segmenting the image to be processed, the image segmentation module 110 is specifically used for:

[0324] The image to be processed is segmented using an object segmentation model;

[0325] The object segmentation model is obtained by training a neural network.

[0326] Optionally, the target region includes the area containing objects of a specified category in the image to be processed.

[0327] Optionally, the image processing apparatus 100 may further include:

[0328] The request receiving module is used to receive style transfer requests, which include the specified category of the object to be style transferred.

[0329] Optionally, the object segmentation model includes PSPNet, where the number of layers in the pyramid pooling module of PSPNet is less than a set number of layers.

[0330] Optionally, when performing style conversion on the target region, the image style conversion module 120 is specifically used for:

[0331] Image style transfer is performed on the target region using a style transfer model, where the style transfer model includes a GAN, and the network structure of the generator of the GAN satisfies any of the following:

[0332] It includes cascaded convolutional layers, residual networks, and deconvolutional layers;

[0333] This includes cascaded convolutional and deconvolutional layers;

[0334] It includes convolutional layers, depthwise convolutional layers, and deconvolutional layers that are cascaded in sequence;

[0335] It includes sequentially cascaded convolutional layers, deep convolutional layers, residual networks, and deconvolutional layers;

[0336] At least one convolutional layer has fewer channels than the set number.

[0337] Optionally, the image processing apparatus 100 further includes an image fusion module, which is used for:

[0338] The image to be processed is segmented using an object segmentation model, resulting in a probability distribution map of objects corresponding to each image style in the style-transformed image. For each image style, each pixel in the probability distribution map represents the probability that the pixel corresponding to that element in the image to be processed belongs to the segmentation category corresponding to that image style. When the segmentation category is converted to the corresponding image style, each element in the probability distribution map represents the probability that the pixel corresponding to that element in the image to be processed belongs to the image style in the target image.

[0339] Based on the object probability distribution map corresponding to each image style, the style images of the image to be processed corresponding to each image style are fused to obtain the target image.

[0340] Optionally, the image fusion module fuses the style images of the image to be processed corresponding to each image style based on the object probability distribution map corresponding to each image style to obtain the target image, including:

[0341] The object probability distribution map corresponding to each image style is used as the weight map of the corresponding style image. The pixel values ​​of each pixel in each style image are weighted and summed to obtain the pixel value of each pixel in the target image.

[0342] Optionally, when the image fusion module fuses the style images corresponding to each image style based on the object probability distribution map corresponding to each image style, it is specifically used for:

[0343] Transform the probability distribution diagrams of each object;

[0344] Based on the object probability distribution map corresponding to each image style after conversion, the style images of the image to be processed corresponding to each image style are fused.

[0345] Based on the same principles as the image processing method and image processing apparatus provided in the embodiments of this application, the embodiments of this application also provide an electronic device, which includes a processor and a memory: the memory is used to store machine-readable instructions; the processor is used to execute the image processing method shown in any embodiment of this application when executing the machine-readable instructions.

[0346] This application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method shown in any embodiment of this application.

[0347] As an example, Figure 21 The diagram shows a structural schematic of an electronic device 4000 to which the solution of this application embodiment applies, such as... Figure 21As shown, the electronic device 4000 may include a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0348] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0349] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 21 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0350] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0351] The memory 4003 stores application code that executes the solution of this application, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the solution shown in any of the foregoing method embodiments.

[0352] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0353] The above description is only a partial embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that, include: Perform style transfer on the target region in the image to be processed; The image to be processed is segmented by an object segmentation model to obtain an object probability distribution map corresponding to each image style in the image after style conversion. For each image style, each element point in the object probability distribution map represents the probability that the pixel point corresponding to that element point in the image to be processed belongs to the segmentation category corresponding to that image style. Based on the object probability distribution map corresponding to each image style, the style images of the image to be processed corresponding to each image style are fused to obtain the target image; wherein, the fusion includes: The object probability distribution map corresponding to each image style is used as the weight map of the corresponding style image. The pixel values ​​of each pixel in each style image are weighted and summed to obtain the pixel value of each pixel in the target image.

2. The method according to claim 1, characterized in that, The segmentation of the image to be processed includes: Based on the rotation angle of the object in the image to be processed, predict the corresponding rotation angle of the image to be processed; The image to be processed is rotated and corrected according to the rotation angle corresponding to the image to be processed, and then the corrected image is segmented.

3. The method according to claim 2, characterized in that, The prediction of the rotation angle corresponding to the image to be processed includes: The rotation angle of the image to be processed is determined based on the rotation angle of the object in the image to be processed.

4. The method according to claim 3, characterized in that, When the image to be processed includes at least two objects, predicting the rotation angle of the image to be processed based on the rotation angle of the objects in the image to be processed includes: The rotation angle of the object with the largest area in the image to be processed is determined as the rotation angle corresponding to the image to be processed; or The rotation angles corresponding to each object in the image to be processed are merged, and the merged rotation angle is determined as the rotation angle corresponding to the image to be processed; or The rotation angle of each object is determined as the rotation angle corresponding to the image to be processed.

5. The method according to claim 2, characterized in that, The prediction of the rotation angle corresponding to the image to be processed includes: Detect the first object region where the object in the image to be processed is located; Predict the rotation angle corresponding to the first object region, and use the rotation angle corresponding to the first object region as the rotation angle corresponding to the image to be processed.

6. The method according to claim 5, characterized in that, The first object region is an object region that meets preset conditions; The preset conditions include at least one of the following: The specified object area, the area where the specified object is located, the area where the specified type of object is located, and the object area that meets the preset area size.

7. The method according to claim 4, characterized in that, If the rotation angle of each object in the image to be processed is determined as the rotation angle corresponding to the image to be processed, the rotation correction of the image to be processed according to the rotation angle corresponding to the image to be processed includes: For each object's region, rotation correction is performed according to the corresponding rotation angle. The segmentation of the corrected image includes: The regions of each object after rotation correction are segmented separately, and then the segmentation results of the regions of each object are merged to obtain the segmentation result of the image to be processed; or The regions of each object after rotation correction are merged, and the merged rotation-corrected region is segmented to obtain the segmentation result of the image to be processed.

8. The method according to any one of claims 2 to 7, characterized in that, The step of predicting the rotation angle of the image to be processed based on the rotation angle of the object in the image to be processed includes: The rotation angle corresponding to the image to be processed is predicted by an angle prediction model, wherein the angle prediction model includes an image feature extraction module, and the number of batch normalized (BN) layers in at least one feature extraction unit of the image feature extraction module is less than a set value. or, The object skeleton key points of the object in the image to be processed are detected by the pose estimation model. The rotation angle of the object is predicted based on the object skeleton key points, and the rotation angle of the object is used as the corresponding rotation angle of the image to be processed.

9. The method according to claim 5 or 6, characterized in that, The step of rotating the image to be processed according to the rotation angle corresponding to the image to be processed includes: Perform rotational correction on the first object region; The segmentation of the corrected image includes: The corrected first object region is segmented; Based on the segmentation result of the first object region, the segmentation result of the image to be processed is obtained.

10. The method according to any one of claims 2 to 7, characterized in that, The segmentation of the corrected image includes: Detect the second object region where each object in the image to be processed is located; Each of the second object regions is segmented separately, or the second object regions are merged and the merged image is segmented. Based on the segmentation result of the second object region, the segmentation result of the image to be processed is obtained.

11. The method according to claim 10, characterized in that, The step of segmenting each of the second object regions includes: Each second object region is enlarged or reduced, and each enlarged or reduced second object region is then segmented. The step of merging the second object regions and segmenting the merged image includes: The second object regions are merged, the merged image is enlarged or reduced, and the enlarged or reduced merged image is segmented.

12. The method according to any one of claims 1 to 7, characterized in that, The object segmentation model is obtained by training a neural network.

13. The method according to claim 12, characterized in that, The target region includes the area containing objects of a specified category in the image to be processed.

14. The method according to claim 13, characterized in that, Also includes: Receive a style transfer request, which includes a specified category of the object to be style transferred.

15. The method according to claim 12, characterized in that, The object segmentation model includes the Pyramid Scene Analysis Network PSPNet, wherein the number of layers in the pyramid pooling module of the PSPNet is less than a set number of layers.

16. The method according to any one of claims 1 to 7, characterized in that, The style transfer of the target region in the image to be processed includes: Image style transfer is performed on the target region using a style transfer model, wherein the style transfer model includes a generative adversarial network (GAN), and the network structure of the generator of the GAN satisfies any one of the following: It includes cascaded convolutional layers, residual networks, and deconvolutional layers; This includes cascaded convolutional and deconvolutional layers; It includes convolutional layers, depthwise convolutional layers, and deconvolutional layers that are cascaded in sequence; It includes sequentially cascaded convolutional layers, deep convolutional layers, residual networks, and deconvolutional layers; At least one convolutional layer has fewer channels than the set number.

17. The method according to any one of claims 1 to 7, characterized in that, The step of using the object probability distribution map corresponding to each image style as the weight map of the corresponding style image, and performing a weighted summation of the pixel values ​​of each pixel in each style image, includes: Preprocess the object probability distribution map corresponding to each image style; The object probability distribution map corresponding to each image style after preprocessing is used as the weight map of the corresponding style image. The pixel values ​​of each pixel in each style image are weighted and summed to obtain the pixel value of each pixel in the target image.

18. The method according to claim 17, characterized in that, Each image style includes the image style of the non-target region of the image to be processed and the image style of the target region. The preprocessing of the object probability distribution map corresponding to each image style includes: For any pixel in the image to be processed, if the probability value of the pixel in the object probability distribution map corresponding to the image style of the target region is not greater than the first set probability value, then the probability value of the pixel is set to the set minimum probability value. For any pixel in the image to be processed, if the probability value of the pixel in the object probability distribution map corresponding to the image style of the target region is not less than the second set probability value, then the probability value of the pixel is set to the set maximum probability value. For any pixel in the image to be processed, if the probability value of the pixel in the object probability distribution map corresponding to the image style of the target region is between the first set probability value and the second set probability value, then the probability value of the pixel is transformed to obtain the transformed probability value of the pixel in the object probability distribution map corresponding to the target region. Among them, the second set probability value is greater than the first set probability value, and the maximum probability value is greater than the minimum probability value.

19. An electronic device, characterized in that, The electronic device includes a processor and a memory: The memory is used to store machine-readable instructions; The processor is configured to execute the method according to any one of claims 1 to 18 when executing the machine-readable instructions.

20. A computer-readable storage medium, characterized in that, The storage medium stores computer instructions that, when executed by a processor, implement the method described in any one of claims 1 to 18.

Images