Image processing method and device, computer device and storage medium

Abstract

The application relates to an image processing method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a target image; performing foreground object recognition on the target image to obtain an object region; removing the object region image from the target image to obtain an intermediate image, wherein the intermediate image comprises a blank image of the object region and a background image of the target image; acquiring a background image block matched with the blank image from the background image, performing background filling, and taking the background image block as a filling image; performing style transformation and plane transformation on the filling image to obtain a transformed image; determining a target visual plane when performing the plane transformation, so that the transformed image is located on the target visual plane; and fusing the transformed image and the object region image to obtain a processed image. The method can make the background of the filled image transition naturally and completely while having corresponding style, and make the foreground and the background of the processed image have a stereoscopic relationship, thereby improving the image processing effect.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, computer device, and storage medium. Background Technology

[0002] With the development of image processing technology, image processing is required in many situations, such as beautifying people in images or adding decorations. For example, users can import images into image processing programs and process them by clicking on the program's function controls. Alternatively, images can be input into image generation models for style transfer to obtain style-transformed images.

[0003] However, current image processing methods suffer from poor image quality. Summary of the Invention

[0004] Therefore, it is necessary to provide an image processing method, apparatus, computer device, and storage medium to address the aforementioned technical problems.

[0005] An image processing method, the method comprising:

[0006] Obtain the target image to be processed;

[0007] Perform foreground object recognition on the target image to obtain the object region corresponding to the foreground object in the target image;

[0008] The image corresponding to the object region is removed from the target image to obtain an intermediate image, which includes a blank image corresponding to the object region and a background image corresponding to the target image;

[0009] Obtain a background image block that matches the blank image from the background image, fill the blank image in the intermediate image with the background image block, and use the filled intermediate image as the filled image.

[0010] The filled image is subjected to style transformation and planar transformation to obtain a transformed image. During the planar transformation, the planar transformation information corresponding to the target view plane is determined, and the filled image is transformed according to the planar transformation information so that the transformed image is an image located on the target view plane.

[0011] The transformed image is fused with the image corresponding to the object region to obtain the processed image corresponding to the target image.

[0012] An image processing apparatus, the apparatus comprising:

[0013] The target image acquisition module is used to acquire the target image to be processed.

[0014] A foreground object recognition module is used to perform foreground object recognition on the target image to obtain the object region corresponding to the foreground object in the target image;

[0015] An intermediate image acquisition module is used to remove the image corresponding to the object region from the target image to obtain an intermediate image, wherein the intermediate image includes a blank image corresponding to the object region and a background image corresponding to the target image;

[0016] The fill image acquisition module is used to acquire a background image block that matches the blank image from the background image, fill the blank image in the intermediate image with the background image block, and use the filled intermediate image as the fill image.

[0017] The transformed image acquisition module is used to perform style transformation and planar transformation on the filled image to obtain a transformed image. When performing planar transformation, the planar transformation information corresponding to the target view plane is determined, and the filled image is transformed according to the planar transformation information so that the transformed image is an image located on the target view plane.

[0018] The image processing module is used to fuse the transformed image with the image corresponding to the object region to obtain the processed image corresponding to the target image.

[0019] In one embodiment, the filling image acquisition module is used to divide the object region into multiple division regions; for the current pixel in the blank image, determine the division region to which the current pixel belongs as a reference division region; acquire the background image block matched by the reference division region, acquire the pixel corresponding to the current pixel in the background image block as a reference background pixel; obtain the filling pixel value corresponding to the current pixel based on the pixel value corresponding to the reference background pixel, and fill the blank image in the intermediate image with background according to the filling pixel value.

[0020] In one embodiment, the filled image is obtained through multiple rounds of background filling. The filled image acquisition module is configured to select one or more candidate background image blocks corresponding to the reference division region in the current background filling round based on an image block selection strategy; acquire the filled image block obtained after the previous round of background filling for the reference division region, and acquire the matching background image block corresponding to the reference division region in the previous round of background filling; calculate the similarity between the candidate background image block and the filled image block as a first similarity; calculate the similarity between the matching background image block and the filled image block as a second similarity; and based on the first similarity and the second similarity, acquire an image block similar to the filled image block from the matching background image block and the candidate background image block as the background image block matched by the reference division region in the current background filling round.

[0021] In one embodiment, the filling image acquisition module is configured to, when there are multiple reference division regions corresponding to the current pixel, acquire the similarity between the background image block matched by the reference division region and the corresponding filling image block in the current background filling round; obtain the pixel weight corresponding to the reference background pixel in the current background filling round based on the similarity, wherein the pixel weight is positively correlated with the similarity; and perform a weighted calculation based on the pixel value corresponding to the reference background pixel and the pixel weight corresponding to the reference background pixel to obtain the filling pixel value corresponding to the current pixel.

[0022] In one embodiment, the filling image acquisition module is configured to: determine a reference division region corresponding to the reference division region among the plurality of division regions; acquire the position of the background image block matched by the reference division region in the previous round of background filling as a reference position; acquire the relative positional relationship between the reference division region and the reference division region; obtain a first image block selection position based on the reference position and the relative positional relationship; and select the background image block corresponding to the first image block selection position as the candidate background image block.

[0023] In one embodiment, the filling image acquisition module is configured to: acquire a randomly generated random offset distance; obtain a corresponding offset distance shrinkage value based on the image block sorting of candidate background image blocks, wherein the shrinkage degree corresponding to the offset distance shrinkage value is positively correlated with the image block sorting; perform shrinkage processing on the random offset distance based on the offset distance shrinkage value to obtain a target offset distance; obtain a second image block selection position based on the position of the background image block matched by the reference division region in the previous round of background filling and the target offset distance; and use the background image block corresponding to the second image block selection position as the candidate background image block.

[0024] In one embodiment, the transformed image acquisition module is configured to perform style transformation on the filled image to obtain an initial style image; acquire a background object image corresponding to the target background object in the initial style image; remove the background object image from the initial style image to obtain an image to be processed in the initial style image; perform color clustering on the image to be processed to obtain a color clustering image; fuse the color clustering image with the background object image, and obtain a target style image based on the fused image.

[0025] In one embodiment, the transformed image acquisition module is used to acquire the color parameter threshold range corresponding to the target background object, wherein the target background object is a background object whose color consistency is greater than the consistency threshold; to select pixels whose color parameter values ​​are within the color parameter threshold range from the initial style image as target pixels; and to use the continuous image blocks corresponding to the target pixels in the initial style image as the background object image corresponding to the target background object.

[0026] In one embodiment, the transformed image acquisition module is used to fuse the color clustering image with the background object image to obtain a fused image; to perform edge extraction on the initial style image to obtain an edge image corresponding to the initial style image; and to remove the edge image from the fused image to obtain a target style image.

[0027] In one embodiment, the transformed image acquisition module is configured to perform edge extraction on the initial style image using an edge extraction operator to obtain a first edge image corresponding to the initial style image; perform edge extraction on the initial style image using an edge extraction model to obtain a second edge image corresponding to the initial style image; obtain the image intersection between the first edge image and the second edge image, and use the image intersection as the edge image corresponding to the initial style image.

[0028] In one embodiment, the target image acquisition module is further configured to receive an image processing request, the image processing request being triggered by the terminal in response to a stereoscopic image generation operation; the transformed image acquisition module is configured to determine the stereoscopic effect of the image displayed by the stereoscopic image generation operation, acquire the view plane corresponding to the stereoscopic effect of the image as the target view plane; determine the plane transformation information corresponding to the target view plane, and perform a plane transformation on the filled image according to the plane transformation information, so that the transformed image is an image located on the target view plane.

[0029] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the image processing method described above.

[0030] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image processing method described above.

[0031] A computer program product includes a computer program that, when executed by a processor, implements the steps of the above-described image processing method.

[0032] The aforementioned image processing method, apparatus, computer equipment, and storage medium acquire a target image to be processed; perform foreground object recognition on the target image to obtain the object region corresponding to the foreground object in the target image; remove the image corresponding to the object region from the target image to obtain an intermediate image, the intermediate image including a blank image corresponding to the object region and a background image corresponding to the target image; obtain a background image block matching the blank image from the background image, fill the blank image in the intermediate image with the background image block, and use the filled intermediate image as the filled image; perform style transformation and planar transformation on the filled image to obtain a transformed image, wherein, when performing planar transformation, the planar transformation information corresponding to the target viewing plane is determined, and the planar transformation is performed according to the planar transformation information so that the transformed image is an image located on the target viewing plane; and fuse the transformed image with the image corresponding to the object region to obtain the processed image corresponding to the target image. This application identifies foreground objects in a target image, removes the foreground object region from the target image, and fills the resulting blank image region with the background of the target image. This results in a natural and complete transition between the filled image and the background. By performing style transformation and planar transformation on the filled image to place it on the target view plane, and then fusing it with the image corresponding to the object region, the image can not only have the corresponding style, but also have a three-dimensional relationship between the foreground and background of the processed image, thus improving the image processing effect. Attached Figure Description

[0033] Figure 1A This is an application environment diagram of an image processing method in one embodiment;

[0034] Figure 1B This is a flowchart illustrating an image processing method in one embodiment;

[0035] Figure 2 This is a schematic diagram illustrating the process of removing an object region from a target image to obtain an intermediate image in one embodiment.

[0036] Figure 3 This is a schematic diagram illustrating the process of filling an object region of an intermediate image to obtain a filled image in one embodiment.

[0037] Figure 4 This is a schematic diagram illustrating the processing of an image in one embodiment;

[0038] Figure 5 This is a schematic diagram illustrating the process of filling the blank image in the intermediate image with background in one embodiment;

[0039] Figure 6 This is a flowchart illustrating the process of obtaining a background image block matched by a reference segmentation region in one embodiment;

[0040] Figure 7 This is a schematic diagram illustrating the positional relationship between the current pixel and the reference division region in one embodiment;

[0041] Figure 8 This is a schematic diagram of the process of performing style transformation on a filled image in one embodiment;

[0042] Figure 9 This is a schematic diagram of the interface of an image processing terminal shown in one embodiment;

[0043] Figure 10 This is a flowchart illustrating the image processing method in another embodiment;

[0044] Figure 11 This is a schematic diagram of the processing flow of a street view pixel filter system in one embodiment.

[0045] Figure 12 This is a schematic diagram of the image filling process in one embodiment;

[0046] Figure 13 This is a structural block diagram of an image processing device in one embodiment;

[0047] Figure 14 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties; correspondingly, this application also provides a corresponding user authorization entry point for users to choose to authorize or refuse.

[0050] The image processing method provided in this application can be applied to, for example... Figure 1AIn the application environment shown, terminal 102 communicates with server 104 via a network. Terminal 102 can send an image processing request to the server, which may carry the target image to be processed or an image identifier of the target image. The server can execute the image processing method provided in this application and return the processed image to the terminal. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, in-vehicle devices, and portable wearable devices. Server 104 can be implemented as a standalone server or a server cluster consisting of multiple servers. It is understood that terminal 102 can also execute the image processing method of this application.

[0051] In one embodiment, such as Figure 1B As shown, an image processing method is provided. This embodiment illustrates the method applied to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0052] Step S102: Obtain the target image to be processed.

[0053] The target image refers to the image material that needs to be processed. It can be an image input by the user in an application used to implement the image processing procedure, or it can be an image pre-stored in an image database. The image processing can be an image processing procedure used to generate a stereoscopic image corresponding to the target image. The processing of the target image can be implemented through user terminals such as various personal computers, laptops, smartphones, tablets, in-vehicle devices, and portable wearable devices, or it can be implemented through a backend server.

[0054] If the image processing flow is implemented by the terminal, when a user needs to process a certain image, the user can input that image as the target image to be processed into the terminal, and the terminal will directly execute the image processing procedure for the target image, thereby outputting the corresponding processed image. If the image processing flow is implemented by the server, the terminal can send the target image to the server after receiving it from the user, and the server can execute the image processing procedure, obtain the corresponding processed image, and then return it to the terminal for output.

[0055] Step S104: Perform foreground object recognition on the target image to obtain the object region corresponding to the foreground object in the target image.

[0056] Foreground objects refer to the objects in the foreground portion of a target image. Generally, a target image consists of a foreground and a background. The foreground typically refers to objects located in front of the subject or even close to the camera lens, used to represent spatial or human relationships. The background, on the other hand, is generally located behind the subject, away from the camera, and is an important part of the environment. For example, in a landscape image with people, the people in the image would typically be the foreground object, while the surrounding scenery would be the background. Similarly, in a landscape image with animals, the animals would be the foreground object, and the surrounding scenery would be the background. The area corresponding to the foreground object refers to the image area occupied by the foreground object within the target image.

[0057] Specifically, after obtaining the target image, the terminal can identify objects in the foreground of the target image. For example, if the uploaded target image is a picture of a person, the terminal can take the person as the foreground object. By recognizing the person in the image, which can be based on a portrait recognition algorithm, the terminal can extract the portrait in the target image as the foreground object and then obtain the area occupied by the portrait in the target image as the object area corresponding to the foreground object.

[0058] Step S106: Remove the image corresponding to the object region from the target image to obtain an intermediate image. The intermediate image includes a blank image corresponding to the object region and a background image corresponding to the target image.

[0059] An intermediate image refers to the image in which the background area is retained after the object area of ​​the target image is removed. In the intermediate image, a blank area will appear after the object area is removed. The image corresponding to this blank area can be used as a blank image. The background image refers to the image corresponding to the background area in the target image. Taking the shooting of a landscape image with people as an example, after determining the area occupied by the people in the image in step S102, the area occupied by the people can be removed from the target image, so that only the landscape image as the background is retained. The image of the people area is represented by a blank image, which serves as the intermediate image corresponding to the target image.

[0060] For reference Figure 2 ,in Figure 2 The 201 part represents the input target image, which may include a person area and a background area. Figure 2 The 202 part represents the intermediate image, which is the image obtained after removing the image corresponding to the person area in the target image. It can be seen that the intermediate image still carries the image corresponding to the background area in the target image, while the person area that was originally in the target image is filled with the corresponding blank image.

[0061] Step S108: Obtain a background image block that matches the blank image from the background image, fill the blank image in the intermediate image with the background image block, and use the filled intermediate image as the filled image.

[0062] Background image blocks refer to image blocks located in the background area of ​​the intermediate image. An intermediate image can consist of multiple image blocks, meaning at least two. That is, an intermediate image can include image blocks located in the blank image and image blocks located in the background area. The background image block is simply the image block in the background area. Filling the image refers to the image obtained by filling the blank image in the intermediate image. Since the intermediate image is obtained by removing the target area from the target image, it carries the blank image corresponding to the target area. Therefore, to achieve a harmonious visual effect, the missing target area in the intermediate image can be filled. For example, background image blocks from the background area can be used to fill the blank image, allowing the blank image to display the corresponding image color, thus ensuring the harmony of the overall image.

[0063] Specifically, the process of filling a blank image can be as follows: First, find a background image block that matches the blank image from the background image. For example, for each image block contained in the blank image, search for the background image block with the highest matching degree in the background. Then, each background image block found can be used as the background image block that matches the blank image. After that, the blank image can be filled with the background of each of the above-mentioned matching background image blocks. This can be done by filling each image block of the blank image with the matching background image block, thereby realizing the background filling of the blank image and using it as the filled image.

[0064] For reference Figure 3 ,in Figure 3 Part 301 in the image represents the intermediate image obtained after removing the object region from the target image. This intermediate image may include a blank image and a background image. Image block A in the blank image matches image block B in the background image. Therefore, when filling the blank image, image block B can be filled into the corresponding position of image block A. The filling process for other image blocks can be similar to the above process until all image blocks in the blank image are filled, resulting in the filled image. Figure 3 As shown in section 302.

[0065] Step S110: Perform style transformation and planar transformation on the filled image to obtain a transformed image. During the planar transformation, determine the planar transformation information corresponding to the target view plane, and perform planar transformation on the filled image according to the planar transformation information so that the transformed image is an image located on the target view plane.

[0066] Style transformation refers to the process of stylizing the filled image obtained in step S104. This can be used to change the background style of the filled image. Stylizing can involve cartoonizing the filled image, transforming it into a cartoon or comic book style, or blacking it out, converting it to a black and white background. Plane transformation refers to converting the current view plane (the horizontal plane of the viewpoint) corresponding to the filled image into a pre-defined view plane or a user-selected view plane (the target view plane). This target view plane can be set by the user based on the desired image processing effect. For example, if the user needs to generate a stereoscopic image corresponding to the target image, and the background is at the bottom of the stereoscopic image, then the view plane corresponding to the bottom of the stereoscopic image can be used as the target view plane. By transforming the background view plane, such as through translation or rotation, different visual effects can be achieved. Plane transformation information refers to the transformation information used for view plane transformation, such as a matrix used to implement the plane transformation. This can be achieved by multiplying the filled image with the aforementioned plane transformation matrix.

[0067] Meanwhile, the style transformation process and the planar transformation process are respectively the process of further image processing of the obtained filled image. The order can be to first perform style transformation on the filled image, and then perform planar transformation on the style-transformed filled image to convert it into a transformed image displayed in the target viewing plane. Alternatively, the filled image can first be transformed into a planar image to be displayed on the target viewing plane, and then style transformation is performed on it to obtain the transformed image.

[0068] Step S112: The transformed image is fused with the image corresponding to the object region to obtain the processed image corresponding to the target image.

[0069] Image processing refers to the image obtained after performing complete image processing on a target image. This can be a stereoscopic image of the target image, where the foreground object in the target image is standing, and the background image, after style transfer, serves as the bottom of the stereoscopic image, presenting a visual effect of separation between the foreground object and the background, and the foreground object appearing to emerge from the background. For example, it can achieve the effect of a character walking out of a cartoon. The generated processed image effect can be as follows: Figure 4 As shown.

[0070] In the above image processing method, the following steps are taken: First, a target image to be processed is acquired. Then, foreground object recognition is performed on the target image to obtain the object region corresponding to the foreground object. Next, the image corresponding to the object region is removed from the target image to obtain an intermediate image, which includes a blank image corresponding to the object region and a background image corresponding to the target image. Finally, a background image block matching the blank image is obtained from the background image, and the blank image in the intermediate image is filled with background based on the background image block. The filled intermediate image is then used as the filled image. Next, style transformation and planar transformation are performed on the filled image to obtain a transformed image. During the planar transformation, planar transformation information corresponding to the target view plane is determined, and planar transformation is performed based on this information to make the transformed image an image located on the target view plane. Finally, the transformed image is fused with the image corresponding to the object region to obtain the processed image corresponding to the target image. This application identifies foreground objects in a target image, removes the foreground object region from the target image, and fills the resulting blank image region with the background of the target image. This results in a natural and complete transition between the filled image and the background. By performing style transformation and planar transformation on the filled image to place it on the target view plane, and then fusing it with the image corresponding to the object region, the image can not only have the corresponding style, but also have a three-dimensional relationship between the foreground and background of the processed image, thus improving the image processing effect.

[0071] In one embodiment, such as Figure 5 As shown, step S108 may further include:

[0072] Step S502: Divide the object region into multiple regions.

[0073] The term "divided region" refers to the multiple regions obtained by dividing the object region in the intermediate image. These regions can be multiple image blocks in the blank image of the object region. In this embodiment, the object region can be divided to obtain multiple divided regions that make up the object region. Alternatively, the blank image in the object region can be divided to obtain multiple image blocks that make up the blank image.

[0074] Step S504: For the current pixel in the blank image, determine the segmented region to which the current pixel belongs, and use it as a reference segmented region.

[0075] The current pixel refers to any pixel within the object region of the blank image. The blank image is composed of multiple pixels, and the segmented regions obtained from its division can also include multiple pixels. The current pixel can be a pixel within a segmented region of the blank image, and the segmented region where the current pixel is located can be used as the reference segmented region corresponding to the current pixel.

[0076] For example, the object region corresponding to a blank image can be divided into region A, region B, and region C. Each region can include 3 pixels. Region A can include pixels 1, 2, and 3; region B can include pixels 4, 5, and 6; and region C can include pixels 7, 8, and 9. The current pixel can be any one of these pixels, such as pixel 4. Since pixel 4 is located in region B, region B can be used as the reference region for the current pixel 4.

[0077] Step S506: Obtain the background image block matched by the reference division region, and obtain the pixel point corresponding to the current pixel point in the background image block as the reference background pixel point.

[0078] The reference background pixel refers to the pixel in the background image block corresponding to the current pixel in the reference division area. It can be the pixel in the matched background image block that corresponds to the position of the current pixel in the reference division area.

[0079] For example, the reference region B corresponding to pixel 4 as the current pixel may include pixels 4, 5, and 6. At the same time, the background image block B' matched by the reference region B may include pixels 4', 5', and 6', and the pixel distribution of pixels 4', 5', and 6' is the same as that of pixels 4, 5, and 6. Then, pixel 4' in the background image block B' can be taken as the current pixel, that is, the reference background pixel corresponding to pixel 4.

[0080] Step S508: Based on the pixel value corresponding to the reference background pixel, obtain the fill pixel value corresponding to the current pixel, and fill the blank image in the intermediate image with background according to the fill pixel value.

[0081] The fill pixel value refers to the pixel value that needs to be filled for the current pixel. Since the current pixel is a pixel in a blank image, its pixel value is 0 when it is not filled. In this embodiment, the fill pixel value that needs to be filled for the current pixel can be obtained based on the pixel value of the reference background pixel corresponding to the current pixel. For example, the pixel value of the reference background pixel can be directly used as the fill pixel value of the current pixel, or the pixel value of the reference background pixel can be changed accordingly to obtain the fill pixel value of each current pixel in the blank image. The blank image is then filled using the fill pixel values ​​of each current pixel to obtain the filled image.

[0082] In this embodiment, the object region of the blank image can be divided to obtain the reference division region corresponding to each pixel in the blank image. The background image block matched by the reference division region can be used to find the reference background pixel corresponding to each pixel. Then, the pixel value can be used to fill the pixel, thereby improving the accuracy of filling the background of the blank image.

[0083] In one embodiment, the filled image is obtained through multiple rounds of background filling, such as... Figure 6 As shown, step S506 may further include:

[0084] Step S602: Select one or more candidate background image blocks corresponding to the reference division region in the current background filling round based on the image block selection strategy.

[0085] In this embodiment, the process of filling the blank image with background can consist of multiple iterations. The current background filling round refers to the current iteration round, and the candidate background image block refers to the background image block that may match the reference division region found in the background image during the current iteration. In this embodiment, each iteration selects one or more background image blocks that may match the reference division region from the background image as candidate background image blocks. The selection process is performed by a pre-set image block selection strategy, such as using image blocks near the matching background image blocks in the adjacent division regions of the reference division region as candidate background image blocks for that reference division region, etc.

[0086] Step S604: Obtain the filled image block of the reference division region after the previous round of background filling, and obtain the matching background image block corresponding to the reference division region in the previous round of background filling.

[0087] The previous background filling round refers to the round before the current one. For example, if the current round is the third, the previous round could be the second. The filled image block refers to the image block corresponding to the reference region after background filling in the previous iteration, which is the image block currently displayed for the reference region. The matching background image block refers to the image block that best matches the reference region in the previous round. If the current round is the first, the terminal can also randomly set a background image block corresponding to each reference region in the background image as the matching background image block for that reference region.

[0088] Step S606: Calculate the similarity between the candidate background image patch and the filled image patch, and use it as the first similarity.

[0089] The first similarity refers to the similarity between the candidate background image block corresponding to the reference division region selected in the current background filling round and the currently displayed image block of the reference division region, that is, the filled image block obtained after the previous round of background filling. This similarity can be calculated by the pixel values ​​of the pixels included in the candidate background image block and the pixel values ​​of the pixels included in the filled image block. For example, the similarity between image blocks can be characterized by calculating the distance between each pixel. The larger the distance, the smaller the similarity.

[0090] Step S608: Calculate the similarity between the matching background image patch and the filling image patch, as the second similarity.

[0091] The second similarity refers to the similarity between the background image block that matches the reference division region determined in the previous background filling round and the aforementioned filling image block. The matching background image block can be one or more. The second similarity can then be calculated in a manner similar to step S606.

[0092] Step S610: Based on the first similarity and the second similarity, obtain an image block similar to the filling image block from the matching background image block and the candidate background image block, and use it as the background image block matched by the reference division region in the current background filling round.

[0093] Finally, based on the determined first and second similarities, image blocks similar to the filling image block can be found from the matching background image blocks and candidate background image blocks. A similarity threshold can be determined based on the first and second similarities; when the similarity threshold is greater than a certain value, the background image block is considered similar to the filling image block, i.e., the background image block matched by the reference division region in the current background filling round. If only one background image block needs to be selected as the background image block matched by the reference division region, then the background image block with the highest similarity to the filling image block can be found from the matching background image blocks and candidate background image blocks. After obtaining the background image block matched by the reference division region in the current background filling round, it can also be used as the matching background image block corresponding to that reference division region in the next background filling round.

[0094] Taking the example of selecting only one background image patch as the background image patch matched by the reference partition region in each round, assuming that the current background filling round is the third, then we can first obtain the candidate background image patches corresponding to the reference partition region according to the image patch selection strategy, such as image patch B, image patch C, and image patch D. At the same time, we can also determine the filled image patch obtained by the reference partition region after the previous round of background filling, that is, the image patch currently displayed by the reference partition region, which can be image patch A, and the image patch matched by the reference partition region found in the previous round of background filling, that is, the second round of background filling, which can be image patch E. Then we can calculate the similarity between image patch A and the candidate background image patches, that is, the similarity between image patch A and image patch B, image patch C, and image patch D respectively, and the similarity between image patch A and the matched background image patch. The similarity between scene image patches, i.e., the similarity between image patch A and image patch E, is used to find the image patch with the highest similarity. For example, if image patch C has the highest similarity to image patch A, then image patch C can be used as the background image patch matched by the reference division region in the current background filling round. At the same time, it can also be used as the matching background image patch corresponding to the reference division region in the next background filling round, i.e., the fourth round of background filling. Furthermore, the terminal can also use image patch C to fill the reference division region. It can be that the image patch C selected in this round is used to fill the reference division region. The image patch obtained after filling can be image patch F, which is the image patch displayed after background filling in the current background filling round, and at the same time, it can be used as the filled image patch corresponding to the reference division region in the next background filling round, i.e., the fourth round of background filling.

[0095] In this embodiment, there can be multiple rounds of background filling for blank images. In each filling round, an image block similar to the image block to be filled is determined based on the similarity of each reference division region. This image block is used as the background image block matched by the reference division region in that round, thereby improving the matching degree of the selected background image block.

[0096] In one embodiment, step S508 may further include: when there are multiple reference partitioning regions corresponding to the current pixel, obtaining the similarity between the background image block matched by the reference partitioning region and the corresponding filling image block in the current background filling round; obtaining the pixel weight corresponding to the reference background pixel in the current background filling round based on the similarity, wherein the pixel weight is positively correlated with the similarity; and performing a weighted calculation based on the pixel value corresponding to the reference background pixel and the pixel weight corresponding to the reference background pixel to obtain the filling pixel value corresponding to the current pixel.

[0097] Pixel weight refers to the influence of the pixel values ​​of each reference background pixel on the fill pixel value of the current pixel. The magnitude of the pixel weight is positively correlated with the similarity between the background image patch matched by each reference segmentation region and its corresponding fill image patch. That is, the greater the similarity, the greater the pixel weight. The terminal can determine the pixel weight corresponding to each reference background pixel based on the relationship between the magnitudes of the similarities. Then, the pixel weight can be used to perform a weighted calculation on the pixel values ​​corresponding to each reference background pixel to obtain the fill pixel value corresponding to the current pixel.

[0098] In this embodiment, it can be as follows: Figure 7 As shown, multiple overlapping image regions are obtained by dividing the object region. When the current pixel falls into an overlapping image region, the current pixel can have multiple reference regions, namely, multiple reference regions corresponding to the current pixel, namely region A and region B. At the same time, region A and region B each have a matching background image block A' and background image block B', and background image block A' and background image block B' each carry a reference background pixel corresponding to the current pixel. Then, the similarity between the filling image block of region A and background image block A' in the current background filling round, and the similarity between the filling image block of region B and background image block B' in the current background filling round can be obtained. Based on the above similarity, the pixel weights corresponding to background image block A' and background image block B' are determined. Then, the pixel values ​​of the reference background pixels carried by background image block A' and background image block B' are weighted and calculated based on the pixel weights, so as to finally obtain the filling pixel value corresponding to the current pixel.

[0099] In this embodiment, the current pixel point can correspond to multiple reference division regions. After determining the reference background pixel points corresponding to each reference division region, the corresponding pixel weights can be determined according to the similarity between each reference division region and the matched background image block. This can make the matching blocks with high similarity have higher credibility and further improve the accuracy of the obtained filling pixel values.

[0100] In one embodiment, step S502 may further include: determining a reference division region corresponding to a reference division region among multiple division regions; obtaining the position of the background image block matched by the reference division region in the previous round of background filling, as a reference position; obtaining the relative positional relationship between the reference division region and the reference division region; obtaining the first image block selection position based on the reference position and the relative positional relationship; and selecting the background image block corresponding to the first image block selection position as a candidate background image block.

[0101] In this embodiment, the reference division region refers to a pre-defined division region that matches the reference division region. For example, it can be a division region adjacent to the reference division region, or it can be a division region with a certain positional relationship to the reference division region, such as a division region diagonally opposite to it. The relative positional relationship refers to the positional relationship between the reference division region and the reference division region, which can be vertically adjacent or horizontally adjacent, etc. The reference position refers to the position of the background image block matched by the reference division region in the previous background filling round. The first image block selection position is the position of the background image block determined according to the reference position and the relative positional relationship. The first image block selection position can be the position of the candidate background image block corresponding to the reference division region.

[0102] Since the background image block that best matches the reference division region is generally located in its adjacent division region, that is, near the background image block that best matches the reference division region, the image block position of the candidate background image block corresponding to the reference division region can be determined by the position of the matching background image block found in the reference division region and the relative positional relationship between the reference division region and the reference division region. That is, the first image block selection position. Then, the background image block at the first image block selection position can be used as the candidate background image block.

[0103] For example, when the reference division region A is above the reference division region B, after determining the position of the background image block A' matched in the previous background filling round of the reference division region, i.e. the reference position, the first image block selection position can be obtained based on the reference position and the positional relationship between the reference division region and the reference division region. That is, the position below the reference position is taken as the first image block selection position, and the background image block corresponding to the first image block selection position is taken as the candidate background image block corresponding to the reference division region. In other words, the image block below A' is taken as one of the candidate background image blocks of the reference division region B.

[0104] In this embodiment, the positional relationship between the baseline division region and the reference division region, as well as the position of the background image block matched by the baseline division region in the previous round of background filling, can be used to obtain the candidate background image block of the reference division region. This can take advantage of the continuity of the image to greatly reduce the complexity of the search and improve the efficiency of obtaining the candidate background image block.

[0105] In one embodiment, step S502 may further include: obtaining a randomly generated random offset distance; obtaining a corresponding offset distance shrinkage value based on the image block sorting of candidate background image blocks, wherein the shrinkage degree corresponding to the offset distance shrinkage value is positively correlated with the image block sorting; performing shrinkage processing on the random offset distance based on the offset distance shrinkage value to obtain a target offset distance; obtaining a second image block selection position based on the position of the background image block matched in the previous round of background filling of the reference division region and the target offset distance; and using the background image block corresponding to the second image block selection position as a candidate background image block.

[0106] The random offset distance is a pre-defined offset distance between a candidate background image patch in the current background filling round and a background image patch matched in the previous round of the reference division region. This distance is positively correlated with the size of the intermediate image; the larger the size of the intermediate image, the larger the set offset distance can be. The offset distance shrinkage value refers to the numerical value used to shrink the random offset distance. This value can be used to characterize the degree of shrinkage of the random offset distance, i.e., the magnitude of shrinkage. For example, this shrinkage value can be a shrinkage ratio for the random offset distance. This shrinkage magnitude is positively correlated with the candidate background image patch to be filtered. That is, the larger the sequence number of the filtered candidate background image patch, the greater its shrinkage ratio. Therefore, the target offset distance obtained after shrinking the random offset distance according to the offset distance shrinkage value will be smaller. The second image patch candidate position refers to the image patch position obtained by offsetting the position of the background image patch matched in the previous round of the reference division region according to the target offset distance. The background image patch corresponding to this position can be used as a candidate background image patch for the reference division region in the current background filling round.

[0107] If the best matching image patch is determined solely based on adjacent matching, it is easy to get trapped in local optima. Therefore, a random search step is also required. The random search step generates a series of candidate locations based on an exponential distribution sampling. The location of the candidate background image patch obtained through the random search can be calculated using the following formula:

[0108]

[0109] Where, x i The x-axis position coordinates of the i-th selected candidate background image patch are represented by the y-axis position coordinates. iLet represent the y-axis position coordinate of the i-th selected candidate background image patch, x0 represent the x-axis position coordinate of the background image patch matched by the reference partitioning region in the previous round of background filling, y0 represent the y-axis position coordinate of the background image patch matched by the reference partitioning region in the previous round of background filling, wp1 and wp2 represent the random offset distances along the x-axis and y-axis, respectively, where w is the maximum diameter of the image, and p1 and p2 are random numbers in the range [-1, 1]. This represents the offset distance shrinkage value. Since i represents the image patch ranking of the selected candidate background image patches, the larger i is, the larger the image patch ranking, the greater the shrinkage of the random offset will be. Then, the target offset distance can be obtained based on the random offset distance and the offset distance shrinkage value. and Therefore, based on the position of the background image patch matched in the previous round of background filling within the reference segmentation region, and the target offset distance, the final candidate background image patch corresponding to the second image patch selection position is obtained.

[0110] In this embodiment, the target offset distance obtained by random offset distance and offset distance shrinkage value can also be used to search for the selection position of the second image block, which can avoid getting trapped in local optima. Since the shrinkage magnitude of the offset distance shrinkage value is positively correlated with the image block sorting, it can avoid the candidate background image blocks being in the same close position when the random offset distances are similar, which can further ensure the hierarchy of candidate background image blocks and thus improve the accuracy of candidate background image block selection.

[0111] In one embodiment, such as Figure 8 As shown, step S510 may further include:

[0112] Step S802: Perform style transformation on the filled image to obtain the initial style image.

[0113] The initial style image refers to the style image directly obtained after style transformation of the filled image. This style transformation can be achieved by cartoonizing the filled image. Specifically, after obtaining the filled image, it can be input into a pre-trained cartoon style transfer model, which outputs the initial style image. This model can classify real images into three representations: appearance, structure, and texture. It is trained using unpaired real images and cartoon-style images through a generative adversarial network.

[0114] Step S804: Obtain the background object image corresponding to the target background object in the initial style image, and remove the background object image from the initial style image to obtain the image to be processed in the initial style image.

[0115] In this context, the target background object refers to the background object in the pre-defined initial style image that is not subject to color clustering, such as the sky portion of the initial style image. The background object image refers to the image region corresponding to the target background object, such as the sky region image in the initial style image. The image to be processed refers to the image corresponding to the region of the initial style image that requires color clustering. In this embodiment, since the color clustering process may cause the sky color in the initial style image to become layered, producing unrealistic effects, the sky region in the initial style image can be used as the target background object. The image corresponding to the sky region can then be identified as the background object image and removed from the initial style image to obtain the corresponding image to be processed that requires color clustering.

[0116] Step S806: Perform color clustering on the image to be processed to obtain a color clustering image.

[0117] Color clustering refers to a method of clustering based on the channel values ​​of colors to achieve the quantization process of images. Specifically, a pre-defined clustering algorithm, such as the k-means clustering algorithm, can be used to perform color clustering on the image to be processed, thereby obtaining a color clustered image. For example, it can be clustered and quantized into an 8-bit image.

[0118] Step S808: The color clustering image is fused with the background object image, and the target style image is obtained based on the fused image.

[0119] The target style image is the image obtained after the final style conversion processing of the filled image. Specifically, after obtaining the background object image corresponding to the target background object in the initial style image in step S804, and the color clustering image of the initial style image after removing the background object image in step S806, the background object image and the color clustering image can be fused. This can be done by weighted summation using a class attention mechanism to obtain the fused image. The final target style image is then obtained based on the fused image. For example, the fused image can be further pixelated by setting the size of the mosaic color block region and setting the pixel value of each pixel in the mosaic color block region to be equal to the pixel value of the center point of the region, thereby achieving pixelation processing of the fused image to obtain the final target style image.

[0120] In this embodiment, the background object image corresponding to the target background object, such as the sky, can also be obtained by style-converting the initial style image of the filled image. The background object image is removed from the initial style image to obtain the image to be processed. Moreover, color clustering is only performed on the image to be processed after removal, which can avoid the color layering of the target background object caused by quantization, thereby improving the realism of the filled image after style conversion.

[0121] In one embodiment, step S804 may further include: obtaining the color parameter threshold range corresponding to the target background object, wherein the target background object is a background object whose color consistency is greater than the consistency threshold; selecting pixels whose color parameter values ​​are within the color parameter threshold range from the initial style image as target pixels; and using the continuous image blocks corresponding to the target pixels in the initial style image as the background object image corresponding to the target background object.

[0122] Color consistency is used to characterize whether the colors of objects are consistent. In this embodiment, the target background object to be filtered is one that meets the color consistency requirement. For example, if the target background object to be filtered is the sky, then the filtered area would be a background object that meets the blue consistency requirement. If the target background object to be filtered is a beach, then the filtered area would be a region that meets the yellow consistency requirement. The filtered color can also be determined by setting a threshold range for the color parameters corresponding to the target background object. If the sky is being filtered, then the threshold range for the blue color parameter can be used as the threshold range for the target background object. If the beach is being filtered, then the threshold range for the yellow color parameter can be used as the threshold range for the target background object. The target pixel refers to the pixel in the initial style image whose color parameter value meets the aforementioned threshold range.

[0123] Specifically, when it is necessary to filter target background objects in the initial style image, a threshold range of color parameters corresponding to the target background object can be preset. For example, if the sky area of ​​the initial style image needs to be filtered, then the threshold range of color parameters corresponding to blue can be used as the threshold range of color parameters corresponding to the target background object. Then, pixels whose color channel values ​​meet the above threshold range of color parameters can be filtered from the initial style image as target pixels, that is, pixels that are displayed as blue are filtered as target pixels, and the continuous image blocks corresponding to the target pixels, that is, the continuous blue image blocks, are used as background object images.

[0124] In this embodiment, the target background object can be selected from the initial style image by setting a threshold range for the color parameters corresponding to the target background object, thereby improving the accuracy of background object image selection.

[0125] In one embodiment, step S808 may further include: fusing the color clustering image with the background object image to obtain a fused image; performing edge extraction on the initial style image to obtain an edge image corresponding to the initial style image; and removing the edge image from the fused image to obtain a target style image.

[0126] A fused image refers to the image obtained by fusing a color clustering image with a background object image. This can be achieved by using a class attention mechanism to perform a weighted summation of the color clustering image and the background object image. An edge image, on the other hand, refers to the image obtained by extracting the edges of an initial style image. This can be achieved by applying an edge detection algorithm to the initial style image to extract the corresponding image edges.

[0127] Specifically, after obtaining the color clustering image and the background object image, a weighted summation of the two images can be performed using a class attention mechanism to obtain a fused image. If the background object image represents the sky region of the initial style image, then the color clustering image can be obtained by clustering only the non-sky regions of the initial style image. Therefore, the sky region of the fused image retains the un-clustered portion of the initial style image, while the non-sky regions can be the color-clustered portion. Simultaneously, edge detection algorithms, such as the Canny edge detection algorithm, can be used to extract edges from the initial style image, obtaining the corresponding edge images, which can be represented as edge grayscale images. Then, the fused image can be subtracted from the edge grayscale images to remove the edge images, resulting in the final target style image.

[0128] In this embodiment, after obtaining the fused image, the edge image obtained by edge extraction can be removed from the fused image to obtain the final target style image, thereby improving the smoothness of the target style image.

[0129] In one embodiment, edge extraction is performed on an initial style image to obtain an edge image corresponding to the initial style image, including: performing edge extraction on the initial style image using an edge extraction operator to obtain a first edge image corresponding to the initial style image; performing edge extraction on the initial style image using an edge extraction model to obtain a second edge image corresponding to the initial style image; obtaining the image intersection between the first edge image and the second edge image, and using the image intersection as the edge image corresponding to the initial style image.

[0130] In this embodiment, the first edge image and the second edge image refer to the edge images obtained after edge extraction of the initial style image using different edge extraction methods. The first edge image is obtained by edge extraction using a pre-defined edge extraction operator, such as the Canny operator, while the second edge image is obtained by edge extraction using a pre-trained edge extraction model, such as the HED model. The image intersection refers to the intersection between the first edge image and the second edge image. This can be achieved by calculating the normalized images corresponding to the first and second edge images respectively, and then using the product of the normalized images as the edge image corresponding to the initial style image.

[0131] Specifically, the first edge image extracted by the Canny operator does not conform to the human visual system and lacks semantic meaning. While the second edge image extracted by the HED (holistically-nested edge detection) model is relatively consistent with human vision, its drawback is that the extracted edges are relatively coarse and produce large grayscale areas at the edges. Therefore, to achieve better edge extraction results, this embodiment uses the Canny operator as the edge operator to obtain the first edge image, and the HED model as the edge extraction model to obtain the second edge image. The intersection of the first and second edge images—that is, the product of the normalized images corresponding to the first and second edge images—is used as the edge image corresponding to the initial style image. This refines the edge image, thereby improving its precision.

[0132] In this embodiment, edge extraction of the initial style image can be achieved by using edge extraction operators and edge extraction models respectively. The intersection of the obtained first edge image and the second edge image is taken as the edge image corresponding to the initial style image. The edge image can be refined to improve the precision of the obtained edge image.

[0133] In one embodiment, step S102 may further include: receiving an image processing request, wherein the image processing request is triggered by the terminal in response to the stereoscopic image generation operation; step S112 may further include: determining the stereoscopic effect of the image displayed by the stereoscopic image generation operation, obtaining the view plane corresponding to the stereoscopic effect of the image as the target view plane; determining the plane transformation information corresponding to the target view plane, and performing a plane transformation on the filling image according to the plane transformation information, so that the transformed image is an image located on the target view plane.

[0134] In this context, an image processing request refers to a request to process a target image. This request can be generated by the terminal responding to a user-triggered operation to generate a stereoscopic image corresponding to the target image. For example, when a user needs to process a target image to generate a stereoscopic image, they can access an image processing application installed on the terminal. The terminal can then display the application's image processing interface, which may offer various image processing functions with corresponding options. The user can trigger the stereoscopic image generation operation by clicking the option to generate a stereoscopic image, at which point the terminal can generate a corresponding image processing request.

[0135] The stereoscopic effect refers to the stereoscopic effect of the image generated by the user. For the same target image, the corresponding stereoscopic image can be generated in different ways, and therefore the stereoscopic effect can also be different. For example, the stereoscopic effect can include using the background object in the target image as the bottom and the foreground object as the top. This could be a person in the target image placed above the background, i.e., the person walking out of the background image. Or it could be using the background object in the target image as the top and the foreground object placed below the background, i.e., the person flying out of the background image. This stereoscopic effect can be achieved by changing the view plane. Therefore, when the user triggers the stereoscopic image generation operation, the terminal can determine the corresponding view plane based on the stereoscopic effect required by the operation, and use the planar transformation information corresponding to the target view plane to perform a planar transformation on the filled background image, i.e., the filled image, to obtain a transformed image that is located on the target view plane.

[0136] like Figure 9 As shown, Figure 9 Section 901 shows the interface of an image processing application displayed on the terminal. The application A interface can contain option windows for various image processing methods, including multiple stereoscopic image processing methods such as Stereoscopic Effect Processing 1 and Stereoscopic Effect Processing 2. When the terminal clicks on the Stereoscopic Effect Processing 1 option window, it can proceed as follows: Figure 9 Section 902 displays the details page of the stereoscopic effect processing 1. At the bottom of the details page, an "Immediately Create" button is displayed. Clicking this button triggers the stereoscopic image generation operation corresponding to stereoscopic effect processing 1. The terminal can then respond to this operation, receiving and responding to the image processing request for the target image. Figure 9Section 903 displays the target image input interface. Users can input a target image here. This can be done by clicking the camera button to take a real-time photo and using it as the target image, or by clicking the image selection button to choose from pre-stored images on the device. After selecting the target image, the user can proceed to... Figure 9 In section 904, after processing the uploaded target image such as image cropping, clicking the "Create Now" button in section 904 will execute the creation of a 3D image corresponding to the target image's 3D effect 1, thus obtaining the image as shown below. Figure 9 The 3D images shown in section 905.

[0137] In this embodiment, an image processing request can be obtained based on the triggered stereo image generation operation. Furthermore, based on the stereo effect of the image corresponding to the stereo image generation operation, the target viewing plane and the plane transformation information corresponding to the target viewing plane can be determined. This enables the transformed image to be located on the target viewing plane. In other words, users can generate stereo images with different stereo effects by triggering different stereo image generation operations, thereby improving the diversity of stereo image stereo effects.

[0138] In one embodiment, such as Figure 10 As shown, an image processing method is provided, which may include the following steps:

[0139] Step S1002: Receive an image processing request, which is triggered by the terminal in response to the stereoscopic image generation operation.

[0140] An image processing request refers to a request to process a target image. A stereoscopic image generation operation refers to an operation triggered by the user to generate a stereoscopic image. The user can trigger the stereoscopic image generation operation by clicking the option to generate a stereoscopic image displayed on the image processing application installed on the terminal. The terminal can then respond to the operation and receive the corresponding image processing request.

[0141] Step S1004: Obtain the target image to be processed; perform foreground object recognition on the target image to obtain the object region corresponding to the foreground object in the target image; remove the image corresponding to the object region from the target image to obtain an intermediate image, which includes the blank image corresponding to the object region and the background image corresponding to the target image.

[0142] The target image refers to the image material that needs to be processed. The foreground object refers to the object corresponding to the foreground part of the target image, which can be a person in the target image, and the object area corresponding to the foreground object can be the area of ​​the person in the target image. The intermediate image refers to the image after removing the foreground object area from the target image, retaining only the background area. It can be an image that only retains the landscape area of ​​the target image, while the portrait area of ​​the target image is removed and presented as a blank image.

[0143] Specifically, after the terminal responds to the image processing request, it can also obtain the image material corresponding to the request as the target image, and extract the foreground object, such as a person object, from the target image. After obtaining the object area corresponding to the foreground object, the area is removed from the target image, thereby obtaining an intermediate image composed of a blank image corresponding to the object area and a background image corresponding to the target image.

[0144] Step S1006: Divide the object region into multiple regions; for the current pixel in the blank image, determine the multiple regions to which the current pixel belongs, and use them as multiple reference regions.

[0145] A region segmentation refers to dividing the object region in an intermediate image into multiple regions. These regions can be multiple image blocks within the blank image of the object region. These regions may overlap. The current pixel refers to any pixel within the object region, while the reference region refers to the multiple regions in which the current pixel resides. Because these regions overlap, the current pixel can also be located in different regions, resulting in multiple reference regions.

[0146] Step S1008: Determine the reference division region corresponding to the reference division region among multiple division regions; obtain the position of the background image block matched by the reference division region in the previous round of background filling, as a reference position; obtain the relative positional relationship between the reference division region and the reference division region; obtain the first image block selection position based on the reference position and the relative positional relationship; and select the background image block corresponding to the first image block selection position as a candidate background image block.

[0147] The process of filling the blank image with background can consist of multiple iterations. The current background filling iteration refers to the current iteration. Candidate background image patches refer to background image patches that may match the reference segmentation region found in the background image during the current iteration. The reference segmentation region refers to a pre-defined segmentation region that matches the reference segmentation region, such as a segmentation region adjacent to the reference segmentation region. The relative positional relationship refers to the positional relationship between the reference segmentation region and the reference segmentation region, which can be vertically adjacent, horizontally adjacent, etc. The reference position refers to the position of the background image patch matched by the reference segmentation region in the previous background filling iteration, and the first image patch selection position is the position of the background image patch determined based on the reference position and the relative positional relationship.

[0148] In this step, the selection of candidate background image blocks for reference region matching can be based on the position of the best-matched background image block in the previous round of the adjacent reference region, i.e., the reference position, and the relative positional relationship between the reference region and the reference region. In other words, the image blocks adjacent to the reference position are used as candidate background image blocks for reference region matching in this round.

[0149] Step S1010: Obtain a randomly generated random offset distance; obtain the corresponding offset distance shrinkage value based on the image block sorting of the candidate background image blocks, and the shrinkage degree corresponding to the offset value and the image block sorting is positively correlated; perform shrinkage processing on the random offset distance based on the offset distance shrinkage value to obtain the target offset distance; obtain the second image block selection position according to the position of the background image block matched in the previous round of background filling of the reference division region and the target offset distance; and take the background image block corresponding to the second image block selection position as the candidate background image block.

[0150] The location of the second image block can be calculated using the following formula:

[0151]

[0152] Where, x i The x-axis position coordinates of the i-th selected candidate background image patch are represented by the y-axis position coordinates. i Let represent the y-axis position coordinate of the i-th selected candidate background image patch, x0 represent the x-axis position coordinate of the background image patch matched by the reference partitioning region in the previous round of background filling, y0 represent the y-axis position coordinate of the background image patch matched by the reference partitioning region in the previous round of background filling, wp1 and wp2 represent the random offset distances along the x-axis and y-axis, respectively, where w is the maximum diameter of the image, and p1 and p2 are random numbers in the range [-1, 1]. This represents the offset distance shrinkage value. Since i represents the image patch ranking of the selected candidate background image patches, the larger i is, the larger the image patch ranking, the greater the shrinkage of the random offset will be. Then, the target offset distance can be obtained based on the random offset distance and the offset distance shrinkage value. and Therefore, based on the position of the background image block matched in the previous round of background filling in the reference division region, and the target offset distance, the second image block selection position corresponding to the final candidate background image block is obtained.

[0153] Step S1012: Obtain the filled image block obtained after the previous round of background filling from the reference segmentation region, and obtain the matching background image block corresponding to the reference segmentation region in the previous round of background filling; calculate the similarity between the candidate background image block and the filled image block as the first similarity; calculate the similarity between the matching background image block and the filled image block as the second similarity; based on the first similarity and the second similarity, obtain the image block similar to the filled image block from the matching background image block and the candidate background image block, as the background image block matched by the reference segmentation region in the current round of background filling.

[0154] The filled image patch refers to the image patch corresponding to the reference segmentation region after background filling in the previous iteration, i.e., the image patch displayed in the current reference segmentation region. The matching background image patch refers to the image patch that best matches the reference segmentation region determined in the previous background filling round. The first similarity refers to the similarity between the candidate background image patch selected in the current background filling round and the filled image patch in the previous round, while the second similarity refers to the similarity between the matching background image patch and the filled image patch in the previous round.

[0155] Specifically, after determining multiple candidate background image blocks of the reference division region, the terminal can calculate the similarity with the filled image block of the reference division region after the previous round of background filling as the first similarity. It can also calculate the similarity between the matching background image block that best matches the reference division region in the previous round of background filling and the filled image block as the second similarity. Based on the first similarity and the second similarity, the terminal finds the image block with the highest similarity as the best matching background image block of the reference division region in the current round of background filling.

[0156] Step S1014: Obtain the similarity between the background image blocks matched by multiple reference division regions and the corresponding filling image blocks in the current background filling round; obtain the pixel weights corresponding to the reference background pixels in the current background filling round based on the similarity, and the pixel weights are positively correlated with the similarity; perform weighted calculation based on the pixel values ​​corresponding to the reference background pixels and the pixel weights corresponding to the reference background pixels to obtain the filling pixel value corresponding to the current pixel, and perform background filling on the blank image in the intermediate image according to the filling pixel value.

[0157] Pixel weight refers to the influence of each reference background pixel value on the fill pixel value of the current pixel. The magnitude of this pixel weight is positively correlated with the similarity between the background image patch matched to each reference region and its corresponding fill image patch; that is, the greater the similarity, the greater the pixel weight. The terminal can determine the pixel weight corresponding to each reference background pixel based on the similarity relationships. Then, it can use these pixel weights to perform a weighted calculation on the pixel values ​​corresponding to each reference background pixel to obtain the fill pixel value corresponding to the current pixel. Finally, the fill pixel value corresponding to each current pixel in the blank image can be used to fill the background of the object region.

[0158] Step S1016: Perform style transformation on the filled image to obtain an initial style image; obtain the color parameter threshold range corresponding to the target background object, where the target background object is a background object with a color consistency greater than the consistency threshold; select pixels with color parameter values ​​within the color parameter threshold range from the initial style image as target pixels; and use the continuous image blocks corresponding to the target pixels in the initial style image as the background object image corresponding to the target background object.

[0159] Here, the initial style image refers to the style image directly obtained after style transformation of the filled image. It can be a cartoonized image obtained by cartoonizing the filled image. The target background object is a background object with a color consistency greater than the consistency threshold, that is, an object that needs to meet the color consistency requirement. It can be the sky area of ​​the cartoonized image. The color parameter threshold range corresponding to the target background object refers to the pre-defined range of color consistency parameters among objects that meet the color consistency requirement. If it is the sky area, then the color parameter threshold range corresponding to the target background object refers to the color parameter threshold range corresponding to blue.

[0160] Specifically, the terminal can filter out pixels in the cartoon image that meet the set blue parameter threshold range for color channel values ​​as target pixels, and use the continuous image blocks corresponding to the target pixels, i.e., the continuous blue image blocks, as background object images, thereby filtering out the blue sky area as the target background object, and obtaining the sky area image corresponding to the sky area as the background object image.

[0161] Step S1018: Remove the background object image from the initial style image to obtain the image to be processed in the initial style image; perform color clustering on the image to be processed to obtain a color clustering image; and fuse the color clustering image with the background object image to obtain a fused image.

[0162] The image to be processed refers to the image corresponding to the region of the initial style image that needs to be processed by color clustering. It can be the non-sky region image remaining after removing the sky region image in a cartoon image. Color clustering refers to a method of clustering based on the channel values ​​of color to realize the quantization process of the image. Color clustered image refers to an 8-bit image quantized by color clustering.

[0163] Specifically, to avoid the color clustering process potentially causing stratification of sky colors in the initial style image and producing unrealistic effects, this embodiment removes the selected sky regions from the initial style image, thus obtaining the corresponding image to be processed for color clustering. Color clustering is performed only on the non-sky regions using the k-means clustering algorithm, resulting in a color-clustered image. Then, the clustered non-sky region images and the unclustered sky region images can be fused, possibly through a weighted summation using a class-attention mechanism, to obtain the fused image.

[0164] Step S1020: Use an edge extraction operator to extract edges from the initial style image to obtain a first edge image corresponding to the initial style image; use an edge extraction model to extract edges from the initial style image to obtain a second edge image corresponding to the initial style image; obtain the image intersection between the first edge image and the second edge image, and use the image intersection as the edge image corresponding to the initial style image; remove the edge image from the fused image to obtain the target style image.

[0165] The first edge image and the second edge image refer to the edge images obtained after edge extraction from the initial style image using different edge extraction methods. The first edge image is obtained by edge extraction using a pre-defined edge extraction operator, such as the Canny operator, while the second edge image is obtained by edge extraction using a pre-trained edge extraction model, such as the HED model. The image intersection is the intersection between the first edge image and the second edge image. The target style image refers to the image obtained after the final style transfer processing of the filled image.

[0166] Specifically, this step can also use the Canny operator and the HED model to extract edges from the initial style image, respectively, to obtain a first edge image and a second edge image, and obtain the image intersection between the first edge image and the second edge image. This can be done by multiplying the normalized images corresponding to the first edge image and the second edge image respectively as the edge image corresponding to the initial style image. Finally, the edge image is removed from the fused image obtained in step S1018 to obtain the final target style image.

[0167] Step S1022: Determine the stereoscopic effect of the image displayed by the stereoscopic image generation operation, obtain the view plane corresponding to the stereoscopic effect of the image as the target view plane; determine the planar transformation information corresponding to the target view plane, perform planar transformation on the target style image according to the planar transformation information, so that the transformed image is an image located on the target view plane; fuse the transformed image with the image corresponding to the object region to obtain the processed image corresponding to the target image.

[0168] The stereoscopic effect refers to the stereoscopic effect of the stereoscopic image that the user needs to generate. This stereoscopic effect can correspond to the triggered stereoscopic image generation operation. The target view plane refers to the view plane corresponding to the stereoscopic effect. The plane transformation information is the transformation information required for the view plane transformation, such as a matrix used to implement the plane transformation. The processed image refers to the stereoscopic image obtained after performing complete image processing on the target image. This stereoscopic image can be a stereoscopic image where the foreground object in the target image is standing, and the image obtained after style conversion of the background part of the target image serves as the bottom of the stereoscopic image, presenting a visual effect of separation between the foreground object and the background.

[0169] Specifically, the terminal can determine the stereoscopic effect of the stereoscopic image to be generated based on the triggered stereoscopic image generation operation, thereby determining the target view plane where the target style image is located, and thus obtaining the corresponding planar transformation matrix. By multiplying the target style image with the planar transformation matrix, the transformed image after the planar transformation of the target style image can be obtained. Furthermore, the transformed image can be fused with the image corresponding to the foreground object region to obtain the final processed image.

[0170] The aforementioned image processing method enables a natural and complete transition of the filled image background, while maintaining a corresponding style and creating a three-dimensional relationship between the foreground and background, thus improving the image processing effect. Furthermore, the filling process utilizes background image blocks matched with reference regions to find the reference background pixel corresponding to each pixel, and then uses its pixel value to achieve pixel filling, which improves the accuracy of filling blank image backgrounds. Multiple rounds of background filling can be performed, and in each round, image blocks similar to the filling image block are determined based on similarity for each reference region, serving as the background image blocks matched with the reference region in that round, thereby improving the matching degree of the selected background image blocks. Simultaneously, to improve the efficiency and accuracy of candidate background image block selection, this application obtains candidate background image blocks for the reference region by considering the positional relationship between the baseline region and the reference region, as well as the position of the background image block matched by the baseline region in the previous round of background filling. Additionally, it searches for candidate image blocks by using a random offset distance and an offset distance contraction value to obtain the target offset distance, further improving the efficiency and accuracy of candidate background image block selection. For pixel value filling at the current pixel, the corresponding pixel weights can be determined according to the similarity between each reference division region and the matched background image patch. This allows matching patches with high similarity to have higher credibility, further improving the accuracy of the obtained filled pixel values. Furthermore, for the style transfer process, color clustering is only performed on the processed image after removal. This avoids color stratification of the target background object caused by quantization, thereby improving the realism of the filled image after style transfer. The target background object is selected by setting a threshold range for the corresponding color parameters, which improves the accuracy of background object image selection. Simultaneously, after obtaining the fused image, the edge images obtained through edge extraction can be removed from the fused image to obtain the final target style image, thereby improving the smoothness of the target style image. Edge extraction is implemented through edge extraction operators and edge extraction models, and the intersection of the extraction results is used as the edge image corresponding to the initial style image, which can refine the edge image and improve its precision. The planar transformation process determines the target view plane based on the stereoscopic effect of the image corresponding to the stereoscopic image generation operation. This allows users to generate stereoscopic images with different stereoscopic effects by triggering different stereoscopic image generation operations, thereby increasing the diversity of stereoscopic image effects.

[0171] In one embodiment, a method for creating stereoscopic images based on street view pixel style is also provided. This method splits the image into foreground and background, performs image folding on the foreground portrait, completes the background, and combines it with a new street view pixel style filter effect to increase the image's layers and transform the image style while making the image into a stereoscopic effect.

[0172] Users can enter the mini program homepage, click on the material to display the material details page, and click "Make Now" to select or take pictures from the album. The pictures are uploaded to the backend, processed by the algorithm model, and the video result is generated and returned to the user.

[0173] The specific process may include the following steps:

[0174] Face detection -- No face detected -- Return image, select new image;

[0175] Face detection -- face detected -- portrait segmentation -- background completion -- street view pixel filter processing (no filter processing on portrait area) -- image folding.

[0176] (1) Street view pixel style filter;

[0177] Street view pixel filter system process as follows Figure 11 As shown, the street view image is first cartoonized to generate a cartoon street view image. Edge extraction and thinning operations are then performed on the cartoon street view image to obtain an edge grayscale image. Sky segmentation is used to obtain a binary segmentation image, and color clustering is used to obtain an 8-bit street view image. Then, using the segmentation image as a mask, the sky region of the original cartoon street view and other regions of the clustered 8-bit street view are retained to obtain a quantized street view. The edge grayscale image and the quantized street view image are then pixelated, and the weighted edge pixel image is subtracted from the street view pixel image to obtain the final result.

[0178] (a) Cartoonization

[0179] The original street scene image is cartoonized using White box Cartoonization. The model divides real images into three representations: appearance, structure, and texture. It is trained using unpaired real images and cartoon-style images.

[0180] (b) Sky segmentation

[0181] To avoid color layering in the quantized sky, large areas of blue are treated separately as the sky. First, the original image is converted to HSV space, the blue areas are selected, and then thresholding is performed. After noise reduction and graphics opening operations, only large blue areas are retained, resulting in a binary segmentation image of the sky area and other regions.

[0182] (c) Edge extraction and refinement

[0183] Collaborative edge detection was performed on cartoonish street scene images using the Canny operator and the Heterogeneous Edge Detection (HED) model. The threshold was adjusted so that the Canny operator extracted dense edges, while HED preserved semantically meaningful edges. Traditional image processing methods using the Canny operator for edge detection do not conform to human visual perception and lack semantic meaning. HED's results are relatively consistent with human vision, but its drawbacks include coarser edges and the generation of large grayscale areas at edges. Using the product of the normalized images of both operators as a thinner edge map allows for better edge rendering of the street scene pixel images.

[0184] (d) Color clustering

[0185] The cartoon-style street view image is quantized using the k-means clustering algorithm to generate an 8-bit street view image. To avoid layering after clustering due to the gradual change in sky color, a binary sky segmentation map is used to perform a weighted summation of the 8-bit street view image and the original cartoon-style street view image using a class attention mechanism to synthesize the quantized street view image, thus preserving the sky region of the original image.

[0186] (e) Pixelation

[0187] The quantized street view image and edge image are pixelated separately. The mosaic size is determined by the image size; the image shows the processing effect with a mosaic size of 3. Specifically, the pixels within each mosaic color block area are set to equal the pixel value of the region's center point. Finally, the weighted edge pixel image is subtracted from the street view pixel image to obtain the final street view pixelated image.

[0188] (2) Image completion

[0189] This invention uses image completion technology, employing background textures to fill in missing areas of the human figure in the background, achieving a harmonious overall image. For example... Figure 12 As shown, the basic assumption of this algorithm is that the background region and the region to be filled have a certain correlation. During the calculation, a series of image patches in the background that are most similar to the image patch at the region to be filled are found, and then these image patches are appropriately weighted and averaged to complete the content filling of the region to be filled.

[0190] In this process, the main computational burden lies in matching image patches. Assume the foreground region contains N... f 1 pixel, background area contains N b For a given number of pixels, a direct brute-force search can achieve a time complexity of O(N). f N b Here we use a random hill-climbing strategy to search for the most similar image patch, which reduces the complexity of the patch matching algorithm to O(N). f logN b ).

[0191] (a) Introduction to the principle of image patch matching

[0192] Image patch matching first randomly assigns a matching image patch to each location in the entire image. Then, the matching patch is continuously improved to increase the similarity. There are two ways to improve this: propagation and random search.

[0193] The propagation step leverages the continuity of the image structure: the best match of a block near any given image patch is most likely to appear in the vicinity of the best match of that image patch. Utilizing this continuity significantly reduces the search cost. We can improve the matching patch at the current position using known matching information from neighboring image patches. If any matching patch at any position has higher similarity, it is improved accordingly. This process needs to be repeated several times, with odd-numbered steps propagating right and down, and even-numbered steps propagating left and up. Experiments show that after several iterations, a good approximation of the best matching patch can be obtained.

[0194] If only a propagation step is used, it is easy to get trapped in local optima. Therefore, a random search step is also required. The random search step generates a series of candidate locations by sampling according to an exponential distribution. Similarly, if any of these locations corresponds to a block with higher similarity, the result is improved accordingly. The positions of the candidate points are generated as follows:

[0195]

[0196] Where w is the maximum diameter of the image, (x0, y0) is the current matching block position, and p1, p2 are random numbers in the range [-1, 1]. The convergence speed of the entire algorithm is related to the initial random matching block position. Therefore, during initialization, the experiment can be repeated multiple times (e.g., 15 times) to improve the matching block. Typically, the above iterative process converges after 2-5 steps.

[0197] (b) Introduction to the principle of image content filling

[0198] After obtaining the nearest neighbor field, the next step is to generate image information for the region to be filled based on the best matching block. The entire filling process is similar to the EM iterative algorithm. For any pixel in the region to be filled, there are many image blocks containing that pixel. Each of these blocks has its own best matching block. Now, the pixel values ​​corresponding to the original pixel in these best matching blocks are weighted and averaged according to their similarity. The higher the similarity of the matching block, the higher its confidence, and the higher its weight.

[0199] (3) Image folding

[0200] After the image is filled, an affine transformation is performed on it to blend it with the foreground, creating a separation effect. Furthermore, a large-radius Gaussian blur is applied to the entire background.

[0201] By combining the three techniques of image completion, image folding, and image stylization through the above image processing methods, the overall gameplay effect is achieved. Image completion fills the background after the portrait is segmented, image stylization transforms the background style of the image, and image folding stands the static image upright to achieve the effect of a dynamic three-dimensional image.

[0202] It should be understood that although the steps in the flowchart of this application are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the diagram may include multiple steps or multiple stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least a portion of steps or stages in other steps.

[0203] In one embodiment, such as Figure 13 As shown, an image processing apparatus is provided, including: a target image acquisition module 1302, a foreground object recognition module 1304, an intermediate image acquisition module 1306, a filled image acquisition module 1308, a transformed image acquisition module 1310, and a processed image acquisition module 1312, wherein:

[0204] The target image acquisition module 1302 is used to acquire the target image to be processed.

[0205] The foreground object recognition module 1304 is used to perform foreground object recognition on the target image and obtain the object region corresponding to the foreground object in the target image;

[0206] The intermediate image acquisition module 1306 is used to remove the image corresponding to the object region from the target image to obtain an intermediate image. The intermediate image includes a blank image corresponding to the object region and a background image corresponding to the target image.

[0207] The fill image acquisition module 1308 is used to acquire a background image block that matches the blank image from the background image, fill the blank image in the intermediate image with the background image block, and use the filled intermediate image as the fill image.

[0208] The transformed image acquisition module 1310 is used to perform style transformation and planar transformation on the filled image to obtain a transformed image. When performing planar transformation, the planar transformation information corresponding to the target view plane is determined, and the filled image is transformed according to the planar transformation information so that the transformed image is an image located on the target view plane.

[0209] The image processing acquisition module 1312 is used to fuse the transformed image with the image corresponding to the object region to obtain the processed image corresponding to the target image.

[0210] In one embodiment, the filling image acquisition module 1308 is used to divide the object region into multiple division regions; for the current pixel in the blank image, determine the division region to which the current pixel belongs as a reference division region; acquire the background image block matched by the reference division region, acquire the pixel corresponding to the current pixel in the background image block as a reference background pixel; obtain the filling pixel value corresponding to the current pixel based on the pixel value corresponding to the reference background pixel, and fill the blank image in the intermediate image with background according to the filling pixel value.

[0211] In one embodiment, the filled image is obtained through multiple rounds of background filling. The filled image acquisition module 1308 is used to select one or more candidate background image blocks corresponding to the reference division region in the current background filling round based on the image block selection strategy; acquire the filled image block obtained after the previous round of background filling of the reference division region, and acquire the matching background image block corresponding to the reference division region in the previous round of background filling; calculate the similarity between the candidate background image block and the filled image block as a first similarity; calculate the similarity between the matching background image block and the filled image block as a second similarity; and based on the first similarity and the second similarity, acquire an image block similar to the filled image block from the matching background image block and the candidate background image block as the background image block matched by the reference division region in the current background filling round.

[0212] In one embodiment, the filling image acquisition module 1308 is used to, when there are multiple reference division regions corresponding to the current pixel, acquire the similarity between the background image block matched by the reference division region and the corresponding filling image block in the current background filling round; obtain the pixel weight corresponding to the reference background pixel in the current background filling round based on the similarity, wherein the pixel weight is positively correlated with the similarity; and perform a weighted calculation based on the pixel value corresponding to the reference background pixel and the pixel weight corresponding to the reference background pixel to obtain the filling pixel value corresponding to the current pixel.

[0213] In one embodiment, the filling image acquisition module 1308 is used to determine a reference division region corresponding to a reference division region among multiple division regions; acquire the position of the background image block matched by the reference division region in the previous round of background filling as a reference position; acquire the relative positional relationship between the reference division region and the reference division region; obtain the first image block selection position according to the reference position and the relative positional relationship; and select the background image block corresponding to the first image block selection position as a candidate background image block.

[0214] In one embodiment, the filling image acquisition module 1308 is used to acquire a randomly generated random offset distance; obtain the corresponding offset distance shrinkage value based on the image block sorting of candidate background image blocks, wherein the shrinkage degree corresponding to the offset value and the image block sorting is positively correlated; perform shrinkage processing on the random offset distance based on the offset distance shrinkage value to obtain the target offset distance; obtain the second image block selection position according to the position of the background image block matched in the previous round of background filling of the reference division region and the target offset distance; and use the background image block corresponding to the second image block selection position as a candidate background image block.

[0215] In one embodiment, the image transformation acquisition module 1310 is used to perform style transformation on the filled image to obtain an initial style image; acquire the background object image corresponding to the target background object in the initial style image; remove the background object image from the initial style image to obtain the image to be processed in the initial style image; perform color clustering on the image to be processed to obtain a color clustering image; fuse the color clustering image with the background object image, and obtain the target style image based on the fused image.

[0216] In one embodiment, the image transformation acquisition module 1310 is used to acquire the color parameter threshold range corresponding to the target background object, wherein the target background object is a background object whose color consistency is greater than the consistency threshold; to select pixels whose color parameter values ​​are within the color parameter threshold range from the initial style image as target pixels; and to use the continuous image blocks corresponding to the target pixels in the initial style image as the background object image corresponding to the target background object.

[0217] In one embodiment, the transform image acquisition module 1310 is used to fuse the color clustering image with the background object image to obtain a fused image; to extract edges from the initial style image to obtain the edge image corresponding to the initial style image; and to remove the edge image from the fused image to obtain the target style image.

[0218] In one embodiment, the transform image acquisition module 1310 is used to perform edge extraction on the initial style image using an edge extraction operator to obtain a first edge image corresponding to the initial style image; perform edge extraction on the initial style image using an edge extraction model to obtain a second edge image corresponding to the initial style image; obtain the image intersection between the first edge image and the second edge image, and use the image intersection as the edge image corresponding to the initial style image.

[0219] In one embodiment, the target image acquisition module 1302 is further configured to receive an image processing request, which is triggered by the terminal in response to the stereoscopic image generation operation; the transformation image acquisition module 1310 is configured to determine the stereoscopic effect of the image displayed by the stereoscopic image generation operation, acquire the view plane corresponding to the stereoscopic effect of the image as the target view plane; determine the plane transformation information corresponding to the target view plane, and perform plane transformation on the filling image according to the plane transformation information so that the transformed image is an image located on the target view plane.

[0220] For specific limitations regarding the image processing apparatus, please refer to the limitations on the image processing method above, which will not be repeated here. Each module in the aforementioned image processing apparatus can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0221] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 14 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad located on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0222] Those skilled in the art will understand that Figure 14The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0223] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0224] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0225] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0226] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0227] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0228] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An image processing method, characterized in that, The method includes: Obtain the target image to be processed; Perform foreground object recognition on the target image to obtain the object region corresponding to the foreground object in the target image; The image corresponding to the object region is removed from the target image to obtain an intermediate image, which includes a blank image corresponding to the object region and a background image corresponding to the target image; Obtain a background image block that matches the blank image from the background image, fill the blank image in the intermediate image with the background image block, and use the filled intermediate image as the filled image. When the target image is a street view image, a style transformation is performed on the filled image to obtain an initial style image; a color parameter threshold range corresponding to the sky region is obtained, wherein the color consistency of the sky region is greater than the consistency threshold; pixels with color parameter values ​​within the color parameter threshold range are selected from the initial style image as target pixels; continuous image blocks corresponding to the target pixels in the initial style image are used as background object images corresponding to the sky region, and the background object images are removed from the initial style image to obtain the image to be processed in the initial style image; color clustering is performed on the image to be processed to obtain a color clustering image; the color clustering image is fused with the background object image, and the target style image is obtained based on the fused image; Determine the planar transformation information corresponding to the target view plane, and perform a planar transformation on the target style image according to the planar transformation information so that the transformed image is an image located on the target view plane; The transformed image is fused with the image corresponding to the object region to obtain the processed image corresponding to the target image.

2. The method according to claim 1, characterized in that, The step of obtaining a background image patch from the background image that matches the blank image, and filling the blank image in the intermediate image with the background image patch, includes: The object region is divided into multiple regions; For the current pixel in the blank image, determine the segmented region to which the current pixel belongs, and use it as a reference segmented region; Obtain the background image block matched by the reference division region, and obtain the pixel point corresponding to the current pixel point in the background image block as the reference background pixel point; Based on the pixel value corresponding to the reference background pixel, the fill pixel value corresponding to the current pixel is obtained, and the blank image in the intermediate image is filled with background according to the fill pixel value.

3. The method according to claim 2, characterized in that, The filled image is obtained through multiple rounds of background filling, and obtaining the background image block matched by the reference segmentation region includes: Based on the image block selection strategy, one or more candidate background image blocks corresponding to the reference division region are selected in the current background filling round; Obtain the filled image block of the reference division region after the previous round of background filling, and obtain the matching background image block corresponding to the reference division region in the previous round of background filling; Calculate the similarity between the candidate background image patch and the filled image patch, and use it as the first similarity. Calculate the similarity between the matching background image patch and the filled image patch, and use it as a second similarity. Based on the first similarity and the second similarity, an image block similar to the filling image block is obtained from the matching background image block and the candidate background image block, and is used as the background image block matched by the reference division region in the current background filling round.

4. The method according to claim 3, characterized in that, The step of obtaining the fill pixel value corresponding to the current pixel based on the pixel value corresponding to the reference background pixel includes: When there are multiple reference division regions corresponding to the current pixel, obtain the similarity between the background image block matched by the reference division region and the corresponding filling image block in the current background filling round; Based on the similarity, the pixel weights corresponding to the reference background pixels in the current background filling round are obtained, and the pixel weights are positively correlated with the similarity. The filling pixel value corresponding to the current pixel is obtained by weighted calculation based on the pixel value corresponding to the reference background pixel and the pixel weight corresponding to the reference background pixel.

5. The method according to claim 3, characterized in that, The method of selecting one or more candidate background image patches corresponding to the reference division region in the current background filling round based on the image patch selection strategy includes: Determine the reference division region among the plurality of division regions; The position of the background image block matched by the baseline division region in the previous round of background filling is obtained as a reference position; Obtain the relative positional relationship between the baseline division region and the reference division region; The first image block selection position is obtained based on the reference position and the relative positional relationship; The background image block corresponding to the selected position of the first image block is used as the candidate background image block.

6. The method according to claim 3, characterized in that, The method of selecting one or more candidate background image patches corresponding to the reference division region in the current background filling round based on the image patch selection strategy includes: Get the randomly generated offset distance; The image patch sorting based on candidate background image patches yields the corresponding offset distance shrinkage value, and the shrinkage degree corresponding to the offset distance shrinkage value is positively correlated with the image patch sorting. Based on the offset distance shrinkage value, the random offset distance is shrunk to obtain the target offset distance; The second image block selection position is obtained based on the position of the background image block matched by the reference division region in the previous round of background filling and the target offset distance; The background image block corresponding to the selected position of the second image block is used as the candidate background image block.

7. The method according to claim 1, characterized in that, The step of fusing the color clustering image with the background object image, and obtaining a target style image based on the fused image, includes: The color clustering image is fused with the background object image to obtain a fused image; Edge extraction is performed on the initial style image to obtain the edge image corresponding to the initial style image; The edge image is removed from the fused image to obtain the target style image.

8. The method according to claim 7, characterized in that, The step of extracting edges from the initial style image to obtain the edge image corresponding to the initial style image includes: The initial style image is subjected to edge extraction using an edge extraction operator to obtain a first edge image corresponding to the initial style image; Edge extraction is performed on the initial style image using an edge extraction model to obtain a second edge image corresponding to the initial style image; Obtain the image intersection between the first edge image and the second edge image, and use the image intersection as the edge image corresponding to the initial style image.

9. The method according to claim 1, characterized in that, The process of acquiring the target image to be processed also includes: Receive an image processing request, which is triggered by the terminal in response to a stereoscopic image generation operation; The step of determining the planar transformation information corresponding to the target view plane, and performing a planar transformation on the target style image based on the planar transformation information so that the transformed image is an image located on the target view plane, includes: Determine the stereoscopic effect of the image displayed by the stereoscopic image generation operation, and obtain the view plane corresponding to the stereoscopic effect of the image as the target view plane; Determine the planar transformation information corresponding to the target view plane, and perform a planar transformation on the target style image according to the planar transformation information so that the transformed image is an image located on the target view plane.

10. An image processing apparatus, characterized in that, The device includes: The target image acquisition module is used to acquire the target image to be processed. A foreground object recognition module is used to perform foreground object recognition on the target image to obtain the object region corresponding to the foreground object in the target image; An intermediate image acquisition module is used to remove the image corresponding to the object region from the target image to obtain an intermediate image, wherein the intermediate image includes a blank image corresponding to the object region and a background image corresponding to the target image; A fill image acquisition module is used to acquire a background image block that matches the blank image from the background image, fill the blank image in the intermediate image with the background image block, and use the filled intermediate image as the fill image. The image transformation acquisition module is used to: when the target image is a street view image, perform style transformation on the filled image to obtain an initial style image; obtain a color parameter threshold range corresponding to the sky region, wherein the color consistency of the sky region is greater than the consistency threshold; filter out pixels with color parameter values ​​within the color parameter threshold range from the initial style image as target pixels; take the continuous image blocks corresponding to the target pixels in the initial style image as the background object image corresponding to the sky region, and remove the background object image from the initial style image to obtain the image to be processed in the initial style image; perform color clustering on the image to be processed to obtain a color clustering image; fuse the color clustering image with the background object image, and obtain the target style image based on the fused image; determine the plane transformation information corresponding to the target view plane, and perform plane transformation on the target style image according to the plane transformation information so that the transformed image is an image located on the target view plane; The image processing module is used to fuse the transformed image with the image corresponding to the object region to obtain the processed image corresponding to the target image.

11. The image processing apparatus according to claim 10, characterized in that, The filled image acquisition module is also used to divide the object region into multiple division regions; for the current pixel in the blank image, the division region to which the current pixel belongs is determined as a reference division region; Obtain the background image block matched by the reference division region, obtain the pixel point corresponding to the current pixel point in the background image block, and use it as the reference background pixel point; obtain the fill pixel value corresponding to the current pixel point based on the pixel value corresponding to the reference background pixel point, and fill the blank image in the intermediate image with the background according to the fill pixel value.

12. The image processing apparatus according to claim 11, characterized in that, The filled image is obtained through multiple rounds of background filling. The filled image acquisition module is also used to select one or more candidate background image blocks corresponding to the reference division region in the current background filling round based on the image block selection strategy. Obtain the filled image block obtained after the previous round of background filling from the reference segmentation region, and obtain the matching background image block corresponding to the reference segmentation region in the previous round of background filling; calculate the similarity between the candidate background image block and the filled image block as the first similarity; Calculate the similarity between the matching background image patch and the filled image patch, and use it as a second similarity. Based on the first similarity and the second similarity, an image block similar to the filling image block is obtained from the matching background image block and the candidate background image block, and is used as the background image block matched by the reference division region in the current background filling round.

13. The image processing apparatus according to claim 12, characterized in that, The filling image acquisition module is further configured to, when there are multiple reference division regions corresponding to the current pixel, acquire the similarity between the background image block matched by the reference division region and the corresponding filling image block in the current background filling round; The pixel weights corresponding to the reference background pixels in the current background filling round are obtained based on the similarity, and the pixel weights are positively correlated with the similarity. The filling pixel values ​​corresponding to the current pixel are obtained by weighted calculation based on the pixel values ​​corresponding to the reference background pixels and the pixel weights corresponding to the reference background pixels.

14. The image processing apparatus according to claim 12, characterized in that, The filled image acquisition module is further configured to determine, among the plurality of segmented regions, the reference segmented region corresponding to the reference segmented region; The position of the background image block matched by the baseline division region in the previous round of background filling is obtained as a reference position; Obtain the relative positional relationship between the baseline division region and the reference division region; obtain the first image block selection position based on the reference position and the relative positional relationship; and use the background image block corresponding to the first image block selection position as the candidate background image block.

15. The image processing apparatus according to claim 12, characterized in that, The filling image acquisition module is further configured to acquire a randomly generated random offset distance; obtain a corresponding offset distance shrinkage value based on the image block sorting of candidate background image blocks, wherein the shrinkage degree corresponding to the offset distance shrinkage value is positively correlated with the image block sorting; perform shrinkage processing on the random offset distance based on the offset distance shrinkage value to obtain a target offset distance; obtain a second image block selection position based on the position of the background image block matched by the reference division region in the previous round of background filling and the target offset distance; and use the background image block corresponding to the second image block selection position as the candidate background image block.

16. The image processing apparatus according to claim 10, characterized in that, The transformed image acquisition module is further configured to fuse the color clustering image with the background object image to obtain a fused image; perform edge extraction on the initial style image to obtain an edge image corresponding to the initial style image; and remove the edge image from the fused image to obtain a target style image.

17. The image processing apparatus according to claim 16, characterized in that, The transformed image acquisition module is further configured to perform edge extraction on the initial style image using an edge extraction operator to obtain a first edge image corresponding to the initial style image; perform edge extraction on the initial style image using an edge extraction model to obtain a second edge image corresponding to the initial style image; obtain the image intersection between the first edge image and the second edge image, and use the image intersection as the edge image corresponding to the initial style image.

18. The image processing apparatus according to claim 10, characterized in that, The target image acquisition module is further configured to receive an image processing request, which is triggered by the terminal in response to the stereoscopic image generation operation; the transformed image acquisition module is further configured to determine the stereoscopic effect of the image displayed by the stereoscopic image generation operation, acquire the view plane corresponding to the stereoscopic effect of the image as the target view plane; determine the planar transformation information corresponding to the target view plane, and perform a planar transformation on the target style image according to the planar transformation information so that the transformed image is an image located on the target view plane.

19. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 9.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 9.

21. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 9.

Images