Image processing method, training method of optical flow prediction model and related device

Abstract

The invention provides an image processing method, a training method of an optical flow prediction model and a related device, and relates to the field of artificial intelligence, in particular to the technical fields of image processing, computer vision, deep learning, data processing and the like. The image processing method comprises the following steps: acquiring a to-be-processed portrait image; performing resolution scaling processing on the to-be-processed portrait image to obtain a target resolution image; according to skeleton points and contour points of a target human body in the target resolution image, obtaining a PAFs image corresponding to the target resolution image; performing optical flow prediction on the target resolution image based on the PAFs image and a skeleton diagram, and obtaining a target optical flow diagram corresponding to the target resolution image, the skeleton diagram being obtained according to the skeleton points; and shaping the to-be-processed portrait image based on the target optical flow graph to obtain a shaped target image corresponding to the to-be-processed portrait image. According to the method and the device, the shaping treatment can be more accurately carried out, and the generalization is better.

Description

Technical Field

[0001] The present disclosure relates to technical fields such as image processing, computer vision, deep learning and data processing in the field of artificial intelligence, and in particular to an image processing method, a training method for an optical flow prediction model and related devices. Background Art

[0002] Artificial Intelligence (AI) adaptive shaping technology for portrait images is a technology that uses artificial intelligence algorithms to automatically adjust and optimize portrait images. It has very important applications in the fields of portrait photography, photo editing, and online social networking.

[0003] At present, a two-dimensional (2D) image deformation scheme based on key points is usually used to reshape portrait images. Specifically, for the portrait image to be processed, the contour points of the human body in the portrait image to be processed are first detected by a contour point detection model; the position coordinates of each contour point after shaping are obtained according to a preset shaping rule, and the transformation matrix is ​​obtained according to the position coordinates of each contour point before and after shaping; according to the transformation matrix, the human body in the portrait image to be processed is shaped to obtain a target image. However, the shaping process performed in the above manner has the problem of poor generalization. Summary of the invention

[0004] The present disclosure provides an image processing method for improving the generalization of shaping processing of portrait images, a training method for an optical flow prediction model, and related devices.

[0005] According to a first aspect of the present disclosure, there is provided an image processing method, comprising:

[0006] Obtain a portrait image to be processed;

[0007] Perform resolution scaling processing on the portrait image to be processed to obtain a target resolution image;

[0008] According to the skeleton points and contour points of the target human body in the target resolution image, the component affinity field PAFs image corresponding to the target resolution image is obtained;

[0009] Based on the PAFs image and the skeleton image, the optical flow of the target resolution image is predicted to obtain the target optical flow map corresponding to the target resolution image. The skeleton map is obtained based on the bone points.

[0010] The portrait image to be processed is reshaped based on the target optical flow map to obtain a shaped target image corresponding to the portrait image to be processed.

[0011] According to a second aspect of the present disclosure, a method for training an optical flow prediction model is provided, comprising:

[0012] Obtaining a training sample pair, the training sample pair comprising a portrait image sample and a label image corresponding to the portrait image sample, the label image being a reference image obtained after shaping the portrait image sample;

[0013] Perform resolution scaling processing on the portrait image sample and the label image respectively to obtain a target resolution image sample and a target resolution label image;

[0014] According to the skeleton points and contour points of the human body sample in the target resolution image sample, the component affinity field PAFs image sample corresponding to the target resolution image sample is obtained;

[0015] Input the target resolution image samples, PAFs image samples and skeleton image samples into the optical flow prediction model to perform optical flow prediction, and obtain the predicted optical flow map output by the optical flow prediction model. The skeleton image samples are obtained based on the skeleton points.

[0016] Apply the predicted optical flow map to perform shaping processing on the target resolution image sample to obtain the shaped image sample corresponding to the target resolution image sample;

[0017] Based on the reshaped image samples and the target resolution label image, adjust the parameters of the optical flow prediction model.

[0018] According to a third aspect of the present disclosure, there is provided an image processing apparatus, comprising:

[0019] A first acquisition unit, used to acquire a portrait image to be processed;

[0020] A resolution processing unit, used for performing resolution scaling processing on the portrait image to be processed to obtain a target resolution image;

[0021] The second acquisition unit is used to acquire a component affinity field PAFs image corresponding to the target resolution image according to the skeleton points and contour points of the target human body in the target resolution image;

[0022] An optical flow prediction unit is used to perform optical flow prediction on a target resolution image based on a PAFs image and a skeleton image, and obtain a target optical flow image corresponding to the target resolution image. The skeleton image is obtained based on bone points.

[0023] The shaping processing unit is used to perform shaping processing on the portrait image to be processed based on the target optical flow map to obtain a shaped target image corresponding to the portrait image to be processed.

[0024] According to a fourth aspect of the present disclosure, a training device for an optical flow prediction model is provided, comprising:

[0025] A first acquisition unit is used to acquire a training sample pair, the training sample pair comprising a portrait image sample and a label image corresponding to the portrait image sample, the label image being a reference image obtained after a shaping process is performed on the portrait image sample;

[0026] A resolution processing unit, used to perform resolution scaling processing on the portrait image sample and the label image respectively to obtain a target resolution image sample and a target resolution label image;

[0027] The second acquisition unit is used to acquire the component affinity field PAFs image sample corresponding to the target resolution image sample according to the skeleton points and contour points of the human body sample in the target resolution image sample;

[0028] An optical flow prediction unit is used to input the target resolution image samples, the PAFs image samples and the skeleton image samples into the optical flow prediction model for optical flow prediction, and obtain the predicted optical flow map output by the optical flow prediction model, where the skeleton image samples are obtained based on the skeleton points;

[0029] A shaping processing unit, used to apply the predicted optical flow map to perform shaping processing on the target resolution image sample to obtain a shaped image sample corresponding to the target resolution image sample;

[0030] The adjustment unit is used to adjust the parameters of the optical flow prediction model based on the reshaped image samples and the target resolution label image.

[0031] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image processing method described in the first aspect or execute the training method of the optical flow prediction model described in the second aspect.

[0032] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the image processing method described in the first aspect or the training method of the optical flow prediction model described in the second aspect.

[0033] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising: a computer program, the computer program being stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, the at least one processor executing the computer program so that the electronic device executes the image processing method described in the first aspect or executes the training method of the optical flow prediction model described in the second aspect.

[0034] The technology disclosed in the present invention solves the problem of poor generalization of the current method of shaping portrait images. The target resolution image is obtained by performing resolution scaling on the portrait image to be processed; the PAFs image corresponding to the target resolution image is obtained according to the skeleton points and contour points of the target human body in the target resolution image, and the information of the obtained PAFs image is richer and more accurate, and can accurately express the position, contour and fatness information of various parts of the target human body; thus, the target resolution image is predicted based on the PAFs image and the skeleton image, and the target optical flow map corresponding to the target resolution image can be obtained more accurately, and the portrait image to be processed is shaped based on the target optical flow map to obtain the shaped target image corresponding to the portrait image to be processed, so as to realize adaptive shaping to different degrees according to different postures and fatness of the human body, and achieve consistent effects with good generalization.

[0035] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0037] Figure 1 is a schematic diagram of an application scenario to which the present disclosure is applicable;

[0038] Figure 2 is a schematic diagram according to a first embodiment of the present disclosure;

[0039] Figure 3 is a schematic diagram according to a second embodiment of the present disclosure;

[0040] Figure 4 is a schematic diagram according to a third embodiment of the present disclosure;

[0041] Figure 5 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0042] Figure 6 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0043] Figure 7 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0044] Figure 8 is a schematic diagram according to a seventh embodiment of the present disclosure;

[0045] Fig. 9 is a schematic diagram according to an eighth embodiment of the present disclosure;

[0046] Fig.10 FIG. 1 is a schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0047] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0048] The present disclosure provides an image processing method, an optical flow prediction model training method and related devices, which are applied to technical fields such as image processing, computer vision, deep learning and data processing in the field of artificial intelligence, so as to achieve the purpose of improving the generalization of shaping processing of portrait images.

[0049] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0050] AI adaptive shaping technology for portrait images (such as portrait photos) has very important applications in the fields of portrait photography, photo editing, and online social networking.

[0051] At present, a two-dimensional (2D) image deformation scheme based on key points is usually used to reshape portrait images. Specifically, for the portrait image to be processed, the contour points of the human body in the portrait image to be processed are first detected by a contour point detection model to obtain, for example, 70 contour points for marking the entire contour of the human body, where, for example, contour point No. 1, for example, represents the contour point on the left side of the human neck, and contour point No. 2, for example, represents the contour point on the right side of the human neck. According to the preset shaping rules (such as the proportion of the left side of the human neck shrinking inward by 30%), the position coordinates of each contour point after shaping are obtained. According to the position coordinates of each contour point before and after shaping, the transformation matrix is ​​obtained, and then the human body in the portrait image to be processed can be shaped according to the transformation matrix to obtain the target image. However, the 2D image deformation scheme based on key points is only applicable to standard postures, so there is a problem of poor generalization.

[0052] In addition, in the related art, the portrait image can be shaped based on the scheme of optical flow. Optical flow (optical flow or optic flow) is a concept in the detection of object motion in the field of view, which is used to describe the movement of the observed target, surface or edge caused by the movement relative to the observer. In the field of computer vision, optical flow refers to the motion vector field of pixels in an image, which describes the displacement of each pixel between two frames of images. The scheme based on optical flow detects human skeleton points with the help of a human posture recognition model to obtain bone structure information, extracts deep features coupled with the bone structure of the input portrait image to be processed, and then predicts the optical flow deformation field and maps it back to the portrait image to be processed. However, the scheme based on optical flow has poor adaptability to different body shapes.

[0053] In summary, due to the complex structure and posture of the human body, as well as the diverse clothing and scenes of portraits, current image processing methods cannot achieve good shaping effects.

[0054] In order to solve the above problems, the present disclosure provides an image processing method, which performs resolution scaling processing on a portrait image to be processed to obtain a target resolution image; according to the skeleton points and contour points of a target human body in the target resolution image, a Part Affinity Fields (PAFs) image corresponding to the target resolution image is obtained, which can accurately express the position, contour and fatness information of each part of the target human body, so as to more accurately obtain a target optical flow map corresponding to the target resolution image; and then based on the target optical flow map, a shaping processing is performed on the portrait image to be processed to obtain a shaped target image corresponding to the portrait image to be processed, so as to realize adaptive shaping to different degrees according to different postures and fatness of the human body, and has good generalization.

[0055] Figure 1 1 is a schematic diagram of an application scenario applicable to the present disclosure. In the application scenario, a user uploads a portrait image to be processed to a server 102 through a client 101. After acquiring the portrait image to be processed, the server 102 performs a shaping process on the portrait image to be processed according to the image processing method provided by the present disclosure, and obtains a shaped target image corresponding to the portrait image to be processed.

[0056] It should be noted that Figure 1 This is only a schematic diagram of an application scenario provided by the embodiment of the present disclosure. Figure 1 does not limit the equipment included in Figure 1 The positional relationship between the devices is limited.

[0057] The following specific embodiments are used to describe in detail the technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0058] Figure 2 Schematic diagram of the first embodiment of the present disclosure. Figure 2 As shown, the image processing method provided in the first embodiment of the present disclosure can be applied to an electronic device, which can be a server or a server cluster, etc. The image processing method provided in the first embodiment of the present disclosure includes:

[0059] S201, obtaining a portrait image to be processed.

[0060] In the disclosed embodiment, the portrait image to be processed may be input by a user to the electronic device executing the method embodiment, or may be sent by other devices to the electronic device executing the method embodiment. The portrait image to be processed may be, for example, a 2D portrait photo containing the target human body, and the portrait image to be processed is a high-resolution image.

[0061] S202 , performing resolution scaling processing on the portrait image to be processed to obtain a target resolution image.

[0062] Exemplarily, the portrait image to be processed is a high-resolution image, and the resolution of the portrait image to be processed is reduced to obtain a target resolution image, the resolution of the target resolution image is lower than the resolution of the portrait image to be processed, and the resolution of the target resolution image is, for example, 384*384. It can be understood that this step is used to perform a unified resolution scaling process on the portrait image to be processed to obtain a target resolution image with the same resolution.

[0063] S203, acquiring a PAFs image corresponding to the target resolution image according to the skeleton points and contour points of the target human body in the target resolution image.

[0064] For example, the skeleton points and contour points of the target human body in the target resolution image are obtained by a human key point detection model, and the PAFs image corresponding to the target resolution image can be obtained by connecting adjacent skeleton points and contour points of the target human body.

[0065] It can be understood that the PAFs image is used to help identify the relationship between key points of the human body. Compared with the current related technologies, the PAFs image in the related technology is obtained by connecting the skeleton points and performing a certain scale of expansion operation on the skeleton point connection according to the size of the bounding box of all the skeleton points, aiming to express the position and fatness information of various parts of the human body (such as shoulders, torso, waist, arms and legs, etc.). However, the line width information after the expansion operation is not enough to accurately express the fatness information of various parts of the human body, which leads to the optical flow map obtained by the PAFs image for shaping processing is not accurate enough. The PAFs image of the embodiment of the present disclosure is obtained based on the skeleton points and contour points of the target human body in the target resolution image. The information of the obtained PAFs image is richer and more accurate, and can accurately express the position, contour and fatness information of various parts of the target human body, which helps to more accurately predict the optical flow.

[0066] S204, performing optical flow prediction on the target resolution image based on the PAFs image and the skeleton image to obtain a target optical flow map corresponding to the target resolution image, where the skeleton image is obtained based on the bone points.

[0067] Exemplarily, the skeleton graph is obtained by connecting the skeletal points of the target human body in the target resolution image. After acquiring the PAFs image, the optical flow prediction of the target resolution image can be performed based on the PAFs image and the skeleton graph to obtain a target optical flow graph corresponding to the target resolution image. For example, the target resolution image, the PAFs image and the skeleton graph are input into the optical flow prediction model to obtain a target optical flow graph output by the optical flow prediction model. Since the PAFs image has richer and more accurate information and can accurately express the position, contour and fatness information of various parts of the target human body, the target optical flow graph obtained by the embodiment of the present disclosure is more accurate.

[0068] S205 , performing shaping processing on the portrait image to be processed based on the target optical flow map to obtain a shaped target image corresponding to the portrait image to be processed.

[0069] Exemplarily, after obtaining the target optical flow map corresponding to the target resolution image, the resolution of the target optical flow map is the same as the resolution of the target resolution image, both of which are low-resolution images. Therefore, the target optical flow map can be resolution-scaled to obtain a resolution-scaled optical flow map, and the resolution of the resolution-scaled optical flow map is the same as the resolution of the portrait image to be processed. For example, the resolution-scaled optical flow map is applied to the portrait image to be processed through a warp function to achieve a shaping process of the portrait image to be processed, thereby obtaining a shaped target image corresponding to the portrait image to be processed. Among them, the warp function is a transformation function used to perform a geometric transformation on an image to change its shape or perspective.

[0070] After obtaining the shaped target image corresponding to the portrait image to be processed, the target image can be output, for example, the target image can be displayed to the user so that the user can view the target image, or the target image can be processed accordingly.

[0071] In the disclosed embodiment, resolution scaling processing is performed on the portrait image to be processed to obtain a target resolution image; based on the skeleton points and contour points of the target human body in the target resolution image, a PAFs image corresponding to the target resolution image is obtained, and the information of the obtained PAFs image is richer and more accurate, and can accurately express the position, contour and fatness information of various parts of the target human body; thereby, optical flow prediction is performed on the target resolution image based on the PAFs image and the skeleton image, and a target optical flow map corresponding to the target resolution image can be more accurately obtained, and a shaping processing is performed on the portrait image to be processed based on the target optical flow map to obtain a shaped target image corresponding to the portrait image to be processed, so as to achieve different degrees of shaping according to different postures and fatness of the human body, so as to achieve a consistent effect and have good generalization.

[0072] Figure 3 FIG. 1 is a schematic diagram of a second embodiment of the present disclosure. Based on the above embodiment, the present disclosure further describes the image processing method. Figure 3 As shown, the image processing method provided by the second embodiment of the present disclosure may include:

[0073] S301, obtaining a portrait image to be processed.

[0074] The implementation principle and technical effect of S301 can be referred to the above Figure 2 The embodiment of step S201 is not described in detail.

[0075] S302 , performing resolution scaling processing on the portrait image to be processed to obtain a target resolution image.

[0076] The implementation principle and technical effect of S302 can be referred to the above Figure 2 In the embodiment of step S202, in the embodiment of the present disclosure, the resolution of the target resolution image is the resolution of the input image corresponding to the optical flow prediction model. The resolution of the target resolution image is, for example, 384*384. By reducing the resolution, the computing power requirement of the optical flow prediction model can be reduced. While achieving the same effect, the performance in terms of memory usage and time consumption can be optimized.

[0077] S303: Input the target resolution image into a human body key point detection model to perform human body key point detection, and obtain the skeleton points and contour points of the target human body output by the human body key point detection model.

[0078] In this step, the human key point detection model can output not only human skeleton points but also human contour points. For details on how to obtain the human key point detection model, reference can be made to current related technologies. For example, the target resolution image is input into the human key point detection model for human key point detection, and multiple skeleton points and multiple contour points of the target human body in the target resolution image can be obtained.

[0079] In the disclosed embodiment, Figure 2 The step S203 may further include the following two steps S304 and S305:

[0080] S304: Connect adjacent skeleton points and contour points according to various preset parts of the target human body to obtain multiple contour lines.

[0081] For example, the preset parts of the target body may include the target body's shoulders, torso, waist, arms, legs, etc. For each part, adjacent skeleton points and contour points are connected to obtain contour lines corresponding to each part, thereby obtaining multiple contour lines. For example, Figure 4 is a schematic diagram according to the third embodiment of the present disclosure. Figure 4 As shown, for the same target human body, the current related technology uses the method of connecting bone points to obtain a bone point connection diagram 401, which mainly includes the shoulders, torso, waist, arms and legs of the target human body; while the embodiment of the present disclosure uses the method of connecting bone points and contour points together to obtain the contour lines corresponding to each part of the target human body, that is, to obtain a contour line diagram 402.

[0082] Optionally, each contour line has a different color, and contour lines of different colors are used to represent contours of different parts. It can be understood that each contour line has a different color, which facilitates the optical flow prediction model to determine the part corresponding to each contour line according to the color.

[0083] S305 . Obtain a PAFs image corresponding to the target resolution image according to the multiple contour lines.

[0084] In this step, after obtaining multiple contour lines corresponding to various preset parts of the target human body, the PAFs image corresponding to the target resolution image can be obtained according to the multiple contour lines. Compared with the current related art, the PAFs image in the related art is obtained by connecting the skeleton points, and the skeleton points are expanded to a certain scale according to the size of the bounding box of all the skeleton points, aiming to express the position and fatness information of various parts of the human body (such as shoulders, torso, waist, arms and legs, etc.). However, the line width information after the expansion operation is not enough to accurately express the fatness information of various parts of the human body, which leads to the optical flow map obtained by the PAFs image for shaping processing is not accurate enough. The PAFs image of the embodiment of the present disclosure is obtained by connecting the skeleton points and contour points of the human body, which can output a PAFs image with richer and more accurate information, so that the position, contour and fatness information of each part can be accurately expressed, and the optical flow map output by the optical flow prediction model can be made more accurate.

[0085] In the present disclosure, Figure 2 The step S204 may further include the following step S306:

[0086] S306, input the target resolution image, the PAFs image and the skeleton image into the optical flow prediction model. The optical flow prediction model performs optical flow prediction on the target resolution image based on the PAFs image and the skeleton image to obtain a target optical flow map. The skeleton image is obtained based on the bone points.

[0087] Among them, the optical flow prediction model is obtained by training based on portrait image samples, predicted optical flow maps corresponding to the portrait image samples and label images. The predicted optical flow maps are obtained based on PAFs image samples, and the PAFs image samples are obtained based on the skeleton points and contour points of the human body samples in the portrait image samples.

[0088] Exemplarily, the optical flow prediction model is obtained by training based on portrait image samples, predicted optical flow maps corresponding to the portrait image samples, and label images. For specific information on how to train and obtain the optical flow prediction model, please refer to the subsequent embodiments. The target resolution image, PAFs image, and skeleton image are input into the optical flow prediction model to obtain the target optical flow map output by the optical flow prediction model. The resolution of the target optical flow map is the same as that of the target resolution image, and both are low-resolution images.

[0089] In the present disclosure, Figure 2 The step S205 may further include the following three steps S307 to S309:

[0090] S307: Smoothing the target optical flow map to obtain a smoothed optical flow map.

[0091] Exemplarily, after obtaining the target optical flow map, Gaussian smoothing can be performed on the target optical flow map, so that the shaping effect map after the optical flow is more smooth and symmetrical, and a smoothed optical flow map is obtained.

[0092] S308 , performing resolution scaling processing on the smoothed optical flow map to obtain a resolution scaled optical flow map, wherein the resolution of the resolution scaled optical flow map is the same as the resolution of the portrait image to be processed.

[0093] In this step, after obtaining the smoothed optical flow map, the smoothed optical flow map can be subjected to resolution scaling to obtain a resolution-scaled optical flow map, and the resolution of the resolution-scaled optical flow map is the same as the resolution of the portrait image to be processed. For example, if the smoothed optical flow map is a low-resolution image, the smoothed optical flow map can be subjected to resolution scaling to obtain a high-resolution resolution-scaled optical flow map having the same resolution as the portrait image to be processed.

[0094] S309 , applying the resolution-scaling optical flow map to perform shaping processing on the portrait image to be processed, so as to obtain a shaped target image corresponding to the portrait image to be processed.

[0095] For example, the resolution-scaled optical flow map can be applied to the portrait image to be processed through a warp function to achieve shaping of the portrait image to be processed, thereby obtaining a shaped target image corresponding to the portrait image to be processed. After obtaining the shaped target image corresponding to the portrait image to be processed, the target image can be output.

[0096] In the disclosed embodiment, the portrait image to be processed is subjected to resolution scaling processing to obtain a target resolution image, which can meet the low resolution requirement of the optical flow prediction model; the target resolution image is input into the human key point detection model to perform human key point detection, and the skeleton points and contour points of the target human body in the target resolution image are obtained; according to each preset part of the target human body, adjacent skeleton points and contour points are connected to obtain a plurality of contour lines; according to the plurality of contour lines, a PAFs image corresponding to the target resolution image is obtained, and the information of the obtained PAFs image is richer and more accurate, and can accurately express the position, contour and fatness information of each preset part of the target human body; thus, the target resolution image, the PAFs image and the skeleton image are input into the optical flow prediction model, so that the optical flow prediction model outputs the target optical flow map more accurately; the target optical flow map is smoothed to obtain a smoother and more symmetrical optical flow map after smoothing processing; the smoothed optical flow map is subjected to resolution scaling processing to obtain a resolution scaled optical flow map with the same resolution as the portrait image to be processed, and then the resolution scaled optical flow map is applied to shape the portrait image to be processed to obtain a shaped target image corresponding to the portrait image to be processed. The disclosed embodiment can adaptively perform different degrees of body shaping according to different postures and fatness of the human body to achieve consistent effects and has good generalization.

[0097] On the basis of the above embodiments, taking into account the situation that the portrait image to be processed contains multiple target human bodies, the image processing method provided by the embodiment of the present disclosure may also include: for each target human body among the multiple target human bodies, obtaining a target optical flow map corresponding to the target human body; performing weighted summation processing on the multiple target optical flow maps to obtain a weighted optical flow map; and performing shaping processing on the portrait image to be processed based on the weighted optical flow map to obtain a shaped target image corresponding to the portrait image to be processed.

[0098] Exemplarily, in the case where a portrait image to be processed contains multiple target human bodies, the transformation degree weight corresponding to each target human body can be pre-set, and the transformation degree weight is, for example, configured by the user. For example, the portrait image to be processed contains two target human bodies, namely target human body A and target human body B, then the user configures the transformation degree weight corresponding to target human body A to be, for example, 50%, and the transformation degree weight corresponding to target human body B to be, for example, 80%. Using the method of the above embodiment, the target optical flow map A corresponding to target human body A is obtained, and the target optical flow map B corresponding to target human body B is obtained. The two target optical flow maps are weighted and summed, that is, the target optical flow map A and the target optical flow map B are multiplied by their respective corresponding transformation degree weights, and the sum is performed to obtain the weighted optical flow map corresponding to the entire image. The portrait image to be processed is shaped based on the weighted optical flow map to obtain the shaped target image corresponding to the portrait image to be processed.

[0099] Based on the above embodiments, Figure 5 is a schematic diagram according to the fourth embodiment of the present disclosure. Figure 5 As shown, the optical flow prediction model training method provided in the fourth embodiment of the present disclosure can be applied to an electronic device, which may be a server or a server cluster. The optical flow prediction model training method provided in the fourth embodiment of the present disclosure includes:

[0100] S501 , obtaining a training sample pair, wherein the training sample pair includes a portrait image sample and a label image corresponding to the portrait image sample, and the label image is a reference image obtained after a shaping process is performed on the portrait image sample.

[0101] In the embodiment of the present disclosure, the portrait image sample is a high-resolution portrait image to be reshaped; the label image is a reference image obtained after the portrait image sample is reshaped. According to the portrait image sample and the label image corresponding to the portrait image sample, multiple training sample pairs can be obtained.

[0102] S502 , performing resolution scaling processing on the portrait image sample and the label image respectively to obtain a target resolution image sample and a target resolution label image.

[0103] For example, the portrait image sample and the label image are respectively reduced in resolution to obtain a target resolution image sample and a target resolution label image. The resolutions of the target resolution image sample and the target resolution label image are both 384*384, which can meet the requirements of the optical flow prediction model for the resolution of the input image. It can be understood that by reducing the resolution, the computing power requirements of the optical flow prediction model can be reduced, and while achieving the same effect, the performance in terms of memory usage and time consumption can be optimized.

[0104] S503: Obtain a PAFs image sample corresponding to the target resolution image sample according to the skeleton points and contour points of the human body sample in the target resolution image sample.

[0105] In this step, the skeleton points and contour points of the human body sample in the target resolution image sample can be combined to obtain the PAFs image sample corresponding to the target resolution image sample.

[0106] Optionally, before obtaining the PAFs image sample corresponding to the target resolution image sample based on the skeleton points and contour points of the human body sample in the target resolution image sample, the training method of the optical flow prediction model provided by the embodiment of the present disclosure may also include: inputting the target resolution image sample into the human body key point detection model to perform human body key point detection, and obtaining the skeleton points and contour points of the human body sample output by the human body key point detection model.

[0107] Exemplarily, the human key point detection model can output not only human skeleton points, but also human contour points. For details on how to obtain the human key point detection model, reference can be made to current related technologies. Exemplarily, the target resolution image sample is input into the human key point detection model for human key point detection, and the skeleton points and contour points of the human sample output by the human key point detection model can be obtained.

[0108] Further, optionally, obtaining a PAFs image sample corresponding to the target resolution image sample based on the bone points and contour points of the human body sample in the target resolution image sample may include: according to various preset parts of the human body sample, connecting adjacent bone points and contour points to obtain multiple contour lines, and obtaining the PAFs image sample based on the multiple contour lines.

[0109] Exemplarily, the preset parts of the human body sample may include, for example, the target human body's shoulders, torso, waist, arms and legs. For each part, adjacent bone points and contour points may be connected to obtain multiple contour lines, thereby obtaining a PAFs image sample corresponding to the target resolution image sample based on the multiple contour lines. Optionally, each contour line has a different color, and contour lines of different colors are used to represent the contours of different parts, so that the optical flow prediction model can determine the part corresponding to each contour line based on the color.

[0110] S504, inputting the target resolution image samples, the PAFs image samples and the skeleton graph samples into the optical flow prediction model for optical flow prediction, and obtaining the predicted optical flow graph output by the optical flow prediction model, wherein the skeleton graph samples are obtained based on the bone points.

[0111] In this step, the skeleton graph samples are obtained by connecting the skeleton points of the human body samples in the target resolution image samples. After obtaining the PAFs image samples, the target resolution image samples, PAFs image samples and skeleton graph samples can be input into the optical flow prediction model for optical flow prediction to obtain the predicted optical flow graph output by the optical flow prediction model. The resolution of the predicted optical flow graph is the same as that of the target resolution image samples, both of which are low-resolution images.

[0112] S505 , applying the predicted optical flow map to perform shaping processing on the target resolution image samples, to obtain shaped image samples corresponding to the target resolution image samples.

[0113] Exemplarily, the predicted optical flow map may be applied to the target resolution image samples through a warp function to implement shaping processing on the target resolution image samples, thereby obtaining shaped image samples corresponding to the target resolution image samples.

[0114] S506: Adjust the parameters of the optical flow prediction model based on the reshaped image sample and the target resolution label image.

[0115] In this step, after obtaining the reshaped image sample corresponding to the target resolution image sample, the reshaped image sample and the target resolution label image can be compared to obtain the loss value, and then the parameters of the optical flow prediction model can be adjusted according to the loss value to obtain the trained optical flow prediction model.

[0116] In the disclosed embodiment, the portrait image sample and the label image are respectively subjected to resolution scaling processing to obtain the target resolution image sample and the target resolution label image, which can reduce the computing power requirement of the optical flow prediction model, and optimize the performance in terms of memory usage and time consumption while achieving the same effect; the PAFs image sample corresponding to the target resolution image sample is obtained according to the skeleton points and contour points of the human body sample in the target resolution image sample, which can accurately express the position, contour and fatness information of each part of the target human body; thus, the target resolution image sample, the PAFs image sample and the skeleton image sample are input into the optical flow prediction model for optical flow prediction, and the predicted optical flow map can be obtained more accurately; the predicted optical flow map is applied to shape the target resolution image sample to obtain the shaped image sample corresponding to the target resolution image sample, and then the parameters of the optical flow prediction model are adjusted based on the shaped image sample and the target resolution label image. The optical flow prediction model obtained by training can output the predicted optical flow map more accurately according to different postures and fatness of the human body, and has good generalization.

[0117] based on Figure 5 The embodiment shown, Figure 6 is a schematic diagram according to the fifth embodiment of the present disclosure. Figure 6 As shown, the optical flow prediction model provided by the fifth embodiment of the present disclosure includes an encoder, an attention module and a decoder. The number of modules included in the encoder is the first number of modules and the number of network layers is the first number of network layers, and the number of modules included in the decoder is the second number of modules and the number of network layers is the second number of network layers. Among them, the encoder is used to encode the image obtained by channel combination of the target resolution image sample and the skeleton image sample to obtain the first implicit feature map; the attention module is used to extract attention information based on the first implicit feature map and the PAFs image sample to obtain the second implicit feature map; the decoder is used to decode the second implicit feature map to obtain a predicted optical flow map.

[0118] Exemplarily, the network structure of the optical flow prediction model of the embodiment of the present disclosure is simpler. For example, the number of modules of the encoder is 3 (i.e., the first number of modules), and the number of network layers is 6 (i.e., the first number of network layers); the number of modules of the decoder is 3 (i.e., the second number of modules), and the number of network layers is 6 (i.e., the second number of network layers). By streamlining the network structure of the optical flow prediction model, the performance of the optical flow prediction model can be further optimized while improving the adaptability and robustness of the effect. For the target resolution image samples, PAFs image samples, and skeleton graph samples input to the optical flow prediction model, the encoder can encode the image obtained by channel combining the target resolution image samples and the skeleton graph samples to obtain a first implicit feature map. The attention module extracts attention information based on the first implicit feature map and the PAFs image samples to obtain a second implicit feature map, and then the decoder decodes the second implicit feature map to obtain a predicted optical flow map.

[0119] Based on the above embodiments, Figure 7 is a schematic diagram according to the sixth embodiment of the present disclosure. Figure 7 As shown, the high-resolution portrait image to be processed is subjected to resolution scaling processing (downsampling processing) to obtain a target resolution image, which is a low-resolution (for example, 384*384) image. The target resolution image is input into the human key point detection model for human key point detection to obtain the skeleton points and contour points of the target human body in the target resolution image. A skeleton image is obtained according to the skeleton points; and a PAFs image corresponding to the target resolution image is obtained according to the skeleton points and contour points. The target resolution image and the skeleton image are channel-combined (concat) to obtain a combined image. The encoder of the optical flow prediction model is used to encode the image to obtain a first implicit feature map. The attention module of the optical flow prediction model is used to extract attention information based on the first implicit feature map and the PAFs image to obtain a second implicit feature map. The decoder of the optical flow prediction model is used to decode the second implicit feature map to obtain a predicted optical flow map, which is a low-resolution optical flow map. The predicted optical flow map is subjected to resolution scaling processing (upsampling processing) to obtain a high-resolution optical flow map, and the resolution of the high-resolution optical flow map is the same as the resolution of the portrait image to be processed. The portrait image to be processed and the high-resolution optical flow map are input into the warp function for shaping processing to obtain a shaped target image corresponding to the portrait image to be processed.

[0120] The disclosed embodiments can achieve better portrait image shaping effects, especially for portrait images of people with different fat and thin body shapes, and can adaptively perform different degrees of shaping to achieve a consistent effect; at the same time, it can optimize the performance in terms of memory usage and time consumption, such as saving the computer's central processing unit (CPU) / graphics processing unit (GPU) calculation and memory overhead, and realize local deployment on the client.

[0121] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0122] Figure 8 is a schematic diagram according to the seventh embodiment of the present disclosure. Figure 8 As shown, the image processing device 800 provided in the seventh embodiment of the present disclosure includes: a first acquisition unit 801, a resolution processing unit 802, a second acquisition unit 803, an optical flow prediction unit 804 and a shaping processing unit 805. Among them:

[0123] The first acquisition unit 801 is used to acquire a portrait image to be processed.

[0124] The resolution processing unit 802 is used to perform resolution scaling processing on the portrait image to be processed to obtain a target resolution image.

[0125] The second acquisition unit 803 is used to acquire a PAFs image corresponding to the target resolution image according to the skeleton points and contour points of the target human body in the target resolution image.

[0126] The optical flow prediction unit 804 is used to perform optical flow prediction on the target resolution image based on the PAFs image and the skeleton image to obtain a target optical flow map corresponding to the target resolution image, where the skeleton image is obtained based on the bone points.

[0127] The shaping processing unit 805 is used to perform shaping processing on the portrait image to be processed based on the target optical flow map to obtain a shaped target image corresponding to the portrait image to be processed.

[0128] In some embodiments, the second acquisition unit 803 includes: a line processing module (not shown in the figure), which is used to connect adjacent bone points and contour points according to various preset parts of the target human body to obtain multiple contour lines; an image acquisition module (not shown in the figure), which is used to obtain a PAFs image based on the multiple contour lines.

[0129] In some embodiments, the optical flow prediction unit 804 includes: an optical flow prediction module (not shown in the figure), which is used to input the target resolution image, the PAFs image and the skeleton map into the optical flow prediction model, and the optical flow prediction model performs optical flow prediction on the target resolution image based on the PAFs image and the skeleton map to obtain a target optical flow map; the optical flow prediction model is trained based on portrait image samples, predicted optical flow maps corresponding to the portrait image samples and label images, and the predicted optical flow map is obtained based on the PAFs image samples, and the PAFs image samples are obtained based on the bone points and contour points of the human body samples in the portrait image samples.

[0130] In some embodiments, the shaping processing unit 805 includes: a smoothing processing module (not shown in the figure), which is used to smooth the target optical flow map to obtain a smoothed optical flow map; a resolution processing module (not shown in the figure), which is used to perform resolution scaling processing on the smoothed optical flow map to obtain a resolution-scaled optical flow map, and the resolution of the resolution-scaled optical flow map is the same as the resolution of the portrait image to be processed; a first shaping processing module (not shown in the figure), which is used to apply the resolution-scaled optical flow map to perform shaping processing on the portrait image to be processed to obtain a shaped target image corresponding to the portrait image to be processed.

[0131] In some embodiments, the portrait image to be processed contains multiple target human bodies, and the shaping processing unit 805 also includes: an optical flow map acquisition module (not shown in the figure), which is used to obtain a target optical flow map corresponding to each target human body in the multiple target human bodies; a weighted processing module (not shown in the figure), which is used to perform weighted summation processing on the multiple target optical flow maps to obtain a weighted optical flow map; a second shaping processing module (not shown in the figure), which is used to perform shaping processing on the portrait image to be processed based on the weighted optical flow map to obtain a shaped target image corresponding to the portrait image to be processed.

[0132] In some embodiments, the second acquisition unit 803 also includes: a key point detection module (not shown in the figure), which is used to input the target resolution image into a human key point detection model to perform human key point detection based on the skeleton points and contour points of the target human body in the target resolution image before acquiring the PAFs image corresponding to the target resolution image, so as to obtain the skeleton points and contour points of the target human body output by the human key point detection model.

[0133] Figure 8 The provided image processing device can execute the steps in the method embodiment corresponding to the above-mentioned image processing method. Its implementation principle and technical effects are similar and will not be repeated here.

[0134] Fig. 9 is a schematic diagram according to the eighth embodiment of the present disclosure. Fig. 9As shown, the optical flow prediction model training device 900 provided in the eighth embodiment of the present disclosure includes: a first acquisition unit 901, a resolution processing unit 902, a second acquisition unit 903, an optical flow prediction unit 904, a shaping processing unit 905 and an adjustment unit 906. Among them:

[0135] The first acquisition unit 901 is used to acquire a training sample pair, where the training sample pair includes a portrait image sample and a label image corresponding to the portrait image sample, where the label image is a reference image obtained after a shaping process is performed on the portrait image sample.

[0136] The resolution processing unit 902 is used to perform resolution scaling processing on the portrait image sample and the label image respectively to obtain a target resolution image sample and a target resolution label image.

[0137] The second acquisition unit 903 is used to acquire the PAFs image samples corresponding to the target resolution image samples according to the skeleton points and contour points of the human body samples in the target resolution image samples.

[0138] The optical flow prediction unit 904 is used to input the target resolution image samples, PAFs image samples and skeleton image samples into the optical flow prediction model for optical flow prediction to obtain a predicted optical flow map output by the optical flow prediction model. The skeleton image samples are obtained based on the bone points.

[0139] The shaping processing unit 905 is used to apply the predicted optical flow map to perform shaping processing on the target resolution image samples to obtain shaped image samples corresponding to the target resolution image samples.

[0140] The adjustment unit 906 is used to adjust the parameters of the optical flow prediction model based on the reshaped image sample and the target resolution label image.

[0141] In some embodiments, the second acquisition unit 903 includes: a connection processing module (not shown in the figure), which is used to connect adjacent bone points and contour points according to various preset parts of the human body sample to obtain multiple contour lines; an image sample acquisition module (not shown in the figure), which is used to obtain PAFs image samples based on multiple contour lines.

[0142] In some embodiments, the second acquisition unit 903 also includes: a key point detection module (not shown in the figure), which is used to input the target resolution image sample into a human key point detection model to perform human key point detection based on the skeleton points and contour points of the human sample in the target resolution image sample before acquiring the PAFs image sample corresponding to the target resolution image sample, so as to obtain the skeleton points and contour points of the human sample output by the human key point detection model.

[0143] In some embodiments, the optical flow prediction model includes an encoder, an attention module and a decoder, the number of modules included in the encoder is the first number of modules and the number of network layers is the first number of network layers, the number of modules included in the decoder is the second number of modules and the number of network layers is the second number of network layers; wherein the encoder is used to encode the image obtained by channel combination of the target resolution image samples and the skeleton map samples to obtain a first implicit feature map; the attention module is used to extract attention information based on the first implicit feature map and the PAFs image samples to obtain a second implicit feature map; the decoder is used to decode the second implicit feature map to obtain a predicted optical flow map.

[0144] Fig. 9 The provided optical flow prediction model training device can execute the steps in the method embodiment corresponding to the above-mentioned optical flow prediction model training method. Its implementation principle and technical effects are similar and will not be repeated here.

[0145] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the solution provided by any of the above embodiments.

[0146] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute a solution provided by any of the above embodiments.

[0147] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0148] Fig.10 1 is a schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0149] like Fig.10 As shown, the electronic device 1000 includes a computing unit 1001, which can store data stored in a read-only memory (ROM) ( Fig.10 A computer program in ROM 1002 is loaded from storage unit 1008 to random access memory (RAM) ( Fig.10 The computer programs in the RAM 1003 are used to perform various appropriate actions and processes. The RAM 1003 may also store various programs and data required for the operation of the electronic device 1000. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. The input / output (I / O) interface ( Fig.10 The I / O interface 1005 is also connected to the bus 1004 .

[0150] Multiple components in the electronic device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0151] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit, a graphics processing unit, various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as image processing methods and training methods for optical flow prediction models. For example, in some embodiments, the image processing methods and training methods for optical flow prediction models may be implemented as computer software programs, which are tangibly included in machine-readable media, such as storage units 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the image processing method and the training method for the optical flow prediction model described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the image processing method and the training method of the optical flow prediction model in any other appropriate manner (for example, by means of firmware).

[0152] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0153] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0154] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0155] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0156] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0157] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0158] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0159] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, fusions, sub-fusions and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. An image processing method, comprising: Obtain a portrait image to be processed; Performing resolution scaling processing on the portrait image to be processed to obtain a target resolution image; According to the skeleton points and contour points of the target human body in the target resolution image, a component affinity field PAFs image corresponding to the target resolution image is obtained; Performing optical flow prediction on the target resolution image based on the PAFs image and the skeleton graph to obtain a target optical flow graph corresponding to the target resolution image, wherein the skeleton graph is obtained based on the skeleton points; The portrait image to be processed is shaped based on the target optical flow map to obtain a shaped target image corresponding to the portrait image to be processed.

2. The image processing method according to claim 1, wherein: The step of obtaining a component affinity field PAFs image corresponding to the target resolution image according to the skeleton points and contour points of the target human body in the target resolution image comprises: According to each preset part of the target human body, adjacent skeleton points and contour points are connected to obtain multiple contour lines; The PAFs image is obtained according to the multiple contour lines.

3. The image processing method according to claim 1, wherein: The performing optical flow prediction on the target resolution image based on the PAFs image and the skeleton image to obtain a target optical flow map corresponding to the target resolution image includes: The target resolution image, the PAFs image and the skeleton map are input into an optical flow prediction model, and the optical flow prediction model performs optical flow prediction on the target resolution image based on the PAFs image and the skeleton map to obtain the target optical flow map; the optical flow prediction model is obtained by training based on portrait image samples, predicted optical flow maps corresponding to the portrait image samples and label images, and the predicted optical flow map is obtained based on PAFs image samples, and the PAFs image samples are obtained based on the bone points and contour points of the human body samples in the portrait image samples.

4. The image processing method according to claim 1, wherein: The step of performing shaping processing on the to-be-processed portrait image based on the target optical flow map to obtain a shaped target image corresponding to the to-be-processed portrait image includes: Smoothing the target optical flow map to obtain a smoothed optical flow map; Performing resolution scaling processing on the smoothed optical flow map to obtain a resolution scaled optical flow map, wherein the resolution of the resolution scaled optical flow map is the same as the resolution of the portrait image to be processed; The resolution-scaling optical flow map is applied to perform shaping processing on the portrait image to be processed to obtain a shaped target image corresponding to the portrait image to be processed.

5. The image processing method according to any one of claims 1 to 4, wherein: The portrait image to be processed contains a plurality of target human bodies, and the method further comprises: For each target human body among the multiple target human bodies, obtaining a target optical flow map corresponding to the target human body; Performing weighted summation processing on the plurality of target optical flow maps to obtain a weighted optical flow map; The portrait image to be processed is shaped based on the weighted optical flow map to obtain a shaped target image corresponding to the portrait image to be processed.

6. The image processing method according to any one of claims 1 to 4, wherein: Before acquiring the component affinity field PAFs image corresponding to the target resolution image according to the skeleton points and contour points of the target human body in the target resolution image, the method further includes: The target resolution image is input into a human body key point detection model to perform human body key point detection, and the skeleton points and contour points of the target human body output by the human body key point detection model are obtained.

7. A method for training an optical flow prediction model, comprising: Acquire a training sample pair, wherein the training sample pair includes a portrait image sample and a label image corresponding to the portrait image sample, wherein the label image is a reference image obtained after shaping the portrait image sample; Performing resolution scaling processing on the portrait image sample and the label image respectively to obtain a target resolution image sample and a target resolution label image; According to the skeleton points and contour points of the human body sample in the target resolution image sample, obtaining the component affinity field PAFs image sample corresponding to the target resolution image sample; Inputting the target resolution image sample, the PAFs image sample and the skeleton graph sample into an optical flow prediction model for optical flow prediction, and obtaining a predicted optical flow graph output by the optical flow prediction model, wherein the skeleton graph sample is obtained based on the skeleton points; Applying the predicted optical flow map to perform shaping processing on the target resolution image sample to obtain a shaped image sample corresponding to the target resolution image sample; Based on the reshaped image samples and the target resolution label image, parameters of the optical flow prediction model are adjusted.

8. The training method according to claim 7, wherein: The step of obtaining the component affinity field PAFs image sample corresponding to the target resolution image sample according to the skeleton points and contour points of the human body sample in the target resolution image sample comprises: According to each preset part of the human body sample, adjacent bone points and contour points are connected to obtain multiple contour lines; The PAFs image samples are obtained according to the multiple contour lines.

9. The training method according to claim 7 or 8, wherein: Before obtaining the component affinity field PAFs image sample corresponding to the target resolution image sample according to the skeleton points and contour points of the human body sample in the target resolution image sample, the method further includes: The target resolution image sample is input into a human key point detection model to perform human key point detection, and the skeleton points and contour points of the human sample output by the human key point detection model are obtained.

10. The training method according to claim 7 or 8, wherein: The optical flow prediction model includes an encoder, an attention module and a decoder, the encoder includes a first number of modules and a first number of network layers, the decoder includes a second number of modules and a second number of network layers; wherein the encoder is used to encode the image obtained by channel combination of the target resolution image samples and the skeleton image samples to obtain a first implicit feature map; the attention module is used to extract attention information based on the first implicit feature map and the PAFs image samples to obtain a second implicit feature map; the decoder is used to decode the second implicit feature map to obtain the predicted optical flow map.

11. An image processing device, comprising: A first acquisition unit, used to acquire a portrait image to be processed; A resolution processing unit, used for performing resolution scaling processing on the portrait image to be processed to obtain a target resolution image; A second acquisition unit is used to acquire a component affinity field PAFs image corresponding to the target resolution image according to the skeleton points and contour points of the target human body in the target resolution image; an optical flow prediction unit, configured to perform optical flow prediction on the target resolution image based on the PAFs image and the skeleton graph to obtain a target optical flow graph corresponding to the target resolution image, wherein the skeleton graph is obtained based on the skeleton points; A shaping processing unit is used to perform shaping processing on the portrait image to be processed based on the target optical flow map to obtain a shaped target image corresponding to the portrait image to be processed.

12. A training device for an optical flow prediction model, comprising: A first acquisition unit is used to acquire a training sample pair, wherein the training sample pair includes a portrait image sample and a label image corresponding to the portrait image sample, wherein the label image is a reference image obtained after the portrait image sample is shaped; A resolution processing unit, used to perform resolution scaling processing on the portrait image sample and the label image respectively to obtain a target resolution image sample and a target resolution label image; A second acquisition unit is used to acquire a component affinity field PAFs image sample corresponding to the target resolution image sample according to the skeleton points and contour points of the human body sample in the target resolution image sample; An optical flow prediction unit, used for inputting the target resolution image samples, the PAFs image samples and the skeleton graph samples into an optical flow prediction model for optical flow prediction, and obtaining a predicted optical flow graph output by the optical flow prediction model, wherein the skeleton graph samples are obtained according to the skeleton points; A shaping processing unit, used to apply the predicted optical flow map to perform shaping processing on the target resolution image sample to obtain a shaped image sample corresponding to the target resolution image sample; An adjustment unit is used to adjust the parameters of the optical flow prediction model based on the reshaped image sample and the target resolution label image.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image processing method according to any one of claims 1 to 6 or the training method of the optical flow prediction model according to any one of claims 7 to 10.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the image processing method according to any one of claims 1 to 6 or the training method of the optical flow prediction model according to any one of claims 7 to 10.

15. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the image processing method according to any one of claims 1 to 6 or the steps of the optical flow prediction model training method according to any one of claims 7 to 10.

Images