Digital human image processing method, electronic device and storage medium

Abstract

The present application provides a digital human image processing method, electronic device and storage medium, belonging to the field of image processing technology, by obtaining at least one abnormal digital human image and a reference image associated with the abnormal digital human image in a digital human video stream; obtaining the abnormal feature boundary line of each abnormal digital human image in at least one abnormal digital human image, segmenting each abnormal digital human image based on the abnormal feature boundary line, obtaining an upper image containing a face image and a lower image containing a body image; determining the transformation parameters of the upper image based on the reference coordinate system of the reference image based on the upper image and the reference image; controlling the lower image to transform according to the transformation parameters based on the reference coordinate system, obtaining a transformed lower image; splicing the upper image and the transformed lower image, obtaining a target digital human image associated with each abnormal digital human image. The present application improves the coordination of the digital human's entire body.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to a digital human image processing method, electronic device and storage medium. Background Art

[0002] Digital human technology uses computers to generate virtual characters with human appearance, behavior and interactive capabilities, and is widely used in the fields of virtual anchors, virtual idols, virtual actors and virtual teachers. The core of this technology is to drive the facial expressions and head movements of digital humans by identifying and matching the feature points of the face and head.

[0003] However, when it comes to the generation and driving of full-body digital humans, the limitations of existing technologies become apparent. Since current algorithms mainly focus on the processing of facial feature points, the processing of other parts of the body such as the neck and shoulders is relatively insufficient. When the head of the digital human moves under the drive of the algorithm, the neck and shoulders of the digital human will follow the movement in a relatively mechanical and unnatural manner, thus destroying the overall coordination and authenticity. Therefore, in order to achieve the natural generation and driving of full-body digital humans, it is urgent to further develop and improve relevant algorithms and technologies to overcome the limitations of existing technologies. Summary of the invention

[0004] The purpose of this application is to provide a digital human image processing method, electronic device and storage medium to solve the above problems.

[0005] In a first aspect, the present application proposes a digital human image processing method, the method comprising:

[0006] Acquire at least one frame of abnormal digital human image in the digital human video stream and a reference image associated with the abnormal digital human image;

[0007] Acquire an abnormal feature boundary line of each abnormal digital human image in the at least one abnormal digital human image, and segment each abnormal digital human image based on the abnormal feature boundary line to obtain an upper image including a face image and a lower image including a body image;

[0008] Based on the upper image and the reference image, determining a transformation parameter of the upper image based on a reference coordinate system of the reference image;

[0009] Based on the reference coordinate system, controlling the lower image to be transformed according to the transformation parameters to obtain a transformed lower image;

[0010] The upper image and the transformed lower image are spliced ​​to obtain a target digital human image associated with each frame of abnormal digital human image.

[0011] In some implementations, the step of obtaining an abnormal feature boundary line of each abnormal digital human image in the at least one abnormal digital human image comprises:

[0012] Determine the abnormal boundary area of ​​each frame of abnormal digital human image based on the fuzzy boundary line of each frame of abnormal digital human image in the at least one frame of abnormal digital human image;

[0013] Extracting characteristic corner points in the abnormal boundary area;

[0014] Determine the position and distribution of abnormal characteristic corner points according to the distribution and density of the characteristic corner points in the abnormal boundary area;

[0015] According to the positions and distribution of the abnormal feature corner points, the abnormal feature boundary line of each frame of abnormal digital human image is determined.

[0016] In some implementations, before extracting the characteristic corner points in the abnormal boundary area, the method further includes:

[0017] Extracting the color channel of each pixel in the abnormal boundary area;

[0018] According to the color channel of each pixel point, the gray value of each pixel point is converted to obtain a grayscale image of the abnormal boundary area;

[0019] The extracting of characteristic corner points in the abnormal boundary area comprises:

[0020] Based on the abnormal boundary region grayscale image, feature corner points in the abnormal boundary region are extracted.

[0021] In some embodiments, the reference coordinate system is a three-dimensional coordinate system, and determining, based on the upper image and the reference image, a transformation parameter of the upper image based on the reference coordinate system of the reference image comprises:

[0022] When the facial image is translated compared to the reference image, determining a translation parameter of the facial image in an X-axis direction and / or a Z-axis direction based on a reference coordinate system of the reference image;

[0023] When the facial image is scaled compared to the reference image, determining a translation parameter of the facial image in a Y-axis direction based on a reference coordinate system of the reference image;

[0024] When the facial image undergoes a rotation transformation compared to the reference image, determining a rotation parameter of the facial image in a Y-axis direction based on a reference coordinate system of the reference image;

[0025] When the facial image is deformed compared to the reference image, a rotation parameter of the facial image in an X-axis direction and / or a Z-axis direction of a reference coordinate system of the reference image is determined.

[0026] In some implementations, controlling the lower image to be transformed according to the transformation parameters based on the reference coordinate system to obtain a transformed lower image includes:

[0027] When the transformation parameters include translation parameters based on the X-axis direction and / or the Z-axis direction of the reference coordinate system, controlling the lower image to perform translation transformation based on the X-axis direction and / or the Z-axis direction of the reference coordinate system;

[0028] When the transformation parameters include a translation parameter based on the Y-axis direction of the reference coordinate system, controlling the lower image to be reduced or enlarged based on the Y-axis direction of the reference coordinate system;

[0029] When the transformation parameters include a rotation parameter based on the Y-axis direction of the reference coordinate system, controlling the lower image to perform a rotation transformation based on the Y-axis direction of the reference coordinate system;

[0030] When the transformation parameters include rotation parameters based on the X-axis direction and / or the Z-axis direction of the reference coordinate system, the lower image is controlled to perform a pitch transformation based on the X-axis direction of the reference coordinate system and / or a turn transformation based on the Z-axis direction of the reference coordinate system.

[0031] In some implementations, before acquiring at least one abnormal digital human image in the digital human video stream and a reference image associated with the abnormal digital human image, the method further includes:

[0032] Receive a digital human video stream, and input each frame of the digital human video stream into an abnormal human portrait recognition model, wherein the abnormal human portrait recognition model is a pre-trained binary classification model, and outputs a calibration result of an abnormal digital human image frame based on each frame of the digital human video stream;

[0033] Based on the calibration result, extracting abnormal digital human image frames from the digital human video stream;

[0034] After the upper image and the transformed lower image are spliced ​​to obtain the target digital human image associated with each frame of abnormal digital human image, the method further comprises:

[0035] The abnormal digital human image frames in the digital human video stream are replaced with the target digital human images associated with each frame of the abnormal digital human image to generate a repaired digital human video stream.

[0036] In some implementations, the reference image includes an original portrait picture for generating the digital human video stream, or a starting frame digital human image in the digital human video stream.

[0037] In some implementations, before acquiring at least one abnormal digital human image in the digital human video stream and a reference image associated with the abnormal digital human image, the method further includes:

[0038] Identifying the outline of the digital human in the abnormal digital human image frame;

[0039] Extracting a digital human image based on the digital human outline;

[0040] Hole filling and / or contour smoothing processing is performed on the extracted digital human image to form the abnormal digital human image.

[0041] In a second aspect, the present application further proposes an electronic device, comprising:

[0042] one or more processors;

[0043] The memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the digital human image processing method as described above.

[0044] In a third aspect, the present application further proposes a storage medium, wherein the storage medium stores executable instructions, and when the instructions are executed by a processor, the processor executes the digital human image processing method as described above.

[0045] Compared with the prior art, the beneficial effects of this application include:

[0046] First, by segmenting the abnormal digital human image and processing the upper and lower images separately, the movement of the neck, shoulders and other parts can be better processed, making the overall digital human more natural and coordinated, and improving the quality of digital human generation. Second, by determining the transformation parameters of the upper image based on the reference image and controlling the transformation of the lower image according to these parameters, the process of motion simulation and image stitching can be simplified, and the processing efficiency can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope of the present application.

[0048] Figure 1 A schematic diagram of a flow chart of an embodiment of the image processing method of the present application;

[0049] Figure 2 A partial flow chart of an embodiment of the image processing method of the present application;

[0050] Figure 3 A partial flow chart of an embodiment of the image processing method of the present application;

[0051] Figure 4 A schematic diagram of a reference image according to an embodiment of the image processing method of the present application;

[0052] Figure 5 A schematic diagram of an abnormal digital human image with a head translation in one embodiment of the image processing method of the present application;

[0053] Figure 6 A schematic diagram of an abnormal digital human image with an enlarged head in one embodiment of the image processing method of the present application;

[0054] Figure 7 A schematic diagram of an abnormal digital human image with a rotated head according to an embodiment of the image processing method of the present application;

[0055] Figure 8 A partial flow chart of an embodiment of the image processing method of the present application;

[0056] Fig. 9 A schematic diagram of an abnormal digital human image with translation transformation in one embodiment of the image processing method of the present application;

[0057] Fig.10 A schematic diagram of an abnormal digital human image that is magnified and transformed according to an embodiment of the image processing method of the present application;

[0058] Fig.11 A schematic diagram of an abnormal digital human image after rotation transformation according to an embodiment of the image processing method of the present application;

[0059] Fig.12 It is a structural schematic diagram of an electronic device involved in the digital human image processing method in any embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] All terms (including technical and scientific terms) used in this application have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used here should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0062] For example, the terms "first", "second", etc. used in this application may be used in this document to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element.

[0063] For another example, the terms “include”, “comprising”, etc. used in this application indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0064] As mentioned above, when it comes to the generation and driving of full-body digital humans, the limitations of the existing technology become apparent. Since the current algorithms mainly focus on the processing of facial feature points, the processing of other parts of the body such as the neck and shoulders is relatively insufficient. When the head of the digital human moves under the drive of the algorithm, the neck and shoulders of the digital human will cause a relatively mechanical and unnatural follow-up movement, thereby destroying the overall coordination and authenticity. Therefore, in order to achieve the natural generation and driving of full-body digital humans, it is urgent to further develop and improve relevant algorithms and technologies to overcome the limitations of the existing technology. To this end, the present application proposes a digital human image processing method, electronic device and storage medium, which can improve the overall coordination and authenticity of the digital human.

[0065] Figure 1 FIG. 1 is a flow chart of an embodiment of the image processing method of the present application. Figure 1 As shown, the digital human image processing method of this embodiment includes the following steps:

[0066] Step S10, obtaining at least one frame of abnormal digital human image and a reference image associated with the abnormal digital human image in the digital human video stream.

[0067] The digital human video stream in this embodiment refers to a video sequence composed of a series of continuous images of a virtual character generated by a computer. The generation of the digital human video stream usually involves combining the user's portrait photo with a specific voice input, and using an advanced deep learning model to predict and synthesize expressions and movements. As a feasible implementation method for generating a digital human video stream, by inputting a single or multiple user photos and corresponding voice data, the model can analyze the emotional and intonation characteristics of the voice, and generate facial expressions and lip movements that match the voice content. These dynamic facial features are then applied to the user's static photos, and through image processing and synthesis technology, a highly realistic digital human video stream that can respond to voice input in real time is generated.

[0068] The abnormal digital human image in this embodiment refers to an image in which the movement of the digital human head lacks natural coordination with the movement of the neck and shoulders due to the lack of fine control of the model over the body parts other than the head of the digital human (especially the neck and shoulders) when the deep learning model is used to drive the digital human to generate facial expressions and lip movements that match the speech content. Specifically, the head of the digital human makes dynamic responses according to the speech emotion and tone, while its neck and shoulders show relatively rigid and mechanical follow-up movements. This unnatural movement pattern destroys the harmony of the overall image of the digital human and the realism of the image.

[0069] The reference image in this embodiment refers to a stable reference image associated with the digital human video stream that is not significantly deformed, and is used to compare with the abnormal digital human image to determine the deformation and displacement of the abnormal digital human image.

[0070] In this embodiment, the application scenarios of the digital human image processing method cover a variety of intelligent terminals, including but not limited to smart phones, personal computers, robots and other electronic devices with data processing, network communication and program running functions. The specific online implementation form can be a website, mobile application (APP), WeChat applet, H5 page and API interface. Users access the Internet through these terminals and interact with the platform that provides digital human image processing services. This may involve opening a website, starting an APP, entering a small program or loading an H5 page in a browser. The user operates on these interfaces and chooses to upload the digital human video stream and the reference image that need to be repaired. Based on the user's operation, the electronic device receives the digital human video stream and the reference image, and automatically detects and extracts at least one frame of abnormal digital human image in the digital human video stream through computer vision technology.

[0071] In one embodiment, the reference image refers to the original portrait image uploaded by the user and used to generate the digital human video stream. This image has not been digitally processed or deformed in any form and represents the most original form of the digital human. By comparing the abnormal digital human image with the original portrait image, the changes caused by the deep learning model can be accurately identified.

[0072] In another embodiment, when the electronic device does not receive the original portrait image that has not been uploaded by the user, it will select the first frame image (i.e., the starting frame digital human image) in the digital human video stream as the reference image. Although the starting frame image has been digitally processed, it represents the initial state of the digital human and has not been significantly affected by facial expressions or lip movements. Therefore, the starting frame image can still be used as a relatively stable and natural reference point for comparison and correction with other digital human images.

[0073] Step S20, obtaining an abnormal feature boundary line of each abnormal digital human image in the at least one abnormal digital human image frame, and segmenting each abnormal digital human image frame based on the abnormal feature boundary line to obtain an upper image including a face image and a lower image including a body image.

[0074] The abnormal feature dividing line in this embodiment reflects the unnatural transition between the dynamic response of the digital human head and the mechanical movement of the neck and shoulders. Based on this dividing line, the image segmentation technology is used to segment each frame of abnormal digital human image into two parts: the upper image contains the face and its dynamic features, and the lower image contains the body and its relatively static features.

[0075] In some embodiments, Figure 2 As shown, the step S20 includes:

[0076] Step S21, determining the abnormal boundary area of ​​each frame of abnormal digital human image based on the fuzzy boundary line of each frame of abnormal digital human image in the at least one frame of abnormal digital human image.

[0077] The fuzzy dividing line in this embodiment refers to a dividing line that can be determined quickly but has low precision. It is used to preliminarily divide dynamic features (such as the head) and static features (such as the body) in the digital human image.

[0078] As an optional implementation for determining the fuzzy boundary line, simple image processing techniques such as thresholding, edge detection, etc. can be used. These techniques can provide a position closer to the actual abnormal feature boundary line at a lower computational cost.

[0079] As another optional implementation method for determining the fuzzy dividing line, this embodiment focuses on processing digital human images with inconsistent head and body. In this specific abnormal mode, the fuzzy dividing line can be defined as a universal dividing position, which can effectively separate dynamic features (such as the head) from static features (such as the body). Since the abnormal mode of inconsistent head and body is consistent, the corresponding fuzzy dividing line position can be quickly determined by analyzing and recording the distribution positions of the head and body in each frame of the abnormal digital human image.

[0080] The abnormal boundary area in this embodiment refers to an area selected by extending a certain range (such as 20 pixels) up and down on the basis of the fuzzy boundary line, and the area covers the possible abnormal feature transition zone.

[0081] Step S22, extracting characteristic corner points in the abnormal boundary area.

[0082] The feature corner points in this embodiment refer to points with obvious angle changes detected in the abnormal boundary area by applying a feature corner detection algorithm, such as Harris corner detection or Shi-Tomasi corner detection. These points may be located at the junction of the head and the body, or at places showing obvious gradient changes in the image.

[0083] In some implementations, before step S22, the method further includes:

[0084] Step A10: extracting the color channel of each pixel in the abnormal boundary area.

[0085] Step A20, converting the grayscale value of each pixel point according to the color channel of each pixel point to obtain a grayscale image of the abnormal boundary area;

[0086] In this embodiment, three color channels (R, G, B) of each pixel in the abnormal boundary area are extracted, and the R, G, B values ​​of each pixel are converted into grayscale values.

[0087] As an optional implementation of grayscale value conversion, the R, G, and B values ​​of each pixel are averaged, and the obtained average value is used as the grayscale value of the pixel.

[0088] As another optional implementation of grayscale value conversion, the R, G, and B values ​​of each pixel are weighted averaged to obtain the corresponding grayscale value. This is to take into account the different sensitivities of the human eye to different colors. Specifically, the human eye is most sensitive to green, followed by red, and least sensitive to blue. Exemplarily, the weights can be set to: R channel weight: 0.299; G channel weight: 0.587; B channel weight: 0.114. For each pixel, its R, G, and B values ​​are multiplied by the weight of the corresponding channel, and then the three products are added to obtain the grayscale value, grayscale value = 0.299 * R + 0.587 * G + 0.114* B.

[0089] As another optional implementation of grayscale value conversion, the maximum value or the minimum value among the R, G, and B values ​​of each pixel is taken as the grayscale value.

[0090] Based on the above implementation, step S20 includes:

[0091] A30, based on the grayscale image of the abnormal boundary area, extract the characteristic corner points in the abnormal boundary area.

[0092] As an optional implementation method for extracting feature corners based on the grayscale image of the abnormal boundary area, the interest value of each pixel, that is, the sum of the squares of the grayscale differences, is calculated, and the minimum value of the sum of the squares of the grayscales of adjacent pixels in four directions (0°, 45°, 90°, and 135°) is taken. If the interest value is greater than a given threshold and is locally maximum, it is considered that a feature corner point has been detected. This implementation method takes into account the grayscale changes in multiple directions rather than a single direction, and can resist noise interference to a certain extent. In addition, by finding the local maximum interest value, it can be ensured that the detected corner points have a certain significance and stability.

[0093] As another optional implementation method for extracting feature corner points based on the grayscale image of the abnormal boundary area, the grayscale image of the abnormal boundary area is divided into sub-areas of equal size. Then, for each pixel in the sub-area, its Roberts gradient (a simple edge detection operator) and the grayscale covariance matrix of the window centered on the pixel are calculated. By analyzing the covariance matrix, an error ellipse can be obtained, which reflects the distribution of grayscale changes around the pixel, and points with error ellipses close to a circle below the threshold are found as feature corner points. This implementation method can better capture the feature corner points in the abnormal boundary area, especially those points with significant grayscale changes in multiple directions. By dividing the area and calculating the covariance matrix, the robustness and accuracy of corner point detection can be improved.

[0094] As another optional implementation method for extracting feature corners based on the grayscale image of the abnormal boundary area, a Gaussian template is used to convolve the grayscale image of the abnormal boundary area to obtain a corner mapping value. This mapping value reflects the possibility of each pixel in the abnormal boundary area becoming a feature corner point. By comparing with a preset threshold, pixels with mapping values ​​higher than the threshold can be extracted as feature corner points. The Gaussian template can smooth the image, reduce the impact of noise, and enhance the corner point features.

[0095] As another optional implementation method for extracting feature corner points based on the grayscale image of the abnormal boundary area, a circular template is used to slide on the image to calculate the difference between the grayscale of the pixels in the template and the grayscale of the center pixel of the template. Pixels that meet certain conditions form a local area, called the USAN area. Calculate the ratio of the size of the USAN area to the area of ​​the template, and set a threshold. When the USAN ratio is less than the threshold, the center pixel of the template is considered to be a feature corner point. This implementation method can effectively detect feature corner points in the abnormal boundary area, especially those points with significant grayscale changes in the local area. By adjusting the threshold, the sensitivity of feature corner point detection can be flexibly controlled.

[0096] Step S23, determining the position and distribution of abnormal feature corner points according to the distribution and density of the feature corner points in the abnormal boundary area.

[0097] In the abnormal boundary area, the detected feature corner points are screened, for example, noise points, duplicate points or unstable points are removed. Through the screening process, unnecessary interference points can be reduced, the accuracy and efficiency of subsequent analysis can be improved, and the screened feature corner points can be more representative.

[0098] By analyzing the distribution and density of feature corner points in the abnormal boundary area, statistical methods (such as mean, variance, histogram analysis, etc.) or machine learning models (such as classifiers, clustering algorithms, etc.) are used to identify the pixels (abnormal feature corner points) that actually cause the inconsistency between the head and the body.

[0099] Step S24, determining the abnormal feature boundary line of each frame of abnormal digital human image according to the position and distribution of the abnormal feature corner points.

[0100] The identified abnormal feature corner points are clustered to determine their concentrated areas. This can be achieved through clustering algorithms such as K-means and DBSCAN, with the goal of clustering scattered abnormal corner points into meaningful areas. Based on the positions of the clustered abnormal feature corner points, one or more dividing lines are determined using geometric calculation methods (such as least squares method, RANSAC, etc.) or curve / surface fitting techniques. These dividing lines should be able to distinguish dynamic feature areas from static feature areas as accurately as possible.

[0101] In some implementations, the initially fitted boundary lines may be optimized through iterative adjustment or morphological operations, etc., to ensure that they can not only reflect the distribution of abnormal feature corner points, but also adapt to local changes and noise in the digital human image.

[0102] Step S30 : determining, based on the upper image and the reference image, transformation parameters of the upper image based on a reference coordinate system of the reference image.

[0103] The reference coordinate system in this embodiment is a coordinate system defined on the reference image, which is used to subsequently describe the transformation parameters of the upper image relative to the reference image. Specifically, it can be a Cartesian coordinate system with three orthogonal axes: X, Y, and Z. Figure 4 , Figure 4 A schematic diagram of a reference image according to an embodiment of the image processing method of the present application.

[0104] In some embodiments, Figure 3 As shown, the step S30 includes:

[0105] Step S31 : when the facial image is translated compared to the reference image, determining a translation parameter of the facial image in an X-axis direction and / or a Z-axis direction based on a reference coordinate system of the reference image.

[0106] By calculating the position difference between the key points of the face image (such as the positions of the eyes, nose, mouth or the geometric center of the face image) and the corresponding key points in the reference image, the translation parameters in the X-axis and Z-axis directions can be determined. Figure 5 This is a schematic diagram of an abnormal digital human image with head translation in one embodiment of the image processing method of the present application. Figure 5 The translation parameter of the face image shown may be -10 pixels in the X-axis direction.

[0107] Step S32: when the facial image is scaled compared to the reference image, determine a translation parameter of the facial image in the Y-axis direction of a reference coordinate system of the reference image.

[0108] In this embodiment, the size change between the face image and the reference image is handled through the concepts of depth perception and perspective transformation. Specifically, when the face image moves in a direction perpendicular to the screen relative to the reference image, this movement is expressed as a translation parameter in the Y-axis direction. There are two main cases here: First, when the face image is translated in the positive direction of the Y-axis as a whole, it means that the face image is enlarged in the vertical direction. In this case, the actual distance between the feature points in the face image (such as eyes, nose, mouth, etc.) will be greater than the corresponding distance in the reference image. Figure 6 , Figure 6 A schematic diagram of an abnormal digital human image with an enlarged head in one embodiment of the image processing method of the present application. Second, when the face image as a whole is translated in the negative direction of the Y axis, it means that the face image is reduced in the vertical direction. In this case, the actual distance between the feature points in the face image will be smaller than the corresponding distance in the reference image.

[0109] By comparing the actual distance between the corresponding feature points in the face image and the reference image, the scaling ratio can be calculated. This scaling ratio reflects the degree of translation of the face image in the Y-axis direction relative to the reference image. Further, by analyzing the position change of the feature points and the scaling ratio, the translation parameters of the specific face image in the Y-axis direction of the reference coordinate system can be determined.

[0110] Step S33, when the facial image is subjected to a rotation transformation compared to the reference image, determining a rotation parameter of the facial image in the Y-axis direction of a reference coordinate system of the reference image;

[0111] In this embodiment, the geometric center of the face image is selected as the rotation center. The rotation center remains unchanged during the rotation transformation. In order to determine the rotation parameters, at least one pair of corresponding points (i.e., a feature point in the face image and its corresponding position in the reference image) is required. Through these corresponding points, the rotation angle (Yaw) of the face image relative to the reference image can be calculated. Figure 7 FIG. 1 is a schematic diagram of an abnormal digital human image with a rotated head according to an embodiment of the image processing method of the present application. Figure 7 The face image shown is rotated about 15 degrees counterclockwise with the Y-axis as the rotation axis.

[0112] Step S34: when the facial image is deformed compared to the reference image, determining a rotation parameter of the facial image in an X-axis direction and / or a Z-axis direction based on a reference coordinate system of the reference image.

[0113] In this embodiment, when the digital human in the face image shows an action of turning its head rightward with the Z axis as the rotation axis, that is, the image area of ​​the face image in the negative direction of the X axis shrinks, and the image area in the positive direction of the X axis expands. It can be considered that the face image has rotated rightward with the Z axis as the rotation axis.

[0114] When the digital human in the face image turns its head to the left with the Z axis as the rotation axis, the image area in the positive X axis direction of the face image will shrink, while the image area in the negative X axis direction will expand. This indicates that the face image is rotating to the left with the Z axis as the rotation axis.

[0115] When the digital human in the face image shows the action of raising its head (i.e. tilting its head back), the image area in the negative direction of the Z axis of the face image will shrink, while the image area in the positive direction of the Z axis will expand. This indicates that the face image is rotating clockwise along the X axis.

[0116] When the digital human in the face image shows a head-down (i.e. leaning forward) action, the image area in the positive Z-axis direction of the face image will shrink, while the image area in the negative Z-axis direction will expand. This indicates that the face image rotates counterclockwise along the X-axis direction.

[0117] The two-dimensional image in this embodiment essentially does not have three-dimensional information, but the head movement of the digital human is driven by an algorithm, that is, in this embodiment, the driving process can be restored by the deformation of the face image, and the rotation parameters of the face image in three-dimensional space can be calculated, which can be described by three Euler angles (Yaw, Pitch, Roll).

[0118] Specifically, use a facial key point detection algorithm (such as the cascade classifier provided by Dlib, OpenCV, or a deep learning-based method) to detect key feature points in the face image and the reference image. Match the feature points in the face image with the feature points in the reference image, ensuring that each feature point has its corresponding point in the other image. For the matched feature point pairs, calculate their displacements on the image plane. These displacements provide information about how the image is deformed. Based on the displacement of the feature points, a pre-trained deformation model is used to describe the deformation of the image. The deformation model is a powerful tool that combines the principles of geometry, mathematics, and physics to simulate and analyze complex shape changes. Estimate the rotation Euler angles based on the deformation model. You can also construct a rotation matrix based on the rotation angles and apply this matrix to the reference image to verify the deformation process.

[0119] Step S40: Based on the reference coordinate system, control the lower image to be transformed according to the transformation parameters to obtain a transformed lower image.

[0120] In some embodiments, Figure 8 As shown, the step S40 includes:

[0121] Step S41, when the transformation parameters include translation parameters based on the X-axis direction and / or the Z-axis direction of the reference coordinate system, control the lower image to perform a translation transformation based on the X-axis direction and / or the Z-axis direction of the reference coordinate system.

[0122] The translation transformation in this embodiment refers to moving the lower image along the coordinate axis in the same translation direction as the face image without changing its shape and size. Fig. 9 , Fig. 9 A schematic diagram of an abnormal digital human image subjected to translation transformation according to an embodiment of the image processing method of the present application.

[0123] Step S42: When the transformation parameters include translation parameters based on the Y-axis direction of the reference coordinate system, the lower image is controlled to be reduced or enlarged based on the Y-axis direction of the reference coordinate system.

[0124] The real meaning of the Y-axis translation parameter in this embodiment is to characterize the change in image size caused by depth perception and perspective transformation. In three-dimensional space, when an object is farther away from the observer, its projection on the two-dimensional image will appear smaller, and vice versa. This effect is caused by perspective projection, in which distant objects appear smaller than nearby objects. The lower image is adjusted through the concepts of depth perception and perspective transformation to compensate for the scaling difference between the lower image and the facial image.

[0125] In this embodiment, the Y-axis translation parameter is used to adjust the lower image to compensate for the scaling difference between the face image and the reference image due to depth perception and perspective transformation. That is, if the face image is different from the reference image in depth (there is a displacement in the Y-axis direction), the lower image needs to be reduced or enlarged in the Y-axis direction accordingly so that the two are visually consistent. Fig.10 , Fig.10 A schematic diagram of an abnormal digital human image that is magnified and transformed according to an embodiment of the image processing method of the present application.

[0126] Step S43: When the transformation parameters include a rotation parameter based on the Y-axis direction of the reference coordinate system, control the lower image to undergo a rotation transformation based on the Y-axis direction of the reference coordinate system.

[0127] The rotation transformation in this embodiment means that the lower image is rotated along the rotation direction of the face image without changing its shape and size. Fig.11 , Fig.11 A schematic diagram of an abnormal digital human image after rotation transformation according to an embodiment of the image processing method of the present application.

[0128] Step S44, when the transformation parameters include rotation parameters based on the X-axis direction and / or the Z-axis direction of the reference coordinate system, control the lower image to perform a pitch transformation based on the X-axis direction of the reference coordinate system and / or a turn transformation based on the Z-axis direction of the reference coordinate system.

[0129] In this embodiment, by introducing depth information for the two-dimensional lower image, a three-dimensional model can be constructed to represent the lower image. According to the determined rotation parameters, the model can be rotated and translated in three-dimensional space. For the rotation parameters in the X-axis direction, this will correspond to a pitch transformation, that is, the model rotates around the X-axis to change its up-down direction. For the rotation parameters in the Z-axis direction, this will correspond to a yaw transformation, that is, the model rotates around the Z-axis to change its left-right direction. After completing the above transformation, the three-dimensional model can be rendered into a two-dimensional image, or it can be projected onto an XZ-axis plane to generate a new two-dimensional image.

[0130] In some implementations, if there are multiple types of transformation parameters at the same time, step S41 is placed before step S43 and step S44, and step S42 is placed before step S44.

[0131] Step S50, splicing the upper image and the transformed lower image to obtain a target digital human image associated with each frame of abnormal digital human image.

[0132] In this embodiment, the transformed lower image has been precisely aligned with the upper image. At this point, the image stitching operation can be directly performed to generate a complete digital human image. The stitching operation usually merges the two images along the abnormal feature boundary to form a complete image. During the stitching process, slight discontinuities or color differences may appear at the junction of the images. These boundaries can be smoothed by using image filtering and fusion technology to make the stitching more natural and seamless.

[0133] In this embodiment, firstly, by segmenting the abnormal digital human image and processing the upper image and the lower image separately, the movement of the neck, shoulders and other parts can be better processed, making the overall digital human more natural and coordinated, and improving the quality of digital human generation. Secondly, by determining the transformation parameters of the upper image based on the reference image and controlling the lower image to transform according to these parameters, the process of motion simulation and image splicing can be simplified, and the processing efficiency can be improved.

[0134] Based on any of the above implementations, in one embodiment, before step S10, the method further includes:

[0135] Step S00, receiving a digital human video stream, and inputting each frame of the digital human video stream into an abnormal human portrait recognition model, wherein the abnormal human portrait recognition model is a pre-trained binary classification model, and outputs a calibration result of an abnormal digital human image frame based on each frame of the digital human video stream.

[0136] The abnormal portrait recognition model in this embodiment is a pre-trained binary classification model. As a feasible implementation method for training the abnormal portrait recognition model, a large number of digital human image frames are collected in advance, and these image frames need to contain samples of normal and abnormal portraits. Abnormal portraits include portrait features that are inconsistent with the head and body. The collected image frames are labeled to mark which image frames contain abnormal portraits and which are normal. Select a model architecture suitable for image classification tasks, such as a convolutional neural network (CNN). Use the labeled data set to train the model. During the training process, the model will learn the characteristics of normal and abnormal portraits and try to make correct classifications in new image frames. A binary classification model means that the model has only two output categories: normal and abnormal. Evaluate the performance of the model on the validation set, and adjust the model parameters based on the results to improve the accuracy and robustness of the model.

[0137] The calibration result in this embodiment can be expressed as a binary value, indicating whether the image frame contains an abnormal human image. For example, "1" can be used to indicate abnormality, and "0" can be used to indicate normality.

[0138] Step S01: extracting abnormal digital human image frames from the digital human video stream based on the calibration result.

[0139] By using the pre-trained abnormal portrait recognition model, abnormal image frames can be automatically identified and extracted from the digital human video stream, without the need for complex segmentation and splicing of each image frame, reducing the amount of calculation, thereby saving computing resources and time. This is particularly advantageous when processing large amounts of video data.

[0140] In this embodiment, after step S50, the following steps are further included:

[0141] Step S60: replacing the abnormal digital human image frames in the digital human video stream with the target digital human image associated with each frame of the abnormal digital human image, to generate a repaired digital human video stream.

[0142] This embodiment effectively solves the problem of head-body incoordination of the digital human in the full-body digital human video stream. This process not only improves the overall quality and viewing experience of the video stream, but also ensures that the image of the digital human is more natural and coordinated, thereby improving the viewing experience and satisfaction of users.

[0143] In this embodiment, before step S10, the following steps are also included:

[0144] Step S02: identifying the outline of the digital human in the abnormal digital human image frame.

[0145] As an optional implementation method for identifying the outline of a digital human, methods such as Canny edge detection and Sobel operator are used to detect edges in an image, thereby preliminarily identifying the outline of a digital human.

[0146] As another optional implementation method of identifying the contour of the digital human, a contour extraction function in a computer vision library such as OpenCV, such as cv2.findContours, is used to further extract the complete contour of the digital human.

[0147] As another optional implementation for identifying the outline of a digital human, a target detection algorithm in deep learning, such as YOLO, SSD, etc., is used to accurately identify the outline of a digital human.

[0148] In some implementations, more complex feature extraction methods, such as local binary pattern (LBP), scale-invariant feature transform (SIFT), etc., may be introduced to enhance the ability to describe the contour of the digital human.

[0149] Step S03: extracting a digital human image based on the digital human outline.

[0150] The "cutting" in this embodiment refers to segmenting the image content inside and outside the outline of the digital human. "Cutting" does not simply cut the digital human out from the background, but refers to accurately separating the human image from the complex background through an algorithm. Segmentation methods based on graph theory, such as the Graph Cut algorithm and the minimum cut maximum flow algorithm, can be used to improve the efficiency and effect of segmentation.

[0151] Step S04, performing hole filling and / or contour smoothing processing on the extracted digital human image to form the abnormal digital human image.

[0152] The hole filling and contour smoothing processing in this embodiment is mainly to improve the quality of the deducted digital human image, making it more complete and smooth. When performing hole filling and contour smoothing processing, the defects of the digital human image are mainly repaired, rather than changing the basic structure of the digital human. Therefore, these processing will not hinder the subsequent upper and lower segmentation and splicing operations. On the contrary, the processed digital human image is more complete and smooth, which helps to improve the accuracy and effect of segmentation.

[0153] In this embodiment, by subtracting the digital human image from the digital human image frame, the portrait area can be more accurately identified, and feature point calculation and matching can be performed, thereby improving the effect of subsequent image stitching.

[0154] In one embodiment, a storage medium is provided, on which executable instructions are stored. When the instructions are executed by a processor, the processor executes the steps in the above-mentioned method embodiments.

[0155] In one embodiment, an electronic device is also provided, including one or more processors; a memory, in which one or more programs are stored, wherein when the one or more programs are executed by one or more processors, the one or more processors execute the steps in the above-mentioned method embodiments.

[0156] In one embodiment, Fig.12 , which shows a schematic diagram of the structure of an electronic device for implementing an embodiment of the present application. The electronic device 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0157] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage section 908 as needed.

[0158] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, including a computer-readable medium carrying instructions, in such an embodiment, the instructions can be downloaded and installed from a network through a communication part 909, and / or installed from a removable medium 911. When the instructions are executed by a central processing unit (CPU) 901, the various method steps described in the present application are executed.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0160] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the above claims, any one of the claimed embodiments may be used in any combination. The information disclosed in this background technology section is intended only to deepen the understanding of the overall background technology of the present application and should not be regarded as an admission or in any form of implication that the information constitutes prior art known to those skilled in the art.

Claims

1. A digital human image processing method, characterized in that: The method comprises: Acquire at least one frame of abnormal digital human image in the digital human video stream and a reference image associated with the abnormal digital human image; Determine the abnormal boundary area of ​​each frame of abnormal digital human image based on the fuzzy boundary line of each frame of abnormal digital human image in the at least one frame of abnormal digital human image; Extracting characteristic corner points in the abnormal boundary area; Determine the position and distribution of abnormal characteristic corner points according to the distribution and density of the characteristic corner points in the abnormal boundary area; Determining the abnormal feature boundary line of each frame of abnormal digital human image according to the position and distribution of the abnormal feature corner points; Segmenting each frame of abnormal digital human image based on the abnormal feature dividing line to obtain an upper image containing a face image and a lower image containing a body image; Based on the upper image and the reference image, determining a transformation parameter of the upper image based on a reference coordinate system of the reference image; Based on the reference coordinate system, controlling the lower image to be transformed according to the transformation parameters to obtain a transformed lower image; The upper image and the transformed lower image are spliced ​​to obtain a target digital human image associated with each frame of abnormal digital human image.

2. The digital human image processing method according to claim 1, characterized in that: Before extracting the characteristic corner points in the abnormal boundary area, the method further includes: Extracting the color channel of each pixel in the abnormal boundary area; According to the color channel of each pixel point, the gray value of each pixel point is converted to obtain a grayscale image of the abnormal boundary area; The extracting of characteristic corner points in the abnormal boundary area comprises: Based on the abnormal boundary region grayscale image, feature corner points in the abnormal boundary region are extracted.

3. The digital human image processing method according to claim 1, characterized in that: The reference coordinate system is a three-dimensional coordinate system, and determining, based on the upper image and the reference image, transformation parameters of the upper image based on the reference coordinate system of the reference image includes: When the facial image is translated compared to the reference image, determining a translation parameter of the facial image in an X-axis direction and / or a Z-axis direction based on a reference coordinate system of the reference image; When the facial image is scaled compared to the reference image, determining a translation parameter of the facial image in a Y-axis direction based on a reference coordinate system of the reference image; When the facial image undergoes a rotation transformation compared to the reference image, determining a rotation parameter of the facial image in a Y-axis direction based on a reference coordinate system of the reference image; When the facial image is deformed compared to the reference image, a rotation parameter of the facial image in an X-axis direction and / or a Z-axis direction of a reference coordinate system of the reference image is determined.

4. The digital human image processing method according to claim 3, characterized in that: The step of controlling the lower image to be transformed according to the transformation parameters based on the reference coordinate system to obtain a transformed lower image comprises: When the transformation parameters include translation parameters based on the X-axis direction and / or the Z-axis direction of the reference coordinate system, controlling the lower image to perform translation transformation based on the X-axis direction and / or the Z-axis direction of the reference coordinate system; When the transformation parameters include a translation parameter based on the Y-axis direction of the reference coordinate system, controlling the lower image to be reduced or enlarged based on the Y-axis direction of the reference coordinate system; When the transformation parameters include a rotation parameter based on the Y-axis direction of the reference coordinate system, controlling the lower image to perform a rotation transformation based on the Y-axis direction of the reference coordinate system; When the transformation parameters include rotation parameters based on the X-axis direction and / or the Z-axis direction of the reference coordinate system, the lower image is controlled to perform a pitch transformation based on the X-axis direction of the reference coordinate system and / or a turn transformation based on the Z-axis direction of the reference coordinate system.

5. The digital human image processing method according to claim 1, characterized in that: Before acquiring at least one frame of abnormal digital human image and a reference image associated with the abnormal digital human image in the digital human video stream, the method further includes: Receive a digital human video stream, and input each frame of the digital human video stream into an abnormal human portrait recognition model, wherein the abnormal human portrait recognition model is a pre-trained binary classification model, and outputs a calibration result of an abnormal digital human image frame based on each frame of the digital human video stream; Based on the calibration result, extracting abnormal digital human image frames from the digital human video stream; After the upper image and the transformed lower image are spliced ​​to obtain the target digital human image associated with each frame of abnormal digital human image, the method further comprises: The abnormal digital human image frames in the digital human video stream are replaced with the target digital human images associated with each frame of the abnormal digital human image to generate a repaired digital human video stream.

6. The digital human image processing method according to claim 5, characterized in that: The reference image includes an original portrait picture for generating the digital human video stream, or a starting frame digital human image in the digital human video stream.

7. The digital human image processing method according to claim 5, characterized in that: Before obtaining at least one abnormal digital human image frame in the digital human video stream and a reference image associated with the abnormal digital human image, the method further includes: Identifying the outline of the digital human in the abnormal digital human image frame; Extracting a digital human image based on the digital human outline; Hole filling and / or contour smoothing processing is performed on the extracted digital human image to form the abnormal digital human image.

8. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the digital human image processing method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium stores executable instructions, and when the instructions are executed by the processor, the processor executes the digital human image processing method according to any one of claims 1 to 7.

Images