3D Image Processing Method, Apparatus, Electronic Device, and Readable Storage Medium

Abstract

The present application discloses a 3D image processing method, apparatus, electronic device, and readable storage medium, which relate to the technical field of image processing. Among them, the 3D image processing method includes: obtaining a target 3D head image and a 3D sticker image; respectively performing primitive assembly on the target 3D head image and the 3D sticker image based on a rendering pipeline to obtain a first primitive corresponding to the target 3D head image and a second primitive corresponding to the 3D sticker image; performing depth testing and back face culling processing of the rendering pipeline on the first primitive and the second primitive to obtain a first front primitive corresponding to the first primitive and a second front primitive corresponding to the second primitive; performing rasterization processing and shading processing on the first front primitive and the second front primitive, and displaying the processed second front primitive.

Description

Technical Field

[0001] This application relates to the field of image processing technologies, and in particular, to a 3D image processing method, apparatus, electronic device, and readable storage medium. Background Art

[0002] A 3D face sticker is to attach a modeled 3D model material, such as 3D glasses, 3D masks, etc., to the face in a two-dimensional picture in a three-dimensional manner. Currently, this technology is widely used in mobile shooting or short video applications. In the 3D face sticker technology, a head occlusion display problem usually needs to be solved. For example, when the user turns their head to the side, bows their head, or raises their head, the 3D sticker display problem of the part occluded by the head. Taking 3D glasses as an example, when the user turns their head to the left, the part of the glasses frame occluded by the face on the left should not be displayed. Currently, the 3D face reconstruction and mask generation and mixing methods are usually used to achieve the cropping of the head occlusion content, but such a method has a large amount of calculation and poor calculation real-time performance. Summary of the Invention

[0003] Embodiments of this application provide a 3D image processing method, apparatus, electronic device, and readable storage medium, which can solve the problem of large 3D image processing calculation amount in related technologies.

[0004] In a first aspect, embodiments of this application provide a 3D image processing method, including:

[0005] Obtain a target 3D head image and a 3D sticker image;

[0006] Based on a rendering pipeline, perform primitive assembly on the target 3D head image and the 3D sticker image respectively to obtain a first primitive corresponding to the target 3D head image and a second primitive corresponding to the 3D sticker image;

[0007] Perform depth testing and back face culling processing on the rendering pipeline of the first primitive and the second primitive to obtain a first front primitive corresponding to the first primitive and a second front primitive corresponding to the second primitive;

[0008] Perform rasterization processing and shading processing on the first front primitive and the second front primitive, and display the processed second front primitive.

[0009] In a second aspect, embodiments of this application provide a 3D image processing apparatus, including:

[0010] An obtaining module, configured to obtain a target 3D head image and a 3D sticker image;

[0011] A primitive assembly module, configured to perform primitive assembly on the target 3D head image and the 3D sticker image respectively based on a rendering pipeline, to obtain a first primitive corresponding to the target 3D head image and a second primitive corresponding to the 3D sticker image;

[0012] A first processing module, configured to perform depth testing and back face culling processing of the rendering pipeline on the first primitive and the second primitive, to obtain a first front primitive corresponding to the first primitive and a second front primitive corresponding to the second primitive;

[0013] A second processing module, configured to perform rasterization processing and shading processing on the first front primitive and the second front primitive, and display the processed second front primitive.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, characterized by including a processor and a memory, the memory stores a program or instruction that can run on the processor, and when the program or instruction is executed by the processor, the steps of the 3D image processing method described in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a readable storage medium, characterized in that a program or instruction is stored on the readable storage medium, and when the program or instruction is executed by a processor, the steps of the 3D image processing method described in the first aspect are implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is configured to run a program or instruction to implement the steps of the 3D image processing method described in the first aspect.

[0017] The technical solution provided by the embodiment of the present application constructs a scene for the 3D sticker image through the target 3D head image, and then determines the parts of the 3D sticker image that may be blocked when the 3D sticker image is attached to the target 3D head image by means of the target 3D head image, so as to implement the processing of the blocked parts of the 3D sticker image when applied to the 3D head image, and finally only display the unblocked parts, that is, implement the cropping of the blocked parts of the 3D sticker image. The solution provided by the embodiment of the present application does not require complex calculation processes such as face reconstruction and mask generation. By taking advantage of the occlusion processing of the rendering pipeline, it can quickly implement the processing of the blocked parts of the 3D sticker image, with lower and simpler computational complexity. Therefore, this method can also be used on electronic devices with lower computing power, which is more conducive to expanding the processing and application scope of 3D sticker images. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flowchart of a 3D image processing method provided by an embodiment of the present application;

[0019] Figure 2 It is a structural diagram of a 3D image processing device provided by an embodiment of the present application;

[0020] Figure 3 It is a structural diagram of an electronic device provided by an embodiment of the present application. Specific embodiments

[0021] Next, the technical solutions in the embodiments of the present application will be clearly described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0022] The terms "first", "second", etc. in the description and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the description and claims means at least one of the connected objects, and the character " / " generally means an "or" relationship between the associated objects before and after.

[0023] Next, in conjunction with the accompanying drawings, the 3D image processing method provided by the embodiments of the present application will be described in detail through specific embodiments and their application scenarios.

[0024] Please refer to Figure 1 , Figure 1 It is a flowchart of a 3D image processing method provided by an embodiment of the present application. As Figure 1 shown, the method includes the following steps:

[0025] Step 101, obtain a target 3D head image and a 3D sticker image.

[0026] Among them, the target 3D head image may refer to the 3D head image to be processed, and the 3D sticker image may be a 3D prop image attached to the target 3D head image. For example, the 3D sticker image may be a 3D glasses image, a 3D hat image, etc.

[0027] It should be noted that the method provided by the embodiments of the present application can be applied to electronic devices such as computers and mobile phones. For better understanding, the method provided by the embodiments of the present application will be specifically described below with an electronic device as the execution subject of the method.

[0028] Step 102: Based on the rendering pipeline, perform primitive assembly on the target 3D head image and the 3D sticker image respectively, to obtain a first primitive corresponding to the target 3D head image and a second primitive corresponding to the 3D sticker image.

[0029] Optionally, the electronic device may determine a rendering scene based on the target 3D head image. For example, the camera that captures the target 3D head image may be placed directly in front of the target 3D head image, and the acquisition field of view of the camera is used as the rendering scene, and the target 3D head image is located in the rendering scene. After determining the rendering scene, initialize the rendering pipeline. For example, initialize the rendering pipeline based on the open interface program OpenGL ES (OpenGL for Embedded Systems), load the target 3D head image and the 3D sticker image into the initialized rendering pipeline, and perform primitive assembly on the target 3D head image and the 3D sticker image based on the rendering pipeline, to obtain a first primitive corresponding to the target 3D head image and a second primitive corresponding to the 3D sticker image.

[0030] Step 103: Perform depth testing and back face culling processing of the rendering pipeline on the first primitive and the second primitive, to obtain a first front primitive corresponding to the first primitive and a second front primitive corresponding to the second primitive.

[0031] In the embodiment of the present application, the 3D sticker image is attached to the surface of the target 3D head image. Based on the principle that depth testing can retain the primitive closest to the camera, through the depth testing function of the rendering pipeline, a first front primitive corresponding to the first primitive, and a second front primitive and a back primitive (or also called a back face primitive) corresponding to the second primitive can be obtained.

[0032] Further, turn on the back face culling function of the rendering pipeline, and perform back face culling processing on the first front primitive corresponding to the first primitive, the second front primitive corresponding to the second primitive, and the back primitive. Then, the back primitive corresponding to the second primitive will be culled, and the remaining first front primitive and second front primitive will be obtained.

[0033] It should be noted that the 3D sticker image is attached to the surface of the target 3D head image. The second primitive is obtained by primitive assembly of the 3D sticker image. The back primitive corresponding to the second primitive is also the primitive in the 3D sticker image that is blocked by the target 3D head image. Through the back face culling function of the rendering pipeline, the primitive in the 3D sticker image that is blocked by the target 3D head image is culled.

[0034] Step 104: Perform rasterization processing and shading processing on the first front primitive and the second front primitive, and display the processed second front primitive.

[0035] In this step, rasterization processing is performed on the first front primitive and the second front primitive, and then the rasterized first front primitive and second front primitive are respectively subjected to shading processing in the fragment shader of the rendering pipeline. Among them, the transparency value of the first front primitive is assigned 0, which means that the first front primitive is a transparent area. Furthermore, the first front primitive will not be displayed, and finally only the second front primitive is displayed. The second front primitive is obtained by primitive assembly of the 3D sticker image, that is, finally only the front primitive of the 3D sticker image is displayed.

[0036] In the embodiment of the present application, a target 3D head image and a 3D sticker image are obtained, and primitive assembly is respectively performed on the target 3D head image and the 3D sticker image based on the rendering pipeline to obtain a first primitive corresponding to the target 3D head image and a second primitive corresponding to the 3D sticker image. Then, depth testing and back face culling processing of the rendering pipeline are performed on the first primitive and the second primitive to obtain a first front primitive corresponding to the first primitive and a second front primitive corresponding to the second primitive. Rasterization processing and shading processing are performed on the first front primitive and the second front primitive, and finally the processed second front primitive is displayed. In this way, the 3D sticker image is used to construct a scene through the target 3D head image, and then the parts that may be blocked when the 3D sticker image is attached to the target 3D head image are determined by means of the target 3D head image, so as to realize the processing of the blocked parts when the 3D sticker image is applied to the 3D head image, and finally only the unblocked parts are displayed, that is, the clipping of the blocked parts of the 3D sticker image is realized. The solution provided by the embodiment of the present application does not require complex calculation processes such as face reconstruction and mask generation. By utilizing the occlusion processing advantage of the rendering pipeline, the processing of the occluded parts of the 3D sticker image can be quickly realized, and the calculation amount is lower and simpler. Furthermore, this method can also be used on electronic devices with low computing power, which is more conducive to increasing the processing and application range of 3D sticker images.

[0037] Optionally, the obtaining of the target 3D head image includes:

[0038] Obtain a to-be-processed 2D head image and a reference 3D head image, and obtain a geometric transformation matrix for fitting the reference 3D head image to the 2D head image;

[0039] Perform vertex transformation on the reference 3D head image based on the geometric transformation matrix to obtain the target 3D head image.

[0040] It should be noted that the reference 3D head image can be a neutral (front face, expressionless) 3D face model clustered from an industry open-source 3D face model database (such as the Basel Face Model (BFM)), which is used as the reference 3D head image. Among them, the front surface of the outer wrapping of the head is the front, and the back surface of the outer wrapping is the back. The 2D head image to be processed can be any 2D head image, or it can also be the 2D head image corresponding to the reference 3D head image.

[0041] It can be understood that the reference 3D head image is a 3D face model obtained by clustering, and the 2D head image to be processed can be any 2D head image. Therefore, the reference 3D head image and the 2D head image to be processed may not be necessarily adaptable. Then, the reference 3D head image is fitted to the 2D head image, and the geometric transformation matrix of the reference 3D head image fitted to the 2D head image is obtained. Based on the geometric transformation matrix, vertex transformation is performed on the reference 3D head image to obtain the target 3D head image, that is, the 3D head image to which the 3D sticker is attached.

[0042] In the embodiment of the present application, by calculating the geometric transformation matrix of the reference 3D head image fitted to the 2D head image, and performing vertex transformation on the reference 3D head image based on the geometric transformation matrix, the target 3D head image for scene construction of the 3D sticker image is obtained. In this way, it is not necessary to perform 3D shape construction of the human face, effectively reducing the amount of calculation and processing complexity.

[0043] Optionally, the obtaining of the 2D head image to be processed and the reference 3D head image, and the obtaining of the geometric transformation matrix of the reference 3D head image fitted to the 2D head image include:

[0044] Obtain the reference 3D head image, and obtain the projection matrix of the reference 3D head image projected onto the 2D plane;

[0045] Obtain the 2D head image to be processed, and obtain the Euler angles of the face pose of the 2D head image to be processed, and convert the Euler angles of the face pose into a rotation matrix;

[0046] Based on the projection matrix and the rotation matrix, obtain the scaling factor and translation vector of the reference 3D head image fitted to the 2D head image to be processed;

[0047] Based on the rotation matrix, the scaling factor, and the translation vector, obtain the geometric transformation matrix of the reference 3D head image fitted to the 2D head image to be processed.

[0048] Specifically, define the projection matrix for projecting the reference 3D head image onto a 2D plane as P, then:

[0049]

[0050] where n is the near clipping plane position parameter, f is the far clipping plane position parameter, θ is the field of view angle position parameter, and aspect is the viewport aspect ratio parameter (image width-to-height ratio).

[0051] After determining the 2D head image to be processed, obtain the Euler angles of the face pose of the 2D head image to be processed, denoted as (α, β, γ), and convert the Euler angles of the face pose into a rotation matrix R:

[0052]

[0053] Based on the above projection matrix and rotation matrix, obtain the scaling factor and translation vector for fitting the reference 3D head image to the 2D head image to be processed.

[0054] For example, use the above projection matrix and rotation matrix to perform 2D projection on two position points in the reference 3D head image, and use the distance between these two position points as a standard, thereby obtaining the scaling factor for fitting the reference 3D head image to the 2D head image to be processed.

[0055] Optionally, for obtaining the translation vector, another position point in the reference 3D head image can be used as a reference point, and the position point corresponding to this reference point in the 2D head image to be processed is back-projected to obtain the 3D coordinates of this position point before projection. Based on the difference between the 3D coordinates and the 3D coordinates of the reference point in the reference 3D head image, the translation vector for fitting the reference 3D head image to the 2D head image to be processed can be obtained.

[0056] Furthermore, based on the rotation matrix, the scaling factor, and the translation vector, obtain the geometric transformation matrix for fitting the reference 3D head image to the 2D head image to be processed. Optionally, the geometric transformation matrix can be the product of the rotation matrix, the scaling factor, and the translation vector.

[0057] In the embodiments of the present application, by performing projection, angle conversion, and fitting on the reference 3D head image and the 2D head image to be processed, the geometric transformation matrix for fitting the reference 3D head image to the 2D head image to be processed is finally obtained, thereby eliminating the need to construct the 3D shape of the human face, effectively reducing the amount of calculation and processing complexity.

[0058] Optionally, the method further includes:

[0059] Perform face key point detection on the to-be-processed 2D head image to obtain a preset number of face key points.

[0060] For example, it can be based on a preset 106-face key point detection model to perform face key point detection on the to-be-processed 2D head image, obtaining 106 2D face key points, denoted as L 2d .

[0061] Optionally, the obtaining of the scaling coefficient for fitting the reference 3D head image to the to-be-processed 2D head image based on the projection matrix and the rotation matrix includes:

[0062] Obtain a first position point and a second position point in the reference 3D head image, where the first position point and the second position point correspond to the positions of two key points among the preset number of face key points;

[0063] Perform 2D projection on the first position point and the second position point based on the projection matrix and the rotation matrix, and obtain a first distance between the first position point and the second position point after 2D projection;

[0064] Obtain the scaling coefficient for fitting the reference 3D head image to the to-be-processed 2D head image based on the first distance.

[0065] In the embodiments of the present application, after obtaining the projection matrix of the reference 3D head image projected onto the 2D plane and the rotation matrix based on the Euler angles of the face pose of the to-be-processed 2D head image, the scaling coefficient for fitting the reference 3D head image to the to-be-processed 2D head image is obtained based on the projection matrix and the rotation matrix.

[0066] Optionally, use the projection matrix and the rotation matrix to perform 2D projection on the left eye pupil center point (corresponding to the 74th point among the 106 face key points, that is, the first position point) and the right eye pupil center point (corresponding to the 77th point among the 106 face key points, that is, the second position point) in the reference 3D head image, and take the inter-pupillary distance (the distance between the left eye pupil center point and the right eye pupil center point) as a standard, and obtain the scaling coefficient S for fitting the reference 3D head image to the to-be-processed 2D head image based on the inter-pupillary distance. The calculation formula of the scaling coefficient S is as follows:

[0067]

[0068] Wherein, P is the above-mentioned projection matrix, R is the above-mentioned rotation matrix, is the 3D coordinate of the 74th point, is the 3D coordinate of the 77th point, is the 2D coordinate of the 74th point, Is the 2D coordinate of point 77.

[0069] In this way, it is also possible to calculate the scaling factor for fitting the reference 3D head image to the 2D head image to be processed by calculating the distance between the first position point and the second position point in the reference 3D head image and combining the above rotation matrix and projection matrix. Furthermore, it is not necessary to construct the 3D shape of the 2D head image, effectively reducing the amount of calculation and complexity.

[0070] Optionally, obtaining the translation vector for fitting the reference 3D head image to the 2D head image to be processed includes:

[0071] Obtain a third position point in the reference 3D head image, where the third position point corresponds to the position of one of the preset number of face key points and is different from the first position point and the second position point;

[0072] Obtain a fourth position point corresponding to the third position point in the 2D head image to be processed, and perform 3D back-projection on the fourth position point to obtain the first 3D coordinate of the third position point in 3D space;

[0073] Obtain the second 3D coordinate corresponding to the third position point in the reference 3D head image;

[0074] Determine the translation vector for fitting the reference 3D head image to the 2D head image to be processed based on the first 3D coordinate and the second 3D coordinate.

[0075] In the embodiment of the present application, after performing face key point detection on the 2D head image to be processed based on a preset 106-face key point detection model and obtaining 106 2D face key points, obtain the fourth position point corresponding to the third position point in the reference 3D head image in the 2D head image to be processed, and perform 3D back-projection on the fourth position point.

[0076] For example, taking the tip center of the nose in the reference 3D head image (corresponding to point 46 in the 106 face key points, that is, the third position point) as a reference point, the 2D coordinate of the tip center position point in the 2D head image to be processed Perform back-projection to obtain the 3D coordinate of this 2D coordinate before projection (in 3D space). Based on the difference between this 3D coordinate and the 3D coordinate of point 46 in the reference 3D head image To obtain the translation vector T for fitting the reference 3D head image to the 2D head image to be processed, the calculation formula of the translation vector T is as follows:

[0077]

[0078] where P is the above projection matrix, is the x-axis coordinate in the 3D coordinates of point No. 46, is the y-axis coordinate in the 3D coordinates of point No. 46, is the z-axis coordinate in the 3D coordinates of point No. 46, is the x-axis coordinate in the 2D coordinates of point No. 46, is the y-axis coordinate in the 2D coordinates of point No. 46, is the z-axis coordinate in the 2D coordinates of point No. 46.

[0079] Further, after calculating the above rotation matrix R, scaling factor S and translation vector T, based on the rotation matrix R, scaling factor S and translation vector T, obtain the geometric transformation matrix M for fitting the reference 3D head image to the 2D head image to be processed, where:

[0080] M = S * R * T

[0081] Further, it is also possible to perform vertex transformation on the reference 3D head image based on the geometric transformation matrix M to obtain the target 3D head image, thereby avoiding the process of 3D face reconstruction and effectively saving computing resources.

[0082] Optionally, based on the above geometric transformation matrix M and projection matrix P, perform geometric transformation on each vertex in the reference 3D head image to obtain the target 3D head image, where the geometric transformation calculation formula for a single vertex is:

[0083] V' = P * M * V

[0084] where V is the vertex in the reference 3D head image, V' is the vertex in the target 3D head image, M is the above geometric transformation matrix, and P is the above projection matrix.

[0085] In the embodiments of the present application, the step of respectively performing primitive assembly on the target 3D head image and the 3D sticker image based on the rendering pipeline to obtain the first primitive corresponding to the target 3D head image and the second primitive corresponding to the 3D sticker image includes:

[0086] Determine the rendering scene according to the image acquisition position of the target 3D head image, and initialize the rendering pipeline;

[0087] Load the target 3D head image into the rendering pipeline, and perform primitive assembly on the target 3D head image based on the rendering pipeline to obtain the first primitive;

[0088] Perform geometric transformation on the 3D sticker image based on the geometric transformation matrix, and load the geometrically transformed 3D sticker image into the rendering pipeline;

[0089] Perform primitive assembly on the geometrically transformed 3D sticker image based on the rendering pipeline to obtain a second primitive.

[0090] Optionally, determining the rendering scene according to the image acquisition position of the target 3D head image may be to place the camera that acquires the target 3D head image in front of the target 3D head image, and use the acquisition field of view of the camera as the rendering scene, and the target 3D head image is located in the rendering scene. After determining the rendering scene, initialize the rendering pipeline, for example, initialize the rendering pipeline based on the open interface program OpenGL ES (OpenGL for Embedded Systems).

[0091] Further, load the target 3D head image into the initialized rendering pipeline, perform primitive assembly on the target 3D head image based on the rendering pipeline to obtain a first primitive corresponding to the target 3D head image. In addition, perform geometric transformation on the 3D sticker image based on the above geometric transformation matrix M, load the relevant rendering resources of the 3D sticker image (such as: lighting, texture mapping, model, etc.), and load the geometrically transformed 3D sticker image and relevant rendering resources into the rendering pipeline, and perform primitive assembly on the geometrically transformed 3D sticker image and relevant rendering resources based on the rendering pipeline to obtain a second primitive corresponding to the 3D sticker image.

[0092] Optionally, performing depth testing and back face culling processing on the rendering pipeline for the first primitive and the second primitive to obtain a first front primitive corresponding to the first primitive and a second front primitive corresponding to the second primitive includes:

[0093] Perform depth testing on the rendering pipeline for the first primitive and the second primitive to obtain a first front primitive corresponding to the first primitive, and a second front primitive and a back face primitive corresponding to the second primitive;

[0094] Perform back face culling processing on the first front primitive, the second front primitive, and the back face primitive based on the rendering pipeline to obtain the first front primitive and the second front primitive.

[0095] In an embodiment of the present application, after performing primitive assembly on the target 3D head image and the 3D sticker image respectively based on a rendering pipeline to obtain a first primitive corresponding to the target 3D head image and a second primitive corresponding to the 3D sticker image, the depth test function of the rendering pipeline is enabled. Since the 3D sticker image is attached to the surface of the target 3D head image, the primitive closest to the camera will be retained based on the depth test function. Furthermore, after performing the depth test, the primitive in the first primitive without facial occlusion (i.e., the first front primitive) can be obtained respectively, and the primitive in the second primitive with facial occlusion (i.e., the reverse primitive) and the primitive without facial occlusion and without 3D sticker image occlusion (i.e., the second front primitive) can be obtained.

[0096] Further, the back face culling function of the rendering pipeline is enabled to perform back face culling on the above-mentioned first front primitive, second front primitive, and reverse primitive. As a result, the primitive with facial occlusion, i.e., the reverse primitive, will be culled, and the remaining first front primitive and second front primitive are obtained.

[0097] In an embodiment of the present application, based on the depth test function and back face culling function of the rendering pipeline, depth test processing and back face culling processing are respectively performed on the first primitive and the second primitive. As a result, the occluded primitives in the first primitive and the second primitive can be culled, thereby realizing the removal of the occluded part in the 3D sticker image.

[0098] Optionally, rasterization processing and shading processing are performed on the first front primitive and the second front primitive, and the processed second front primitive is displayed, including:

[0099] Rasterization processing is performed on the first front primitive and the second front primitive, and shading processing is performed on the rasterized first front primitive and second front primitive based on the shading function and alpha blending function of the rendering pipeline, and the processed second front primitive is displayed.

[0100] In an embodiment of the present application, after culling the occluded reverse primitive in the 3D sticker image, rasterization processing is performed on the remaining first front primitive and second front primitive, and then the pixel results after rasterization of the first front primitive and the second front primitive are respectively shaded in the fragment shader of the rendering pipeline. The transparency value of the first front primitive is set to 0, which means that the first front primitive is a transparent area. As a result, the first front primitive will not be displayed, and finally only the second front primitive is displayed.

[0101] Understandably, the second front primitive is obtained by assembling primitives of the 3D sticker image. That is, only the front primitive of the 3D sticker image is finally displayed. The target 3D head image provides the scene construction for the 3D sticker image. Then, the parts that may be occluded when the 3D sticker image is attached to the target 3D head image can be determined with the help of the target 3D head image, so as to process the occluded parts of the 3D sticker image when it is applied to the 3D head image, and finally only the non-occluded parts are displayed, that is, the cropping of the occluded parts of the 3D sticker image is realized.

[0102] Optionally, rasterizing the first front primitive and the second front primitive, and performing a shading process on the rasterized first front primitive and second front primitive based on the shading function and the alpha blending function of the rendering pipeline, and displaying the processed second front primitive, includes:

[0103] Rasterize the first front primitive and the second front primitive to obtain a first pixel result after rasterizing the first front primitive and a second pixel result after rasterizing the second front primitive;

[0104] Process the first pixel result based on the shader of the rendering pipeline, and the transparency value of the processed first pixel result is 0;

[0105] Perform a display process on the processed first pixel result and the second pixel result based on the alpha blending function of the rendering pipeline;

[0106] Display the second pixel result corresponding to the second front primitive.

[0107] Specifically, rasterize the first front primitive and the second front primitive based on the rendering pipeline to obtain a first pixel result after rasterizing the first front primitive and a second pixel result after rasterizing the second front primitive. Further, different processes are performed on the first pixel result and the second pixel result respectively in the fragment shader of the rendering pipeline. The second pixel result may not be processed, or remains the same after processing. The first pixel result is set to (x, y, z, w), where (x, y, z) is the color value, such as the RGB value, which can take any value, such as white (1, 1, 1), and w is the transparency value, with a value of 0, indicating that the area of the first front primitive corresponding to the first pixel result is a transparent area.

[0108] Further, based on the color mixing model of the rendering pipeline, the blending factor is configured as alpha blending. Based on the principle that after pixel alpha blending in the rendering pipeline, pixels with a transparency value of 0 will not be displayed, the first pixel result corresponding to the first front primitive will not be displayed either. Consequently, only the second pixel result corresponding to the second front primitive will be finally displayed, which is the front primitive part of the 3D sticker image. In this way, the cropping of the occluded part of the 3D sticker image and the display of the unoccluded part are completed.

[0109] The embodiment of the present application provides a fast facial occlusion content cropping solution. Based on the face key point detection technology and combined with the depth test, face culling, and blending technologies of the rendering pipeline, the 3D sticker image of the occluded part of the head is cropped, and then the unoccluded part of the 3D sticker image is displayed. In this way, complex calculation processes such as face reconstruction and mask generation are not required, and the calculation amount is lower and simpler. Therefore, this method can also be used on electronic devices with lower computing capabilities and has a wider application range.

[0110] Please refer to Figure 2 , Figure 2 which is the structural diagram of a 3D image processing device provided by the embodiment of the present application. As Figure 2 shown, the 3D image processing device 200 includes:

[0111] An acquisition module 201, configured to acquire a target 3D head image and a 3D sticker image;

[0112] A primitive assembly module 202, configured to perform primitive assembly on the target 3D head image and the 3D sticker image respectively based on the rendering pipeline to obtain a first primitive corresponding to the target 3D head image and a second primitive corresponding to the 3D sticker image;

[0113] A first processing module 203, configured to perform depth test and face culling processing of the rendering pipeline on the first primitive and the second primitive to obtain a first front primitive corresponding to the first primitive and a second front primitive corresponding to the second primitive;

[0114] A second processing module 204, configured to perform rasterization processing and shading processing on the first front primitive and the second front primitive, and display the processed second front primitive.

[0115] Optionally, the acquisition module 201 includes:

[0116] An acquisition unit, configured to acquire a to-be-processed 2D head image and a reference 3D head image, and acquire a geometric transformation matrix for fitting the reference 3D head image to the 2D head image;

[0117] A transformation unit, configured to perform vertex transformation on the reference 3D head image based on the geometric transformation matrix to obtain the target 3D head image.

[0118] Optionally, the primitive assembly module 202 is further configured to:

[0119] Determine a rendering scene according to the image acquisition position of the target 3D head image, and initialize a rendering pipeline;

[0120] Load the target 3D head image into the rendering pipeline, and perform primitive assembly on the target 3D head image based on the rendering pipeline to obtain a first primitive;

[0121] Perform geometric transformation on the 3D sticker image based on the geometric transformation matrix, and load the geometrically transformed 3D sticker image into the rendering pipeline;

[0122] Perform primitive assembly on the geometrically transformed 3D sticker image based on the rendering pipeline to obtain a second primitive.

[0123] Optionally, the first processing module 203 is further configured to:

[0124] Perform depth testing on the rendering pipeline for the first primitive and the second primitive to obtain a first front primitive corresponding to the first primitive, and a second front primitive and a back primitive corresponding to the second primitive;

[0125] Perform back face culling on the first front primitive, the second front primitive, and the back primitive based on the rendering pipeline to obtain the first front primitive and the second front primitive.

[0126] Optionally, the second processing module 204 is further configured to:

[0127] Perform rasterization on the first front primitive and the second front primitive, and perform shading processing on the rasterized first front primitive and second front primitive based on the shading function and the alpha blending function of the rendering pipeline, and display the processed second front primitive.

[0128] Optionally, the second processing module 204 is further configured to:

[0129] Perform rasterization on the first front primitive and the second front primitive, and obtain a first pixel result after rasterization of the first front primitive and a second pixel result after rasterization of the second front primitive;

[0130] The shader based on the rendering pipeline processes the first pixel result, and the transparency value of the processed first pixel result is 0;

[0131] Based on the alpha blending function of the rendering pipeline, perform display processing on the processed first pixel result and the second pixel result;

[0132] Display the second pixel result corresponding to the second front primitive.

[0133] Optionally, the obtaining unit is further configured to:

[0134] Obtain a reference 3D head image, and obtain a projection matrix for projecting the reference 3D head image onto a 2D plane;

[0135] Obtain a 2D head image to be processed, and obtain the face pose Euler angles of the 2D head image to be processed, and convert the face pose Euler angles into a rotation matrix;

[0136] Based on the projection matrix and the rotation matrix, obtain a scaling coefficient and a translation vector for fitting the reference 3D head image to the 2D head image to be processed;

[0137] Based on the rotation matrix, the scaling coefficient, and the translation vector, obtain a geometric transformation matrix for fitting the reference 3D head image to the 2D head image to be processed.

[0138] Optionally, the obtaining unit is further configured to:

[0139] Perform face key point detection on the 2D head image to be processed to obtain a preset number of face key points;

[0140] Obtain a first position point and a second position point in the reference 3D head image, where the first position point and the second position point correspond to the positions of two key points among the preset number of face key points;

[0141] Perform 2D projection on the first position point and the second position point based on the projection matrix and the rotation matrix, and obtain a first distance between the first position point and the second position point after 2D projection;

[0142] Obtain a scaling coefficient for fitting the reference 3D head image to the 2D head image to be processed based on the first distance.

[0143] Optionally, the obtaining unit is further configured to:

[0144] Obtain a third position point in the reference 3D head image, where the third position point corresponds to the position of one key point among the preset number of face key points and is different from the first position point and the second position point;

[0145] Obtain a fourth position point corresponding to the third position point in the to-be-processed 2D head image, and perform 3D inverse projection on the fourth position point to obtain the first 3D coordinate of the third position point in 3D space;

[0146] Obtain the second 3D coordinate corresponding to the third position point in the reference 3D head image;

[0147] Determine the translation vector for fitting the reference 3D head image to the to-be-processed 2D head image based on the first 3D coordinate and the second 3D coordinate.

[0148] The 3D image processing device 200 provided in this embodiment constructs a scene for the 3D sticker image through the target 3D head image, and then determines the parts that may be blocked when the 3D sticker image is attached to the target 3D head image by means of the target 3D head image, so as to implement the processing of the blocked parts of the 3D sticker image when applied to the 3D head image, and finally only display the parts that will not be blocked, that is, the cropping of the blocked parts of the 3D sticker image is realized. The solution provided in the embodiment of the present application does not require complex calculation processes such as face reconstruction and mask generation. By utilizing the occlusion processing advantage of the rendering pipeline, the processing of the blocked parts of the 3D sticker image can be quickly realized, and the calculation amount is lower and simpler.

[0149] The 3D image processing device 200 in the embodiment of the present application may be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal or other devices other than the terminal. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a laptop computer, a handheld computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., which is not specifically limited in the embodiment of the present application.

[0150] The 3D image processing device 200 in the embodiments of the present application may be a device with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

[0151] The 3D image processing device 200 provided in the embodiments of the present application can implement Figure 1 each process implemented in the method embodiments. To avoid repetition, details are not described herein again.

[0152] The embodiments of the present application also provide an electronic device. Please refer to Figure 3 , Figure 3 FIG. is a structural diagram of an electronic device provided in the embodiments of the present application. As Figure 3 shown, the electronic device includes: a processor 300, a memory 320, and a program or instruction stored in the memory 320 and executable on the processor 300. The processor 300 is configured to read the program or instruction in the memory 320. The electronic device further includes a bus interface and a transceiver 310.

[0153] The transceiver 310 is configured to receive and send data under the control of the processor 300.

[0154] Among them, in Figure 3 the bus architecture may include any number of interconnected buses and bridges. Specifically, various circuits represented by one or more processors represented by the processor 300 and the memory represented by the memory 320 are linked together. The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art. Therefore, further description thereof is not provided herein. The bus interface provides an interface. The transceiver 310 may be multiple components, that is, including a transmitter and a transceiver, and provides a unit for communicating with various other devices on the transmission medium. The processor 300 is responsible for managing the bus architecture and general processing, and the memory 320 can store data used by the processor 300 when performing operations.

[0155] Among them, the processor 300 is configured to read the program or instruction in the memory 320 and perform the following steps:

[0156] Obtain a target 3D head image and a 3D sticker image;

[0157] Based on a rendering pipeline, perform primitive assembly on the target 3D head image and the 3D sticker image respectively to obtain a first primitive corresponding to the target 3D head image and a second primitive corresponding to the 3D sticker image;

[0158] Perform depth testing and back face culling on the first primitive and the second primitive through a rendering pipeline to obtain a first front primitive corresponding to the first primitive and a second front primitive corresponding to the second primitive;

[0159] Perform rasterization and shading on the first front primitive and the second front primitive, and display the processed second front primitive.

[0160] Optionally, the processor 300 is further configured to read a program or instruction in the memory 320 and execute Figure 1 each step in the method embodiments, which will not be elaborated in the embodiments of the present application.

[0161] In the embodiments of the present application, the electronic device constructs a scene for the 3D sticker image through the target 3D head image, and then determines the parts that may be occluded when the 3D sticker image is attached to the target 3D head image by means of the target 3D head image, so as to implement the processing of the occluded parts of the 3D sticker image when applied to the 3D head image, and finally only display the parts that will not be occluded, that is, the cropping of the occluded parts of the 3D sticker image is realized. Furthermore, the electronic device does not need to perform complex calculation processes such as face reconstruction and mask generation. By utilizing the occlusion processing advantages of the rendering pipeline, the processing of the occluded parts of the 3D sticker image can be quickly realized, and the calculation amount is lower and simpler.

[0162] The embodiments of the present application further provide a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements the above Figure 1 each process of the method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0163] Wherein, the processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc, etc.

[0164] The embodiments of the present application further provide a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is configured to run a program or instruction to implement the above Figure 1 each process of the method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0165] It should be understood that the chip mentioned in the embodiments of the present application may also be referred to as a system-on-chip, a system chip, a chip system, or a system-on-chip, etc.

[0166] It should be noted that, in this text, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising one..." does not exclude the presence of additional identical elements in the process, method, article or device comprising such element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, the features described with reference to certain examples may be combined in other examples.

[0167] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such understanding, the technical solution of the present application, in essence or the part that contributes to the prior art, can be embodied in the form of a computer software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present application.

[0168] The embodiments of the present application have been described above with reference to the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Those of ordinary skill in the art, under the inspiration of the present application and without departing from the spirit of the present application and the scope protected by the claims, can still make many forms, all of which fall within the protection scope of the present application.

Claims

1. A 3D image processing method, characterized in that, Including: Obtain a target 3D head image and a 3D sticker image; Based on a rendering pipeline, perform primitive assembly on the target 3D head image and the 3D sticker image respectively to obtain a first primitive corresponding to the target 3D head image and a second primitive corresponding to the 3D sticker image; Perform depth testing and back face culling processing of the rendering pipeline on the first primitive and the second primitive to obtain a first front primitive corresponding to the first primitive and a second front primitive corresponding to the second primitive; Perform rasterization processing and shading processing on the first front primitive and the second front primitive, and display the processed second front primitive; Among them, the performing rasterization processing and shading processing on the first front primitive and the second front primitive, and displaying the processed second front primitive includes: Perform rasterization processing on the first front primitive and the second front primitive to obtain a first pixel result after rasterization processing of the first front primitive and a second pixel result after rasterization processing of the second front primitive; Process the first pixel result based on the shader of the rendering pipeline, and the transparency value of the processed first pixel result is 0; Perform display processing on the processed first pixel result and the second pixel result based on the alpha blending function of the rendering pipeline; Display the second pixel result corresponding to the second front primitive.

2. The method according to claim 1, wherein The obtaining the target 3D head image includes: Obtain a to-be-processed 2D head image and a reference 3D head image, and obtain a geometric transformation matrix for fitting the reference 3D head image to the 2D head image; Perform vertex transformation on the reference 3D head image based on the geometric transformation matrix to obtain the target 3D head image.

3. The method according to claim 2, wherein The performing primitive assembly on the target 3D head image and the 3D sticker image respectively based on the rendering pipeline to obtain a first primitive corresponding to the target 3D head image and a second primitive corresponding to the 3D sticker image includes: Determine a rendering scene according to the image acquisition position of the target 3D head image, and initialize the rendering pipeline; Load the target 3D head image into the rendering pipeline, and perform primitive assembly on the target 3D head image based on the rendering pipeline to obtain a first primitive; Perform geometric transformation on the 3D sticker image based on the geometric transformation matrix, and load the geometrically transformed 3D sticker image into the rendering pipeline; Perform primitive assembly on the geometrically transformed 3D sticker image based on the rendering pipeline to obtain a second primitive.

4. The method according to claim 1, wherein The performing depth testing and back face culling processing of the rendering pipeline on the first primitive and the second primitive to obtain a first front primitive corresponding to the first primitive and a second front primitive corresponding to the second primitive includes: Perform depth testing of the rendering pipeline on the first primitive and the second primitive to obtain a first front primitive corresponding to the first primitive, and a second front primitive and a back face primitive corresponding to the second primitive; Performing back-face culling on the first front primitive, the second front primitive, and the back primitive based on the rendering pipeline to obtain the first front primitive and the second front primitive.

5. The method according to claim 2, wherein The obtaining of the to-be-processed 2D head image and the reference 3D head image, and the obtaining of the geometric transformation matrix for fitting the reference 3D head image to the 2D head image includes: Obtaining a reference 3D head image, and obtaining a projection matrix for projecting the reference 3D head image onto a 2D plane; Obtaining a to-be-processed 2D head image, and obtaining the Euler angles of the face pose of the to-be-processed 2D head image, and converting the Euler angles of the face pose into a rotation matrix; Based on the projection matrix and the rotation matrix, obtaining a scaling factor and a translation vector for fitting the reference 3D head image to the to-be-processed 2D head image; Based on the rotation matrix, the scaling factor, and the translation vector, obtaining a geometric transformation matrix for fitting the reference 3D head image to the to-be-processed 2D head image.

6. The method according to claim 5, wherein The method further includes: Performing face key point detection on the to-be-processed 2D head image to obtain a preset number of face key points; The obtaining of the scaling factor for fitting the reference 3D head image to the to-be-processed 2D head image based on the projection matrix and the rotation matrix includes: Obtaining a first position point and a second position point in the reference 3D head image, where the first position point and the second position point correspond to the positions of two key points among the preset number of face key points; Performing 2D projection on the first position point and the second position point based on the projection matrix and the rotation matrix, and obtaining a first distance between the first position point and the second position point after 2D projection; Obtaining a scaling factor for fitting the reference 3D head image to the to-be-processed 2D head image based on the first distance.

7. The method according to claim 6, characterized in that, The obtaining of the translation vector for fitting the reference 3D head image to the to-be-processed 2D head image includes: Obtaining a third position point in the reference 3D head image, where the third position point corresponds to the position of one key point among the preset number of face key points and is different from the first position point and the second position point; Obtaining a fourth position point corresponding to the third position point in the to-be-processed 2D head image, and performing 3D inverse projection on the fourth position point to obtain a first 3D coordinate of the third position point in 3D space; Obtaining a second 3D coordinate corresponding to the third position point in the reference 3D head image; Determining a translation vector for fitting the reference 3D head image to the to-be-processed 2D head image based on the first 3D coordinate and the second 3D coordinate.

8. A 3D image processing device, characterized in that, Includes: An obtaining module, configured to obtain a target 3D head image and a 3D sticker image; A primitive assembly module, configured to perform primitive assembly on the target 3D head image and the 3D sticker image respectively based on a rendering pipeline to obtain a first primitive corresponding to the target 3D head image and a second primitive corresponding to the 3D sticker image; A first processing module, configured to perform depth testing and back face culling processing on the rendering pipeline for the first primitive and the second primitive, to obtain a first front face primitive corresponding to the first primitive and a second front face primitive corresponding to the second primitive; A second processing module, configured to perform rasterization processing and shading processing on the first front face primitive and the second front face primitive, and display the processed second front face primitive; Wherein, the second processing module is further configured to: Perform rasterization processing on the first front face primitive and the second front face primitive, to obtain a first pixel result after rasterization processing of the first front face primitive and a second pixel result after rasterization processing of the second front face primitive; Process the first pixel result based on the shader of the rendering pipeline, and the transparency value of the processed first pixel result is 0; Perform display processing on the processed first pixel result and the second pixel result based on the alpha blending function of the rendering pipeline; Display the second pixel result corresponding to the second front face primitive.

9. An electronic device, characterized in that, Comprising a processor and a memory, the memory stores programs or instructions that can run on the processor, and when the programs or instructions are executed by the processor, the steps of the 3D image processing method according to any one of claims 1-7 are implemented.

10. A readable storage medium, characterized in that, Programs or instructions are stored on the readable storage medium, and when the programs or instructions are executed by a processor, the steps of the 3D image processing method according to any one of claims 1-7 are implemented.

Images