Digital portrait processing method and device

Abstract

The embodiment of the invention provides a digital portrait processing method and device, and the method comprises the steps: obtaining the human face image data of a target user, carrying out the target detection of the human face image data, obtaining a target detection result which comprises the coordinate data of human face key points, and carrying out the processing of a digital portrait according to the coordinate data of the human face key points. The method comprises the following steps: acquiring face image data of a target user, performing head posture detection, and when a head posture detection result is a specified head posture, generating a digital portrait of the target user according to the face image data so as to present the digital portrait in an intelligent cabin, thereby realizing head posture detection in a cabin scene so as to generate and present the digital portrait. The accuracy and reliability of user head posture detection are ensured, and the duplicating effect of the digital human image in the cabin scene is improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent cockpit interaction, and in particular to a method and device for processing digital portraits. Background Art

[0002] In cockpit interaction scenarios, the reproduction of digital human images provides an innovative solution to improve the driving experience and the quality of human-computer interaction. High-quality facial photos are the basis for accurate digital human reproduction. However, in existing technologies, even after obtaining high-quality facial photos, the reproduction effect of digital human images in cockpit scenarios is still poor. Summary of the invention

[0003] In view of the above problems, a method and device for processing a digital portrait is proposed to overcome the above problems or at least partially solve the above problems, including:

[0004] Obtain facial image data of the target user;

[0005] Performing target detection on the face image data to obtain a target detection result; wherein the target detection result includes coordinate data of key points of the face;

[0006] Performing head posture detection according to the coordinate data of the key points of the face;

[0007] When the result of the head posture detection is a designated head posture, a digital portrait of the target user is generated according to the facial image data, so as to present the digital portrait in the smart cockpit.

[0008] Optionally, performing head posture detection according to the coordinate data of the key points of the face includes:

[0009] The facial orientation of the target user is detected according to the coordinate data of the facial key points.

[0010] Optionally, the detecting the facial orientation of the target user according to the coordinate data of the facial key points includes:

[0011] According to the coordinate data of the facial key points, determine the foot point of the perpendicular from the nose tip key point to the line between the two eyes; wherein the line between the two eyes is the line between the left eye key point and the right eye key point;

[0012] According to the vertical foot point, the face of the target user is detected from the front or side, and whether the user is looking up or looking down.

[0013] Optionally, the detecting the face of the target user from the front or side according to the perpendicular foot point includes:

[0014] Determine a first distance from the nose tip key point to the perpendicular foot point according to the perpendicular foot point;

[0015] Determining a second distance between the two eyes according to the connecting line between the two eyes;

[0016] A detection result of whether the face of the target user is in the front or side is determined according to the first distance and the second distance.

[0017] Optionally, determining, according to the first distance and the second distance, a detection result of whether the face of the target user is in the front or side view includes:

[0018] When the ratio of the first distance to the second distance is greater than a preset threshold, determining that the face of the target user is at a side view;

[0019] When the ratio of the first distance to the second distance is less than or equal to a preset threshold, it is determined that the face of the target user is in the front.

[0020] Optionally, the detecting, based on the vertical foot point, whether the target user's face is looking up or looking down includes:

[0021] According to the perpendicular foot point, establishing a first straight line equation with the nose tip key point;

[0022] According to the coordinate data of the facial key points, establish a second straight line equation of the left mouth corner key points and the right mouth corner key points, and obtain the intersection of the first straight line equation and the second straight line equation;

[0023] According to the perpendicular foot point, determining a first distance from the nose tip key point to the perpendicular foot point, and a third distance from the intersection point to the perpendicular foot point;

[0024] A detection result of whether the target user's face is lowering the head or raising the head is determined according to the first distance and the third distance.

[0025] Optionally, determining the detection result of whether the target user's face is in the state of lowering the head or raising the head according to the first distance and the third distance includes:

[0026] When the ratio of the first distance to the third distance is not within a preset range, determining that the face of the target user is in a posture of lowering the head or raising the head;

[0027] When the ratio of the first distance to the third distance is within a preset range, it is determined that the face of the target user is not in a posture of lowering the head or raising the head.

[0028] Optionally, performing target detection on the facial image data to obtain a target detection result includes:

[0029] Extracting features from the face image data through a target feature extraction network to obtain target feature data;

[0030] Performing feature fusion on the target feature data through a target feature fusion network to obtain target feature fusion data; wherein the target feature fusion network includes a deconvolution operation and a deep feature extraction network;

[0031] Performing feature enhancement extraction on the target feature fusion data through a target feature enhancement extraction network to obtain target feature enhancement data;

[0032] The data is enhanced according to the target features to obtain a target detection result.

[0033] Optionally, the facial image data of the target user is acquired through a camera in the smart cockpit.

[0034] A digital portrait processing device, comprising:

[0035] A face data acquisition module is used to acquire face image data of a target user;

[0036] A target detection module, used to perform target detection on the face image data to obtain a target detection result; wherein the target detection result includes coordinate data of key points of the face;

[0037] A head posture detection module, used for performing head posture detection based on the coordinate data of the key points of the face;

[0038] The digital human reproduction module is used to generate a digital portrait of the target user according to the facial image data when the result of the head posture detection is a specified head posture, so as to present the digital portrait in the smart cockpit.

[0039] An electronic device includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method for processing user face data as described above when executed by the processor.

[0040] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for processing user face data as described above is implemented.

[0041] The embodiments of the present application have the following advantages:

[0042] In an embodiment of the present application, by acquiring facial image data of a target user, target detection is performed on the facial image data to obtain a target detection result, wherein the target detection result includes coordinate data of facial key points, and head posture detection is performed based on the coordinate data of the facial key points. When the result of the head posture detection is a specified head posture, a digital portrait of the target user is generated based on the facial image data to present the digital portrait in the smart cockpit, thereby realizing head posture detection in a cockpit scene to generate and present a digital human image, which not only ensures the accuracy and reliability of the user's head posture detection, but also improves the reproduction effect of the digital human image in the cockpit scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 is a flowchart of a method for processing a digital portrait provided in some embodiments of the present application;

[0045] Figure 2 is an overall flow chart of a visual target detection model provided by some embodiments of the present application;

[0046] Figure 3 is a flowchart of the steps of performing feature enhancement extraction in a single-stage headless module provided in some embodiments of the present application;

[0047] Figure 4 It is a flowchart of a context module in a single-stage headless module provided by some embodiments of the present application;

[0048] Figure 5 is a schematic diagram of key points of a human face provided by some embodiments of the present application;

[0049] Figure 6 This is a flowchart of steps for digital human reproduction based on face data provided by some embodiments of the present application;

[0050] Figure 7 This is a structural block diagram of a device for detecting drowsiness of a user in a cockpit provided in some embodiments of the present application. DETAILED DESCRIPTION

[0051] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0052] In the process of taking photos for generating digital human images, the determination of head posture is crucial, which directly affects the effect of digital human image reproduction. Only when the head posture is correct can the key features of the face be captured, thus generating a realistic and highly restored digital human image.

[0053] In the related technologies, the main methods for head posture detection tasks include technologies based on conventional detection algorithms, such as YOLO (You Only Look Once, a real-time target detection algorithm), CenterNet (an anchor free target detection algorithm) and Faster R-CNN (Faster Region-based Convolutional Neural Network, a deep learning model for real-time target detection), as well as detection algorithms based on visual Transformer, etc. These methods have achieved good recall and accuracy to a certain extent.

[0054] However, during the actual implementation of the project, when these conventional detection algorithms were deployed on the vehicle side, the accuracy and reliability of judging the user's head posture in the cockpit scenario were insufficient, and the head posture of the user in the smart cockpit could not be detected well, thereby improving the reproduction effect of the digital human image in the cockpit scenario.

[0055] In an embodiment of the present application, target detection can be performed on the photo used to generate the digital human image through a visual target detection model designed for head posture detection, so that the head posture can be detected based on the result of the target detection, and then a photo that meets the specified head posture can be obtained, and a digital human image can be generated based on the photo.

[0056] Reference Figure 1 , shows a flowchart of a method for processing a digital portrait provided by some embodiments of the present application, which is applied to a vehicle terminal and may specifically include the following steps:

[0057] Step 101, obtaining facial image data of a target user.

[0058] In the cockpit interaction scenario, the image reproduction of digital humans provides an innovative solution to improve the driving experience and the quality of human-computer interaction. High-quality facial photos are the basis for accurate digital human reproduction. Therefore, before digital human reproduction, it is necessary to obtain the facial image data of the target user for digital human reproduction.

[0059] In some embodiments of the present application, the facial image data of the target user is acquired through a camera in the smart cockpit.

[0060] In practical applications, the facial image data may be RGB (Red Green Blue) facial image data collected by a camera in a vehicle intelligent cockpit, or facial image data acquired through a mobile device such as a mobile phone or a tablet computer.

[0061] Step 102, performing target detection on the face image data to obtain a target detection result; wherein the target detection result includes coordinate data of key points of the face.

[0062] After acquiring the facial image data of the target user, target detection can be performed on the facial image data using a preset visual target detection model to obtain a target detection result of the facial image data, which may include coordinate data of facial key points.

[0063] In some embodiments of the present application, before performing target detection on the face image data to obtain the target detection result, the following steps may also be included:

[0064] Sub-step 11, scaling the face image data in proportion to a preset ratio, collecting negative sample data, and sampling the negative sample data according to a preset probability to obtain preliminary processed data.

[0065] Before performing target detection on face image data by using a preset visual target detection model and obtaining target detection results, the face image data may be preprocessed. Specifically, the acquired face image data may be scaled proportionally according to a preset ratio, and then negative sample data may be collected and sampled according to a preset probability to obtain preliminary processed data.

[0066] In practical applications, negative sample data can be data in face image data that does not belong to the face category. The data in these images are usually called negative sample data.

[0067] As an example, after a batch of negative sample data is collected, the preset probability may be 0.3, and the batch of negative sample data may be sampled in a dataloader with a probability of 0.3.

[0068] Sub-step 12, performing data enhancement and normalization processing on the preliminary processed data to obtain pre-processed facial image data.

[0069] The preliminary processed data may be subjected to data enhancement and normalization to obtain preprocessed face image data.

[0070] In one example, the data enhancement operation may include adding random noise, Mosiac data enhancement, and ColorJitter operation.

[0071] In some embodiments of the present application, performing target detection on the face image data to obtain a target detection result includes:

[0072] Sub-step 21, extracting features from the face image data through a target feature extraction network to obtain target feature data.

[0073] In order to perform target detection on face image data, a visual target detection model is preset, and target detection can be performed through the visual target detection model. The model may include a target feature extraction network, a target feature fusion network and a target feature enhancement extraction network.

[0074] After acquiring the face image data, the face image data may be first subjected to feature extraction through a target feature extraction network to obtain target feature data. The face image data may be subjected to target detection after being preprocessed.

[0075] Sub-step 22, performing feature fusion on the target feature data through a target feature fusion network to obtain target feature fusion data; wherein the target feature fusion network includes a deconvolution operation and a deep feature extraction network.

[0076] After obtaining the target feature data, the target feature data can be subjected to feature fusion through a target feature fusion network to obtain target feature fusion data. The target feature fusion network can include a deconvolution operation and a deep feature extraction network. The deep feature extraction network can be a state space conversion module based on SSM (State Space Model).

[0077] Sub-step 23, performing feature enhancement extraction on the target feature fusion data through a target feature enhancement extraction network to obtain target feature enhancement data.

[0078] The acquired target feature fusion data is used to perform feature enhancement extraction through the target feature enhancement extraction network to obtain target feature enhancement data.

[0079] Sub-step 24, enhancing the data according to the target features to obtain a target detection result.

[0080] After acquiring the target feature enhancement data, the preset visual target detection model can process the target feature enhancement data to output a target detection result, which may include the coordinate data of the key points of the face.

[0081] Specifically, Figure 2 , take a 480*288*3 cockpit face image data as an example to explain:

[0082] 1. Feature extraction stage

[0083] ODMamba Backbone (a feature extraction module built based on the state space of SSM) is the target feature extraction network. In ODMamba Backbone, there are Simple Stem module, Vision Clue Merge module, SPPF (Spatial Pyramid Pooling-Fast) module and ODSSBlocks (a state space conversion module based on SSM), which is the deep feature extraction network.

[0084] The Simple Stem module is used to preprocess the data, including using convolutional layers with larger strides to reduce the spatial size of the image, thereby reducing the computational burden of subsequent processing, and preliminarily extracting image features.

[0085] The Vision Clue Merge module is used to further fuse and optimize features to ensure that features extracted from different levels can be effectively combined to enhance the ability to detect targets.

[0086] The SPPF module can be used to generate a fixed-size feature vector by performing pooling operations on feature maps at different scales.

[0087] The ODSSBlock module is used for deep feature extraction and applies the SSM structure to the field of target detection, so that the state space model technology can be used to effectively capture features at different levels and process the temporal dynamic information in the image. The ODSSBlock module includes the SS2D (Single Shot2D, single two-dimensional target detection) module, LSBlock (LocalSpatial Block, local spatial block) and RGBlock (ResGated Block, residual gated block).

[0088] The SS2D module can be responsible for preliminary feature conversion and enhancement through its linear layers, layer normalization, scanning operations, etc., and then LSBlock can process specific local spatial features through depthwise separable convolution and batch normalization to help the model better understand the details of the image. RGBlock can then use residual connections and gating mechanisms to further refine and enhance the features extracted by SS2D and LS Block, making the feature flow more stable and effective in the model.

[0089] The face image data is input into ODMamba Backbone to obtain feature maps of different scales, namely, target feature data.

[0090] 2. Multi-scale feature fusion stage

[0091] After obtaining the target feature data, a multi-scale feature fusion module constructed based on FPN (Feature Pyramid Network), namely, a target feature fusion module, is used to perform feature fusion and obtain target feature fusion data.

[0092] In the FPN module, the deconvolution operation (Deconv) and the ODSSBlock module are introduced. The deconvolution operation can better process low-resolution images to better obtain image features.

[0093] 3. Strengthen the feature extraction stage

[0094] After obtaining the target feature fusion data, a convolution operation can be performed on it, and then it can be input into the SSH (Single Stage Headless) module, that is, the target feature enhancement extraction network for feature enhancement extraction, so as to obtain the target feature enhancement data.

[0095] See attached Figure 3 , showing a flow chart of the steps of performing feature enhancement extraction in a single-stage headless module provided in some embodiments of the present application. After the feature map is input, feature extraction can be performed through CBAModule (convolutional attention module) and ContextModule (context module) respectively, and finally the feature maps extracted by the two sub-modules are merged according to the channel dimension.

[0096] The CBAModule module can be composed of a 3*3 convolution kernel with a step size of 1 and a padding pixel of 1, a BN (BatchNormalization), and a RELU (Rectified Linear Unit) activation function.

[0097] See attached Figure 4, shows a flow diagram of the context module in the single-stage headless module provided by some embodiments of the present application. The ContextModule module is designed by four submodules: Inconv module (input convolution module), Upconv module (upper convolution module), Downconv module (down convolution module), and Downconv2 module (down convolution module 2). Each submodule is composed of a 3*3 convolution kernel with a step size of 1 and a fill pixel of 1, BN, and RELU activation function. First, the Inconv module is used to extract the input feature map to obtain a feature map with an output channel number of one-fourth the number of input channels, and then the Upconv module, Downconv module, and Downconv2 module are used for feature extraction, and finally the feature maps are merged according to the channel dimension.

[0098] 4. Target detection result acquisition stage

[0099] After acquiring the target feature enhancement data, the three output heads are decoupled to output the target detection result, which includes the coordinate data of the key points of the face.

[0100] exist Figure 2 In the figure, from left to right, the first output is the Gaussian heat map corresponding to the center of the face target box, the second output is the deviation of the upper left, lower right, and relative box (bounding box) center of the target box, and the third output is the output of the face key points, which outputs the 68 key points corresponding to the face, that is, the coordinate data of the face key points. See Appendix Figure 5 , showing a schematic diagram of facial key points provided in some embodiments of the present application.

[0101] Specifically, in terms of decoupling, regression based on Gaussian heat maps is used to locate the center point of the target box, which provides a basis for subsequent target box positioning and facial key point detection. The deviation of the upper left and lower right parts of the target box relative to the box center can be used to determine the size and position of the target box, ensuring that the target box can accurately cover the target area and provide support for the acquisition of coordinate data of facial key points. Facial key points are used to represent the position of each feature point of the face, that is, coordinate data, which provides important information for tasks such as face recognition, expression analysis, and posture estimation.

[0102] Step 103: Perform head posture detection based on the coordinate data of the facial key points.

[0103] After obtaining the coordinate data of the key points of the face, head posture detection can be performed based on the coordinate data of the key points of the face.

[0104] In some embodiments of the present application, the head posture detection is performed according to the coordinate data of the key points of the face, including:

[0105] The facial orientation of the target user is detected according to the coordinate data of the facial key points.

[0106] In some embodiments of the present application, the detecting of the facial orientation of the target user according to the coordinate data of the facial key points includes:

[0107] Sub-step 31, determining the foot point of the perpendicular from the nose tip key point to the line between the two eyes according to the coordinate data of the facial key points; wherein the line between the two eyes is the line between the left eye key point and the right eye key point.

[0108] Specifically, the coordinate data of the nose tip key point and the coordinate data of the left eye key points and the right eye key points can be obtained from the coordinate data of the acquired facial key points, so that the connecting line between the left eye key point and the right eye key point, that is, the connecting line between the two eyes, can be obtained, and then the foot point of the perpendicular from the nose tip key point to the connecting line between the two eyes can be determined.

[0109] In one example, the line between the two eyes may be a line connecting one of the left eye eye key points and one of the right eye eye key points.

[0110] Sub-step 32, based on the vertical foot point, the face of the target user is detected from the front or side, and when the user is looking up or looking down.

[0111] After the vertical foot point is obtained, the face of the target user can be detected from the front or side, and whether the user is looking up or looking down, based on the vertical foot point.

[0112] In some embodiments of the present application, the detecting the face of the target user from the front or side according to the perpendicular foot point includes:

[0113] Sub-step 41, determining a first distance from the nose tip key point to the perpendicular foot point according to the perpendicular foot point.

[0114] After the perpendicular foot point is determined, the coordinate data of the perpendicular foot point can be obtained, so as to determine the distance from the nose tip key point to the perpendicular foot point, ie, the first distance, based on the coordinate data of the perpendicular foot point and the coordinate data of the nose tip key point.

[0115] Sub-step 42, determining a second distance between the two eyes according to the line between the two eyes.

[0116] Specifically, the coordinate data of the left eye key point and the right eye key point for connection are obtained, and then the distance between the two key points, that is, the second distance, can be determined through the two coordinate data.

[0117] Sub-step 43: determining, based on the first distance and the second distance, a detection result of whether the face of the target user is in the front or side view.

[0118] After obtaining the first distance and the second distance, a ratio of the first distance to the second distance is obtained and normalized, and then a detection result of whether the face of the target user is in the front or side is determined according to the normalized ratio.

[0119] In some embodiments of the present application, determining, according to the first distance and the second distance, a detection result of whether the face of the target user is in the front or side view includes:

[0120] When the ratio of the first distance to the second distance is greater than a preset threshold, it is determined that the face of the target user is at the side.

[0121] When the ratio of the first distance to the second distance is less than or equal to a preset threshold, it is determined that the face of the target user is in the front.

[0122] As an example, a line connecting the left eye corner key points and the right eye corner key points is used as the line between the two eyes, and then the face of the target user is detected from the front or side according to the perpendicular foot point. The calculation method can be as follows:

[0123] We can first obtain the coordinates of the left eye corner key point: p1 = (x1, y1), the right eye corner key point coordinates: p2 = (x2, y2) and the nose tip key point coordinates: p3 = (x3, y3), and assume that the vertical foot point is A = (a x ,a y ), then the coordinate data of the vertical foot point A and the coordinate data of the left eye corner key point, the right eye corner key point and the nose tip key point can satisfy the following formula:

[0124] (x2- x1)(x3-a x ) + (y2- y1)(y3-a y )=0 (1)

[0125]

[0126] a y =k*a x +b (3)

[0127] Among them, k and b in formula (3) are the slope and intercept of the straight line equation y=kx+b determined by the coordinate data of the left and right corners of the eye, and the coordinate data of the foot point can be obtained according to formulas (1), (2) and (3).

[0128] Then, according to the coordinate data of the foot of the perpendicular point, we can obtain the distance from the nose tip key point to the foot of the perpendicular point, ‖A-p3‖, that is, the first distance. The distance from the left eye corner key point to the right eye corner key point is ‖p1-p2‖, so the normalized vertical distance ratio can be dis ftont , the calculation formula is as follows:

[0129]

[0130] After the ratio is obtained by formula (4), if the ratio is greater than a preset threshold, it is determined to be a profile face, otherwise it is determined to be a front face. In this example, the preset threshold can be 0.5, such as dis front >0.5, the target user is judged to be in profile, otherwise the target user is judged to be in front of the face. In practical applications, the preset threshold can be adaptively adjusted accordingly.

[0131] In some embodiments of the present application, the step of detecting whether the target user is looking up or looking down based on the vertical foot point includes:

[0132] Sub-step 51, establishing a first straight line equation with the nose tip key point according to the perpendicular foot point.

[0133] After the foot point of perpendicular is determined, the coordinate data of the foot point of perpendicular can be obtained, so as to establish the equation of the straight line connecting the foot point of perpendicular and the key point of the nose tip, that is, the first straight line equation, through the coordinate data of the foot point of perpendicular and the coordinate data of the key point of the nose tip.

[0134] Sub-step 52, establishing a second straight line equation of the left mouth corner key point and the right mouth corner key point according to the coordinate data of the face key point, and obtaining the intersection of the first straight line equation and the second straight line equation.

[0135] From the coordinate data of the key points of the face, the coordinate data of the left corner of the mouth key points and the right corner of the mouth key points can be obtained, and based on these two coordinate data, the equation of the straight line connecting the left corner of the mouth key points and the right corner of the mouth key points, that is, the second straight line equation, can be obtained, and then the intersection of the first straight line equation and the second straight line equation can be obtained.

[0136] Sub-step 53, determining, based on the perpendicular foot point, a first distance from the nose tip key point to the perpendicular foot point, and a third distance from the intersection point to the perpendicular foot point.

[0137] According to the coordinate data of the foot point of the perpendicular, combined with the coordinate data of the key point of the nose tip, the first distance from the key point to the foot point of the perpendicular can be determined. Similarly, according to the coordinate data of the foot point of the perpendicular, combined with the coordinate data of the intersection of the first straight line equation and the second straight line equation, the third distance from the intersection to the foot point of the perpendicular can be determined.

[0138] Sub-step 54: determining, based on the first distance and the third distance, a detection result of whether the target user's face is lowering or raising its head.

[0139] After the first distance and the third distance are obtained, the detection result of whether the target user's face is lowering or raising the head can be determined according to the ratio of the first distance to the third distance.

[0140] In some embodiments of the present application, determining the detection result of whether the target user's face is in the state of lowering the head or raising the head according to the first distance and the third distance includes:

[0141] When the ratio of the first distance to the third distance is not within a preset range, it is determined that the face of the target user is in a posture of lowering the head or raising the head.

[0142] When the ratio of the first distance to the third distance is within a preset range, it is determined that the face of the target user is not in a posture of lowering the head or raising the head.

[0143] As an example, the coordinate representation of the perpendicular foot point can be obtained by the aforementioned method, which will not be described in detail here. Then, based on the coordinate data of the perpendicular foot point, the face of the target user is detected to be looking up or down. The calculation method can be as follows:

[0144] The first straight line equation y=k1x+c1 can be obtained by first using the coordinate data of the nose tip key point and the coordinate data of the perpendicular point, wherein k1 and c1 are the corresponding slope and intercept calculated using the coordinate data of the nose tip key point p3 and the perpendicular point A.

[0145] Then, the coordinates of the left corner of the mouth key point: p4 = (x4, y4) and the right corner of the mouth key point coordinates: p5 = (x5, y5) can be obtained, so that the equation of the straight line passing through the left corner of the mouth key point p4 and the right corner of the mouth key point p5 can be obtained: y = k2x + c2, where k2 and c2 are the corresponding slope and intercept calculated by the coordinate data of the left corner of the mouth key point p4 and the right corner of the mouth key point p5.

[0146] Furthermore, the coordinates of the intersection point B of the two straight lines can be calculated by the equation of the first straight line and the equation of the second straight line: y B =k1x B +c1. Thus, we can obtain the distance from the nose tip key point to the foot point of the perpendicular ‖p3-A‖, that is, the first distance, and the distance from the intersection of the first straight line equation and the second straight line equation to the foot point of the perpendicular ‖BA‖, that is, the third distance.

[0147] Finally, the normalized ratio dis can be obtained according to the first distance and the third distance bow , the calculation formula is as follows:

[0148]

[0149] The ratio dis is obtained by formula (5): bow After that, if dis bow If the target user is not within the preset range, it is determined that the target user is in a head-up or head-down posture, that is, the face is facing up or down; if the bow If the target user is within the preset range, it is determined that the target user is not in a head-up or head-down posture, that is, the face is not facing up or down. In this example, the preset range can be, that is, if the dis bow If it is not within [0.2,0.7], the target user is judged to be in a head-up or head-down posture; if bow If the value is within [0.2, 0.7], it is determined that the target user is not in the posture of raising the head or lowering the head. In practical applications, the preset range can be adaptively adjusted accordingly.

[0150] It should be noted that in the present application, the detection of the target user's face from the front or side and the detection of the target user's face raising or lowering the head can be performed simultaneously to improve the efficiency of the detection. The user's head posture may be in multiple postures such as lowering the head from the side, raising the head from the side, raising the head from the front, raising the head from the front, not raising the head from the front or lowering the head, and not raising the head from the side or lowering the head.

[0151] Step 104 : When the result of the head posture detection is a specified head posture, a digital portrait of the target user is generated according to the facial image data, so as to present the digital portrait in the smart cockpit.

[0152] After obtaining the result of the head posture detection, when the result of the head posture detection is a specified head posture, a digital portrait of the target user may be generated according to the facial image data to present the digital portrait in the smart cockpit.

[0153] As an example, after obtaining the head posture detection result, that is, performing front or side detection and head-up or head-down detection on the target user's face, after obtaining the corresponding detection results, when the target user is facing forward and is not in a head-up or head-down posture, a digital portrait of the target user is generated based on the acquired facial image data, that is, a digital human image of the target user is replicated to present the digital portrait in the smart cockpit.

[0154] In an embodiment of the present application, by acquiring facial image data of a target user, target detection is performed on the facial image data to obtain a target detection result, wherein the target detection result includes coordinate data of facial key points, and head posture detection is performed based on the coordinate data of the facial key points. When the result of the head posture detection is a specified head posture, a digital portrait of the target user is generated based on the facial image data to present the digital portrait in the smart cockpit, thereby realizing head posture detection in a cockpit scene to generate and present a digital human image, which not only ensures the accuracy and reliability of the user's head posture detection, but also improves the reproduction effect of the digital human image in the cockpit scene.

[0155] Reference Figure 6 , shows a flowchart of steps for digital human reproduction based on face data provided by some embodiments of the present application, which may specifically include the following steps:

[0156] 1. Obtain RGB face image data collected by the camera in the vehicle intelligent cockpit;

[0157] 2. Perform target detection on the acquired face image data;

[0158] 3. Obtain the target detection results, that is, obtain the coordinate data of the key points of the face;

[0159] 4. According to the coordinate data of the key points of the face, the head posture is judged. When the head posture is in the front and not raised or lowered, that is, the face is in the front, step 6 is executed. Otherwise, it is considered that the face image of the target user is not in the front, and step 5 is executed;

[0160] 5. Remind the target user through voice reminder that the facial image data is not in the front view and needs to be re-photographed.

[0161] 6. Create a digital human image of the target user.

[0162] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0163] Reference Figure 7 , shows a schematic diagram of the structure of a digital portrait processing device provided by some embodiments of the present application, which may specifically include the following modules:

[0164] The face data acquisition module 701 is used to acquire the face image data of the target user;

[0165] The target detection module 702 is used to perform target detection on the face image data to obtain a target detection result; wherein the target detection result includes coordinate data of key points of the face;

[0166] A head posture detection module 703, used for performing head posture detection according to the coordinate data of the key points of the face;

[0167] The digital human reproduction module 704 is used to generate a digital portrait of the target user according to the facial image data when the result of the head posture detection is a specified head posture, so as to present the digital portrait in the smart cockpit.

[0168] In one embodiment of the present application, the head posture detection module 703 includes:

[0169] The head posture detection submodule is used to detect the facial orientation of the target user based on the coordinate data of the key points of the face.

[0170] In one embodiment of the present application, the head posture detection submodule includes:

[0171] A perpendicular point acquisition unit, used to determine the perpendicular point from the nose tip key point to the line between the two eyes according to the coordinate data of the face key points; wherein the line between the two eyes is the line between the left eye key point and the right eye key point;

[0172] The face orientation detection unit is used to detect the front or side view of the target user's face and detect whether the user is looking up or looking down according to the vertical foot point.

[0173] In one embodiment of the present application, the facial orientation detection unit includes:

[0174] A first distance acquisition subunit is used to determine a first distance from the nose tip key point to the perpendicular foot point according to the perpendicular foot point;

[0175] A second distance acquisition subunit, configured to determine a second distance between the two eyes according to a line between the two eyes;

[0176] The front or side face determination subunit is used to determine the detection result of whether the face of the target user is in the front or side face according to the first distance and the second distance.

[0177] In an embodiment of the present application, determining the detection result of whether the face of the target user is in the front or side according to the first distance and the second distance includes:

[0178] When the ratio of the first distance to the second distance is greater than a preset threshold, determining that the face of the target user is at a side view;

[0179] When the ratio of the first distance to the second distance is less than or equal to a preset threshold, it is determined that the face of the target user is in the front.

[0180] In one embodiment of the present application, the facial orientation detection unit includes:

[0181] A first straight line equation obtaining subunit is used to establish a first straight line equation with the nose tip key point according to the perpendicular foot point;

[0182] An intersection point acquisition subunit, used to establish a second straight line equation of the left mouth corner key point and the right mouth corner key point according to the coordinate data of the face key point, and obtain the intersection point of the first straight line equation and the second straight line equation;

[0183] A third distance acquisition subunit is used to determine, according to the perpendicular foot point, a first distance from the nose tip key point to the perpendicular foot point, and a third distance from the intersection point to the perpendicular foot point;

[0184] The head-down / head-up determination subunit is used to determine the detection result of whether the target user's face is in the state of lowering the head or raising the head according to the first distance and the third distance.

[0185] In an embodiment of the present application, determining the detection result of whether the face of the target user is lowering or raising the head according to the first distance and the third distance includes:

[0186] When the ratio of the first distance to the third distance is not within a preset range, determining that the face of the target user is in a posture of lowering the head or raising the head;

[0187] When the ratio of the first distance to the third distance is within a preset range, it is determined that the face of the target user is not in a posture of lowering the head or raising the head.

[0188] In one embodiment of the present application, the target detection module 702 includes:

[0189] A feature extraction submodule, used to extract features from the face image data through a target feature extraction network to obtain target feature data;

[0190] A feature fusion submodule, used for performing feature fusion on the target feature data through a target feature fusion network to obtain target feature fusion data; wherein the target feature fusion network includes a deconvolution operation and a deep feature extraction network;

[0191] The enhanced feature extraction submodule is used to perform feature enhanced extraction on the target feature fusion data through a target feature enhanced extraction network to obtain target feature enhanced data;

[0192] The target detection result acquisition submodule is used to enhance the data according to the target features and obtain the target detection result.

[0193] In one embodiment of the present application, the facial image data of the target user is acquired through a camera in the smart cockpit.

[0194] In an embodiment of the present application, by acquiring facial image data of a target user, target detection is performed on the facial image data to obtain a target detection result, wherein the target detection result includes coordinate data of facial key points, and head posture detection is performed based on the coordinate data of the facial key points. When the result of the head posture detection is a specified head posture, a digital portrait of the target user is generated based on the facial image data to present the digital portrait in the smart cockpit, thereby realizing head posture detection in a cockpit scene to generate and present a digital human image, which not only ensures the accuracy and reliability of the user's head posture detection, but also improves the reproduction effect of the digital human image in the cockpit scene.

[0195] Some embodiments of the present application also provide an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and when the computer program is executed by the processor, the above-mentioned digital portrait processing method is implemented.

[0196] Some embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned digital portrait processing method is implemented.

[0197] Some embodiments of the present application further provide a computer program product, including a computer program, which implements the above digital portrait processing method when executed by a processor.

[0198] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0199] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0200] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0201] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0202] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0203] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0204] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0205] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0206] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0207] The above is a detailed introduction to the provided digital portrait processing method and device. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for processing a digital portrait, characterized in that: The method comprises: Obtain facial image data of the target user; Performing target detection on the face image data to obtain a target detection result; wherein the target detection result includes coordinate data of key points of the face; Performing head posture detection according to the coordinate data of the key points of the face; When the result of the head posture detection is a designated head posture, a digital portrait of the target user is generated according to the facial image data, so as to present the digital portrait in the smart cockpit.

2. The method according to claim 1, characterized in that The step of performing head posture detection according to the coordinate data of the key points of the face includes: The facial orientation of the target user is detected according to the coordinate data of the facial key points.

3. The method according to claim 2, characterized in that The detecting of the facial orientation of the target user according to the coordinate data of the facial key points includes: According to the coordinate data of the facial key points, determine the foot point of the perpendicular from the nose tip key point to the line between the two eyes; wherein the line between the two eyes is the line between the left eye key point and the right eye key point; According to the vertical foot point, the face of the target user is detected from the front or side, and whether the user is looking up or looking down.

4. The method according to claim 3, characterized in that The detecting the face of the target user from the front or side according to the vertical foot point includes: Determine a first distance from the nose tip key point to the perpendicular foot point according to the perpendicular foot point; Determining a second distance between the two eyes according to the connecting line between the two eyes; A detection result of whether the face of the target user is in the front or side is determined according to the first distance and the second distance.

5. The method according to claim 4, characterized in that The step of determining, based on the first distance and the second distance, a detection result of whether the face of the target user is in the front or side view includes: When the ratio of the first distance to the second distance is greater than a preset threshold, determining that the face of the target user is at a side view; When the ratio of the first distance to the second distance is less than or equal to a preset threshold, it is determined that the face of the target user is in the front.

6. The method according to claim 3, characterized in that The detecting of raising or lowering the face of the target user according to the vertical foot point includes: According to the perpendicular foot point, establishing a first straight line equation with the nose tip key point; According to the coordinate data of the facial key points, establish a second straight line equation of the left mouth corner key points and the right mouth corner key points, and obtain the intersection of the first straight line equation and the second straight line equation; According to the perpendicular foot point, determining a first distance from the nose tip key point to the perpendicular foot point, and a third distance from the intersection point to the perpendicular foot point; A detection result of whether the target user's face is lowering the head or raising the head is determined according to the first distance and the third distance.

7. The method according to claim 6, characterized in that The step of determining, based on the first distance and the third distance, a detection result of whether the face of the target user is in the state of lowering the head or raising the head comprises: When the ratio of the first distance to the third distance is not within a preset range, determining that the face of the target user is in a posture of lowering the head or raising the head; When the ratio of the first distance to the third distance is within a preset range, it is determined that the face of the target user is not in a posture of lowering the head or raising the head.

8. The method according to any one of claims 1 to 7, characterized in that: The performing target detection on the face image data to obtain a target detection result includes: Extracting features from the face image data through a target feature extraction network to obtain target feature data; Performing feature fusion on the target feature data through a target feature fusion network to obtain target feature fusion data; wherein the target feature fusion network includes a deconvolution operation and a deep feature extraction network; Performing feature enhancement extraction on the target feature fusion data through a target feature enhancement extraction network to obtain target feature enhancement data; The data is enhanced according to the target features to obtain a target detection result.

9. The method according to any one of claims 1 to 7, characterized in that: The facial image data of the target user is acquired through a camera in the smart cockpit.

10. A digital portrait processing device, characterized in that: The device comprises: A face data acquisition module is used to acquire face image data of a target user; A target detection module, used to perform target detection on the face image data to obtain a target detection result; wherein the target detection result includes coordinate data of key points of the face; A head posture detection module, used for performing head posture detection based on the coordinate data of the key points of the face; The digital human reproduction module is used to generate a digital portrait of the target user according to the facial image data when the result of the head posture detection is a specified head posture, so as to present the digital portrait in the smart cockpit.

11. An electronic device, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the digital portrait processing method according to any one of claims 1 to 9 is implemented.

12. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the digital portrait processing method according to any one of claims 1 to 9 is implemented.

Images