Face recognition method, system and equipment based on depth map self-adaption and medium

Through the depth map adaptive method, combined with UNet network and timing characteristics, the problems of incomplete portrait extraction and insufficient stability of video clipping in the prior art are solved, and accurate recognition of human bodies and objects and high-quality image generation are achieved.

CN119068374AActive Publication Date: 2024-12-03SHENZHEN GUANGJIAN TECH CO LTD +2
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202310613442.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-29
Publication Date
2024-12-03
Estimated Expiration
2043-05-29

AI Technical Summary

Technical Problem

The existing cutout technology is difficult to accurately extract portraits and their complete prospects with objects without relying on user guidance information, especially in video processing, with insufficient stability and robustness.

Method used

Adaptive method based on depth maps is adopted, and the UNet network and Skip-Connection with different dimensional features are used, combined with timing characteristics, and the portrait range is identified and adjusted through preset distance parameters and depth value distribution, and the information of weak related objects is eliminated, and high Quality high resolution images.

Benefits of technology

It realizes complete recognition of the human body and interactive objects, improves detection accuracy and stability of video clipping, meets the needs of people and clipping, and generates high-quality images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068374A_ABST
    Figure CN119068374A_ABST
Patent Text Reader

Abstract

The invention discloses a depth map self-adaption-based portrait recognition method. The method comprises the steps of S1, acquiring a first frame image in a video; wherein the first frame image comprises a first RGB image and a first depth image; s2, obtaining a first portrait range on the first depth image by using a first preset distance parameter; s3, on the basis of a UNet network structure and Skip-Consection with different dimension features, in combination with the time sequence features of the first frame image, identifying the first portrait range to obtain a second portrait range; s4, calculating a depth value distribution range of the second portrait range, and obtaining a second preset distance parameter and a third preset distance parameter according to a confidence interval a; and S5, adjusting the second portrait range by adopting the second preset distance parameter and the third preset distance parameter to obtain a third portrait range. According to the invention, the human body and the interaction object can be completely and effectively identified, the high-quality and high-resolution image is generated, and the real-time processing of the video can be realized.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] Image matting (image background removal) refers to extracting the foreground portrait of interest from an image or video (image sequence) while filtering out the background. An image can be simply considered to consist of two parts, namely the foreground and the background. In short, image matting is to distinguish the foreground from the background of a given image. It is one of the key technologies in many image editing. The research on this technology has a history of more than 20 years. The mathematical definition of image background removal was first proposed by foreign scholars Porter and Duff in 1984. They first introduced the concept of transparent (alpha) channel to control the linear interpolation ratio of the foreground and background colors when the foreground and background are merged. In the end, the image matting problem is defined as the task of estimating the alpha value (foreground-background color ratio) of each image pixel. In this task, the input is the original image, and the output will be the alpha value of each pixel. Since the known information is the color information of each pixel, the unknown information is the alpha and foreground and background colors, and the number of unknown variables is greater than the number of known variables, it is a difficult problem with insufficient constraints in mathematics. Most matting methods rely on the user to provide guidance information or prior assumptions to constrain the problem and obtain good estimates of unknown variables. The use of guidance information and prior assumptions is destined to have weak generalization capabilities of the model, and the prediction accuracy will be very poor under certain conditions. Therefore, providing high-confidence prior information in the matting process can significantly improve its effect. A more classic method is an interactive matting technology called Bayesian-based matting, that is, humans participate in this process and manually mark the foreground and background to achieve foreground extraction. In the present invention, there is no need to rely on other information, and the algorithm can independently judge and achieve the separation of foreground and background.

[0003] In addition, currently common cutout algorithms, such as those provided by Tencent Conference and Zoom, can only meet the needs of portrait cutouts, that is, extracting the foreground of the portrait, while other foregrounds associated with the portrait cannot be completely extracted, such as handheld objects, attached tables, computers, etc. For users who have needs such as live broadcasts and displaying items in meetings, the existing methods cannot meet expectations. However, the present invention can extract the foreground of the portrait or the portrait plus the object based on the depth map according to user needs on the basis of completely extracting the foreground of the portrait. The depth image is also called a range image, which refers to an image that uses the distance (depth) from the image collector to each point in the scene as the pixel value, which directly reflects the geometry of the visible surface of the scene.

[0004] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0005] To this end, the present invention uses a first preset distance parameter to obtain a first portrait range, and then uses the UNet network and Skip-Connection of different dimensional features combined with timing features to identify a second portrait image, and adaptively adjusts the second portrait range according to the depth distribution. It can completely and effectively identify the human body and interactive objects, eliminate weakly related object information, and generate high-quality and high-resolution images, which can realize real-time processing of videos.

[0006] In a first aspect, the present invention provides a method for portrait recognition based on depth map adaptation, characterized in that it comprises the following steps:

[0007] Step S1: Acquire a first frame image in a video; wherein the first frame image includes a first RGB image and a first depth image;

[0008] Step S2: obtaining a first portrait range on the first depth image using a first preset distance parameter;

[0009] Step S3: Based on the UNet network structure and Skip-Connection of different dimensional features, combined with the temporal features of the first frame image, the first portrait range is identified to obtain the second portrait range;

[0010] Step S4: calculating the depth value distribution range of the second portrait range, and obtaining a second preset distance parameter and a third preset distance parameter according to the confidence interval a;

[0011] Step S5: using the second preset distance parameter and the third preset distance parameter to adjust the second portrait range to obtain a third portrait range.

[0012] Optionally, the method for portrait recognition based on depth map adaptation is characterized in that step S2 comprises:

[0013] Step S21: filtering the first depth image to obtain a second depth image;

[0014] Step S22: extracting a first portrait range from the second depth image according to a first preset distance parameter;

[0015] Step S23: performing an erosion operation on the first portrait range, and marking the eroded area as 1, the eroded area as 0.5, and the other areas as 0.

[0016] Optionally, the portrait recognition method based on depth map adaptation is characterized in that the erosion operation is determined according to a pixel offset between the second depth map and the first RGB image.

[0017] Optionally, the method for portrait recognition based on depth map adaptation is characterized in that step S3 comprises:

[0018] Step S31: using an encoder to extract features and condense semantic information from the first portrait range;

[0019] Step S32: using a decoder to restore image pixels;

[0020] Step S33: using Skip-Connection with different dimensional features to supplement information;

[0021] Step S34: iterating through the video timing correlation loss function to obtain a second portrait range.

[0022] Optionally, the depth map adaptive portrait recognition method is characterized in that, in step S32, the decoder includes a GRU for inter-frame information transmission.

[0023] Optionally, the depth map-based adaptive portrait recognition method is characterized by further comprising:

[0024] Step S6: identifying the hair region of the third portrait range, and adjusting the hair region according to the time sequence feature.

[0025] Optionally, the method for portrait recognition based on depth map adaptation is characterized in that step S6 comprises:

[0026] Step S61: identifying the hair area of ​​the third portrait range and classifying the hair types;

[0027] Step S62: selecting a motion trajectory for the hair according to the hair type;

[0028] Step S63: adjusting the hair area according to the motion trajectory and the time sequence characteristics.

[0029] In a second aspect, the present invention provides a depth map-based adaptive portrait recognition system, which is used to implement any of the depth map-based adaptive portrait recognition methods described above, and is characterized by comprising:

[0030] An acquisition module, used to acquire a first frame image in a video; wherein the first frame image includes a first RGB image and a first depth image;

[0031] A preprocessing module, configured to obtain a first portrait range on the first depth image using a first preset distance parameter;

[0032] A cutout module is used to identify the first portrait range based on the UNet network structure and Skip-Connection of different dimensional features, combined with the temporal features of the first frame image, to obtain a second portrait range;

[0033] A confidence module, used to calculate the depth value distribution range of the second portrait range, and obtain a second preset distance parameter and a third preset distance parameter according to a confidence interval a;

[0034] The adjustment module is used to adjust the second portrait range by using the second preset distance parameter and the third preset distance parameter to obtain a third portrait range.

[0035] In a third aspect, the present invention provides a portrait recognition device based on depth map adaptation, characterized in that it includes:

[0036] processor;

[0037] a memory storing executable instructions of the processor;

[0038] The processor is configured to execute any one of the steps of the above-mentioned depth map-based adaptive portrait recognition method by executing the executable instructions.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium for storing a program, characterized in that when the program is executed, the steps of any one of the aforementioned depth map-based adaptive portrait recognition methods are implemented.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention uses the first preset distance parameter reference image for preliminary processing, which greatly reduces the interference of background information, reduces the amount of data processing, is conducive to improving detection accuracy, and improves efficiency.

[0042] The present invention improves the stability of video cutout through the temporal characteristics of the image, and integrates the human segmentation function. Through the multi-task training strategy, the cutout algorithm can take into account the integrity and robustness of the human foreground while achieving refined foreground prediction. It can realize real-time foreground cutout based on RGB images and depth images, and meet the needs of cutout of people and objects at the same time.

[0043] The present invention analyzes the depth value distribution of the second portrait range, adjusts the data using the confidence of the depth value, obtains the final cut-out result, effectively removes abnormal data interference such as weakly associated interactive objects and flying points, and improves the cut-out quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings in the following descriptions are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:

[0045] Figure 1 A flowchart of a method for portrait recognition based on depth map adaptation in an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of a depth value distribution range in an embodiment of the present invention;

[0047] Figure 3 This is another schematic diagram of the depth value distribution range in an embodiment of the present invention;

[0048] Figure 4 This is a flow chart of steps for obtaining a first portrait range in an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram of a portrait range in an embodiment of the present invention;

[0050] Figure 6 is a flow chart of steps for obtaining a second portrait range in an embodiment of the present invention;

[0051] Figure 7 This is a flowchart of another method for portrait recognition based on depth map adaptation in an embodiment of the present invention;

[0052] Figure 8 A schematic diagram of a hair region in an embodiment of the present invention;

[0053] Fig. 9 This is a flow chart of steps for adjusting a hair area in an embodiment of the present invention;

[0054] Fig.10 Schematic diagram of the structure of a depth map-based adaptive portrait recognition system in an embodiment of the present invention;

[0055] Fig.11 is a structural schematic diagram of a depth map-based adaptive portrait recognition device in an embodiment of the present invention; and

[0056] Fig.12 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0058] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0059] An embodiment of the present invention provides a depth map-based adaptive portrait recognition method, which aims to solve the problems existing in the prior art.

[0060] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0061] The present invention uses a first preset distance parameter to obtain a first portrait range, and then uses the UNet network and Skip-Connection of different dimensional features combined with time series features to identify a second portrait image, and adaptively adjusts the second portrait range according to the depth distribution. It can completely and effectively identify the human body and interactive objects, eliminate weakly related object information, and generate high-quality and high-resolution images, which can realize real-time processing of videos.

[0062] Figure 1 The present invention is a flowchart of a method for portrait recognition based on depth map adaptation in an embodiment of the present invention.

[0063] like Figure 1 As shown, the steps of a depth map-based adaptive portrait recognition method in an embodiment of the present invention include:

[0064] Step S1: Get the first frame image in the video.

[0065] In this step, the first frame image includes a first RGB image and a first depth image. In some application scenarios, the first RGB image and the first depth image will be described as RGBD images, which have the same function as in this embodiment. The first RGB image and the first depth image in this step are pixel-aligned, and describing them separately is to more clearly illustrate the processing process of the image data. The first frame image is acquired by taking a depth camera. Subsequent steps are performed on the mask to obtain the final portrait area range. Figure 5 a is an RGB image, Figure 5 b is the depth image.

[0066] Step S2: obtaining a first portrait range on the first depth image using a first preset distance parameter.

[0067] In this step, the first preset distance parameter is a constant and is used to extract the effective area. The range less than the first preset distance parameter is the first portrait range. For example, if the first preset distance parameter is 1.2m, all pixels less than 1.2m constitute the first portrait range. For the same video, the first preset distance parameter is a fixed value. For different videos, the first preset distance parameter can be different values. The setting of the first preset distance parameter can be selected according to the depth camera and the specific application scenario.

[0068] Step S3: Based on the UNet network structure and Skip-Connection of different dimensional features, combined with the temporal features of the first frame image, the first portrait range is identified to obtain the second portrait range.

[0069] In this step, the UNet structure includes three components: encoder, decoder and Skip-Connection in the same layer. The encoder performs downsampling. For example, the input image size is 572*572, the feature map size after two rounds of 3*3 convolution is 568*568, and the output size after 22 maximum pooling is 284*284. The decoder performs upsampling. For example, the decoder is composed of 4 groups of 2*2 transposed convolutions, 3*3 convolutions and a ReLU activation function, and a 1*1 convolution is added to the final output layer. Finally, the Skip-Connection in the same layer is to crop the output of each layer during downsampling and connect it to the upsampling layer of the same layer for fusion. This step also processes the image in combination with the temporal features of the first frame image to ensure the stability of the portrait recognition between different frames. To some extent, the temporal features can also be regarded as a kind of deep supervision. The second portrait range identified in this step is the range of the human body, including the head, body, clothes, etc., and includes objects that interact with the human body.

[0070] Step S4: Calculate the depth value distribution range of the second portrait range, and obtain the second preset distance parameter and the third preset distance parameter according to the confidence interval a.

[0071] In this step, the depth value is analyzed and processed to obtain the second preset distance parameter and the third preset distance parameter. Figure 2 As shown, the depth range where the target object exists can be intuitively observed through the depth value distribution. Figure 2 The medium depth value of 0 is the result of resetting the relevant flying points, noise, etc. Figure 2 yes Figure 5 Distribution of depth values ​​in b.

[0072] Figure 3 yes Figure 5 The depth distribution diagram in d. Figure 3As shown, after resetting the background area (depth value of 1500-2500) to 0, the number of pixels with a depth value of 0 increases significantly, and the depth value range of the second portrait range is (300, 600). Due to the characteristics of the portrait, the depth value is continuously distributed. The second preset distance parameter is located on the left side of the depth value range of the second portrait range, and the third preset distance parameter is located on the right side of the depth value range of the second portrait range. The two endpoints of the confidence interval a are the second preset distance parameter and the third preset distance parameter, respectively. The confidence level of the confidence interval a is not less than 90%. The confidence interval a is not fixed, but is determined comprehensively based on the continuity of the depth value distribution and the number of depth values. On the depth value distribution graph, the closer the depth value is to the two sides, the smaller the confidence level is; the confidence level of the part that is discontinuous with the middle depth value is even smaller; the fewer the pixels with the same depth value, the smaller the confidence level is. When determining the confidence interval a, the continuity of the depth value distribution is maintained, and the confidence level is made as large as possible.

[0073] Step S5: using the second preset distance parameter and the third preset distance parameter to adjust the second portrait range to obtain a third portrait range.

[0074] In this step, different from the screening performed in step S2, this step resets the area within the partial depth value range to optimize the second portrait range and obtain the third portrait range. In this step, the pixel points whose depth values ​​are located at the second preset distance parameter and the third preset distance parameter are set to "display", that is, set to 1 on the mask. The third portrait range is larger than the second portrait range.

[0075] Figure 4 FIG. 1 is a flow chart of steps for obtaining a first portrait range in an embodiment of the present invention. Figure 4 As shown, in an embodiment of the present invention, a step of obtaining a first portrait range includes:

[0076] Step S21: performing filtering processing on the first depth image to obtain a second depth image.

[0077] In this step, the filtering process includes processes such as flicker removal, flying spot removal, and missing area filling. Figure 5 b is the TOF depth map obtained by the 3D camera. As can be seen from the figure, there are flying spots and missing areas in the depth map. The differences between different frame images will cause flickering and other phenomena. This step can improve the quality of the depth image and ensure the integrity of the image content by processing the single frame image and the previous image, so that the extraction of the foreground is more complete.

[0078] Step S22: extracting a first portrait range from the second depth image according to a first preset distance parameter.

[0079] In this step, the first preset distance parameter is a constant. By comparing the depth value of the pixel point with the preset distance parameter, all the pixels whose depth values ​​are less than the preset distance parameter can be obtained, that is, the first portrait range. The first portrait range is the set of all pixels whose depth values ​​are less than the preset distance parameter.

[0080] Step S23: performing an erosion operation on the first portrait range, and marking the eroded area as 1, the eroded area as 0.5, and the other areas as 0.

[0081] In this step, the image is marked in three categories according to the first portrait range. The first portrait range is eroded to obtain the core area, which is marked as 1, and has a high credibility of being the foreground. The eroded part of the first portrait range is marked as 0.5, and has a low credibility of being the foreground. The other areas are marked as 0, which are the background. The part marked as 0.5 may be either the foreground or the background, and needs to be judged in the subsequent steps.

[0082] In some embodiments, the erosion operation is determined according to the pixel offset between the second depth map and the first RGB image. Different from the erosion operation using a convolutional template for a single RGB image in the prior art, this embodiment uses the offset between the depth map and the RGB map to perform the erosion operation, and uses the depth change to further confirm the confidence of the portrait object, which can greatly improve the scope and accuracy of the erosion, which is of great significance for obtaining a confirmed portrait object and can greatly improve the recognition effect.

[0083] In some embodiments, the area filtered in step S21 is also marked as 0.5. The area processed in step S21 is not the original data, but a low confidence area, which is also marked as 0.5. The subsequent step makes the next judgment on it, and a more accurate result can be obtained.

[0084] This embodiment performs filtering processing on the image, then extracts the first portrait range according to the preset distance parameter, and then marks the image area as 1, 0.5 and 0 according to the corrosion operation, and directly uses the depth value to determine part of the foreground and background, thereby reducing the amount of calculation of the subsequent algorithm, improving efficiency, and avoiding the determined foreground or background being misidentified by the algorithm, thereby ensuring quality.

[0085] Figure 6 FIG. 1 is a flow chart of steps for obtaining a second portrait range in an embodiment of the present invention. Figure 6 As shown, in an embodiment of the present invention, a step of obtaining a second portrait range includes:

[0086] Step S31: using an encoder to extract features and condense semantic information from the first portrait range.

[0087] In this step, the encoder is a downsampling module composed of two 3x3 convolutional layers (RELU) plus a 2x2 maxpooling layer. After multiple downsampling, the low-resolution information is obtained to obtain the portrait features and can provide the contextual semantic information of the segmented portrait in the entire image.

[0088] Step S32: using a decoder to restore image pixels.

[0089] In this step, the decoder consists of an upsampling convolutional layer (deconvolutional layer), a feature concatenation layer, and two 3x3 convolutional layers (ReLU).

[0090] Step S33: Use Skip-Connection with different dimensional features to supplement information.

[0091] In this step, the U-Net with skipconnection structure is added, which enables the network to fuse the feature maps of the corresponding position of the encoder on the channel during each level of upsampling. By fusing the underlying features with the high-level features, the network can retain more high-resolution detail information contained in the high-level feature maps, thereby improving the image segmentation accuracy. While the cross-layer feature reconstruction enhances the transmission of feature information between layers, it further utilizes the rich detail information in the high-level convolutional feature layer, thereby maximizing the utilization of feature information in each layer of the network. This step introduces the feature information at the corresponding scale into the upsampling or deconvolution process, providing multi-scale and multi-level information for the later image segmentation, thereby obtaining a more refined segmentation effect.

[0092] Step S34: iterating through the video timing correlation loss function to obtain a second portrait range.

[0093] In this step, the information of the previously processed images is compared with the first frame image, and the loss function is iteratively optimized to obtain the final second portrait range.

[0094] In some embodiments, the decoder includes a GRU for inter-frame information transmission. The input to the GRU is a three-dimensional variable, and the conv layer input variable changes, which can be well used for inter-frame information transmission and greatly reduces defects and improves the perceived quality by predicting more coherent results. At the same time, the GRU can also improve the robustness of extinction and better guess the boundaries when individual frames may be blurred.

[0095] This embodiment uses UNet to process the first portrait range, supplements it with Skip-Connection, and uses the video timing-related loss function for iterative optimization, so that the final second portrait range can obtain a high-accuracy and high-precision image in a single frame, and can ensure the continuity of the portrait object in the video, and prevent problems such as video jitter.

[0096] In some embodiments, step S3 is performed using a pre-trained model, and the pre-trained model is trained by alternately using single-frame images and continuous videos. The training input is the image and video processed in step S2.

[0097] Figure 7 FIG. 1 is a flowchart of another method for portrait recognition based on depth map adaptation in an embodiment of the present invention. Figure 7 As shown, compared with the above-mentioned embodiment, another method for portrait recognition based on depth map adaptation in the embodiment of the present invention further includes:

[0098] Step S6: identifying the hair region of the third portrait range, and adjusting the hair region according to the time sequence feature.

[0099] In this step, the hair region of the third portrait range can be identified at the same time as the above step, or it can be identified separately in this step. When the above step identifies the hair region, the same method as the above step is used, but this step does not identify the hair region separately. When the above step identifies the hair region, the same model as the above step can be used, or a special hair region identification model can be used, which is not limited in this embodiment. When adjusting the hair region according to the time series feature, a variety of solutions can be used. For example, Figure 8 As shown in , the hair area is divided into a fixed area and a moving area. The fixed area is the area that remains stationary when the hair area is adjusted. The moving area is the area that needs to change position. Figure 8 As shown, the position of the motion area can be changed from position 1 to position 2, thereby adjusting the hair area. By adjusting the hair area, different effects can be achieved, such as increasing agility by swinging the hair, and overcoming environmental interference by keeping the hair within a certain range. If this embodiment is combined with a replacement background, the swing of the hair can be adapted to the background, and a better fusion of the foreground and the replacement background can be achieved.

[0100] This embodiment identifies the hair area of ​​a person and adjusts the hair area so that the hair of the person can achieve the expected swaying effect or be still, so that it can be better applied to scenes such as background replacement, so that the person and the background after the background replacement are more natural, and avoid the conflict problem that the background is windless but the person's hair flutters, or the background is breezy but the person's hair is still.

[0101] Fig. 9 This is a flow chart of steps for adjusting a hair area in an embodiment of the present invention;

[0102] Step S61: identifying the hair region of the third portrait range and classifying the hair types.

[0103] In this step, the hair area is classified so that different adjustment strategies can be adopted in subsequent steps. For example, the hair is divided into long hair, medium hair, and short hair to distinguish the range of hair adjustment. Long hair refers to hair that droops more, such as over the shoulders, and has the largest adjustment range. Medium hair refers to hair that droops less, such as over the earlobe but not over the shoulders, and has a larger adjustment range. Short hair refers to hair that droops less, such as not over the earlobe, and is little affected by the wind, with only a very small adjustment range.

[0104] Step S62: selecting a motion trajectory for the hair according to the hair type.

[0105] In this step, different hair types correspond to different motion trajectories. For example, for short hair, only the outermost part of the hair is slightly moved; for medium hair, the majority of the hair is adjusted as a whole; for long hair, the majority of the hair is divided into three sections for position and curvature adjustment to better simulate the natural state of the hair.

[0106] Step S63: adjusting the hair area according to the motion trajectory and the time sequence characteristics.

[0107] In this step, the motion trajectory is different for different hair types. The motion trajectory is closed, such as a circle, an ellipse, an "8" ring, etc. The specific point of the hair moves along the motion trajectory. There can be multiple motion trajectories, and different motion trajectories are set for different parts of the hair. If the motion trajectory is set only for some specific points of the hair, when adjusting, the specific points are first placed in the preset position, and then the other points are adjusted accordingly to ensure the integrity of the hair. The position of the hair in the current frame image is determined by the motion trajectory and the time sequence characteristics, so as to determine the position of the hair after the change.

[0108] This embodiment classifies the hair types, selects the corresponding motion trajectory, and adjusts the hair region in combination with the time series feature, so that the hair can move in a manner suitable for the hair type. Furthermore, this embodiment can also control the adjustment of the hair region by specifying the motion trajectory, thereby achieving various types of hair movements, which is particularly suitable for scenes where the background is replaced, so that the movement of the hair is adapted to the background, and the video after the background is replaced is more realistic.

[0109] Fig.10The figure is a schematic diagram of the structure of a portrait recognition system based on depth map adaptation in an embodiment of the present invention.

[0110] like Fig.10 As shown, in an embodiment of the present invention, a portrait recognition system based on depth map adaptation includes:

[0111] An acquisition module, used to acquire a first frame image in a video; wherein the first frame image includes a first RGB image and a first depth image;

[0112] A preprocessing module, configured to obtain a first portrait range on the first depth image using a first preset distance parameter;

[0113] A cutout module is used to identify the first portrait range based on the UNet network structure and Skip-Connection of different dimensional features, combined with the temporal features of the first frame image, to obtain a second portrait range;

[0114] A confidence module, used to calculate the depth value distribution range of the second portrait range, and obtain a second preset distance parameter and a third preset distance parameter according to a confidence interval a;

[0115] The adjustment module is used to adjust the second portrait range by using the second preset distance parameter and the third preset distance parameter to obtain a third portrait range.

[0116] Specifically, the acquisition module obtains the first frame image with depth information, and the subsequent modules process it on the mask to obtain the final portrait area. The preprocessing module directly uses the depth information to obtain the first portrait range. The image clipping module uses the UNet-based network structure to perform portrait recognition in the first portrait range while ensuring different dimensional features and inter-frame information to obtain the second portrait range. The confidence module uses the depth value distribution range for analysis to obtain the second preset distance parameter and the third preset distance parameter. The adjustment module then uses the second preset distance parameter and the third preset distance parameter to fine-tune the second portrait range to ensure the range of the core foreground and background.

[0117] This embodiment uses a first preset distance parameter to obtain a first portrait range, and then uses the UNet network and Skip-Connection of different dimensional features combined with timing features to identify a second portrait image, and adaptively adjusts the second portrait range according to the depth distribution. It can completely and effectively identify the human body and interactive objects, eliminate weakly related object information, and generate high-quality and high-resolution images, which can realize real-time processing of videos.

[0118] The embodiment of the present invention also provides a portrait recognition device based on depth map adaptation, including a processor and a memory, wherein executable instructions of the processor are stored. The processor is configured to execute the steps of a portrait recognition method based on depth map adaptation by executing the executable instructions.

[0119] As mentioned above, this embodiment uses the first preset distance parameter to obtain the first portrait range, and then uses the UNet network and the Skip-Connection of different dimensional features, combined with the timing characteristics, to identify the second portrait image, and adaptively adjust the second portrait range according to the depth distribution. It can completely and effectively identify the human body and interactive objects, eliminate weakly related object information, and generate high-quality and high-resolution images, which can realize real-time processing of video.

[0120] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0121] Fig.11 Schematic diagram of a depth map-based adaptive portrait recognition device in an embodiment of the present invention. Fig.11 The electronic device 600 according to this embodiment of the present invention is described. Fig.11 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0122] like Fig.11 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0123] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps of various exemplary embodiments of the present invention described in the above-mentioned method for portrait recognition based on depth map adaptation. For example, the processing unit 610 can execute the following steps: Figure 1 Follow the steps shown in .

[0124] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0125] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0126] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0127] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 through the bus 630. It should be understood that although Fig.11 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0128] In an embodiment of the present invention, a computer-readable storage medium is also provided for storing a program, and when the program is executed, the steps of a method for portrait recognition based on depth map adaptation are implemented. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above-mentioned method for portrait recognition based on depth map adaptation of this specification.

[0129] As shown above, this embodiment uses the first preset distance parameter to obtain the first portrait range, and then uses the UNet network and the Skip-Connection of different dimensional features, combined with the timing characteristics, to identify the second portrait image, and adaptively adjust the second portrait range according to the depth distribution. It can completely and effectively identify the human body and interactive objects, eliminate weakly related object information, and generate high-quality and high-resolution images, which can realize real-time processing of video.

[0130] Fig.12 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Fig.12 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0131] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0132] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0133] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0134] This embodiment uses a first preset distance parameter to obtain a first portrait range, and then uses the UNet network and Skip-Connection of different dimensional features combined with timing features to identify a second portrait image, and adaptively adjusts the second portrait range according to the depth distribution. It can completely and effectively identify the human body and interactive objects, eliminate weakly related object information, and generate high-quality and high-resolution images, which can realize real-time processing of videos.

[0135] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this article, but will comply with the widest range consistent with the principles and novel features disclosed herein.

[0136] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A portrait recognition method based on depth map adaptation, characterized in that: The steps include: Step S1: Acquire a first frame image in a video; wherein the first frame image includes a first RGB image and a first depth image; Step S2: obtaining a first portrait range on the first depth image using a first preset distance parameter; Step S3: Based on the UNet network structure and Skip-Connection of different dimensional features, combined with the temporal features of the first frame image, the first portrait range is identified to obtain the second portrait range; Step S4: calculating the depth value distribution range of the second portrait range, and obtaining a second preset distance parameter and a third preset distance parameter according to the confidence interval a; Step S5: using the second preset distance parameter and the third preset distance parameter to adjust the second portrait range to obtain a third portrait range.

2. The method for portrait recognition based on depth map adaptation according to claim 1, characterized in that: Step S2 includes: Step S21: filtering the first depth image to obtain a second depth image; Step S22: extracting a first portrait range from the second depth image according to a first preset distance parameter; Step S23: performing an erosion operation on the first portrait range, and marking the eroded area as 1, the eroded area as 0.5, and the other areas as 0.

3. The method for portrait recognition based on depth map adaptation according to claim 2, characterized in that: The erosion operation is determined according to a pixel offset between the second depth map and the first RGB image.

4. The method for portrait recognition based on depth map adaptation according to claim 1, characterized in that: Step S3 includes: Step S31: using an encoder to extract features and condense semantic information from the first portrait range; Step S32: using a decoder to restore image pixels; Step S33: using Skip-Connection with different dimensional features to supplement information; Step S34: iterating through the video timing correlation loss function to obtain a second portrait range.

5. The method for portrait recognition based on depth map adaptation according to claim 4, characterized in that: In step S32, the decoder includes a GRU for inter-frame information transmission.

6. The method for portrait recognition based on depth map adaptation according to claim 1, characterized in that: Also includes: Step S6: identifying the hair region of the third portrait range, and adjusting the hair region according to the time sequence feature.

7. The method for portrait recognition based on depth map adaptation according to claim 6, characterized in that: Step S6 includes: Step S61: identifying the hair area of ​​the third portrait range and classifying the hair types; Step S62: selecting a motion trajectory for the hair according to the hair type; Step S63: adjusting the hair area according to the motion trajectory and the time sequence characteristics.

8. A depth map-based adaptive portrait recognition system, used to implement the depth map-based adaptive portrait recognition method according to any one of claims 1 to 7, characterized in that: include: An acquisition module, used to acquire a first frame image in a video; wherein the first frame image includes a first RGB image and a first depth image; A preprocessing module, configured to obtain a first portrait range on the first depth image using a first preset distance parameter; A cutout module is used to identify the first portrait range based on the UNet network structure and Skip-Connection of different dimensional features, combined with the temporal features of the first frame image, to obtain a second portrait range; A confidence module, used to calculate the depth value distribution range of the second portrait range, and obtain a second preset distance parameter and a third preset distance parameter according to a confidence interval a; The adjustment module is used to adjust the second portrait range by using the second preset distance parameter and the third preset distance parameter to obtain a third portrait range.

9. A portrait recognition device based on depth map adaptation, characterized in that: include: processor; a memory storing executable instructions of the processor; The processor is configured to execute the steps of the depth map adaptive portrait recognition method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed, the steps of the depth map adaptive portrait recognition method described in any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Face recognition method and device

    CN109190539A

  • Low-illumination image enhancement method based on zero-order learning

    CN115953321A

  • Video processing method, system and device and storage medium

    CN116029893A

  • Human image recognition method, system and equipment based on depth map double guidance and medium

    CN119048946A

  • Method of domain adaptation applied to image segmentation, apparatus, and storage medium

    JP2022164597A