Curling player face reconstruction method based on rebroadcast video

Abstract

The invention provides a curling player face reconstruction method based on a rebroadcast video. According to the method, the facial features of the curling player in the rebroadcast video are optimized, the deep learning technology is utilized, a mask network strategy is combined, the limitation of a traditional method in processing reflective and weak texture areas is overcome, and a more real and vivid digital human image is provided for the fields of element universe, digital human, sports event rebroadcast and the like.

Description

Technical Field

[0001] The present invention relates to the fields of metaverse, digital human and artificial intelligence technology, and in particular to a method for reconstructing the face of a curling player based on broadcast video. Background Art

[0002] In the curling game broadcast video, the facial expressions and interactive details of the athletes are crucial to the audience's emotional involvement and the creation of an immersive game atmosphere. However, due to the influence of venue lighting and ice reflection in curling games, the texture information of the athletes' faces is particularly complex and difficult to capture.

[0003] Traditional 3D modeling methods, such as manual modeling with software, high-precision instrument scanning, and multi-view image fusion, can achieve good results in specific scenarios, but they generally have problems such as high equipment prices, complex operating procedures, and strict dependence on environmental conditions. In particular, the multi-view image fusion modeling method, although excellent in reconstruction accuracy and edge processing, is often difficult to avoid problems such as incomplete models and missing textures when processing the facial expressions of curling athletes with high reflectivity and weak texture features in the broadcast video of the game, which seriously affects the realism and practicality of the digital image of the curling athlete. Summary of the invention

[0004] The purpose of the present invention is to solve the problems in the prior art and propose a method for reconstructing the face of a curling player based on a broadcast video. The method takes into account that the weak texture areas of the curling player's facial image in the broadcast video mainly appear in the cheeks, forehead, hair and other parts. By extracting the correct weak texture areas in the curling player's facial image, combining traditional multi-view 3D reconstruction and deep learning multi-view 3D reconstruction methods, a reconstruction method based on evaluation is proposed, which improves the integrity and usability of multi-view facial 3D reconstruction.

[0005] The present invention is implemented by the following technical scheme. The present invention proposes a method for reconstructing the face of a curling player based on a broadcast video, and the method comprises the following steps:

[0006] Step 1: Collect multi-view images of curling athletes in the throwing stage, sliding stage, ice-wiping stage and decision-making stage according to the broadcast video, solve the internal and external parameters of the broadcast camera at different stages, extract the key points of the curling athletes' faces, determine the facial features, use the weak texture detection algorithm to extract the weak texture areas in the rest, and construct the facial depth map merge mask;

[0007] Step 2: Use a depth estimation method to obtain a first depth map of multiple perspectives, use a deep learning depth estimation method to obtain a second depth map of multiple perspectives, align and normalize the first depth map and the second depth map, and merge them according to the mask to obtain a third depth map;

[0008] Step 3: Fuse the third depth map from multiple perspectives, use the multi-perspective reconstruction method to obtain a facial 3D model with texture mapping, quantitatively evaluate the facial 3D model reconstruction effect based on the model smoothness evaluation method and the model integrity evaluation method, and update the depth map merge mask according to the evaluation results until a complete and correct curling athlete facial model is reconstructed.

[0009] Furthermore, in step 1, the key points of the curling players' faces are extracted, and the facial features are determined as follows: multi-perspective images of the curling players in the throwing stage, gliding stage, ice-wiping stage and decision-making stage are collected from the broadcast video, and the non-facial parts are replaced with a gray background; by matching the feature points of the images, the matches that satisfy the geometric constraints are retained, and the internal and external parameters of the broadcast camera are restored using the SFM method; the key points of the facial features, including eyebrows, eyes, nose and mouth, are extracted, the convex hull is drawn and feathered to obtain the facial features, and when the facial features are removed, the weak texture areas of the cheeks, forehead and hair are obtained.

[0010] Furthermore, in step 1, the weak texture area is extracted based on the grayscale gradient threshold method as follows: the Prewitt operator convolution in the horizontal and vertical directions is used to obtain the gradient g in the x and y directions of the pixel point x (x,y),g y (x,y), then the gradient value of this pixel is Where (x, y) is the pixel position; if this pixel belongs to the weak texture area, it satisfies: g(x, y) < g t , where g t is the gradient threshold, and the obtained pixel position is the segmented weak texture area.

[0011] Furthermore, in step 1, the mask for depth map merging is obtained as follows: after detecting the weak texture area in the facial image, the mask value of the weak texture area in the image is set to 1, and the mask value of the rest of the image is set to 0, that is:

[0012]

[0013] Among them, face_mask(x,y) is the depth merge mask.

[0014] Furthermore, the step 2 is specifically as follows:

[0015] Step 2.1, use the PatchMatch method to perform depth estimation, take the broadcast camera posture and the curling athlete's facial image as input, and obtain the first depth map depthmap1 through the matching cost construction, accumulation, estimation and optimization process, which effectively distinguishes the background and facial area; use the deep learning network PVA-MVSNet, based on multi-metric aggregation of multi-scale pyramid image information, and use multi-scale information to estimate the depth value of the weak texture area to obtain the second depth map depthmap2;

[0016] Step 2.2: Align and normalize the first depth map with the second depth map; normalize the first depth map to the depth range of the second depth map, and design the normalization coefficient as:

[0017]

[0018] Among them, Max{} refers to the maximum depth value in the depth map, and Min{} refers to the minimum depth value in the depth map;

[0019] The normalized first depth map Depthmap1 is:

[0020] Depthmap1=Min{depthmap2}+k*(depthmap1-Min{depthmap1});

[0021] Step 2.3: merge the first depth map and the second depth map according to the mask to obtain a third depth map depthmap3, and calculate the depth difference at the edge. If it exceeds the threshold, perform weighted filtering to smooth the depth map; wherein the merging method is:

[0022] depthmap3=face_mask*depthmap2+(1-face_mask)*Depthmap1

[0023] The method for determining the depth difference at the edge is:

[0024] ∫|Depthmap1(edge)-depthmap2(edge)| <th

[0025] Among them, edge is the edge area of ​​the depth merge mask, th is the depth difference threshold; the weighted filtering method selects the convolution sliding average filtering algorithm with a window size of 5*5.

[0026] Furthermore, in step 3, the multi-perspective third depth maps of the curling player in the throwing phase, gliding phase, ice-wiping phase and decision-making phase in the broadcast video are integrated, and a dense point cloud set of the curling player's facial model in the three-dimensional space is calculated. The point cloud set is tetrahedral meshed and surface extracted to obtain a facial mesh model, and a mapping-based color map mesh is used to generate a three-dimensional facial model of the curling player after texture mapping.

[0027] Furthermore, in step 3, the model smoothness evaluation is specifically as follows: the Gaussian curvature of the vertices of the facial three-dimensional model except the facial features is calculated:

[0028]

[0029] Among them, k G is the Gaussian curvature, v i is the i-th vertex, N(i) is the number of all vertex v i The number of triangle patches, θ ij is the angle difference between the jth triangle patch and the vertex normal vector;

[0030] If the number of points whose Gaussian curvature is greater than the curvature threshold is greater than the number threshold, the depth value of the edge area changes greatly when the depth map is merged. In this case, the window size of the weighted filter is increased.

[0031] Furthermore, in step 3, the model integrity evaluation is specifically as follows: construct a virtual camera according to the internal and external parameters of the broadcast camera solved in step 1, render the images of the curling athlete's facial texture model from each perspective, and calculate the structural similarity SSIM index and peak signal-to-noise ratio PSNR index between the images from each perspective and the images taken by the broadcast camera;

[0032] Specifically, the structural similarity SSIM index is

[0033]

[0034] Among them, μ x ,μ y are the average values ​​of x and y respectively. are the variances of x and y, σ xy is the covariance of x and y; c1 and c2 are constants used to maintain stability;

[0035] For a given original object of size m*n Figure I And the noise map K, the mean square error is expressed as follows:

[0036]

[0037] The peak signal-to-noise ratio PSNR indicator is: Among them MAX Iis the maximum pixel value of the image;

[0038] When the structural similarity SSIM index and the peak signal-to-noise ratio PSNR index are lower than the threshold, the evaluation model is incomplete. At this time, the non-zero part in the depth map merge mask is expanded or contracted to update the depth merge mask; the expansion of the non-zero part optimizes the incomplete model caused by weak texture; the contraction of the non-zero part optimizes the incomplete model caused by edge performance and accuracy problems.

[0039] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method for reconstructing the face of a curling player based on a broadcast video are implemented.

[0040] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the method for reconstructing the face of a curling player based on broadcast video.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention proposes a method for reconstructing the face of a curling player based on broadcast video, which effectively solves the problems of incomplete model and texture loss encountered by traditional 3D modeling methods when dealing with highly reflective and weakly textured facial features. By combining deep learning technology and mask network strategy, the present invention can optimize the reconstruction of the facial features of curling players in broadcast videos, significantly improving the realism and practicality of digital human images. This progress brings more realistic and vivid visual effects to the fields of metaverse, digital humans, and sports event broadcasts, while reducing dependence on high-cost equipment and complex operating procedures, and improving modeling efficiency and flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0044] Figure 1 The present invention is a flow chart of a method for reconstructing the face of a curling player based on broadcast video. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Combination Figure 1 The present invention proposes a method for reconstructing the face of a curling player based on a broadcast video, the method comprising the following steps:

[0047] Step 1: Collect multi-view images of curling athletes in the throwing stage, sliding stage, ice-wiping stage and decision-making stage according to the broadcast video, solve the internal and external parameters of the broadcast camera at different stages, extract the key points of the curling athletes' faces, determine the facial features, use the weak texture detection algorithm to extract the weak texture areas in the rest, and construct the facial depth map merge mask;

[0048] In step 1, the key points of the curling players' faces are extracted, and the facial features are determined as follows: multi-view images of the curling players in the throwing stage, sliding stage, ice-wiping stage and decision-making stage are collected from the broadcast video, and the non-facial parts are replaced with a gray background; by matching the image feature points, the matches that satisfy the geometric constraints are retained, and the internal and external parameters of the broadcast camera are restored using the SFM method; the key points of the facial features, including eyebrows, eyes, nose and mouth, are extracted, and the convex hull is drawn and feathered to obtain the facial features. When the facial features are removed, weak texture areas such as cheeks, forehead and hair can be obtained.

[0049] In step 1, the weak texture area is extracted based on the grayscale gradient threshold method: the Prewitt operator convolution in the horizontal and vertical directions is used to obtain the gradient g in the x and y directions of the pixel point x (x,y),g y (x,y), then the gradient value of this pixel is Where (x, y) is the pixel position; if this pixel belongs to the weak texture area, it satisfies: g(x, y) < g t , where g t is the gradient threshold, and the obtained pixel position is the segmented weak texture area.

[0050] In step 1, the mask for depth map merging is obtained as follows: after detecting the weak texture area in the facial image, the mask value of the weak texture area in the image is set to 1, and the mask value of the rest of the image is set to 0, that is:

[0051]

[0052] Among them, face_mask(x,y) is the depth merge mask.

[0053] Step 2: Use a traditional depth estimation method to obtain a first depth map of multiple perspectives, use a deep learning depth estimation method to obtain a second depth map of multiple perspectives, align and normalize the first depth map and the second depth map, and merge them according to the mask to obtain a third depth map;

[0054] The step 2 is specifically as follows:

[0055] Step 2.1, use the PatchMatch method to perform depth estimation, take the broadcast camera posture and the curling athlete's facial image as input, and obtain the first depth map depthmap1 through the matching cost construction, accumulation, estimation and optimization process, which effectively distinguishes the background and facial area; use the deep learning network PVA-MVSNet, based on multi-metric aggregation of multi-scale pyramid image information, and use multi-scale information to estimate the depth value of the weak texture area to obtain the second depth map depthmap2;

[0056] Step 2.2: Align and normalize the first depth map with the second depth map; normalize the first depth map to the depth range of the second depth map, and design the normalization coefficient as:

[0057]

[0058] Among them, Max{} refers to the maximum depth value in the depth map, and Min{} refers to the minimum depth value in the depth map;

[0059] The normalized first depth map Depthmap1 is:

[0060] Depthmap1=Min{depthmap2}+k*(depthmap1-Min{depthmap1});

[0061] Step 2.3: merge the first depth map and the second depth map according to the mask to obtain a third depth map depthmap3, and calculate the depth difference at the edge. If it exceeds the threshold, perform weighted filtering to smooth the depth map; wherein the merging method is:

[0062] depthmap3=face_mask*depthmap2+(1-face_mask)*Depthmap1

[0063] The method for determining the depth difference at the edge is:

[0064] ∫|Depthmap1(edge)-depthmap2(edge)| <th

[0065] Among them, edge is the edge area of ​​the depth merge mask, th is the depth difference threshold; the weighted filtering method selects the convolution sliding average filtering algorithm with a window size of 5*5.

[0066] Step 3: Fuse the third depth map from multiple perspectives, use the multi-perspective reconstruction method to obtain a facial 3D model with texture mapping, quantitatively evaluate the facial 3D model reconstruction effect based on the model smoothness evaluation method and the model integrity evaluation method, and update the depth map merge mask according to the evaluation results until a complete and correct curling athlete facial model is reconstructed.

[0067] In step 3, the multi-perspective third depth maps of the curling player in the throwing phase, sliding phase, ice-wiping phase and decision-making phase in the broadcast video are integrated, and a dense second point cloud set of the curling player's facial model in the three-dimensional space is calculated. The second point cloud set is tetrahedral meshed and surface extracted to obtain a facial mesh model, and a mapping-based color map mesh is used to generate a three-dimensional facial model of the curling player after texture mapping.

[0068] In step 3, the model smoothness evaluation is specifically as follows: the Gaussian curvature of the vertices of the facial 3D model excluding the facial features is calculated:

[0069]

[0070] Among them, k G is the Gaussian curvature, v i is the i-th vertex, N(i) is the number of all vertex v i The number of triangle patches, θ ij is the angle difference between the jth triangle patch and the vertex normal vector;

[0071] If the number of points whose Gaussian curvature is greater than the curvature threshold is greater than the number threshold, the depth value of the edge area changes greatly when the depth map is merged. In this case, the window size of the weighted filter is increased.

[0072] In step 3, the model integrity evaluation is specifically as follows: construct a virtual camera according to the internal and external parameters of the broadcast camera calculated in step 1, render the images of the curling athlete's facial texture model from each perspective, and calculate the structural similarity SSIM index and peak signal-to-noise ratio PSNR index between the images from each perspective and the images taken by the broadcast camera;

[0073] Specifically, the structural similarity SSIM index is

[0074]

[0075] Among them, μ x ,μ y are the average values ​​of x and y respectively. are the variances of x and y, σxy is the covariance of x and y; c1 and c2 are constants used to maintain stability;

[0076] For a given original object of size m*n Figure I And the noise map K, the mean square error is expressed as follows:

[0077]

[0078] The peak signal-to-noise ratio PSNR indicator is: Among them MAX I is the maximum pixel value of the image;

[0079] When the structural similarity SSIM index and the peak signal-to-noise ratio PSNR index are lower than the threshold, the evaluation model is incomplete. At this time, the non-zero part in the depth map merge mask is expanded or contracted to update the depth merge mask; the expansion of the non-zero part optimizes the incomplete model caused by weak texture; the contraction of the non-zero part optimizes the incomplete model caused by edge performance and accuracy problems.

[0080] The present invention proposes a method for reconstructing the faces of curling players based on broadcast videos. The method optimizes the facial features of curling players in broadcast videos, uses deep learning technology and combines mask network strategies to overcome the limitations of traditional methods in processing reflective and weak texture areas, and provides more realistic and vivid digital human images for the fields of metaverse, digital humans, and sports event broadcasts.

[0081] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method for reconstructing the face of a curling player based on a broadcast video are implemented.

[0082] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the method for reconstructing the face of a curling player based on broadcast video.

[0083] The memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DRRAM). It should be noted that the memory of the method described in the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0084] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disc (SSD)).

[0085] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0086] It should be noted that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor can be combined and performed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0087] The above is a detailed introduction to the method for facial reconstruction of curling players based on broadcast video proposed by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for reconstructing curling players' faces based on broadcast video, characterized in that: The method comprises the following steps: Step 1: Collect multi-view images of curling athletes in the throwing stage, sliding stage, ice-wiping stage and decision-making stage according to the broadcast video, solve the internal and external parameters of the broadcast camera at different stages, extract the key points of the curling athletes' faces, determine the facial features, use the weak texture detection algorithm to extract the weak texture areas in the rest, and construct the facial depth map merge mask; Step 2: Use a depth estimation method to obtain a first depth map of multiple perspectives, use a deep learning depth estimation method to obtain a second depth map of multiple perspectives, align and normalize the first depth map and the second depth map, and merge them according to the mask to obtain a third depth map; Step 3: Fuse the third depth map from multiple perspectives, use the multi-perspective reconstruction method to obtain a facial 3D model with texture mapping, quantitatively evaluate the facial 3D model reconstruction effect based on the model smoothness evaluation method and the model integrity evaluation method, and update the depth map merge mask according to the evaluation results until a complete and correct curling athlete facial model is reconstructed.

2. The method according to claim 1, characterized in that In step 1, the key points of the curling players' faces are extracted, and the facial features are determined as follows: multi-view images of the curling players in the throwing stage, sliding stage, ice-wiping stage and decision-making stage are collected from the broadcast video, and the non-facial parts are replaced with a gray background; by matching the image feature points, the matches that satisfy the geometric constraints are retained, and the internal and external parameters of the broadcast camera are restored using the SFM method; the key points of the facial features, including eyebrows, eyes, nose and mouth, are extracted, and the convex hull is drawn and feathered to obtain the facial features. When the facial features are removed, the weak texture areas of the cheeks, forehead and hair are obtained.

3. The method according to claim 2, characterized in that In step 1, the weak texture area is extracted based on the grayscale gradient threshold method: the Prewitt operator convolution in the horizontal and vertical directions is used to obtain the gradient g in the x and y directions of the pixel point x (x,y),g y (x,y), then the gradient value of this pixel is Where (x, y) is the pixel position; if this pixel belongs to the weak texture area, it satisfies: g(x, y) < g t , where g t is the gradient threshold, and the obtained pixel position is the segmented weak texture area.

4. The method according to claim 3, characterized in that: In step 1, the mask for depth map merging is obtained as follows: after detecting the weak texture area in the facial image, the mask value of the weak texture area in the image is set to 1, and the mask value of the rest of the image is set to 0, that is: Among them, face_mask(x,y) is the depth merge mask.

5. The method according to claim 4, characterized in that The step 2 is specifically as follows: Step 2.1, use the PatchMatch method to perform depth estimation, take the broadcast camera posture and the curling athlete's facial image as input, and obtain the first depth map depthmap1 through the matching cost construction, accumulation, estimation and optimization process, which effectively distinguishes the background and facial area; Using the deep learning network PVA-MVSNet, based on multi-metric aggregation of multi-scale pyramid image information, the depth value of the weak texture area is estimated using multi-scale information to obtain the second depth map depthmap2; Step 2.2: Align and normalize the first depth map with the second depth map; normalize the first depth map to the depth range of the second depth map, and design the normalization coefficient as: Where Max{} refers to the maximum depth value in the depth map, and Min{} refers to the minimum depth value in the depth map; The normalized first depth map Depthmap1 is: Depthmap1=Min{depthmap2}+k*(depthmap1-Min{depthmap1}); Step 2.3: merge the first depth map and the second depth map according to the mask to obtain a third depth map depthmap3, and calculate the depth difference at the edge. If it exceeds the threshold, perform weighted filtering to smooth the depth map; wherein the merging method is: depthmap3=face_mask*depthmap2+(1-face_mask)*Depthmap1 The method for determining the depth difference at the edge is: ∫|Depthmap1(edge)-depthmap2(edge)| <th Among them, edge is the edge area of ​​the depth merge mask, th is the depth difference threshold; the weighted filtering method selects the convolution sliding average filtering algorithm with a window size of 5*5.

6. The method according to claim 5, characterized in that In step 3, the multi-perspective third depth maps of the curling player in the throwing phase, sliding phase, ice-wiping phase and decision-making phase in the broadcast video are integrated, and the dense point cloud set of the curling player's facial model in the three-dimensional space is calculated. The point cloud set is tetrahedral meshed and surface extracted to obtain the facial mesh model, and the mapping-based color map mesh is used to generate the three-dimensional facial model of the curling player after texture mapping.

7. The method according to claim 6, characterized in that In step 3, the model smoothness evaluation is specifically as follows: the Gaussian curvature of the vertices of the facial 3D model excluding the facial features is calculated: Among them, k G is the Gaussian curvature, v i is the i-th vertex, N(i) is the number of all vertex v i The number of triangle patches, θ ij is the angle difference between the jth triangle patch and the vertex normal vector; If the number of points whose Gaussian curvature is greater than the curvature threshold is greater than the number threshold, the depth value of the edge area changes greatly when the depth map is merged. In this case, the window size of the weighted filter is increased.

8. The method according to claim 7, characterized in that In step 3, the model integrity evaluation is specifically as follows: construct a virtual camera according to the internal and external parameters of the broadcast camera calculated in step 1, render the images of the curling athlete's facial texture model from each perspective, and calculate the structural similarity SSIM index and peak signal-to-noise ratio PSNR index between the images from each perspective and the images taken by the broadcast camera; Specifically, the structural similarity SSIM index is Among them, μ x ,μ y are the average values ​​of x and y respectively. are the variances of x and y, σ xy is the covariance of x and y; c1 and c2 are constants used to maintain stability; For a given original image I and noise image K of size m*n, the mean square error is expressed as follows: The peak signal-to-noise ratio PSNR indicator is: Among them MAX I is the maximum pixel value of the image; When the structural similarity SSIM index and the peak signal-to-noise ratio PSNR index are lower than the threshold, the evaluation model is incomplete. At this time, the non-zero part in the depth map merge mask is expanded or contracted to update the depth merge mask; the expansion of the non-zero part optimizes the incomplete model caused by weak texture; the contraction of the non-zero part optimizes the incomplete model caused by edge performance and accuracy problems.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images