Fall scene restoration method and system realized by utilizing three-dimensional modeling

Abstract

The invention provides a tumble scene restoration method and system realized by using three-dimensional modeling, and the method comprises the steps: obtaining an original video comprising a tumble object, extracting a plurality of 3D human body key points of the tumble object from the original video, generating a human body two-dimensional contour image of the tumble object based on the 3D human body key points, and obtaining a tumble scene restoration result based on the human body two-dimensional contour image. Determining a multi-view two-dimensional image; generating a normal map corresponding to the multi-view two-dimensional image; generating a three-dimensional human body grid model according to the multi-view two-dimensional images and the corresponding normal chartlets; matching the human body key points in the three-dimensional virtual object corresponding to each image with the 3D human body key points corresponding to each image to obtain successfully matched target human body key points; based on the time sequence of the original video, the positions of the target human body key points in the three-dimensional virtual objects are adjusted, a three-dimensional video is obtained, and the three-dimensional video is used for restoring the falling scene of the falling object. According to the invention, the scene application flexibility of fall scene restoration can be improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for reconstructing fall scenarios using three-dimensional modeling. Background Technology

[0002] Currently, fall incidents have become a crucial scenario requiring precise analysis in areas such as home-based elderly care, medical monitoring, and forensic evidence collection. Accurately reconstructing the three-dimensional dynamic details of a fall is not only crucial evidence for determining the cause of the fall and clarifying liability, but also provides a scientific basis for optimizing subsequent preventive measures and developing rehabilitation programs.

[0003] Currently, the analysis of surveillance videos mainly relies on two-dimensional image data captured by camera equipment. However, this type of data cannot provide sufficient depth information, which limits its ability to accurately reconstruct the details of three-dimensional human fall movements. To address this issue, existing technologies typically employ limb movement recognition methods based on the Kinect 3D skeleton model. These methods utilize the Kinect camera to acquire three-dimensional coordinate data of the human skeleton, extract geometric features such as joint angles, and after preprocessing, use classifiers such as random forests to complete movement recognition, thereby reconstructing the fall scenario.

[0004] However, the above methods rely on Kinect cameras, which limits the application scenarios and flexibility of fall scene reconstruction. Furthermore, because they only use geometric features such as joint angles for motion recognition, they cannot fully capture the detailed spatiotemporal changes of complex movements like falls, resulting in low accuracy in fall scene reconstruction. Summary of the Invention

[0005] This invention provides a method and system for reconstructing fall scenarios using 3D modeling, which addresses the shortcomings of existing technologies in the application flexibility and accuracy of detail in fall scenario reconstruction. This invention not only improves the application flexibility of fall scenario reconstruction but also enhances the accuracy of detail in fall scenario reconstruction.

[0006] This invention provides a method for reconstructing fall scenarios using 3D modeling, comprising: Obtain the original video, which includes the fallen object; Extract multiple 3D human key points of the fallen object from each frame of the original video; For each frame of image, a two-dimensional outline image of the fallen object is generated based on the 3D human key points corresponding to the image. Based on the two-dimensional outline image of the human body, a multi-view two-dimensional image of the fallen object is determined, the multi-view two-dimensional image including a front view, a left view, a right view and a rear view; Generate normal maps for each of the 2D images from multiple perspectives; A three-dimensional human body mesh model is generated based on the multi-view two-dimensional images and their corresponding normal maps. The three-dimensional human body mesh model includes the three-dimensional virtual object corresponding to the fallen object. The human body key points in the three-dimensional virtual objects corresponding to each image are matched with the 3D human body key points corresponding to each image to obtain the target human body key points that are successfully matched in each three-dimensional virtual object. Based on the time series of all images in the original video, the positions of key points of each target human body in each of the three-dimensional virtual objects are adjusted to obtain a three-dimensional video, which is used to recreate the falling scene of the fallen object.

[0007] According to the present invention, a method for reconstructing a fall scene using three-dimensional modeling is provided, wherein the original video also includes a tripping object that trips the fallen object; The method further includes: Extract multiple 3D object key points of the tripped object in each of the images; Based on the key points of 3D objects in each of the images, a three-dimensional object mesh model is determined, and the three-dimensional object mesh model includes the three-dimensional virtual object corresponding to the tripped object; The key points of the objects in the three-dimensional virtual objects corresponding to each of the images are matched with the key points of the 3D objects corresponding to each of the images to obtain the target object key points that are successfully matched in each of the three-dimensional virtual objects. Based on the time series of all images in the original video, the positions of key points of each target item in each of the three-dimensional virtual items are adjusted to update the three-dimensional video, thereby obtaining a target three-dimensional video. The target three-dimensional video is used to recreate the scene where the falling object is tripped by the tripping item.

[0008] According to the present invention, a method for reconstructing a fall scenario using 3D modeling is provided, wherein generating a 2D contour image of the fallen object based on 3D human key points corresponding to the image includes: The coordinate data of each 3D human body key point are standardized to obtain the standardized coordinate data of each 3D human body key point. The coordinate data of each standardized 3D human body key point are input into the diffusion model, and the human body contour boundary is added through the diffusion model to obtain the human body two-dimensional contour image output by the diffusion model.

[0009] According to the present invention, a method for reconstructing a fall scenario using 3D modeling is provided, wherein determining a multi-view 2D image of the fallen object based on the 2D contour image of the human body includes: The two-dimensional human body contour image is input into the multi-view diffusion model to obtain the initial two-dimensional image with multiple views output by the multi-view diffusion model. The initial two-dimensional images from all perspectives are stitched together to obtain a stitched two-dimensional image; The stitched two-dimensional image is input into the multi-view perception model to obtain a multi-view perceptual two-dimensional image output by the multi-view perception model. The resolution of the perceptual two-dimensional image is higher than that of the initial two-dimensional image. The multi-view perception model is used to obtain a high-resolution image based on multiple low-resolution images. The multi-view perceptual 2D images are input into a single-view super-resolution model to obtain the multi-view 2D images output by the single-view super-resolution model. The resolution of the 2D images is higher than that of the perceptual 2D images. The single-view super-resolution model is used to convert the low-resolution perceptual 2D images into high-resolution 2D images.

[0010] According to the present invention, a method for reconstructing a fall scenario using 3D modeling is provided, wherein generating normal maps corresponding to 2D images from multiple perspectives includes: Two-dimensional images from multiple perspectives are input into a normal diffusion model to obtain initial normal maps corresponding to each perspective output by the normal diffusion model. The normal diffusion model is used to generate normal maps of two-dimensional images. The initial normal maps corresponding to each viewpoint are input into the single-view super-resolution model to obtain the normal maps of each viewpoint output by the single-view super-resolution model. The resolution of the normal maps is higher than that of the initial normal maps.

[0011] According to the present invention, a method for reconstructing a fall scenario using 3D modeling is provided, wherein generating a 3D human body mesh model based on the multi-view 2D images and their corresponding normal maps includes: Integrate the positions of each pixel in the normal map of the front view to obtain the first depth map corresponding to the front view, and integrate the positions of each pixel in the normal map of the back view to obtain the second depth map corresponding to the back view. Each pixel in the first depth map and the second depth map is mapped to a spatial location to obtain a frontal mesh model and a negative mesh model; The positive mesh model and the negative mesh model are processed by Poisson reconstruction to obtain the initial mesh model; The three-dimensional human body mesh model is generated based on the multi-view two-dimensional images, the normal maps, and the initial mesh model.

[0012] According to the present invention, a method for reconstructing a fall scenario using 3D modeling is provided, wherein generating the 3D human body mesh model based on the multi-view 2D images, each of the normal maps, and the initial mesh model includes: Based on the multi-view 2D images and the normal maps, each vertex in the initial mesh model is moved using a differentiable rendering method to obtain an optimized mesh model. Based on the multi-view 2D images, the visibility of each vertex in the optimized mesh model in the 2D images of each view and the color of each vertex in the 2D images of each view are determined, wherein the visibility includes visible or invisible. Based on the normal maps, the normal direction of each vertex in the optimized mesh model is determined in the normal map of each viewpoint; The target color of each vertex is determined based on the visibility of each vertex in the two-dimensional image of each viewpoint, the color of each vertex in the two-dimensional image of each viewpoint, and the normal direction of each vertex in the normal map of each viewpoint. Based on the target color of each vertex, the optimized mesh model is rendered to obtain the three-dimensional human body mesh model.

[0013] According to the present invention, a method for reconstructing a fall scenario using 3D modeling is provided, wherein generating a 2D contour image of the fallen object based on 3D human key points corresponding to the image includes: Obtain a target image of the fallen object, the target image containing the human features of the fallen object; Based on the human body features of the fallen object and the 3D human body key points corresponding to each image, a two-dimensional human body contour image of the fallen object is generated.

[0014] The present invention also provides a fall scene reconstruction system using three-dimensional modeling, comprising: The acquisition module is used to acquire the original video, which includes the fallen object; An extraction module is used to extract multiple 3D human key points of the fallen object from each frame of the original video. The generation module is used to generate a two-dimensional human contour image of the fallen object based on the 3D human key points corresponding to the image for each frame. The determination module is used to determine multi-view two-dimensional images of the fallen object based on the two-dimensional outline image of the human body, wherein the multi-view two-dimensional images include a front view, a left view, a right view, and a rear view. The generation module is also used to generate normal maps corresponding to the two-dimensional images from multiple perspectives. The generation module is also used to generate a three-dimensional human body mesh model based on the multi-view two-dimensional images and their respective corresponding normal maps. The three-dimensional human body mesh model includes a three-dimensional virtual object corresponding to the fallen object. The matching module is used to match the human body key points in the three-dimensional virtual objects corresponding to each image with the 3D human body key points corresponding to each image, so as to obtain the target human body key points that are successfully matched in each three-dimensional virtual object. The adjustment module is used to adjust the position of each target human body key point in each of the three-dimensional virtual objects based on the time series of all images in the original video to obtain a three-dimensional video, which is used to recreate the falling scene of the falling object.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the fall scene reconstruction method using three-dimensional modeling as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the fall scene reconstruction method using three-dimensional modeling as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the fall scene reconstruction method using three-dimensional modeling as described above.

[0018] This invention provides a method and system for reconstructing a fall scenario using 3D modeling. The method involves acquiring an original video containing the fallen object and extracting multiple 3D human key points from each frame of the original video. For each frame, a 2D human contour image of the fallen object is generated based on the corresponding 3D human key points. Based on the 2D human contour image, multi-view 2D images of the fallen object are determined, and normal maps corresponding to each of the multi-view 2D images are generated. A 3D human mesh model is generated based on the multi-view 2D images and their corresponding normal maps. This 3D human mesh model includes a 3D virtual object corresponding to the fallen object. The human key points in the 3D virtual object corresponding to each image are matched with the corresponding 3D human key points in each image. After obtaining the successfully matched target human key points in each 3D virtual object, the positions of the target human key points in each 3D virtual object are adjusted based on the time series of all images in the original video to obtain a 3D video. This 3D video is used to reconstruct the fall scenario of the fallen object. Because 3D information is reconstructed from ordinary 2D videos using artificial intelligence algorithms, fall scenarios can be recreated without relying on specific camera equipment, improving the flexibility of fall scenario reconstruction applications. Furthermore, by employing multi-view normal mapping constraints and iterative optimization with differentiable rendering, the details of fall scenario reconstruction are refined layer by layer, from contours and surface orientation to texture details, thus enhancing the accuracy of detail in fall scenario reconstruction. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the fall scenario reconstruction method using 3D modeling provided by the present invention.

[0021] Figure 2 This is a schematic diagram of the fall scene reconstruction system implemented using three-dimensional modeling, as provided in an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] Currently, human motion analysis based on surveillance video typically relies on capturing two-dimensional images using camera equipment and then analyzing these images to recognize human movements. However, two-dimensional images cannot provide sufficient depth information, thus limiting the level of detail when reconstructing human fall movements based on two-dimensional images.

[0025] To address this issue, a 3D model needs to be constructed to accurately reconstruct the fall scenario. Current technologies typically utilize a Kinect camera to acquire 3D coordinate data of the human skeleton, extract geometric features such as joint angles from this data, and then preprocess it before using a classifier like a random forest to perform motion recognition, thereby reconstructing the fall scenario.

[0026] However, the above methods rely on Kinect cameras, which limits the application scenarios and flexibility of fall scene reconstruction. Furthermore, because they only use geometric features such as joint angles for motion recognition, they cannot fully capture the detailed spatiotemporal changes of complex movements like falls, resulting in low accuracy in fall scene reconstruction.

[0027] In view of the above problems, this invention proposes a fall scenario reconstruction method using 3D modeling, which can not only improve the flexibility of fall scenario reconstruction in terms of scene application, but also improve the accuracy of details in fall scenario reconstruction.

[0028] This invention can be applied to any scenario requiring the reconstruction of a fall, such as nursing homes, shopping malls, and schools. The implementer of this method can be an electronic device such as a camera, terminal device, computer, server, server cluster, or a specially designed fall scene reconstruction device using 3D modeling. Alternatively, it can be a fall scene reconstruction device using 3D modeling installed within the electronic device, which can be implemented through software, hardware, or a combination of both.

[0029] Figure 1 This is a flowchart illustrating the fall scenario reconstruction method using 3D modeling provided by the present invention, as shown below. Figure 1 As shown, the method includes: Step 101: Obtain the original video, which includes the fallen object.

[0030] In this step, the original video is a video stream file captured by a camera device, which records the entire falling process of the object, from standing up, losing balance, to hitting the ground. The falling object can include a human body. The camera device can be a regular camera device without depth information acquisition, or it can be a camera device with depth information acquisition; this embodiment of the invention does not limit the type of camera device.

[0031] Step 102: Extract multiple 3D human key points of the fallen object from each frame of the original video.

[0032] In this step, for each frame of the original video, the human keypoint extraction method can be used to extract multiple 3D human keypoints of the fallen object, resulting in a set of 3D human keypoints for each frame.

[0033] Specifically, based on AlphaPose's human joint detection algorithm, a top-down approach can be used to detect the coordinates of 17 joints and the 2D coordinates of 5 limb endpoints of the fallen object in each frame of the image. Further, by combining monocular depth prediction, temporal triangulation, or human model fitting, the depth values ​​of each keypoint can be inferred, thereby obtaining the 3D coordinates of each keypoint. The 17 extracted joints include the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left foot, and right foot. The 5 limb endpoints include the top of the head, the end of the left hand, the end of the right hand, the end of the left foot, and the end of the right foot.

[0034] Furthermore, by combining the coordinate data of the 3D human body key points obtained above with time information, a time-stamped time series data K can be constructed. 3 (t). Among them, , Let represent the three-dimensional coordinates of the i-th 3D human keypoint in the t-th frame image, where i represents the index of the 3D human keypoint and t represents the time frame sequence.

[0035] Step 103: For each frame of image, generate a two-dimensional outline image of the fallen object based on the corresponding 3D human key points.

[0036] In this step, for each frame, the coordinate data of each 3D human keypoint detected in the image are standardized, and the standardized coordinate data is input into a pre-trained diffusion model to obtain a 2D human contour image of the fallen object output by the diffusion model. This diffusion model is used to add human contour boundaries to each joint region. Additionally, the diffusion model can also be used to calculate bone length and bone rotation angle based on joint point positions.

[0037] Step 104: Based on the two-dimensional outline image of the human body, determine the two-dimensional images of the fallen object from multiple perspectives, including the front view, left view, right view and rear view.

[0038] In this step, a multi-view diffusion model can be used to process a single 2D human contour image to generate four orthogonal 2D images, including a front view, left view, right view, and rear view, to achieve a complete surround view of the human posture. The multi-view diffusion model is a model that learns the distribution of a 2D image through training and expands that 2D image into 3D space, thereby generating images from multiple different perspectives.

[0039] By generating multi-view 2D images, the fallen object can be observed from different directions, solving the problems of single-view occlusion and lack of depth information, and providing complete visual information for subsequent 3D reconstruction.

[0040] Step 105: Generate normal maps corresponding to the two-dimensional images from multiple perspectives.

[0041] In this step, a normal diffusion model can be used to generate corresponding normal maps for the front, left, right, and rear views to record the normal vector information of each pixel on the human body surface from each viewpoint. The normal map is a texture used to store the normal direction of the object's surface, enhancing the lighting effects and detail of the 3D model through normal information. Each pixel records the normal vector of its corresponding 3D point (i.e., the perpendicular direction of the surface at that point), used to upgrade 2D visual information to computable 3D geometric constraints.

[0042] Step 106: Generate a 3D human body mesh model based on the 2D images from multiple perspectives and their corresponding normal maps. The 3D human body mesh model includes the 3D virtual object corresponding to the fallen object.

[0043] In this step, a mesh reconstruction algorithm is used to fuse multi-view 2D images with corresponding normal maps. After initial mesh estimation, differentiable rendering iterative optimization, and color completion, a 3D human body mesh model is generated. This mesh reconstruction algorithm is an efficient 3D mesh reconstruction algorithm used to reconstruct high-precision 3D meshes from 2D images and normal maps. The 3D human body mesh model contains the complete geometric shape and texture information of the fallen object.

[0044] Step 107: Match the human body key points in the 3D virtual object corresponding to each image with the 3D human body key points corresponding to each image to obtain the target human body key points that are successfully matched in each 3D virtual object.

[0045] In this step, when performing keypoint matching, if the order of the human body keypoints in the 3D virtual object corresponds one-to-one with the order of the 3D human body keypoints, then matching can be performed directly through the keypoint index.

[0046] If the order or number of human key points in the 3D virtual object is inconsistent with the 3D human key points, dynamic programming or nearest neighbor search can be used to find the optimal match. Specifically, the key point distance optimization function shown in formula (1) can be used to minimize the Euclidean distance between the coordinates of the human key points in the 3D virtual object and the coordinates of the 3D human key points extracted in step 102, thereby achieving key point matching. That is, the human key point in the 3D virtual object that is closest to the 3D human key point is determined as the target human key point of successful matching. The formula (1) is as follows: (1) in, This represents the coordinates of the i-th human body key point in a 3D virtual object. This represents the coordinates of the i-th 3D human keypoint, and n represents the number of human keypoints, or the number of 3D human keypoints.

[0047] Step 108: Based on the time series of all images in the original video, adjust the position of each target human body key point in each 3D virtual object to obtain a 3D video. The 3D video is used to recreate the falling scene of the fallen object.

[0048] In this step, since the 3D human key points extracted in step 102 are time series with timestamps, the positions of each target human key point in each 3D virtual object can be dynamically adjusted according to the time series to adjust the motion trajectory of the 3D virtual object so that it is consistent with the movement of the 3D human key points, thereby making the adjusted motion trajectory consistent with the actual falling process of the falling object.

[0049] Specifically, a 3D skeletal skinning animation method can be used, where each key point of the target human body is used as a skeletal control point to drive the movement of the vertices of the 3D human body mesh model. The skeletal movement follows the movement trajectory of the joints, thus making the falling action of the 3D human body mesh model more natural.

[0050] Furthermore, the trajectory can be smoothed by minimizing the motion trajectory optimization function (2) to reduce noise and errors, and finally generate a three-dimensional video that can be played back frame by frame to completely restore the falling scene of the falling object: (2) Where T represents the total number of frames in the 3D video. This represents the coordinates of the i-th key point of the target human body at time t+1. This represents the coordinates of the i-th key point of the target human body at time t.

[0051] The fall scene reconstruction method using 3D modeling provided in this invention involves acquiring an original video including the fallen object, extracting multiple 3D human key points from each frame of the original video, generating a 2D human contour image of the fallen object based on the corresponding 3D human key points for each frame, determining multi-view 2D images of the fallen object based on the 2D human contour images, generating normal maps corresponding to each of the multi-view 2D images, generating a 3D human mesh model based on the multi-view 2D images and their corresponding normal maps, including a 3D virtual object corresponding to the fallen object, matching the human key points in the 3D virtual object corresponding to each image with the corresponding 3D human key points in each image, obtaining the successfully matched target human key points in each 3D virtual object, and then adjusting the positions of each target human key point in each 3D virtual object based on the time series of all images in the original video to obtain a 3D video, which is used to reconstruct the fall scene of the fallen object. Because 3D information is reconstructed from ordinary 2D videos using artificial intelligence algorithms, fall scenarios can be recreated without relying on specific camera equipment, improving the flexibility of fall scenario reconstruction applications. Furthermore, by employing multi-view normal mapping constraints and iterative optimization with differentiable rendering, the details of fall scenario reconstruction are refined layer by layer, from contours and surface orientation to texture details, thus enhancing the accuracy of detail in fall scenario reconstruction.

[0052] For example, based on the above embodiments, when the original video also includes the tripping object that tripped the fallen object, multiple 3D object key points of the tripping object in each image can be extracted. Based on the 3D object key points in each image, a three-dimensional object mesh model is determined. The three-dimensional object mesh model includes the three-dimensional virtual object corresponding to the tripping object. The object key points in the three-dimensional virtual object corresponding to each image are matched with the 3D object key points corresponding to each image to obtain the successfully matched target object key points in each three-dimensional virtual object. Based on the time series of all images in the original video, the position of each target object key point in each three-dimensional virtual object is adjusted to update the three-dimensional video and obtain the target three-dimensional video. The target three-dimensional video is used to recreate the scene of the fallen object being tripped by the tripping object.

[0053] Specifically, tripping objects can include ground obstacles, wheelchairs, steps, etc. Object detection algorithms can be used to identify and locate tripping objects in each frame of the image. Then, using 3D object keypoint detection methods similar to those used for human keypoint extraction, such as PVNet-based 6D pose estimation algorithms, 3D keypoints such as corner points and center points of the object can be extracted. Based on the extracted 3D object keypoints, a 2D contour image of the tripping object is generated using the same diffusion model process as for generating 2D human contours. Based on this 2D contour image, multi-view 2D images of the object and corresponding normal maps are generated. A 3D object mesh model is then constructed using a mesh reconstruction algorithm. This 3D object mesh model includes the corresponding 3D virtual object.

[0054] Furthermore, the key points of the 3D virtual objects in each frame image are matched with the corresponding 3D object key points. The registration is achieved by minimizing the Euclidean distance, and the target object key points that are successfully matched are determined.

[0055] Based on the original video time series, the positions of key points of the target object are adjusted, thereby adjusting the position and posture of the 3D virtual object to match the actual movement trajectory of the tripped object. Finally, the updated 3D virtual object and the 3D virtual object are merged and rendered in a unified 3D scene to generate the target 3D video, which can completely restore the entire process of the falling object being collided with and tripped by the tripped object.

[0056] In this embodiment, by detecting and reconstructing the motion trajectory of the tripped object, and rendering the three-dimensional virtual object and the three-dimensional virtual object into the target three-dimensional video in a unified manner, the entire process of human-object interaction dynamics can be accurately reproduced, further improving the accuracy of fall scene reconstruction.

[0057] For example, based on the above embodiments, when generating a two-dimensional human contour image of a fallen object based on the 3D human key points corresponding to the image, the coordinate data of each 3D human key point can be standardized to obtain the standardized coordinate data of each 3D human key point. The standardized coordinate data of each 3D human key point is then input into the diffusion model, and the human contour boundary is added through the diffusion model to obtain the two-dimensional human contour image output by the diffusion model.

[0058] Specifically, the coordinate data of each 3D human body key point can be standardized according to the following formula (3): (3) in, Represents the x-coordinate, y-coordinate, or depth coordinate of the i-th 3D human keypoint after standardization. Represents the x-coordinate, y-coordinate, or depth coordinate of the i-th 3D human keypoint. This represents the maximum value of the x-coordinate, y-coordinate, or depth coordinate of the i-th 3D human keypoint across all frames. This represents the minimum x-coordinate, y-coordinate, or depth coordinate of the i-th 3D human keypoint across all frames.

[0059] Formula (3) can be used to standardize the horizontal, vertical or depth coordinates of each 3D human key point to obtain the standardized coordinate data of each 3D human key point.

[0060] The standardized coordinate data of each 3D human key point is input into the diffusion model, which can add human contour boundaries to each joint region to obtain a two-dimensional human contour image output by the diffusion model.

[0061] In this approach, by standardizing the key points of each 3D human body, the scale differences between individuals of different heights and body types can be eliminated, the data distribution can be unified, and the diffusion model can stably learn the posture structure in the normalized space. Moreover, when generating a 2D human body contour image through the diffusion model, the generation position and shape of the contour of each joint area can be precisely controlled, ensuring that the output 2D human body contour is aligned with the real 3D posture, and avoiding structural misalignment and limb loss.

[0062] In addition, the diffusion model can also calculate bone length and bone rotation angle by the position of each 3D human key point. In order to eliminate the geometric scale differences between different individuals and unify the motion representation space, the bone length can also be standardized according to formula (3). At this time, the bone length in formula (3) is... This represents the length of the i-th bone after standardization. This represents the length of the i-th bone. This represents the maximum length of the i-th bone across all frames. This represents the minimum length of the i-th bone across all frames. The bone rotation angle is standardized according to formula (4): (4) in, This represents the standardized bone rotation angle. This indicates the bone rotation angle before standardization. This represents the maximum value of the bone rotation angle across all frames before normalization. This represents the minimum bone rotation angle across all frames before standardization.

[0063] By determining the bone length and bone rotation angle, constraints that conform to the laws of human movement can be applied during subsequent optimization of the 3D human body mesh model and adjustment of motion trajectory. This prevents the generation of unnatural shapes such as reverse joints and excessive limb stretching, ensuring the physiological rationality of virtual object movements and the robustness of cross-scale reconstruction.

[0064] For example, based on the above embodiments, when determining a multi-view two-dimensional image of a fallen object based on a two-dimensional human body contour image, it can be done in the following way: A 2D human contour image is input into a multi-view diffusion model to obtain initial 2D images from multiple perspectives. These initial 2D images from all perspectives are then stitched together to obtain a stitched 2D image. This stitched 2D image is then input into a multi-view perception model to obtain a multi-view perceptual 2D image from multiple perspectives. The resolution of the perceptual 2D image is higher than that of the initial 2D image. The multi-view perception model is used to obtain a high-resolution image based on multiple low-resolution images. The multi-view perceptual 2D images are then input into a single-view super-resolution model to obtain multi-view 2D images from multiple perspectives. The resolution of these 2D images is higher than that of the perceptual 2D images. The single-view super-resolution model is used to convert the low-resolution perceptual 2D images into high-resolution 2D images.

[0065] Specifically, the multi-view diffusion model is based on the Stable Diffusion pre-trained weight initialization. By encoding multi-view dependencies, the parameters of the initialized model are fine-tuned so that it can generate four orthogonal initial two-dimensional images with a resolution of 256×256 from a single human two-dimensional contour image.

[0066] After generating four initial 2D images of orthogonal views, these four images are juxtaposed and stitched together. This stitched image is then input into the ControlNet multi-view perception model as control information. This model utilizes the constraints of the four juxtaposed RGB images to generate clearer and more accurate multi-view perception 2D images through fine-tuning, increasing the resolution to 512×512. The juxtaposition mentioned above can be understood as stitching the four initial 2D images from different perspectives into a single large image. There are generally two methods: horizontal juxtaposition and matrix juxtaposition. Horizontal juxtaposition arranges the four initial 2D images horizontally, forming a row containing all four perspectives. Matrix juxtaposition arranges the four initial 2D images in a 2×2 pattern, forming a 2x2 image matrix.

[0067] Furthermore, the perceptual 2D images from each viewpoint are input into a single-view super-resolution model, which independently processes each perceptual 2D image and magnifies it four times, ultimately outputting a multi-view 2D image with a resolution of 2048×2048.

[0068] In this embodiment, by continuously improving the resolution of the image, the geometric consistency and texture details of the fallen object can be optimized in stages, avoiding structural distortion and noise amplification caused by directly generating ultra-high resolution images, thus ensuring the accuracy of details when reconstructing the fall scene.

[0069] For example, based on the above embodiments, when generating normal maps corresponding to two-dimensional images from multiple perspectives, the two-dimensional images from multiple perspectives can be input into a normal diffusion model to obtain the initial normal maps corresponding to each perspective output by the normal diffusion model. The normal diffusion model is used to generate normal maps of two-dimensional images, and the initial normal maps corresponding to each perspective are input into a single-view super-resolution model to obtain the normal maps of each perspective output by the single-view super-resolution model. The resolution of the normal maps is higher than the resolution of the initial normal maps.

[0070] Specifically, the normal diffusion model is based on the Stable Diffusion pre-trained weight initialization and the model obtained by the deep surface normal estimation method. This method performs hierarchical fusion with the fusion module at different scales through RGB-D branch and depth branch. The confidence map estimation is used for depth adjustment inside the fusion module. Finally, the two-dimensional images of each viewpoint are calibrated by minimizing the mixing loss function (5) to generate the initial normal map corresponding to each viewpoint: (5) in, This represents the normal vector of the predicted normal map at pixel position (x, y). This represents the normal vector of the real normal map at pixel position (x, y).

[0071] After generating the initial normal map of the 2D image through the normal diffusion model, the initial normal maps of each viewpoint are input into the single-view super-resolution model. The single-view super-resolution model can independently process each normal map and enlarge it by four times, and finally output the normal maps of each viewpoint with a resolution higher than that of the initial normal map, thereby accurately capturing the microscopic geometric orientation information of each pixel on the surface of the human body and objects.

[0072] In this embodiment, after generating the initial normal map using the diffusion model, each initial normal map can be independently magnified by four times the resolution based on the single-view super-resolution model, capturing more microscopic geometric details while maintaining the accuracy of the normal vector direction.

[0073] For example, based on the above embodiments, when generating a 3D human body mesh model from multiple perspective 2D images and their corresponding normal maps, it can be done in the following way: By integrating the positions of each pixel in the normal map of the front view, a first depth map corresponding to the front view is obtained. Similarly, by integrating the positions of each pixel in the normal map of the back view, a second depth map corresponding to the back view is obtained. Each pixel in the first and second depth maps is then mapped to a spatial location to obtain a frontal mesh model and a negative mesh model. Finally, the frontal mesh model and the negative mesh model are processed by Poisson reconstruction to obtain an initial mesh model. Thus, a three-dimensional human body mesh model is generated based on the multi-view two-dimensional images, the normal maps, and the initial mesh model.

[0074] Specifically, a 3D human body mesh model can be generated using a mesh reconstruction method based on multi-view 2D images and their corresponding normal maps. The mesh reconstruction algorithm includes initial mesh estimation, mesh optimization, explicit target optimization, and color completion.

[0075] For the initial mesh estimation, the integral formula (6) can be applied to the normal maps of the front and back views respectively. The depth value is calculated by accumulating the normal components along the integral path and taking the average value, thus generating the first depth map. With the second depth map : (6) Where n(x) represents the gradient of the depth map in the x-direction. After obtaining the first depth map and the second depth map, each pixel in the first depth map and the second depth map is back-projected into the three-dimensional spatial coordinate system according to its depth value, and the front mesh model and the back mesh model are constructed respectively.

[0076] Furthermore, the surface continuity equation can be solved using the Poisson reconstruction algorithm, which merges the front and back mesh models at the side seams to ensure a smooth transition, generating a closed initial 3D mesh model, completing mesh initialization, and providing a geometrically correct initial shape for subsequent optimization.

[0077] After determining the initial mesh model, it is necessary to perform mesh optimization, explicit target optimization, and color completion on the initial mesh model based on multi-view 2D images and normal maps of each view to obtain the final 3D human body mesh model.

[0078] In the above embodiments, by determining the first depth map corresponding to the front view and the second depth map corresponding to the back view, and based on the first depth map and the second depth map, the front mesh model and the negative mesh model are determined. Then, the front mesh model and the negative mesh model are processed by Poisson reconstruction, thereby generating a closed and geometrically continuous three-dimensional initial mesh model, automatically filling the side holes and gaps between the front and back sides, and providing a reliable shape reference for subsequent mesh optimization.

[0079] For example, when generating a 3D human body mesh model based on multi-view 2D images, various normal maps, and an initial mesh model, it can be done in the following way: Based on multi-view 2D images and various normal maps, each vertex in the initial mesh model is moved using a differentiable rendering method to obtain an optimized mesh model. Based on the multi-view 2D images, the visibility and color of each vertex in the optimized mesh model in each view 2D image are determined. Visibility includes visible or invisible. Based on each normal map, the normal direction of each vertex in the normal map of each view in the optimized mesh model is determined. Based on the visibility, color, and normal direction of each vertex in the 2D images of each view, the target color of each vertex is determined. Finally, based on the target color of each vertex, the optimized mesh model is rendered to obtain a 3D human body mesh model.

[0080] Specifically, after generating an initial mesh model through depth estimation and Poisson reconstruction of the front and rear views, preliminary prediction masks can be rendered at different viewpoints k based on this initial mesh model. and prediction normal map .

[0081] Based on the reverse rendering method, this embodiment iteratively optimizes the initial mesh model by minimizing the loss function (7). In each optimization step, the mesh undergoes differentiable rendering to calculate the loss and gradient, and then the vertices are moved according to the gradient. Specifically, in the differentiable rendering optimization stage, the initial mesh model is rendered to four views respectively, and the mask-based loss is calculated for each view based on formulas (8) and (9). Compared with loss based on normals Among them, the mask-based loss The mesh profile is constrained, and the loss is based on the normal. The direction of the normal in the visible area can be optimized: (7) (8) (9) in, Indicates the total loss. This represents the rendering mask under view k. This represents the prediction mask under view k. This represents the rendered normal map under view k. This represents the predicted normal map under view k, used to optimize the normal direction of the visible area. This represents element-wise product.

[0082] Loss based on mask By comparing the error between the rendered mask and the real mask, the mesh contour is constrained, driving vertex movement to match the human body shape in multi-view 2D images; loss based on normals... By comparing the differences between the rendered normal map and the predicted normal map, the surface orientation is optimized only in the visible area to ensure that the geometric details are consistent with the normal map. Figure 1 To.

[0083] After multiple iterations from coarse to fine, the mesh vertices converge to the target shape, resulting in an optimized mesh model. Then, the color completion stage begins. For each vertex v in the optimized mesh model, the visibility of each vertex in the 2D image from each viewpoint and its color in the 2D image from each viewpoint can be determined. Visibility can be represented numerically, such as 0 for invisible and 1 for visible.

[0084] In addition, based on each normal map, the normal direction of each vertex in the optimized mesh model in each viewpoint can be determined, thereby determining the weight factor based on the normal direction and the viewing direction of each viewpoint, and calculating the target color of vertex v according to formula (10): (10) in, Indicates the target color of vertex v. This represents the color of vertex v in a two-dimensional image at viewpoint k. This represents the visibility of vertex v in a 2D image with viewpoint k; a value of 1 indicates visibility, and a value of 0 indicates invisibility. Weighting factor. , Represents a grid Mid-vertices The direction of the normal, Indicates perspective The direction of the line of sight.

[0085] In addition, the weighted average function in formula (10) It can be calculated as follows: The above formula is a weighted average function. The general calculation formula is as follows: when calculating formula (10), replace A in the above formula with Replace B with That's all.

[0086] Furthermore, since the projected area is positively correlated with the cosine value, the prediction accuracy is also positively correlated with the cosine value. The color rendering error is minimized by defining a target loss function (11) for the explicit target, and the final optimized loss function (12) is synthesized to minimize it: (11) (12) in, The grid M is formed by the k-th viewpoint. Generated, This represents the target loss function value. This represents the loss based on the mask.

[0087] In this embodiment, by optimizing the initial mesh model and color rendering, vertex color assignment can be completed while maintaining the mask contour constraints, thereby obtaining a textured and geometrically accurate 3D human body mesh model.

[0088] For example, based on the above embodiments, when generating a two-dimensional human contour image of the fallen object based on the 3D human key points corresponding to the image, a target image of the fallen object can also be obtained. The target image contains the human features of the fallen object, and a two-dimensional human contour image of the fallen object is generated based on the human features of the fallen object and the 3D human key points corresponding to each image.

[0089] Specifically, when generating a 2D human contour image using the diffusion model, a target image of the fallen object can also be obtained. This target image is a clear, unobstructed full-body frontal image, either pre-shot or selected from a video, which can serve as an identity reference template. The target image and the coordinates of 3D human key points are simultaneously input into the diffusion model. The model extracts high-dimensional features such as human shape, proportions, and clothing texture from the target image through an encoder and injects these features as conditional vectors into the diffusion process. This ensures that the target template is preferentially referenced when generating the contour. Especially in cases of occlusion, motion blur, or low resolution in video frames, the target image can be used to complete the contour curves of occluded parts, maintaining cross-frame identity consistency. This results in the generation of a 2D human contour image with a high degree of similarity to the real object, improving the accuracy of the 2D human contour image.

[0090] Similarly, for tripped objects, a separate image can be input, which allows the model to more accurately capture the geometric features and texture details of the object, improving the accuracy of subsequent modeling.

[0091] The fall scene reconstruction method using 3D modeling provided in this invention can achieve high-fidelity reconstruction of fall scenes without relying on manually labeled key points or simple kinematic models. Compared to methods that rely on multiple camera views or LiDAR to obtain precise depth information, this invention is low-cost and highly efficient. Furthermore, the inputs and outputs of each part in this invention are interconnected, extracting and integrating different features, improving efficiency and retaining more original information in the generated results. Moreover, this invention employs a multi-stage, refined processing flow to ensure higher accuracy and realism, using normal mapping generation technology to calculate depth information instead of traditional multi-view triangulation, simplifying the process and improving detail reconstruction capabilities.

[0092] The fall scenario reconstruction system based on three-dimensional modeling provided by the present invention will be described below. The fall scenario reconstruction system based on three-dimensional modeling described below can be referred to in correspondence with the fall scenario reconstruction method based on three-dimensional modeling described above.

[0093] Figure 2 This is a schematic diagram of the fall scene reconstruction system implemented using three-dimensional modeling, as provided in an embodiment of the present invention. Figure 2 As shown, the fall scenario reconstruction system 200 implemented using 3D modeling includes: Acquisition module 11 is used to acquire the original video, which includes the fallen object; Extraction module 12 is used to extract multiple 3D human key points of the fallen object from each frame of the original video. The generation module 13 is used to generate a two-dimensional human contour image of the fallen object based on the 3D human key points corresponding to the image for each frame. The determining module 14 is used to determine a multi-view two-dimensional image of the fallen object based on the two-dimensional outline image of the human body. The multi-view two-dimensional image includes a front view, a left view, a right view, and a rear view. The generation module 13 is also used to generate normal maps corresponding to the two-dimensional images from multiple perspectives. The generation module 13 is also used to generate a three-dimensional human body mesh model based on the multi-view two-dimensional images and their respective corresponding normal maps. The three-dimensional human body mesh model includes a three-dimensional virtual object corresponding to the fallen object. Matching module 15 is used to match the human body key points in the three-dimensional virtual objects corresponding to each image with the 3D human body key points corresponding to each image, so as to obtain the target human body key points that are successfully matched in each three-dimensional virtual object. The adjustment module 16 is used to adjust the position of each target human body key point in each of the three-dimensional virtual objects based on the time series of all images in the original video to obtain a three-dimensional video, which is used to recreate the falling scene of the falling object.

[0094] In one example embodiment, the original video also includes a tripping object that trips the fallen object; The extraction module 12 is also used to extract multiple 3D object key points of the tripped object in each of the images; The determining module 14 is further configured to determine a three-dimensional object mesh model based on the 3D object key points in each of the images, wherein the three-dimensional object mesh model includes the three-dimensional virtual object corresponding to the tripped object; The matching module 15 is further configured to match the key points of the objects in the three-dimensional virtual objects corresponding to each image with the key points of the 3D objects corresponding to each image, so as to obtain the target object key points that are successfully matched in each three-dimensional virtual object. The adjustment module 16 is also used to adjust the position of key points of each target item in each of the three-dimensional virtual items based on the time series of all images in the original video, update the three-dimensional video, and obtain a target three-dimensional video. The target three-dimensional video is used to recreate the scene of the falling object being tripped by the tripping item.

[0095] In one example embodiment, the generation module 13 is specifically used for: The coordinate data of each 3D human body key point are standardized to obtain the standardized coordinate data of each 3D human body key point. The coordinate data of each standardized 3D human body key point are input into the diffusion model, and the human body contour boundary is added through the diffusion model to obtain the human body two-dimensional contour image output by the diffusion model.

[0096] In one example embodiment, the determining module 14 is specifically used for: The two-dimensional human body contour image is input into the multi-view diffusion model to obtain the initial two-dimensional image with multiple views output by the multi-view diffusion model. The initial two-dimensional images from all perspectives are stitched together to obtain a stitched two-dimensional image; The stitched two-dimensional image is input into the multi-view perception model to obtain a multi-view perceptual two-dimensional image output by the multi-view perception model. The resolution of the perceptual two-dimensional image is higher than that of the initial two-dimensional image. The multi-view perception model is used to obtain a high-resolution image based on multiple low-resolution images. The multi-view perceptual 2D images are input into a single-view super-resolution model to obtain the multi-view 2D images output by the single-view super-resolution model. The resolution of the 2D images is higher than that of the perceptual 2D images. The single-view super-resolution model is used to convert the low-resolution perceptual 2D images into high-resolution 2D images.

[0097] In one example embodiment, the generation module 13 is specifically used for: Two-dimensional images from multiple perspectives are input into a normal diffusion model to obtain initial normal maps corresponding to each perspective output by the normal diffusion model. The normal diffusion model is used to generate normal maps of two-dimensional images. The initial normal maps corresponding to each viewpoint are input into the single-view super-resolution model to obtain the normal maps of each viewpoint output by the single-view super-resolution model. The resolution of the normal maps is higher than that of the initial normal maps.

[0098] In one example embodiment, the generation module 13 is specifically used for: Integrate the positions of each pixel in the normal map of the front view to obtain the first depth map corresponding to the front view, and integrate the positions of each pixel in the normal map of the back view to obtain the second depth map corresponding to the back view. Each pixel in the first depth map and the second depth map is mapped to a spatial location to obtain a frontal mesh model and a negative mesh model; The positive mesh model and the negative mesh model are processed by Poisson reconstruction to obtain the initial mesh model; The three-dimensional human body mesh model is generated based on the multi-view two-dimensional images, the normal maps, and the initial mesh model.

[0099] In one example embodiment, the generation module 13 is specifically used for: Based on the multi-view 2D images and the normal maps, each vertex in the initial mesh model is moved using a differentiable rendering method to obtain an optimized mesh model. Based on the multi-view 2D images, the visibility of each vertex in the optimized mesh model in the 2D images of each view and the color of each vertex in the 2D images of each view are determined, wherein the visibility includes visible or invisible. Based on the normal maps, the normal direction of each vertex in the optimized mesh model is determined in the normal map of each viewpoint; The target color of each vertex is determined based on the visibility of each vertex in the two-dimensional image of each viewpoint, the color of each vertex in the two-dimensional image of each viewpoint, and the normal direction of each vertex in the normal map of each viewpoint. Based on the target color of each vertex, the optimized mesh model is rendered to obtain the three-dimensional human body mesh model.

[0100] In one example embodiment, the generation module 13 is specifically used for: Obtain a target image of the fallen object, the target image containing the human features of the fallen object; Based on the human body features of the fallen object and the 3D human body key points corresponding to each image, a two-dimensional human body contour image of the fallen object is generated.

[0101] The system in this embodiment can be used in any of the methods in the side embodiment of the fall scene reconstruction method implemented by 3D modeling. Its specific implementation process and technical effects are similar to those in the side embodiment of the fall scene reconstruction method implemented by 3D modeling. For details, please refer to the detailed description in the side embodiment of the fall scene reconstruction method implemented by 3D modeling, which will not be repeated here.

[0102] Figure 3 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 3As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a fall scene reconstruction method implemented using 3D modeling. This method includes: acquiring an original video, the original video including a falling object; extracting multiple 3D human key points of the falling object from each frame of the original video; generating a 2D human contour image of the falling object for each frame based on the corresponding 3D human key points; determining multi-view 2D images of the falling object based on the 2D human contour images, the multi-view 2D images including a front view, left view, right view, and rear view; and generating a multi-view 2D image. Each image has its corresponding normal map; based on the multi-view 2D images and their corresponding normal maps, a 3D human body mesh model is generated, the 3D human body mesh model including the 3D virtual object corresponding to the fallen object; the human body key points in the 3D virtual object corresponding to each image are matched with the 3D human body key points corresponding to each image to obtain the successfully matched target human body key points in each 3D virtual object; based on the time series of all images in the original video, the positions of each target human body key point in each 3D virtual object are adjusted to obtain a 3D video, the 3D video being used to recreate the falling scene of the fallen object.

[0103] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the fall scene reconstruction method using 3D modeling provided by the above methods. The method includes: acquiring an original video, the original video including a falling object; extracting multiple 3D human key points of the falling object from each frame of the original video; generating a two-dimensional human contour image of the falling object based on the 3D human key points corresponding to each frame; and determining a multi-view two-dimensional image of the falling object based on the two-dimensional human contour image. The two-dimensional image of the angle includes a front view, a left view, a right view, and a rear view; normal maps corresponding to each of the two-dimensional images from multiple perspectives are generated; based on the two-dimensional images from multiple perspectives and their corresponding normal maps, a three-dimensional human body mesh model is generated, the three-dimensional human body mesh model including a three-dimensional virtual object corresponding to the fallen object; the human body key points in the three-dimensional virtual object corresponding to each image are matched with the 3D human body key points corresponding to each image to obtain the successfully matched target human body key points in each of the three-dimensional virtual objects; based on the time sequence of all images in the original video, the positions of each target human body key point in each of the three-dimensional virtual objects are adjusted to obtain a three-dimensional video, the three-dimensional video being used to recreate the falling scene of the fallen object.

[0105] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs the fall scene reconstruction method using 3D modeling provided by the methods described above. The method includes: acquiring an original video, the original video including a falling object; extracting multiple 3D human key points of the falling object from each frame of the original video; generating a 2D human contour image of the falling object for each frame based on the corresponding 3D human key points; and determining multi-view 2D images of the falling object based on the 2D human contour images, the multi-view 2D images including a front view, a left view, and a right view. The video generates a 2D image with a right view and a rear view; it generates normal maps corresponding to each of the 2D images from multiple perspectives; based on the 2D images from multiple perspectives and their corresponding normal maps, it generates a 3D human body mesh model, which includes a 3D virtual object corresponding to the fallen object; it matches the human body key points in the 3D virtual object corresponding to each image with the 3D human body key points corresponding to each image to obtain the successfully matched target human body key points in each 3D virtual object; based on the time series of all images in the original video, it adjusts the positions of each target human body key point in each 3D virtual object to obtain a 3D video, which is used to recreate the falling scene of the fallen object.

[0106] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0107] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fall scenario reconstruction method implemented using three-dimensional modeling, characterized by, The method comprises: obtaining an original video, wherein the original video comprises a falling object; extracting a plurality of 3D human body key points of the falling object from each frame image of the original video; generating a human body two-dimensional contour image of the falling object based on the 3D human body key points corresponding to each frame image; determining a multi-view two-dimensional image of the falling object based on the human body two-dimensional contour image, wherein the multi-view two-dimensional image comprises a front view, a left view, a right view and a back view; generating a normal map corresponding to each of the multi-view two-dimensional images; generating a three-dimensional human body mesh model based on the multi-view two-dimensional images and the normal maps corresponding to the multi-view two-dimensional images, wherein the three-dimensional human body mesh model comprises a three-dimensional virtual object corresponding to the falling object; matching the human body key points in the three-dimensional virtual object corresponding to each of the images with the 3D human body key points corresponding to each of the images to obtain target human body key points matched successfully in the three-dimensional virtual object; adjusting the positions of the target human body key points in each of the three-dimensional virtual objects based on the time sequence of all the images in the original video to obtain a three-dimensional video, wherein the three-dimensional video is used to restore a falling scene of the falling object. 2.The fall scenario reconstruction method using three-dimensional modeling according to claim 1, wherein, The original video further comprises a stumbling object that stumbles the falling object. The method further comprises: extracting a plurality of 3D object key points of the stumbling object in each of the images; determining a three-dimensional object mesh model based on the 3D object key points in each of the images, wherein the three-dimensional object mesh model comprises a three-dimensional virtual object corresponding to the stumbling object; matching the object key points in the three-dimensional virtual object corresponding to each of the images with the 3D object key points corresponding to each of the images to obtain target object key points matched successfully in the three-dimensional virtual object; updating the three-dimensional video based on the time sequence of all the images in the original video to adjust the positions of the target object key points in each of the three-dimensional virtual objects, and obtaining a target three-dimensional video, wherein the target three-dimensional video is used to restore a scene in which the falling object is stumbled by the stumbling object. 3.The fall scenario reconstruction method using three-dimensional modeling according to claim 1, wherein, The method of generating the human body two-dimensional contour image of the falling object based on the 3D human body key points corresponding to each of the images comprises: standardizing the coordinate data of each 3D human body key point to obtain standardized coordinate data of each 3D human body key point; inputting the standardized coordinate data of each 3D human body key point into a diffusion model to add a human body contour boundary through the diffusion model, and obtaining the human body two-dimensional contour image output by the diffusion model. 4.The fall scenario reconstruction method using three-dimensional modeling according to claim 1, wherein, The method of determining the multi-view two-dimensional image of the falling object based on the human body two-dimensional contour image comprises: inputting the human body two-dimensional contour image into a multi-view diffusion model to obtain an initial multi-view two-dimensional image output by the multi-view diffusion model; splicing all the initial two-dimensional images of different views to obtain a spliced two-dimensional image; inputting the spliced two-dimensional images into a multi-view perception model to obtain multi-view perception two-dimensional images output by the multi-view perception model, the resolution of the perception two-dimensional images being higher than the resolution of the initial two-dimensional images, the multi-view perception model being used to obtain high-resolution images based on multiple low-resolution images; inputting the multi-view perception two-dimensional images into single-view super-resolution models respectively to obtain multi-view two-dimensional images output by the single-view super-resolution models, the resolution of the two-dimensional images being higher than the resolution of the perception two-dimensional images, the single-view super-resolution models being used to convert low-resolution perception two-dimensional images into high-resolution two-dimensional images.

5. The fall scenario reconstruction method using three-dimensional modeling according to any one of claims 1-4, characterized in that, the generating of the normal maps corresponding to the multi-view two-dimensional images respectively comprises: inputting the multi-view two-dimensional images into normal diffusion models respectively to obtain initial normal maps corresponding to the respective views output by the normal diffusion models, the normal diffusion models being used to generate normal maps of two-dimensional images; inputting the initial normal maps corresponding to the respective views into single-view super-resolution models to obtain normal maps of the respective views output by the single-view super-resolution models, the resolution of the normal maps being higher than the resolution of the initial normal maps.

6. The fall scenario reconstruction method using three-dimensional modeling according to any one of claims 1-4, characterized in that, the generating of the three-dimensional human body mesh model according to the multi-view two-dimensional images and the normal maps corresponding to the respective views comprises: integrating the positions of the pixel points in the normal map of the front view to obtain a first depth map corresponding to the front view, and integrating the positions of the pixel points in the normal map of the rear view to obtain a second depth map corresponding to the rear view; mapping the pixel points in the first depth map and the second depth map to spatial positions respectively to obtain a front mesh model and a rear mesh model; processing the front mesh model and the rear mesh model through Poisson reconstruction to obtain an initial mesh model; generating the three-dimensional human body mesh model based on the multi-view two-dimensional images, the normal maps, and the initial mesh model.

7. The fall scenario reconstruction method using three-dimensional modeling according to claim 6, characterized in that, the generating of the three-dimensional human body mesh model based on the multi-view two-dimensional images, the normal maps, and the initial mesh model comprises: moving the vertices in the initial mesh model through a differentiable rendering method based on the multi-view two-dimensional images and the normal maps to obtain an optimized mesh model; determining the visibility of the vertices in the optimized mesh model in the two-dimensional images of the respective views and the colors of the vertices in the two-dimensional images of the respective views based on the multi-view two-dimensional images, the visibility including visible or invisible; determining the normal directions of the vertices in the optimized mesh model in the normal maps of the respective views based on the normal maps; determining target colors of the vertices based on the visibility of the vertices in the two-dimensional images of the respective views, the colors of the vertices in the two-dimensional images of the respective views, and the normal directions of the vertices in the normal maps of the respective views; rendering the optimized mesh model based on the target colors of the vertices to obtain the three-dimensional human body mesh model.

8. The fall scenario reconstruction method using three-dimensional modeling according to any one of claims 1-4, characterized in that, The generating, based on the 3D human key points corresponding to the images, of the human body two-dimensional contour image of the falling object includes: Obtaining a target image of the falling object, the target image containing human body features of the falling object; Generating, based on the human body features of the falling object and the 3D human key points corresponding to each of the images, a human body two-dimensional contour image of the falling object.

9. A fall scenario reconstruction system implemented using three-dimensional modeling, the system comprising: Comprise: An acquisition module configured to acquire an original video, the original video containing a falling object; An extraction module configured to extract a plurality of 3D human key points of the falling object from each image of the original video; A generation module configured to generate, for each image, a human body two-dimensional contour image of the falling object based on 3D human key points corresponding to the image; A determination module configured to determine, based on the human body two-dimensional contour image, a multi-view two-dimensional image of the falling object, the multi-view two-dimensional image including a front view, a left view, a right view, and a back view; The generation module is further configured to generate a normal map corresponding to each of the multi-view two-dimensional images; The generation module is further configured to generate a three-dimensional human body mesh model according to the multi-view two-dimensional images and the normal maps corresponding thereto, the three-dimensional human body mesh model including a three-dimensional virtual object corresponding to the falling object; A matching module configured to match human key points in the three-dimensional virtual objects corresponding to each of the images with the 3D human key points corresponding to each of the images to obtain target human key points in each of the three-dimensional virtual objects that are successfully matched; An adjustment module configured to adjust positions of the target human key points in each of the three-dimensional virtual objects based on a time sequence of all images in the original video to obtain a three-dimensional video, the three-dimensional video being used to restore a falling scenario of the falling object.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to implement the falling scenario restoration method using three-dimensional modeling according to any one of claims 1 to 8.

Images