A machine learning-based three-dimensional model video fusion method, device and medium

Abstract

The present application relates to a kind of three-dimensional model video fusion method, equipment and medium based on machine learning, belong to computer vision technical field, the method includes the following steps: obtaining camera imaging parameter, target scene video frame image and target scene three-dimensional model collected by camera;Three-dimensional scene video collaborative fusion model is constructed, including video structure analysis layer, three-dimensional model structure analysis layer, video three-dimensional model alignment layer and fusion output layer;Video structure analysis layer is used to convert target scene video frame image into video mapping structured feature;Three-dimensional model structure analysis layer is used to convert three-dimensional model into discrete structured feature by camera imaging parameter;Video three-dimensional model alignment layer is aligned projection to video mapping structured feature and discrete structured feature, and obtains alignment result;Fusion output layer is used to generate three-dimensional model video fusion result by alignment result.

Description

Technical Field

[0001] This invention relates to a method, device, and medium for fusion of three-dimensional models and videos based on machine learning, belonging to the field of computer vision technology. Background Technology

[0002] With the development of digital twins, smart buildings, virtual simulation, industrial visualization and augmented reality technologies, more and more application scenarios need to integrate real scene videos with target scene 3D models for display, so as to more intuitively restore the real scene status in 3D space, display equipment operation status or conduct remote monitoring and analysis.

[0003] In existing technologies, a common approach is to directly attach video images captured by a camera as a 2D texture to the surface of a 3D model, or to project video images onto the corresponding area of ​​the 3D model based on camera calibration results, thereby obtaining a visual image that combines virtual and real elements. Another approach is to use image registration, feature matching, or simple depth constraints to ensure that the video image and the 3D model roughly correspond in terms of viewpoint, and then overlay the video content onto the 3D scene.

[0004] However, the existing methods described above generally still have the following problems: Firstly, real video frame images contain both static background information and dynamic object information. However, existing solutions often lack effective differentiation between dynamic objects and static backgrounds, which can easily lead to incorrect mapping of dynamic content such as pedestrians, vehicles, and mobile devices onto the surface of 3D models, resulting in afterimages, ghosting, or incorrect texture solidification. Secondly, existing methods mostly process the entire image by projection or mapping, failing to perform discrete structural analysis of the 3D model from multiple levels such as 3D model objects, surface fragments, and mesh topology. This results in a coarse spatial mapping relationship between video content and 3D model, which is prone to problems such as inaccurate mapping position, discontinuous surface texture, and local misalignment. Third, existing solutions often fail to comprehensively consider factors such as camera imaging parameters, 3D model surface orientation, surface visibility, and surface material, thus failing to accurately determine which 3D model surfaces are suitable for receiving video textures, which can easily lead to incorrect projection, invalid projection, or cross-surface contamination. Fourth, in the process of fusion output of video and 3D model, the existing methods are insufficient in modeling the fine-grained relationship between video pixels, surface fragments and mesh vertices, making it difficult to balance spatial consistency and texture continuity in complex scenes. Fifth, existing methods typically lack an adaptive fusion mechanism based on the correlation between pixels and surface fragments when generating the final fusion result, leading to uneven fusion effects between different surface regions and affecting the realism and stability of the 3D model video fusion result. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention proposes a three-dimensional model video fusion method, device, and medium based on machine learning.

[0006] The technical solution of the present invention is as follows: On the one hand, this invention provides a machine learning-based method for fusion of 3D model videos, comprising the following steps: Acquire camera imaging parameters, video frame images of the target scene captured by the camera, and a 3D model of the target scene; Construct a collaborative fusion model for 3D scene video, including a video structure analysis layer, a 3D model structure analysis layer, a video 3D model alignment layer, and a fusion output layer; The video structure parsing layer is used to convert video frame images of the target scene into video-mapped structured features; The 3D model structure parsing layer is used to convert the 3D model into discrete structured features using camera imaging parameters; The video 3D model alignment layer performs alignment projection on the video mapping structured features and discrete structured features to obtain the alignment result; The fusion output layer is used to generate a 3D model video fusion result from the alignment results.

[0007] Preferably, the video structure parsing layer includes a video multi-scale decomposition coding unit and a video background decoupling unit; The video multiscale decomposition coding unit includes a local texture coding branch, a temporal motion coding branch, and a deep structure coding branch; Local texture coding branch is used to extract texture features from video frame images; The temporal motion coding branch is used to extract the motion features of dynamic objects in video frame images; The deep structure coding branch is used to extract deep structure features from video frame images; The comprehensive features of video frame images are constructed by combining texture features, dynamic object motion features, and deep structural features.

[0008] Preferably, the video background decoupling unit is used to extract the background features of the video frame image through the comprehensive features of the video frame image.

[0009] Preferably, the 3D model structure resolution layer includes object encoding units, surface encoding units, and mesh topology encoding units; The object encoding unit is used to extract all object features from the 3D model; The surface encoding unit is used to divide the 3D model of the target scene into multiple surface fragments and extract the structural properties of each surface fragment; Mesh topology coding units are used to encode each vertex by means of the connection relationship between vertices of a 3D model mesh, thereby obtaining the topological features of each mesh vertex; By combining all object features, the structural properties of each surface fragment, and the topological features of each mesh vertex, a comprehensive 3D scene feature is obtained.

[0010] Preferably, the video 3D model alignment layer includes a 3D model surface projectible analysis unit, a view frustum mesh alignment unit, and a video texture 3D surface construction unit; The 3D model surface projectibility analysis unit calculates the projectibility score of each surface fragment based on camera imaging parameters and the structural properties of each surface fragment.

[0011] Preferably, the view frustum mesh alignment unit receives the comprehensive features of the video frame image, the background features of the video frame image, and the comprehensive features of the three-dimensional scene, and establishes the mapping relationship between the pixels of the video frame image, surface fragments, and the vertices of the three-dimensional mesh.

[0012] Preferably, the video texture 3D surface construction unit is used to reconstruct the texture features of the video frame image into each surface fragment based on the mapping relationship between the video frame image pixels, surface fragments, and 3D mesh vertices.

[0013] Preferably, the fusion output layer calculates attention between each pixel and each surface fragment of the video frame image by performing attention calculation on the comprehensive features of the video frame image and the texture features of each surface fragment; The 3D model video fusion result is generated based on attention weights and the texture features of each surface fragment.

[0014] On the other hand, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the present invention.

[0015] In another aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in the present invention.

[0016] The present invention has the following beneficial effects: 1. This invention sets up a video structure parsing layer, and extracts texture features, dynamic object motion features, and depth structure features from video frame images by a video multi-scale decomposition and coding unit, and then extracts background features from video frame images by a video background decoupling unit. This enables the differentiation between static background information and dynamic object information in target scene video frame images, reduces the situation where dynamic object textures are incorrectly mapped to the surface of 3D models, and improves the effectiveness and stability of subsequent video texture fusion.

[0017] 2. This invention sets up a 3D model structure analysis layer and a video 3D model alignment layer, and uses object coding units, surface coding units, and mesh topology coding units to perform discrete structured analysis of the target scene 3D model. It also establishes the mapping relationship between video frame image pixels, surface fragments, and 3D mesh vertices through 3D model surface projectible analysis units, view frustum alignment units, and video texture 3D surface construction units. This allows the texture features of the video frame image to be reconstructed into each surface fragment, thereby improving the spatial correspondence accuracy between video content and 3D model and reducing the problems of local misalignment, texture discontinuity, and incorrect projection that are prone to occur under overall projection or simple mapping methods.

[0018] 3. This invention sets up a fusion output layer to perform attention calculations on the comprehensive features of video frame images and the texture features of each surface fragment, thereby obtaining the attention weights between each pixel and each surface fragment in the video frame image. Based on the attention weights and the texture features of each surface fragment, a three-dimensional model video fusion result is generated. This allows for adaptive adjustment of the contribution of different surface fragments to the final output result according to the degree of correlation between pixels and surface fragments, improving the realism, naturalness of transitions, and display stability of the three-dimensional model video fusion result. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be understood that the step numbers used in the text are for ease of description only and are not intended to limit the order in which the steps are performed.

[0022] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” are intended to include the plural forms.

[0023] The terms “comprising” and “including” indicate the presence of the described feature, whole, step, operation, element and / or component, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0024] The term “and / or” refers to any combination of one or more of the associated listed items, as well as all possible combinations, and includes these combinations.

[0025] See Figure 1 A machine learning-based 3D model video fusion method includes the following steps: Acquire camera imaging parameters, video frame images of the target scene captured by the camera, and a 3D model of the target scene; Construct a collaborative fusion model for 3D scene video, including a video structure analysis layer, a 3D model structure analysis layer, a video 3D model alignment layer, and a fusion output layer; The video structure parsing layer is used to convert video frame images of the target scene into video-mapped structured features; The 3D model structure parsing layer is used to convert the 3D model into discrete structured features using camera imaging parameters; The video 3D model alignment layer performs alignment projection on the video mapping structured features and discrete structured features to obtain the alignment result; The fusion output layer is used to generate a 3D model video fusion result from the alignment results.

[0026] In one specific embodiment, a machine learning-based 3D model video fusion method is provided, applicable to smart buildings, digital twin parks, factory inspections, exhibition hall security, or urban space visualization scenarios. This method maps and fuses real target scene video frame images captured by cameras into a 3D model of the target scene, enabling the real video content to be fused and displayed according to the spatial structure, surface fragments, occlusion relationships, and texture distribution of the 3D model. In this embodiment, the camera is fixedly installed at a preset location in the target scene, such as in a building lobby, equipment room, production workshop, or building corridor. The 3D model of the target scene can be generated by 3D modeling software, BIM model, laser scanning reconstruction model, or 3D engine such as UE5. The 3D model includes at least walls, floors, ceilings, doors and windows, columns, equipment, pipelines, or other spatial objects. In one specific embodiment, camera imaging parameters, video frame images of the target scene captured by the camera, and a 3D model of the target scene are acquired. Camera imaging parameters include camera intrinsic matrix, camera extrinsic matrix, and camera distortion parameters. The intrinsic matrix describes the mapping relationship between pixel coordinates and camera coordinates, while the extrinsic matrix describes the pose relationship between the camera coordinate system and the 3D model coordinate system. The camera intrinsic parameter matrix is ​​represented as follows: ; in: This represents the camera intrinsic parameter matrix; This indicates the focal length parameter of the camera in the horizontal direction of the image; This indicates the focal length parameter of the camera in the vertical direction of the image; This represents the coordinates of the principal point of the image in the horizontal direction; Represents the coordinates of the principal point of the image in the vertical direction; The camera extrinsic matrix is ​​represented as follows: ; in: This represents the extrinsic parameter matrix of the camera. This represents the rotation matrix of the camera coordinate system relative to the 3D model coordinate system; This represents the translation vector of the camera coordinate system relative to the 3D model coordinate system; The target scene video frame image is represented as follows: ; in: Indicates the first A frame-centric video frame sequence; Indicates the first Frame target scene video frame image; Indicates the number of frames selected forward and backward; The target scene 3D model is represented as follows: ; in: Represents a 3D model of the target scene; Represents the set of mesh vertices in a 3D model; Represents the set of mesh edges in a 3D model; Represents a set of mesh surfaces in a 3D model; This refers to a collection of objects in a 3D model. In this embodiment, the objects include walls, floors, ceilings, doors and windows, equipment, columns, or pipelines.

[0027] In some embodiments, the video structure parsing layer includes a video multi-scale decomposition coding unit and a video background decoupling unit; The video multiscale decomposition coding unit includes a local texture coding branch, a temporal motion coding branch, and a deep structure coding branch; Local texture coding branch is used to extract texture features from video frame images; The temporal motion coding branch is used to extract the motion features of dynamic objects in video frame images; The deep structure coding branch is used to extract deep structure features from video frame images; The comprehensive features of video frame images are constructed by combining texture features, dynamic object motion features, and deep structural features.

[0028] In one specific embodiment, the local texture coding branch performs multi-scale convolutional coding on the current video frame image to extract texture features, edge features, and color distribution features, as shown in the following formula: ; in: Indicates the first Texture features of video frame images of the target scene; Indicates a local texture coding branch; In one implementation, the local texture encoding branch includes convolutional blocks at three scales, corresponding to small-scale edge texture, medium-scale local structure, and large-scale region appearance, respectively. The outputs of the three scales are then concatenated to obtain the texture features, as shown in the following equation: ; in: Indicates feature concatenation operation; Indicates the kernel size as Convolution operation; Indicates the kernel size as Convolution operation; Indicates the kernel size as The convolution operation.

[0029] In one specific embodiment, the temporal motion coding branch is used to extract motion features of dynamic objects in video frame images; Specifically, the optical flow field is first calculated based on adjacent video frames, then the dynamic object region is determined by combining inter-frame difference and dynamic object mask, and finally the motion direction, motion amplitude and motion boundary within the dynamic object region are encoded to obtain the motion characteristics of the dynamic object.

[0030] The inter-frame difference is represented as follows: ; in: Indicates the first Frame relative to the first Adjacent difference graph of frames; express and The absolute value of pixel-by-pixel subtraction; The optical flow field is represented as: ; in: Indicates by the first Frame to the The optical flow field of the frame; Represents the optical flow estimation function; The dynamic object mask is specifically: Calculate the probability that each pixel belongs to the dynamic object region: ; ; ; in: Represents pixels The probability that it belongs to the dynamic object region; This represents the sigmoid activation function; , , Indicates the weight coefficient of a dynamic object; Represents pixels The amplitude of motion at that point; Indicates by the first Frame to the The optical flow field of a frame at a pixel The transverse optical flow component at that location; Indicates by the first Frame to the The optical flow field of a frame at a pixel The longitudinal optical flow component at that location; Indicates at pixel point First Frame relative to the first The adjacent difference value of the frame; Indicates the first Frame pixels Motion boundary response value at the location; This indicates a normalization operation; Indicates the first Frame pixels The gradient of motion amplitude at that point; Indicates the first Frame pixels The optical flow gradient at that location; Denotes the Frobenius norm; Represents pixels First Frame relative to the first Gradient of adjacent difference values ​​of frames; This indicates the preset dynamic region discrimination bias term; Pixels with a probability greater than a preset value are used to construct a dynamic object mask; The motion characteristics of a dynamic object are represented as follows: ; in: Indicates the first Dynamic motion characteristics of video frame images; Indicates the temporal motion coding branch; Represents the motion amplitude graph, consisting of all constitute; Indicates the first Dynamic object masks for frame-by-frame video and image frames; This indicates element-wise multiplication; In one implementation, the temporal motion coding branch compresses and encodes the motion coding input using a lightweight convolutional encoder to obtain the motion features of the dynamic object.

[0031] In one specific embodiment, the deep structure coding branch is used to extract deep structure features from video frame images; Specifically, monocular depth estimation is performed on video frame images or depth maps are obtained based on existing depth sensors, and normal vector maps are calculated based on the depth maps; The depth map normal vector diagram is represented as: ; in: Indicates the first Frame pixels The depth map normal vector at that location; Represents a vector normalization function; Indicates the first Frame pixels The depth value at that location; Indicates the first Frame pixels Gradient in the horizontal direction of a pixel; Indicates the first Frame pixels Gradient in the vertical direction of the pixel; Deep structural features are represented as: ; in: Indicates the first The deep structural features of video frame images; Indicates the first Depth map of a video frame image; Indicates the first Frame depth map normal vector map; Indicates a deep structure encoding branch; In one implementation, the deep structure encoding branch concatenates the input data and then sequentially passes it through convolutional encoding and average pooling operations to obtain deep structure features.

[0032] In one specific embodiment, texture features, dynamic object motion features, and deep structural features are combined to construct comprehensive features of video frame images: ; in: Indicates the first Image synthesis features of video frame frames; Represents a non-linear activation function; , , These represent the trainable weights for texture features, dynamic object motion features, and deep structural features, respectively. This indicates the bias in video feature fusion.

[0033] In some embodiments, the video background decoupling unit is used to extract background features of the video frame image through the comprehensive features of the video frame image.

[0034] In one specific embodiment, the video background decoupling unit, based on the comprehensive features of video frame images and dynamic object masks, suppresses the influence of dynamic object regions on background texture and extracts background features of video frame images, as shown in the following formula: ; in: Indicates the first Background features of video frame images; In a further embodiment, to avoid texture contamination caused by the boundaries of dynamic objects, the dynamic object mask is dilated to obtain an expanded dynamic mask: ; in: Indicates the expanded first... Extended dynamic mask for frame video frames; Indicates morphological dilation operation; Indicates the size of the expansion core; In one implementation, the morphological dilation operation is specifically as follows: Let the dynamic object mask be Set structural elements (size is) A set of structural elements, such as those centered at the current pixel. Neighborhood (specifically, the structuring element is a set of pixel offsets centered on the current pixel, used to define the neighborhood range of the morphological dilation operation), then the extended dynamic mask is represented as: ; in: Indicates the first Pixels in a frame The extended dynamic mask value after morphological dilation; Represents structural element Offset pixels within; like For a binary mask, if a dynamic object pixel exists in the neighborhood of a pixel, then the dilated result will be... It is set as a dynamic region; In one specific embodiment, the size of the expansion kernel is adaptively set, as shown in the following formula: ; in: Indicates the first Pixels in a frame The size of the expansion core at that location; Indicates the size of the base expansion core; This represents the floor function; This represents the motion amplitude adjustment coefficient.

[0035] In some embodiments, the three-dimensional model structure resolution layer includes object encoding units, surface encoding units, and mesh topology encoding units; The object encoding unit is used to extract all object features from the 3D model; The surface encoding unit is used to divide the 3D model of the target scene into multiple surface fragments and extract the structural properties of each surface fragment; Mesh topology coding units are used to encode each vertex by means of the connection relationship between vertices of a 3D model mesh, thereby obtaining the topological features of each mesh vertex; By combining all object features, the structural properties of each surface fragment, and the topological features of each mesh vertex, a comprehensive 3D scene feature is obtained.

[0036] In one specific embodiment, the object encoding unit is used to extract all object features in the 3D model; Specifically, the 3D model is first decomposed into a set of objects based on the object labels, component IDs, material IDs, bounding boxes, and spatial poses in the 3D model file. Then, the category, spatial dimensions, centroid coordinates, orientation direction, material type, and adjacency relationship of each object are extracted to form object features.

[0037] For any object, its object characteristics are represented as follows: ; in: Indicates the first The object characteristics of an object; This represents an object encoding unit, which encodes the input data after concatenation based on training parameters; Indicates the first Category coding for each object; Indicates the first The centroid coordinates of the object; Indicates the first The geometric dimensions of the object; Indicates the first The orientation encoding of an object; Indicates the first Material code of an object; Indicates the first Encoding the adjacency relationship between an object and its neighboring objects; In this embodiment, the centroid coordinates of the object are represented as: ; in: Indicates belonging to the first A collection of mesh vertices for an object; express The number of vertices in the array; Represents vertices Three-dimensional coordinates in the three-dimensional model coordinate system; In this embodiment, the geometric dimensions of the object are represented as follows: ; in: Indicates the first The maximum value of the horizontal coordinate of an object in the 3D model coordinate system; Indicates the first The minimum value of the horizontal coordinate of an object in the 3D model coordinate system; Indicates the first The maximum value of the vertical coordinate of an object in the 3D model coordinate system; Indicates the first The minimum value of the vertical coordinate of an object in the 3D model coordinate system; Indicates the first The maximum value of the height coordinate of an object in the 3D model coordinate system; Indicates the first The minimum value of the height coordinate of an object in the 3D model coordinate system; The adjacency relationships include spatial contact relationships, spatial proximity relationships, inclusion or subordinate relationships, support or bearing relationships, occlusion relationships, and visual adjacency relationships; Spatial contact relationship indicates that two objects in a 3D model are in direct contact or connected by sharing an edge, face, or point. For example: The wall is in contact with the ground; the wall is in contact with the ceiling; the door frame is in contact with the wall; the equipment base is in contact with the ground; the pipeline support is in contact with the wall.

[0038] Spatial proximity indicates that two objects, although not in direct contact, are spatially close and may jointly influence video mapping or occlusion detection. For example: The equipment and nearby walls in the camera's field of view; pipelines and adjacent equipment; pillars and nearby ground areas; doors and nearby passageways.

[0039] The inclusion or dependency relationship indicates that one object belongs to another object spatially, or is installed, embedded, or attached to another object. For example: Doors are openings in the wall; windows are embedded in the wall; cameras are installed on the ceiling or wall; signs are attached to the wall; and pipes are fixed to the wall or ceiling.

[0040] A support or bearing relationship indicates that one object geometrically supports or bears another object, for example: Ground-mounted equipment; wall-mounted display screens; brackets supporting pipelines; columns supporting beams; platforms supporting server racks.

[0041] Occlusion relationships indicate that, from the camera's perspective, one object may occlude another object. For example: Columns obscure walls; equipment obscures the ground; door frames obscure the area behind the door; pipelines in front obscure the wall behind.

[0042] Visual adjacency means that two objects are simultaneously visible from the same camera's viewpoint and their projected areas are adjacent. For example: The area where the wall meets the ground from the same perspective; the door and the door frame; the equipment and the background wall; the pipeline and the wall behind it.

[0043] In one specific embodiment, the surface encoding unit divides the 3D model of the target scene into multiple surface fragments; Specifically, based on a preset threshold, mesh surfaces belonging to the same object, with similar normal vector directions, the same material properties, and spatial continuity are aggregated into a single surface fragment. For any surface fragment, its structural properties are represented as follows: ; in: Indicates the first Structural properties of surface fragments; This represents a surface coding unit, which encodes the input data after concatenation based on training parameters; Indicates the first The coordinates of the center point of each surface fragment; Indicates the first The normal vector of each surface fragment; Indicates the first The area of ​​each surface element; Indicates the first Texture coordinates of each surface fragment; Indicates the first Material properties of each surface element; Indicates the first The curvature of each surface element; The coordinates of the center point of the surface fragment are represented as follows: ; in: Indicates the first A set of mesh surfaces for surface fragments; express The number of grid faces in the data; Represents the mesh surface The coordinates of the center point; The normal vector of a surface fragment is represented as: ; in: Represents the mesh surface The area; Represents the mesh surface The normal vector; In one specific embodiment, the mesh topology coding unit encodes each vertex by means of the connection relationship between the vertices of the three-dimensional model mesh, thereby obtaining the topological features of each mesh vertex; The topological features of the mesh vertices are represented as follows: ; in: Indicates the first Topological features of each grid vertex; This represents a network topology coding unit, which encodes input data after concatenation based on training parameters; Indicates the first The three-dimensional coordinates of each grid vertex; Indicates the first The set of vertices adjacent to each other in a grid; Representing adjacent vertices 3D coordinates; Indicates the first The connectivity of a grid vertex is the number of vertices in the set of its adjacent vertices; Indicates the first The curvature of each grid vertex.

[0044] The object features, surface fragment structure attributes, and mesh vertex topological features are weighted and stitched together to obtain the comprehensive features of the 3D scene.

[0045] In some embodiments, the video 3D model alignment layer includes a 3D model surface projectible analysis unit, a view frustum mesh alignment unit, and a video texture 3D surface construction unit; The 3D model surface projectibility analysis unit calculates the projectibility score of each surface fragment based on camera imaging parameters and the structural properties of each surface fragment.

[0046] In one specific embodiment, the 3D model surface projectibility analysis unit calculates the projectibility score of each surface fragment using camera imaging parameters and the structural properties of each surface fragment: Calculate the line-of-sight direction from the camera center to the center point of each surface fragment: ; in: Indicates that the center of the camera is pointing to the first The line of sight at the center point of each surface element; This indicates the coordinates of the camera center in the 3D model coordinate system; Calculate the angle of view: ; in: Indicates the direction of the camera's line of sight and the first The angle between the viewpoints of the surface fragment normal vectors; Indicates the transpose operation; Calculate the projectibility score of surface fragments: ; in: Indicates the first Projectability score of each surface element; Indicates the center of the camera and the first Euclidean distance between the center points of each surface element; Indicates the first The material of each surface fragment can receive texture coefficients, which characterize whether the surface fragment is suitable as a surface for carrying real video textures. For example, walls, floors, and device casings are higher, while glass, mirrors, or luminous screens are lower; Indicates the first The visible state of each surface element; This indicates a bias in the projectibility score; , This represents the weighting coefficient for projectibility scoring.

[0047] In some embodiments, the view frustum alignment unit receives video frame image synthesis features, video frame image background features, and 3D scene synthesis features to establish a mapping relationship between video frame image pixels, surface fragments, and 3D mesh vertices.

[0048] In one specific embodiment, the view frustum alignment unit receives video frame image comprehensive features, video frame image background features, and 3D scene comprehensive features to establish a mapping relationship between video frame image pixels, surface fragments, and 3D mesh vertices. For each pixel in a video frame image, a corresponding 3D ray is generated based on the camera's intrinsic and extrinsic parameter matrices: ; in: Represents pixels The corresponding three-dimensional ray; Indicates the ray depth parameter; Represents the homogeneous coordinates of a pixel; The steps for calculating the ray depth parameter are as follows: Based on the depth map of the video frame image, determine the initial depth value of each pixel, and calculate the ray depth parameter based on the initial depth value: ; in: Represents pixels Ray depth parameters; This represents the unit vector indicating the direction of the optical axis in the camera coordinate system. This represents the stability constant to prevent the denominator from being zero; Calculate the geometric distance between the 3D ray corresponding to the pixel and the surface fragment: ; in: Represents pixels The corresponding three-dimensional ray and the first Geometric distance between surface fragments; Indicates the first The set of mesh vertices of a surface fragment; express With grid vertices The Euclidean distance between them; Construct matching weights between pixels and surface fragments based on geometric distance and projectibility scores: ; in: Represents pixels With the Matching weights between surface fragments; Indicates the geometric distance attenuation coefficient; Indicates the total number of surface flakes; To simultaneously establish the relationship between pixels and 3D mesh vertices, centroid weights are further assigned to the mesh vertices within each surface fragment: ; in: Represents pixels With the Mapping weights between grid vertices; This represents the vertex distance decay coefficient; Indicates the first The three-dimensional coordinates of each grid vertex; Represents pixels The three-dimensional ray in the first The three-dimensional coordinates of the intersection point or approximate projection point of each surface fragment; Thus, a continuous mapping relationship is formed between video frame image pixels, surface fragments, and 3D mesh vertices.

[0049] In some embodiments, the video texture 3D surface construction unit is used to reconstruct the texture features of a video frame image into each surface fragment based on the mapping relationship between video frame image pixels, surface fragments, and 3D mesh vertices.

[0050] In one specific embodiment, the video texture 3D surface construction unit reconstructs the texture features of the video frame image into each surface fragment based on the mapping relationship between pixels, surface fragments and 3D mesh vertices; For any surface fragment, the fragment texture features obtained by mapping in the current frame are: ; in: Indicates the first Frame mapping to the Texture features of each surface fragment; Furthermore, to improve cross-frame stability, fragment texture features are recursively updated: ; in: Indicates the first The first frame after the update The historical texture memory of each surface element; Indicates the first The surface fragment in the first The texture memory update coefficient of the frame; Indicates the first The first frame The historical texture memory of each surface element; The texture memory update coefficient can be expressed as: ; in: Indicates the offset of the update coefficients; Indicates the first The change in the projectibility score of a surface fragment between adjacent frames; Indicates mapping to the first The amount of variation in the texture features of a surface fragment between adjacent frames; , This indicates that the updated coefficients are used to calculate the weight coefficients.

[0051] In some embodiments, the fusion output layer obtains the attention weights between each pixel and each surface fragment of the video frame image by performing attention calculations on the comprehensive features of the video frame image and the texture features of each surface fragment. The 3D model video fusion result is generated based on attention weights and the texture features of each surface fragment.

[0052] In one specific embodiment, the fusion output layer calculates attention between the comprehensive features of the video frame image and the texture features of each surface fragment to obtain the attention weights between each pixel of the video frame image and each surface fragment; based on the attention weights, the texture features of each surface fragment are weighted and summed to obtain the 3D model video fusion result, specifically: Geometric attention scores are constructed for pixels and surface fragments: ; in: Represents pixels With the Semantic geometric attention score between surface fragments; This indicates a query vector mapping function; Represents pixels Comprehensive features of video frame images at specific points; Represents the key vector mapping function; Indicates the dimension of the attention vector; , , This represents the weighting coefficient used to calculate the geometric attention score; The attention weights are then expressed as: ; in: Represents pixels With the Attention weights between surface fragments; Specifically, during the fusion output process, to avoid erroneous penetration between foreground components of the 3D model and the actual video content, an occlusion visibility adjustment factor is calculated based on video depth estimation data and 3D model rendering depth information: ; in: Represents pixels Point occlusion visibility adjustment factor; This represents the depth value of the 3D model rendered from the current viewpoint (first, a virtual camera is constructed based on the camera's imaging parameters; then, the vertices of the 3D model's mesh are projected onto the image plane; for multiple mesh vertices projected to the same pixel, the visible mesh vertex closest to the virtual camera is selected; the depth of this visible mesh vertex in the virtual camera coordinate system is used as...). ); Represents pixels Optimal matching surface fragment Attention weights between them; This indicates the occlusion / disappearance adjustment factor; , This indicates the weighting coefficient for the occlusion visibility adjustment factor. The texture features of each surface fragment are weighted and summed based on attention weights to generate the 3D model video fusion features corresponding to each pixel: ; in: Indicates the first The results of 3D model video fusion at the pixel level The output values ​​at this point include the red channel value, green channel value, blue channel value, and transparency.

[0053] In some embodiments, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method as described in any embodiment of the present invention.

[0054] In some embodiments, a computer-readable storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the method as described in any embodiment of the present invention.

[0055] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0056] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0057] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0058] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0059] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A machine learning-based method for fusion of 3D model videos, characterized in that, Includes the following steps: Acquire camera imaging parameters, video frame images of the target scene captured by the camera, and a 3D model of the target scene; Construct a collaborative fusion model for 3D scene video, including a video structure analysis layer, a 3D model structure analysis layer, a video 3D model alignment layer, and a fusion output layer; The video structure parsing layer is used to convert the target scene video frame image into video mapping structured features. The video mapping structured features include texture features, dynamic object motion features, depth structure features and background features extracted from the target scene video frame image. These features are used to characterize the image content, motion state, spatial depth relationship and background region information of the target scene video frame image that participates in the 3D model mapping. The 3D model structure parsing layer is used to convert the 3D model into discrete structured features through camera imaging parameters. The discrete structured features include object features, surface fragment structure attributes and mesh vertex topology features obtained by parsing the 3D model of the target scene. These features are used to characterize the discrete spatial structure information of the object level, surface level and mesh topology level in the 3D model of the target scene. The video 3D model alignment layer performs alignment projection on the video mapping structured features and discrete structured features to obtain the alignment result; The fusion output layer is used to generate a 3D model video fusion result from the alignment results.

2. The machine learning-based 3D model video fusion method according to claim 1, characterized in that, The video structure parsing layer includes a video multi-scale decomposition coding unit and a video background decoupling unit; The video multiscale decomposition coding unit includes a local texture coding branch, a temporal motion coding branch, and a deep structure coding branch; Local texture coding branch is used to extract texture features from video frame images; The temporal motion coding branch is used to extract the motion features of dynamic objects in video frame images; The deep structure coding branch is used to extract deep structure features from video frame images; The comprehensive features of video frame images are constructed by combining texture features, dynamic object motion features, and deep structural features.

3. The machine learning-based 3D model video fusion method according to claim 2, characterized in that, The video background decoupling unit is used to extract the background features of video frame images through the comprehensive features of video frame images.

4. The machine learning-based 3D model video fusion method according to claim 3, characterized in that, The 3D model structure resolution layer includes object encoding units, surface encoding units, and mesh topology encoding units; The object encoding unit is used to extract all object features from the 3D model; The surface encoding unit is used to divide the 3D model of the target scene into multiple surface fragments and extract the structural properties of each surface fragment; Mesh topology coding units are used to encode each vertex by means of the connection relationship between vertices of a 3D model mesh, thereby obtaining the topological features of each mesh vertex; By combining all object features, the structural properties of each surface fragment, and the topological features of each mesh vertex, a comprehensive 3D scene feature is obtained.

5. The machine learning-based 3D model video fusion method according to claim 4, characterized in that, The video 3D model alignment layer includes 3D model surface projectible analysis units, view frustum mesh alignment units, and video texture 3D surface construction units; The 3D model surface projectibility analysis unit calculates the projectibility score of each surface fragment based on camera imaging parameters and the structural properties of each surface fragment.

6. The machine learning-based 3D model video fusion method according to claim 5, characterized in that, The view frustum alignment unit receives the comprehensive features of the video frame image, the background features of the video frame image, and the comprehensive features of the 3D scene, and establishes the mapping relationship between the pixels of the video frame image, surface fragments, and vertices of the 3D mesh.

7. The machine learning-based 3D model video fusion method according to claim 6, characterized in that, The video texture 3D surface construction unit is used to reconstruct the texture features of a video frame image into each surface fragment based on the mapping relationship between video frame image pixels, surface fragments, and 3D mesh vertices.

8. The machine learning-based 3D model video fusion method according to claim 7, characterized in that, The fusion output layer calculates attention between each pixel and each surface fragment of the video frame image by performing attention calculation on the comprehensive features of the video frame image and the texture features of each surface fragment. The 3D model video fusion result is generated based on attention weights and the texture features of each surface fragment.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 8.

Images