Video picture and virtual information projection registration and fusion method and system based on digital twin scene

Abstract

The application provides a video picture and virtual information projection registration and fusion method and system based on a digital twin scene, relates to the technical field of digital twinning, and obtains a virtual three-dimensional coordinate in a digital twin scene and a time sequence video stream and an actual three-dimensional coordinate sequence synchronously collected by multiple cameras; then, a space pose solution is performed through a Perspective-n-Point algorithm to obtain an initial mapping relationship; subsequently, a time sequence sensing support vector machine regression model is used to perform deviation registration on the relationship to obtain an accurate target mapping relationship; then, a target fusion region is determined and a visual feature thereof is extracted; finally, the visual feature and an image feature of virtual information are input into a random forest model, a fusion control parameter is generated by combining a dynamic change index, and then the virtual information is dynamically projected and fused into the video picture, which can improve the accuracy of virtual-real projection registration and the naturalness of dynamic fusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital twinning, in particular to a video picture and virtual information projection registration and fusion method and system based on a digital twinning scene. BACKGROUND

[0002] In the fields of smart city management, industrial simulation and interactive display, it has important application prospects to accurately superimpose virtual information in a digital twinning scene onto a real-time video picture to realize intuitive virtual-real fusion interaction.

[0003] At present, the common way to realize such virtual-real fusion is usually to determine the projection position of virtual information based on the preliminary positioning result of real-time picture features, and then to synthesize it with the video picture according to fixed superimposition rules.

[0004] However, when dealing with dynamic real scenes, this kind of method often regards position alignment and picture synthesis as two separate steps, and fails to effectively utilize continuous time series information to cooperatively optimize the projection position and fusion effect, resulting in easy drift of the superimposed position of virtual information in dynamic video and relatively harsh fusion visual perception. SUMMARY

[0005] The purpose of the present application is to provide a video picture and virtual information projection registration and fusion method and system based on a digital twinning scene to solve the problems of insufficient virtual-real combination stability and poor visual fusion effect in the prior art.

[0006] To solve the above technical problems, in a first aspect, the present application provides a video picture and virtual information projection registration and fusion method based on a digital twinning scene, comprising:

[0007] Obtaining virtual three-dimensional coordinates of virtual information under a digital twinning scene, and time series video streams and corresponding actual three-dimensional coordinate sequences collected by multiple sensors, the multiple sensors including at least two cameras with different viewing angles;

[0008] Taking a system processing clock as a reference, obtaining or synthesizing a real-time video picture at a corresponding time from the time series video streams, and taking the three-dimensional coordinates in the actual three-dimensional coordinate sequence that are spatiotemporally synchronized with the real-time video picture as corresponding actual three-dimensional coordinates;

[0009] Based on the actual three-dimensional coordinates and the virtual three-dimensional coordinates, performing space pose solving through a Perspective-n-Point (PnP) algorithm to obtain an initial mapping relationship;

[0010] Using a time series perception support vector machine regression model to perform deviation registration on the initial mapping relationship to obtain a target mapping relationship;

[0011] According to the target mapping relationship, a target fusion area of the virtual information in the real-time video picture is determined, and a visual feature of the target fusion area is extracted;

[0012] The visual feature and an image feature corresponding to the virtual information are input into a random forest model for decision analysis, a dynamic change index of a digital twin scene is combined, a fusion control parameter is generated, and the virtual information is dynamically projected and fused into the target fusion area according to the fusion control parameter.

[0013] Optionally, the time sequence perception-based support vector machine regression model is used to perform deviation registration on the initial mapping relationship to obtain a target mapping relationship, including:

[0014] Based on the initial mapping relationship, the actual three-dimensional coordinates are projected to the real-time video picture to obtain a projection position of a current frame; meanwhile, based on a historical mapping relationship of at least a previous frame obtained from the time sequence video stream, corresponding historical actual three-dimensional coordinates are projected to corresponding historical video pictures to obtain historical projection positions;

[0015] Based on the projection position of the current frame and the historical projection positions, and in combination with a corresponding relationship of the actual three-dimensional coordinates between continuous frames, a deviation set is calculated, the deviation set including projection deviations of a same three-dimensional coordinate point between the current frame and the historical frames;

[0016] According to a spatial distribution of the actual three-dimensional coordinates and a spatiotemporal evolution law of the deviations in the deviation set, a spatial correlation feature is constructed, the spatiotemporal evolution law including a deviation amplitude and a deviation change trend;

[0017] The spatial correlation feature is input into a time sequence perception-based support vector machine regression model for error prediction to obtain an error registration field, wherein the time sequence perception-based support vector machine regression model is embedded with a time sequence smoothing constraint during prediction;

[0018] According to the error registration field, the initial mapping relationship is compensated to generate a target mapping relationship.

[0019] Optionally, the spatial correlation feature is constructed according to the spatial distribution of the actual three-dimensional coordinates and the spatiotemporal evolution law of the deviations in the deviation set, including:

[0020] Based on spatial distances between the coordinate points in the actual three-dimensional coordinates, an adjacency graph structure is constructed, each graph node in the adjacency graph structure corresponding to an actual three-dimensional coordinate point;

[0021] Based on the deviation of the amplitude and the trend of the deviation of each coordinate point corresponding to the bias, the correlation strength of the corresponding edge in the adjacent graph structure is calculated, and the edge weight of the adjacent graph structure is dynamically adjusted according to the correlation strength, and a weighted adjacent graph is obtained.

[0022] At least one layer of graph convolution operation is performed on the weighted adjacent graph, and the deviation correlation features of each graph node are extracted and updated through information propagation and aggregation between nodes.

[0023] The deviation correlation features of all graph nodes are globally aggregated to obtain spatial correlation features.

[0024] Optionally, the visual features and the image features corresponding to the virtual information are input into a random forest model for decision analysis, and a fusion control parameter is generated in combination with a dynamic change index of the digital twin scene, including:

[0025] Based on the time sequence video stream, the scene motion feature change quantity of the real-time video picture relative to the previous frame is calculated to obtain a first dynamic index; at the same time, based on the state update information of the object associated with the virtual information in the digital twin scene, a second dynamic index is obtained.

[0026] The first dynamic index and the second dynamic index are fused to generate a dynamic change index of the digital twin scene.

[0027] The visual features, the image features corresponding to the virtual information, and the dynamic change index are jointly input into a pre-trained random forest decision model, and the random forest decision model outputs an initial parameter set corresponding to the current frame according to the input features, the initial parameter set at least including a fusion transparency, a size scaling factor, and a position adjustment vector.

[0028] According to the visual saliency value of the target fusion region in the real-time video picture, the parameters in the initial parameter set are fine-tuned to generate a fusion control parameter.

[0029] Optionally, the virtual information is dynamically projected and fused into the target fusion region according to the fusion control parameter, including:

[0030] According to the position adjustment vector in the fusion control parameter, the target mapping relationship is dynamically adjusted to generate a real-time projection matrix.

[0031] Based on the real-time projection matrix and the size scaling factor in the fusion control parameter, the virtual information is rendered to a size and position matching the target fusion region to generate a virtual layer to be fused.

[0032] According to the fusion transparency in the fusion control parameter, an adaptive weight mixing algorithm is used to perform pixel-by-pixel mixing between the virtual layer to be fused and the target fusion region in the real-time video picture.

[0033] The output picture after mixing is subjected to smoothing filtering processing to ensure that the fusion effect of the virtual information in the continuous video frame remains visually coherent.

[0034] Optionally, based on the actual three-dimensional coordinates and the virtual three-dimensional coordinates, an initial mapping relationship is obtained by using a Perspective-n-Point algorithm for spatial pose solving, comprising:

[0035] Two-dimensional pixel coordinates of a plurality of feature points are extracted from the real-time video picture, and three-dimensional space coordinates corresponding to the feature points are obtained from the digital twin scene;

[0036] Based on the two-dimensional pixel coordinates and the three-dimensional space coordinates, an initial coordinate set is constructed, and the initial coordinate set is subjected to outlier screening to obtain a target coordinate set;

[0037] The target coordinate set is subjected to Perspective-n-Point solving to obtain spatial position parameters and orientation parameters of a projection device;

[0038] The spatial position parameters and the orientation parameters are used to calculate an initial mapping relationship of the virtual information to the real-time video picture.

[0039] Optionally, according to the target mapping relationship, a target fusion region of the virtual information in the real-time video picture is determined, and visual features of the target fusion region are extracted, comprising:

[0040] Based on the target mapping relationship, a projection range of the virtual information in the real-time video picture is calculated to obtain a region mask;

[0041] The region mask is used to crop the real-time video picture to obtain a preliminary picture, and the preliminary picture is subjected to perceptual segmentation to obtain a target picture;

[0042] A superpixel segmentation method is used to divide the target picture into regions to obtain a plurality of image subblocks, and color histograms and texture descriptors of the image subblocks are extracted;

[0043] All the color histograms and the texture descriptors are aggregated and sorted to obtain visual features.

[0044] In a second aspect, the present application provides a video picture and virtual information projection registration and fusion system based on a digital twin scene, comprising:

[0045] The first acquisition module is configured to acquire virtual three-dimensional coordinates of virtual information in a digital twin scene and time-series video streams collected by multiple sensors and corresponding actual three-dimensional coordinate sequences, wherein the multiple sensors include at least two cameras with different visual angles.

[0046] The second acquisition module is configured to acquire or synthesize real-time video pictures at corresponding moments from the time-series video streams based on a system processing clock, and take three-dimensional coordinates in the actual three-dimensional coordinate sequences that are spatiotemporally synchronized with the real-time video pictures as corresponding actual three-dimensional coordinates.

[0047] The solving module is configured to perform space pose solving based on the actual three-dimensional coordinates and the virtual three-dimensional coordinates by using a Perspective-n-Point algorithm to obtain an initial mapping relationship.

[0048] The registration module is configured to perform deviation registration on the initial mapping relationship by using a time-series perception support vector machine regression model to obtain a target mapping relationship.

[0049] The extraction module is configured to determine a target fusion area of the virtual information in the real-time video pictures according to the target mapping relationship, and extract visual features of the target fusion area.

[0050] The fusion module is configured to input the visual features and image features corresponding to the virtual information into a random forest model for decision analysis, combine a dynamic change index of the digital twin scene to generate a fusion control parameter, and dynamically project and fuse the virtual information into the target fusion area according to the fusion control parameter.

[0051] In a third aspect, the present application provides an electronic device, comprising:

[0052] The memory is configured to store a computer program.

[0053] The processor is configured to implement the steps of the video picture and virtual information projection registration and fusion method based on a digital twin scene as described in the first aspect when executing the computer program.

[0054] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program can implement the steps of the video picture and virtual information projection registration and fusion method based on a digital twin scene as described in the first aspect when executed by a processor.

[0055] The method for video picture and virtual information projection registration and fusion based on a digital twin scene has the following beneficial effects: firstly, the virtual coordinates in the digital twin model, the time sequence video stream collected by the multiple cameras and the actual coordinate sequence can provide accurate and synchronous multi-source data basis for subsequent processing; secondly, the Perspective-n-Point algorithm is used for space pose solution to quickly obtain the initial space relationship of virtual information projection; thirdly, the time sequence perception support vector machine regression model is used for deviation registration of the initial relationship, which can improve the continuity and accuracy of the projection position in a dynamic scene; then, the target fusion area is determined according to the optimized mapping relationship, and visual features are extracted, which can provide specific picture information for virtual-real fusion; finally, the visual features and virtual information features are combined with a scene dynamic index, and then input into a random forest model to generate fusion control parameters, and dynamic projection fusion is performed according to the parameters, so that the virtual information and real-time video picture can be adaptively and visually consistent.

[0056] Further, the deviation set is constructed by comparing the projection deviation of the current frame and the historical frame, and the features reflecting the spatial distribution and the spatio-temporal evolution rule are extracted from the deviation set, which can provide more abundant learning basis for the deviation registration model; meanwhile, the time sequence smoothing constraint is embedded in the support vector machine regression model to predict and compensate errors, which can effectively suppress frame jitter and make the projection position more smooth and stable in the time dimension, thereby improving the robustness and visual comfort of virtual-real registration in a dynamic scene. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0058] Figure 1 A flowchart of a method for video picture and virtual information projection registration and fusion based on a digital twin scene provided by an embodiment of the present application;

[0059] Figure 2 A specific implementation diagram of a method for video picture and virtual information projection registration and fusion based on a digital twin scene provided by an embodiment of the present application;

[0060] Figure 3 A structure diagram of a system for video picture and virtual information projection registration and fusion based on a digital twin scene provided by an embodiment of the present application;

[0061] Figure 4A structural schematic diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0062] In digital twin and augmented reality applications, it is the key to realize high-quality interaction to stabilize and naturally integrate virtual information into real-time video pictures, but the common practice of the current mainstream technical solutions is to perform one-time position estimation through real-time pictures first, and then superimpose virtual content according to fixed rules, however, this kind of method is easy to cause projection position drift in dynamic scenes due to ignoring the time sequence association, and the picture fusion effect is rigid, which is difficult to adapt to the continuous changes of light, view angle and object motion in real environment.

[0063] Therefore, the present application proposes a video picture and virtual information projection registration and fusion method based on a digital twin scene, the core of the method is that: an initial projection relationship is established by synchronously acquiring virtual coordinates and multi-view time sequence video streams, then a model with time sequence perception capability is used to continuously correct the projection position, and finally the fusion parameters are generated adaptively combined with the dynamic characteristics of the scene and the virtual-real superposition is completed. This scheme can improve the stability of the projection position and the naturalness of the visual fusion in dynamic changing scenes, thereby effectively overcoming the problems of easy position drift, rigid fusion effect and difficulty in adapting to real-time changes in the prior art.

[0064] In order to enable personnel in the technical field to better understand the present application scheme, the present application will be further described in detail below in combination with the drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0065] The core of the present application is to provide a video picture and virtual information projection registration and fusion method based on a digital twin scene, and a flowchart of one specific embodiment thereof is shown in Figure 1 The method comprises:

[0066] S101, acquiring virtual three-dimensional coordinates of virtual information under a digital twin scene, and time sequence video streams and corresponding actual three-dimensional coordinate sequences collected by multiple sensors, the multiple sensors including at least two cameras with different view angles.

[0067] The virtual three-dimensional coordinates refer to coordinate data output by a digital twin model software and used to describe a position of virtual information to be superimposed in its own three-dimensional space; the actual three-dimensional coordinate sequence refers to a series of coordinate points obtained by processing time-series video streams collected by multiple sensors through three-dimensional reconstruction technology, and used to describe spatial positions of object surface key points in a same world coordinate system and changes thereof over time in a real scene; and the digital twin scene refers to a virtualized and digitized mirror image of a real physical environment.

[0068] In step S101, first, communicate with the digital twin model software to read or receive in real time three-dimensional coordinates of virtual information determined and output by the software according to preset logic or user interaction.

[0069] Then, start a hardware array composed of at least two different-view cameras, and synchronously collect multiple videos based on a unified system clock to form time-series video streams; subsequently, process the video streams through a real-time three-dimensional reconstruction algorithm, for example, a stereo vision method based on multi-view geometry, specifically including: matching feature points between pictures collected by different cameras at the same time, and then calculating positions of the feature points in three-dimensional space through a triangulation principle to generate actual three-dimensional coordinate sequences corresponding to actual scene structures one by one.

[0070] The application synchronously obtains accurate data from virtual models and real environments that can be correlated and aligned in time and space, and can lay a reliable data foundation for subsequent accurate superimposition of virtual information into real-time video pictures.

[0071] S102, based on a system processing clock, acquire or synthesize a real-time video picture at a corresponding time from the time-series video streams, and take three-dimensional coordinates in the actual three-dimensional coordinate sequence that are time and space synchronized with the real-time video picture as corresponding actual three-dimensional coordinates.

[0072] The system processing clock refers to a unified and continuous time reference signal generated and maintained inside a computer system, used to ensure that all data collection and processing links are aligned in time; and the time and space synchronization refers to ensuring that real-time video pictures and actual three-dimensional coordinate data acquired from different sources not only correspond to a same physical time, but also match each other in a spatial coordinate system.

[0073] In step S102, first, select single-frame pictures of each camera from the multiple time-series video streams that are closest to the current time, and if a specific view picture is needed, generate a synthetic view at the corresponding time by using multi-view image synthesis technology, to serve as the real-time video picture.

[0074] Secondly, in the generated actual three-dimensional coordinate sequence, find and extract the set of three-dimensional coordinate points that match the current time stamp, because these coordinate points and real-time video picture originate from the same time sensor data, so they can be defined as the actual three-dimensional coordinates corresponding to the current time that are "spatiotemporally synchronized" with the picture.

[0075] The application can effectively eliminate the time dislocation and space mismatch between multi-source data through the above-mentioned synchronization mechanism based on a unified clock, thereby ensuring the high consistency of video pictures and three-dimensional space information in the time-space dimension for subsequent processing, and further providing necessary conditions for precise projection registration.

[0076] S103, based on the actual three-dimensional coordinates and the virtual three-dimensional coordinates, performing space pose solving through a Perspective-n-Point algorithm to obtain an initial mapping relationship.

[0077] In a specific implementation process, S103 includes:

[0078] Step 1031, extracting two-dimensional pixel coordinates of a plurality of feature points from the real-time video picture, and obtaining three-dimensional space coordinates corresponding to the feature points from the digital twin scene.

[0079] Among them, the feature point refers to a pixel position that is recognized from the real-time video picture and has significant visual characteristics, such as a corner point, an edge intersection point, and is easy to track stably between different frames; the two-dimensional pixel coordinates refer to the row value and column value used to describe the specific position of the feature point in the two-dimensional plane of the video picture to which it belongs; the three-dimensional space coordinates refer to the three-dimensional position information of the actual object surface position point corresponding to the feature point in the world coordinate system of the digital twin scene, including the numerical values in the X-axis, Y-axis and Z-axis directions.

[0080] In step 1031, first, a feature detection algorithm is run on the real-time video picture, which locates the center points of regions with rich texture or obvious contour transitions in the picture by analyzing the gradient changes of pixel brightness in multiple directions, and records the row and column positions of these points in the pixel coordinate system of the frame picture to form a two-dimensional pixel coordinate set.

[0081] At the same time, according to the pre-established correspondence database between the visual features in the video picture and the three-dimensional model surface points in the digital twin scene, the three-dimensional model points matched with each extracted two-dimensional feature point are queried and obtained, so as to obtain the corresponding three-dimensional space coordinates.

[0082] Step 1032, based on the two-dimensional pixel coordinates and the three-dimensional space coordinates, constructing an initial coordinate set and performing outlier point screening on the initial coordinate set to obtain a target coordinate set.

[0083] wherein, the initial coordinate set refers to a set of unoptimized two-dimensional and three-dimensional point pair matching data generated by step 1031; the outlier screening refers to a process of identifying and removing from the initial coordinate set those false matching point pairs which are obviously inconsistent with the spatial geometric relationship reflected by the majority of other point pairs due to feature mismatching, motion blur or occlusion, etc.

[0084] In step 1032, first, each output two-dimensional pixel coordinate is paired with its corresponding three-dimensional spatial coordinate, and combined into an initial two-dimensional and three-dimensional corresponding point set.

[0085] Subsequently, consistency test is performed on the initial point set, that is, a candidate spatial transformation model is calculated by iteratively randomly selecting a subset, and the number of point pairs that meet the model, that is, the re-projection error is less than the pre-set tolerance, for example, 5 pixels, is counted, and those false matching point pairs that cannot maintain consistent geometric relationship in multiple iterations are determined as outliers and removed, and finally the point pair set that remains consistent with the geometric constraint relationship constitutes the target coordinate set.

[0086] Step 1033, Perspective-n-Point is used to solve the target coordinate set, to obtain the spatial position parameter and the orientation parameter of the projection device.

[0087] wherein, Perspective-n-Point solving refers to a mathematical calculation method for solving the position and orientation of a camera in a three-dimensional world coordinate system for shooting an image, under the condition that a three-dimensional spatial point set and its projected point set on a two-dimensional image plane are known; the spatial position parameter refers to the three-dimensional coordinates of the optical center of the projection device in the world coordinate system; the orientation parameter refers to the rotation angle or equivalent direction vector of the optical axis direction of the projection device relative to the three coordinate axes of the world coordinate system.

[0088] In step 1033, the target coordinate set is input into a PnP solver, which establishes a geometric equation describing how three-dimensional points are projected to a two-dimensional image plane based on the pinhole perspective projection model of camera imaging.

[0089] Subsequently, a numerical optimization algorithm, such as Levenberg-Marquardt algorithm, is used to iteratively adjust the assumed spatial position and orientation parameters of the projection device, so that the re-projection error between the theoretical two-dimensional position calculated according to the current assumed parameters and the actually observed two-dimensional pixel coordinates reaches a minimum; when the optimization process converges, the obtained optimal position and rotation parameters are the spatial position parameter and the orientation parameter of the projection device, wherein the content of the convergence condition is not specifically limited in the embodiments of the present application.

[0090] Step 1034, using the spatial position parameter and the orientation parameter, calculating an initial mapping relationship of the virtual information to the real-time video picture.

[0091] The initial mapping relationship refers to an initial geometric transformation rule to be followed for transforming the three-dimensional coordinates of the virtual information in the digital twin scene to the two-dimensional pixel coordinate space of the real-time video picture, and the mathematical form can be a composite matrix including rotation, translation and projection.

[0092] In step 1034, a complete perspective projection matrix is constructed according to the spatial position parameter and the orientation parameter of the projection device, combined with the internal parameters of the projection device such as focal length and principal point coordinates, which defines the continuous transformation from the world coordinate system, i.e. the coordinate system of the digital twin scene, to the camera coordinate system, and then to the image pixel coordinate system; then the matrix is applied to the three-dimensional coordinates of the virtual information, and the corresponding theoretical pixel position in the current frame of real-time video picture is calculated, and the rule used in the calculation process is defined as the initial mapping relationship.

[0093] The present application uses the screened reliable feature point pairs to robustly solve the device pose under the current view angle by PnP algorithm, and establishes an accurate initial projection geometric model accordingly, which provides an accurate and stable spatial reference for the positioning of virtual information in the video picture, and effectively supports the subsequent fine registration operation.

[0094] S104, using a time sequence perception support vector machine regression model, performing deviation registration on the initial mapping relationship to obtain a target mapping relationship.

[0095] In a specific implementation process, as shown in Figure 2 S104 includes:

[0096] Step 1041, projecting the actual three-dimensional coordinates to the real-time video picture based on the initial mapping relationship to obtain a projection position of the current frame; at the same time, projecting the corresponding historical actual three-dimensional coordinates to the corresponding historical video picture based on the historical mapping relationship of at least the previous frame obtained from the time sequence video stream to obtain a historical projection position.

[0097] The projection position of the current frame refers to the two-dimensional pixel coordinates in the current video picture calculated by using the initial mapping relationship just solved for the actual three-dimensional coordinate points collected at the same time; the historical projection position refers to the two-dimensional pixel coordinates obtained by projecting the historical actual three-dimensional coordinates collected and stored at the previous time to the corresponding historical frame picture by using the target mapping relationship optimized by the computer system for at least the previous frame.

[0098] In step 1041, first, the current frame initial mapping relationship is called to project the acquired all actual three-dimensional coordinate points strictly time-synchronized with the current frame through the matrix one by one, and the two-dimensional projection position of each point in the current frame real-time video picture is calculated; at the same time, the computer system reads the previous frame, for example, the target mapping relationship obtained after the processing of the t-1 frame is completed as the historical mapping relationship, and acquires the historical actual three-dimensional coordinate point set stored synchronously with the t-1 frame in the timestamp, and then calculates the projection position of these coordinate points in the t-1 frame historical video picture, that is, the historical projection position, by using the historical mapping relationship.

[0099] For example, the computer system is processing the 100th frame, wherein the coordinates of an actual three-dimensional coordinate point A are (1.0, 2.0, 10.0) meters; then the projection position of point A in the 100th frame real-time picture is (200.5, 150.3) pixels, which is obtained by calculation through the initial mapping relationship of the 100th frame; at the same time, the computer system reads the optimized target mapping relationship of the 99th frame, and projects the historical actual three-dimensional coordinates of point A stored in the 99th frame, which are also (1.0, 2.0, 10.0) meters, to obtain the historical projection position of point A in the 99th frame picture, which is (199.8, 151.0) pixels.

[0100] Step 1042, based on the projection position of the current frame and the historical projection position, and in combination with the corresponding relationship of the actual three-dimensional coordinates between the continuous frames, a deviation set is calculated, and the deviation set contains the projection deviation of the same three-dimensional coordinate point between the current frame and the historical frame.

[0101] The deviation set refers to a set composed of the coordinate difference between the projection position of the same three-dimensional coordinate point in the tth frame and the historical projection position of the same three-dimensional coordinate point in the t-1th frame, and the difference is the projection deviation, which quantitatively reflects the pixel-level position deviation of the point between the two continuous frames due to camera motion, target micro-motion or inaccurate initial mapping.

[0102] In step 1042, for each three-dimensional coordinate point with a corresponding relationship, the difference between the current frame projection position ( , ) and the historical projection position ( , ) in each pixel direction is calculated, and the projection deviations in the horizontal direction and the vertical direction are obtained , Since the actual three-dimensional coordinate sequence has ensured the correspondence of the data points across frames when collected through feature tracking or timestamp, the projection deviation of all tracking points can be calculated, and the deviation set of the current frame is formed by summarizing.

[0103] For example, if point A's projected position in frame 100 is (200.5, 150.3) and its historical projected position in frame 99 is (199.8, 151.0), then its projection deviation is calculated to be (0.7, -0.7) pixels. Subsequently, the same calculation is performed on all tracking points in the scene, ultimately forming a deviation set containing the projection deviations of all points, for example: {point A: (0.7, -0.7), point B: (0.5, -0.3), point C: (-0.2, 0.4), ...}.

[0104] Step 1043: Based on the spatial distribution of the actual three-dimensional coordinates and the spatiotemporal evolution law of the deviation concentration, construct spatial correlation features, wherein the spatiotemporal evolution law includes the deviation amplitude and the deviation change trend.

[0105] Among them, spatial correlation features refer to a comprehensive feature representation that simultaneously encodes the spatial proximity relationship of three-dimensional coordinate points, the magnitude of the projection deviation of each point, and the direction of change of the deviation in a short period of time, aiming to reveal the spatial correlation and temporal evolution pattern of the projection deviation.

[0106] Step 1043 may specifically include the following steps:

[0107] Step a1: Based on the spatial distance between each coordinate point in the actual three-dimensional coordinates, construct an adjacency graph structure, where each graph node in the adjacency graph structure corresponds to an actual three-dimensional coordinate point.

[0108] The adjacency graph structure refers to a data structure used to represent the connection relationship between objects. In this step, each three-dimensional coordinate point is regarded as a node in the graph. If the Euclidean distance between two points in three-dimensional space is less than a preset threshold, an edge is established between the two points to indicate that they are spatial neighbors.

[0109] In step a1, the computer system calculates the Euclidean distance between all pairs of actual 3D coordinate points. For a given distance threshold, such as 0.5 meters, it traverses all point pairs. If the distance between point i and point j is less than the threshold, an edge is added to the two points in the adjacency graph. Finally, each actual 3D coordinate point becomes a node in the graph, and the edges between nodes represent the proximity relationship between two points in physical space.

[0110] For example, there are three actual 3D coordinate points in the scene: A(0, 0, 0), B(0.3, 0, 0), and C(1, 0, 0) (unit: meters); then the distance threshold is set to 0.5 meters; the distances are calculated as follows: the distance between AB is 0.3 meters and less than 0.5; the distance between AC is 1 meter and greater than 0.5; the distance between BC is 0.7 meters and greater than 0.5. Therefore, in the constructed adjacency graph structure, there is an edge between node A and node B, and no edge between node A and C, or between node B and C.

[0111] Step a2: Based on the deviation magnitude and deviation change trend corresponding to each coordinate point in the deviation set, calculate the association strength of the corresponding edges in the adjacency graph structure, and dynamically adjust the edge weights of the adjacency graph structure according to the association strength to obtain a weighted adjacency graph.

[0112] Among them, the association strength refers to an index that measures the similarity of the projection deviation patterns of two adjacent nodes. The deviation pattern includes the magnitude and direction of the deviation. The higher the association strength, the more similar the deviations of the two points are, and the more important the edge between the two points is in subsequent information transmission.

[0113] In step a2, for each edge in the adjacency graph, the two nodes i and j connected by it are provided with their corresponding projected deviations from the deviation set. First, the deviation magnitude of each point is calculated. and the direction vector of the deviation Where r refers to the magnitude of the projection deviation, and i refers to the node. It is the projection deviation of node i in the horizontal direction. It is the projection deviation of node i in the longitudinal direction; then, the association strength of the edge is calculated using the formula. ,For example: ,in, and This refers to the magnitude of the deviation between nodes i and j. and This refers to the normalized deviation direction vector between nodes i and j, where exp() is an exponential function, and the calculated value is then used to... As the new weight of this edge, the original adjacency graph becomes a weighted adjacency graph where the edge weights reflect the deviation similarity.

[0114] Step a3: Perform at least one layer of graph convolution operation on the weighted adjacency graph, and extract and update the deviation association features of each graph node through information propagation and aggregation between nodes.

[0115] Graph convolution refers to a neural network layer specifically designed for processing graph-structured data. It allows the feature information of each node to propagate and aggregate along the edges of the graph. After graph convolution, the new features of each node will not only contain its own original information but also incorporate the information of its neighboring nodes.

[0116] In step a3, the weighted adjacency graph is input into a graph convolutional layer, and the initial features of each node are set as its projection bias vector (Δu, Δv), or a combination of features composed of magnitude and direction, etc. The graph convolutional layer then performs the following operation on each node according to the weighted adjacency matrix: the features of the node itself are weighted and averaged with the features of all its neighboring nodes, and the weights are the association strength of the edges. Then, the weighted average result is passed through a learnable linear transformation, such as multiplying by a weight matrix, and a non-linear activation function, such as the ReLU function, is applied to finally obtain the updated features of the node, which are called bias association features.

[0117] For example, taking node A as an example, its initial features are [0.7, -0.7]. Assuming it has only one neighbor node B, whose features are [0.5, -0.3], and the weight of edge AB is 1.633; further assuming that the linear transformation weight matrix of the graph convolutional layer is the identity matrix, i.e., no transformation, and the activation function is the identity function, then the updated features of node A are calculated as: the weighted sum of its own features and the features of its neighbors. If a simple average aggregation is used, then the new features of node A = [0.7, -0.7] + 1.633 × [0.5, -0.3] = [1.5165, -1.1899]. The above feature aggregation process is exemplary, and in practice there can be more complex aggregation and transformation methods. The embodiments of this application can be set according to the actual situation, which will not be elaborated here.

[0118] Step a4: Globally aggregate the deviation correlation features of all graph nodes to obtain spatial correlation features.

[0119] Global aggregation refers to an operation that integrates the features of all nodes in a graph into a fixed-length vector. Common methods include global average pooling, which takes the average value of the features of all nodes in each dimension, or global max pooling, which takes the maximum value.

[0120] In step a4, for each dimension of the feature vector, the arithmetic mean of the values ​​of all N nodes in that dimension is calculated, and finally a k-dimensional vector is obtained. This vector is the final spatial correlation feature, which summarizes the deviation correlation pattern of all three-dimensional observation points in the entire scene.

[0121] For example, in a graph with three nodes A, B, and C, after graph convolution, the bias correlation features of the three nodes are: A: [1.5, -1.2], B: [0.8, -0.5], C: [-0.2, 0.4]. Then, global average pooling is performed, such as: the average value of the first dimension = (1.5 + 0.8 - 0.2) / 3 = 0.7; the average value of the second dimension = (-1.2 - 0.5 + 0.4) / 3 = -1.3 / 3 ≈ -0.433. Therefore, the spatial correlation feature obtained is [0.7, -0.433].

[0122] Step 1044: Input the spatial correlation features into the time-aware support vector machine regression model for error prediction to obtain the error registration field, wherein the time-aware support vector machine regression model embeds time-series smoothing constraints during prediction.

[0123] Among them, the error registration field refers to the projection error correction amount predicted by the model that needs to be applied to the entire scene or each three-dimensional coordinate point; the time-aware support vector machine regression model is based on the classic support vector regression machine, and its loss function is modified to add a "time-series smoothing constraint".

[0124] In step 1044, the model is first trained offline. The training data can come from historical frame sequences, that is, the spatial correlation features of each frame are used as input and the corresponding "true" projection error is used as the target label. Then, when the model is trained, its optimization objective is not only to make the predicted value as close as possible to the true value, but also to minimize the error between the predicted value of the current frame and the predicted value of the previous frame.

[0125] Next, after the model is trained, when used online, the spatial correlation features of the current frame are input into the model, and the model outputs the prediction error corresponding to the current frame. This output can be a global error correction quantity, such as a 2D vector representing the average offset of the entire image, or it can be an error field associated with each point through features.

[0126] For example, a simple linear model can be set up, which can take the form: Prediction Error = W × Spatial Correlation Features + b, where W is the weight matrix and b is the bias; subsequently, temporal smoothing constraints are reflected in the training process, i.e., the loss function is: , where λ is the smoothing coefficient.

[0127] Step 1045: Based on the error registration field, compensate for the initial mapping relationship to generate the target mapping relationship.

[0128] Among them, compensation refers to using the predicted error registration field to adjust the parameters of the initial mapping relationship in order to offset the predicted systematic deviation, thereby obtaining a more accurate and stable new mapping relationship, namely the target mapping relationship.

[0129] In step 1045, the initial mapping relationship is usually represented as a 3x4 projection matrix P, and the error registration field provides the expected correction amount. The goal of the compensation is to find an adjustment amount so that when the 3D point is projected using the adjusted new matrix P', its projected position is closer to the ideal position, that is, the initial projection position plus the prediction error.

[0130] For example, a simplified implementation in this application assumes that the predicted global error is mainly caused by the inaccuracy of the camera's extrinsic parameters. Therefore, the translation adjustment in the three-dimensional space can be calculated in reverse based on the predicted global error correction amount and the camera's intrinsic parameters, and the translation part in the projection matrix can be updated accordingly. A more general approach is to construct an optimization problem: minimize the difference between the position of all points after projection through the new matrix P' and the "ideal position", and solve for the optimal P' through an iterative algorithm.

[0131] This application, through the above series of operations, can effectively overcome projection position drift and inter-frame jitter caused by factors such as camera movement and environmental interference in dynamically changing digital twin scenarios, and ultimately achieve accurate, stable and visually coherent projection registration between virtual information and real-time video images.

[0132] S105. Based on the target mapping relationship, determine the target fusion region of the virtual information in the real-time video frame, and extract the visual features of the target fusion region.

[0133] Among them, visual features refer to a high-dimensional numerical vector that can comprehensively and orderly describe the visual attributes of the entire target fusion region. It is formed by organizing and splicing the color and texture features of all image sub-blocks in the region according to the spatial order of the sub-blocks in the image.

[0134] In a specific implementation process, step S105 includes:

[0135] Step 1051: Based on the target mapping relationship, calculate the projection range of the virtual information in the real-time video frame to obtain the region mask.

[0136] In this context, a region mask is a binary image whose size is exactly the same as the real-time video image. The region with a pixel value of "1" (such as white) marks the precise range of virtual information that should be projected and covered in the image, while the region with a pixel value of "0" (such as black) marks the background and does not project.

[0137] In step 1051, the target mapping relationship obtained after optimization in S104 is first used to project the coordinates of all vertices of the virtual information to be projected onto the two-dimensional pixel plane of the current frame; then, a minimum convex polygon or closed boundary that can completely enclose the outline of the virtual information is calculated based on these projection points; next, a blank binary image with the same resolution as the current video frame is created in memory, and all pixel values ​​inside the polygon are set to 1, and the pixel values ​​outside are set to 0, thereby generating a clear and well-defined region mask.

[0138] For example, in a digital twin scene of an exhibition hall, the virtual information is a three-dimensional "information display board" model. After the target mapping relationship is calculated, its projection on the video screen forms an irregular quadrilateral outline. Then, the area enclosed by this outline is calculated, and a corresponding binary mask is generated. In this mask, all pixels belonging to the quadrilateral are marked as white with a value of 1, while other parts of the exhibition hall background are marked as black with a value of 0.

[0139] Step 1052: Use the region mask to crop the real-time video frame to obtain a preliminary frame, and perform perceptual segmentation on the preliminary frame to obtain the target frame.

[0140] The preliminary image refers to a local image extracted from a complete real-time video frame using a region mask, containing only the target fusion area and a small amount of surrounding background; perceptual segmentation is a computer vision technique aimed at understanding image content and classifying pixels in the image into different semantic categories; the target image refers to the final image after perceptual segmentation, in which only background environment pixels are retained in the preliminary image, while all foreground dynamic objects are removed or identified.

[0141] In step 1052, a region mask is used as a selection area and applied to the original real-time video image. All pixels corresponding to the white area of ​​the mask are extracted, and the boundary is usually extended outward by a few pixels to include a small amount of the surrounding environment, thus obtaining a preliminary image. This preliminary image is then input into a pre-trained semantic segmentation neural network model, such as DeepLab or MaskR-CNN. The model then analyzes the image, identifies and labels the semantic category of each pixel. Based on the labeling results, pixel regions of the foreground dynamic category that are not related to the virtual information fusion are removed from the preliminary image, and only pixels of the static background category are retained, finally generating a clean target image that represents the background of the fusion target.

[0142] For example, by using a region mask of the "information display board", a rectangular image containing the display board area and the surrounding ground is cropped from the complete exhibition hall video, which is the preliminary image. Then, the image is input into a semantic segmentation model, which recognizes that the image contains a "pedestrian" and a "booth background". Then, the pixel area corresponding to the "pedestrian" is set to transparent or filled with the background color, leaving only the pixels of the "booth background", thus obtaining the target image with foreground interference eliminated.

[0143] Step 1053: Using a superpixel segmentation method, the target image is divided into regions to obtain multiple image sub-blocks, and the color histogram and texture descriptor of the image sub-blocks are extracted.

[0144] Among them, superpixel segmentation refers to the process of dividing an image into multiple sets of pixels that are visually coherent and similar in color and texture. Each set is called a superpixel or image sub-block. Color histogram is a statistical feature used to describe the frequency distribution of various colors in an image region. Texture descriptor is a numerical feature used to quantify the spatial arrangement pattern of pixel intensity or color in an image region, such as roughness, contrast, and directionality.

[0145] In step 1053, a superpixel segmentation algorithm, such as the SLIC algorithm, is applied to the target image. This algorithm automatically clusters the image into dozens to hundreds of irregular but visually coherent image sub-blocks based on the color similarity and spatial proximity of pixels. Then, for each image sub-block, its color features and texture features are calculated. The color features are obtained by calculating the histogram of the hue and saturation components of all pixels in the sub-block in the HSV color space. The texture features are obtained by calculating the local binary mode descriptor. This descriptor generates a binary mode string by comparing the gray values ​​of each pixel with its surrounding neighboring pixels, and then counts the histograms of these mode strings to characterize the texture.

[0146] For example, the target image obtained after segmentation is superpixel segmented to obtain 50 image sub-blocks; then, for a sub-block representing the surface of a wooden textured display stand, its HSV color histogram is calculated, which shows a concentrated distribution of brown tones; then its LBP texture descriptor is calculated, which shows a histogram that shows a regular repeating pattern reflecting the wood texture.

[0147] Step 1054: Aggregate and sort all the color histograms and texture descriptors to obtain visual features.

[0148] In step 1054, the color histogram vector and texture descriptor vector are concatenated end to end to form the "combined feature vector" of the sub-block; then, according to the spatial position relationship of all image sub-blocks in the original target image, the combined feature vectors of each sub-block are concatenated in sequence; finally, all the concatenated sub-block feature vectors are concatenated again to form a long, one-dimensional global visual feature vector. In order to eliminate the problem of inconsistent dimensions caused by different numbers of sub-blocks, the combined feature vectors of all sub-blocks can be first subjected to average pooling or max pooling, and then sorted and concatenated.

[0149] For example, there are 50 sub-blocks, each with a combined feature vector of 128 dimensions, including 64 dimensions of color and 64 dimensions of texture. After arranging them in spatial order, they can be directly concatenated to obtain a 6400-dimensional vector. Alternatively, the 128-dimensional vectors of all sub-blocks can be averaged separately to obtain a 128-dimensional global average feature. At the same time, a 128-dimensional global maximum feature can also be generated. Then, these two vectors are concatenated to form a 256-dimensional visual feature vector.

[0150] This application ensures that the extracted features can accurately reflect the background environment of the location where the virtual-real fusion occurs, thus providing a crucial visual basis for subsequent intelligent decision-making to generate a natural and harmonious fusion effect.

[0151] S106. Input the visual features and the image features corresponding to the virtual information into a random forest model for decision analysis. Combine the dynamic change index of the digital twin scene to generate fusion control parameters. Based on the fusion control parameters, dynamically project and fuse the virtual information into the target fusion region.

[0152] In a specific implementation process, step S106 includes:

[0153] Step 1061: Based on the time-series video stream, calculate the change in scene motion features of the real-time video frame relative to the previous frame to obtain a first dynamic index; simultaneously, based on the state update information of the objects associated with the virtual information in the digital twin scene, obtain a second dynamic index.

[0154] The first dynamic index is a scalar value used to characterize the intensity of motion in a real physical scene, calculated by analyzing the overall pixel motion between consecutive video frames; the second dynamic index is a scalar value used to characterize the activity of the virtual information itself, obtained by monitoring the state update frequency of virtual objects in the digital twin model and normalizing it.

[0155] In step 1061, during the process of calculating the first dynamic index, firstly, the optical flow algorithm is used to analyze the current frame and the previous frame of video, and the optical flow field describing the motion speed and direction of each pixel is calculated; then, the average amplitude of all motion vectors in the entire optical flow field is statistically analyzed; finally, this average motion amplitude is input into a preset normalization function, such as an S-shaped function, and mapped to the range of 0 to 1, and the resulting value is the first dynamic index.

[0156] In the process of calculating the second dynamic index, the state update event stream from the digital twin model data interface is monitored in real time. These events are triggered when the position, attributes, or visibility of the virtual object changes. Then, the number of such events occurring within a fixed time window is counted. This number is then divided by the theoretically maximum number of events allowed within the aforementioned time window, or simply normalized using a maximum frequency, to obtain the second dynamic index between 0 and 1.

[0157] For example, in a showroom monitoring scenario, the optical flow algorithm calculates that the average pixel motion amplitude of the current frame compared to the previous frame is 4.5 pixels. Then, through normalization function processing, the first dynamic index Im is obtained as 0.6. At the same time, the virtual exhibit "engine model" in the digital twin model is triggered by user interaction twice in the last second to update its rotation state. Assuming that the maximum state update frequency is 5 times per second, the second dynamic index Is is 2 divided by 5, which equals 0.4.

[0158] Step 1062: Merge the first dynamic index with the second dynamic index to generate the dynamic change index of the digital twin scene.

[0159] Among them, the dynamic change index refers to a single comprehensive index obtained by weighted fusion of the first dynamic index and the second dynamic index. It comprehensively represents the overall dynamic level of the digital twin scenario at the current moment.

[0160] In step 1062, a linear weighted fusion model is used to combine the first dynamic index and the second dynamic index into a unified dynamic change index. Specifically, a weighting coefficient is assigned to the first dynamic index. Assign a weighting coefficient to the second dynamic index. ,and and The sum is 1, and the weighting coefficient reflects the prior judgment of the importance of different dynamic sources to the final fusion effect; subsequently, the dynamic change index is used. The calculation formula is: And calculated using this formula This is the final indicator representing the overall dynamic level of the current frame.

[0161] For example, setting weights It is 0.7. The value is 0.3, and the first dynamic index is known. The second dynamic index is 0.6. The value is 0.4. Substitute this value into the fusion formula to calculate: =0.7×0.6+0.3×0.4=0.42+0.12=0.54, that is, the dynamic change index is 0.54.

[0162] Step 1063: Input the visual features, the image features corresponding to the virtual information, and the dynamic change index into a pre-trained random forest decision model. The random forest decision model outputs an initial parameter set corresponding to the current frame based on the input features. The initial parameter set includes at least fusion transparency, size scaling factor, and position adjustment vector.

[0163] Among them, the image features corresponding to virtual information refer to the visual description vectors extracted from the image representation of the virtual information to be projected itself, such as its color distribution, shape contour features, etc.; the initial parameter set refers to a set of basic fusion control parameters directly inferred by the random forest model based on all the features of the current input.

[0164] In step 1063, firstly, the training of the random forest decision model has been completed in the offline stage. Its training data comes from a large number of historical scene samples. Each sample contains the visual features at that time, virtual information image features, dynamic change index, and the optimal fusion parameter label derived by manual evaluation or optimization algorithm.

[0165] Subsequently, the model learns the complex mapping relationship between these features and the optimal parameters. When applied online, the extracted visual feature vector F is used... V Pre-calculated virtual information image feature vector I V and the dynamic change index generated by the steps The vectors are concatenated to form a complete input feature vector. This vector is then fed into a trained random forest model, where multiple decision trees within the model independently perform inferences, with each tree outputting a predicted parameter value.

[0166] Finally, by calculating the average of all decision tree outputs, the initial fusion parameter set for the current frame is obtained, including the initial fusion transparency T0, the initial size scaling factor S0, and the initial position adjustment vector D0.

[0167] For example, visual feature vector F V The virtual information image feature vector I is 200-dimensional. V The dimensions are 50, and this information is then compared with the dynamic change index. =0.54 concatenated, resulting in a 251-dimensional feature vector [FV I V [0.54] Subsequently, after inputting the feature vector into the pre-trained random forest model, the model outputs the initial parameters: fusion transparency T0=0.8, size scaling factor S0=1.1, and position adjustment vector D0=(Δx0, Δy0)=(2.0, -1.5).

[0168] Step 1064: Based on the visual saliency value of the target fusion region in the real-time video frame, fine-tune the parameters in the initial parameter set to generate fusion control parameters.

[0169] Among them, the visual saliency value refers to the quantitative value of the degree to which the target fusion region attracts human visual attention in the entire video frame, which is evaluated by the computational model; fine-tuning refers to the refined adaptive correction of the initial parameters predicted based on global features based on local saliency information.

[0170] In step 1064, a pre-trained visual saliency detection model is first used to process the entire real-time video frame to obtain the saliency value of each pixel. Then, using the region mask obtained in step 1051, the arithmetic mean of the saliency values ​​of all pixels in the target fusion region is calculated and denoted as the visual saliency value Sal of the region.

[0171] Subsequently, the initial parameters are fine-tuned according to the visual saliency value. Generally, highly saliency areas should be avoided from being excessively occluded or interfered with. Therefore, the initial fusion transparency T0 needs to be adjusted downwards. The adjustment formula can be T = T0 × K. t , where K t It is a decay coefficient based on Sal, for example, K t =1-η×Sal, where η is the attenuation factor; the position adjustment vector D0 may also be slightly offset according to the centroid of the saliency distribution, and after fine-tuning, the final fusion control parameters are obtained: fusion transparency T, size scaling factor S and position adjustment vector D.

[0172] For example, the visual saliency value Sal of the target fusion region was calculated to be 0.3, and the attenuation factor η was set to 0.4. Subsequently, the transparency attenuation coefficient K was calculated. t =1-0.4×0.3=0.88, then the final blending transparency T=0.8×0.88=0.704. Then, assuming that the scaling factor and position vector are finely adjusted according to the preset rules, the final blending control parameters are: T=0.704, S=1.08, D=(Δx, Δy)=(2.2, -1.3).

[0173] Step 1065: Based on the position adjustment vector in the fusion control parameters, dynamically adjust the target mapping relationship to generate a real-time projection matrix.

[0174] The real-time projection matrix refers to the geometric transformation matrix that is ultimately used to render virtual information from three-dimensional space to two-dimensional video images. It is a composite matrix formed by fusing the pixel-level offset represented by the position adjustment vector on the basis of the target mapping relationship matrix obtained in step S104.

[0175] In step 1065, the target mapping relationship is essentially a 3x4 projection matrix. This defines the basic projection rules from the three-dimensional world coordinate system to the two-dimensional image coordinate system. The position adjustment vector D = (Δx, Δy) represents the additional translation to be applied to the final two-dimensional image plane. To integrate this two-dimensional translation into the entire three-dimensional projection process, a two-dimensional translation transformation can be cascaded at the end of the projection transformation. Mathematically, this can be achieved by constructing a corresponding translation transformation matrix. and with the original projection matrix This is achieved through matrix multiplication, i.e., real-time projection matrix. ,in, It is a 3x3 matrix whose function is to transform the image point (x, y, 1) in homogeneous coordinates into (x + Δx, y + Δy, 1), and is generated in this way. The matrix is ​​a real-time projection matrix that integrates all geometric calibration and position fine-tuning information.

[0176] For example, the projection matrix corresponding to the known target mapping relationship is: The final position adjustment vector D is (2.2, -1.3), and then a translation matrix is ​​constructed. This results in an increase of 2.2 pixels in the x-direction and a decrease of 1.3 pixels in the y-direction. The real-time projection matrix is ​​then calculated using matrix multiplication. = × .

[0177] Step 1066: Based on the real-time projection matrix and the size scaling factor in the fusion control parameters, render the virtual information to a size and position that matches the target fusion region, and generate a virtual layer to be fused.

[0178] The virtual layer to be merged refers to a bitmap with an alpha transparency channel that is the same size as the video frame. It contains virtual information images that have undergone the correct geometric transformation, while other areas are completely transparent.

[0179] In step 1066, the graphics rendering engine receives the original material data of the virtual information and the real-time projection matrix. Including the size scaling factor S in the fusion control parameters, the rendering process is divided into two steps: the first step is to perform geometric transformations, using the engine... The matrix performs coordinate transformation on all vertices of the virtual information to determine its precise position and shape in the final image. Then, a scaling operation is performed, and the projected virtual information image is enlarged or reduced according to the scaling factor S. In the second step, the engine draws the virtual information that has undergone the above geometric processing onto a newly created, fully transparent bitmap layer. This layer, which contains virtual content of the correct size and position, is the virtual layer to be merged.

[0180] For example, the virtual message is a "Keep Away from High Temperatures" warning icon, which the rendering engine then uses. The matrix calculates the coordinates of the four corners of the icon in the image; then, based on the scaling factor S=1.08, the size of the icon is uniformly enlarged to 108% of its original size; finally, this enlarged icon is drawn onto a new transparent layer, generating a virtual layer to be blended.

[0181] Step 1067: Based on the fusion transparency in the fusion control parameters, an adaptive weighted fusion algorithm is used to blend the virtual layer to be fused with the target fusion area in the real-time video frame pixel by pixel.

[0182] Among them, the adaptive weight blending algorithm refers to a synthesis technique that can dynamically adjust the blending weight of each pixel according to the local features of the virtual layer and the background layer when blending images, aiming to achieve a more natural edge blending effect than simple transparency blending.

[0183] In step 1067, the virtual layer to be blended, the corresponding target background area in the real-time video frame, and the final blend transparency T in the blending control parameters are first obtained. Then, the basic blending formula is used as follows: ,in, and These are the colors of a specific pixel on the virtual layer and the background layer, respectively. w is the output color, and w is the actual blending weight.

[0184] Its adaptive algorithm is reflected in the calculation of w, which is analyzed as follows: w is not directly equal to T, but is modulated according to the edge intensity at the pixel's location; then, the gradient magnitude G of the virtual layer or target background region in the neighborhood of that pixel is calculated, and the higher the gradient, the more likely it is to be an edge; then, the formula is used. Calculate the actual weights, where λ is a factor that controls the modulation intensity. In the edge region, the mixing weight w is lower than T to make the transition smoother; in the flat region, w is closer to T.

[0185] For example, for a pixel within the target area, its virtual layer color The background color is red (255, 0, 0). The pixel is gray (100, 100, 100), and the final blending transparency T = 0.704. Then, assuming the pixel is located at an edge, the calculated gradient magnitude G is 0.8, and the modulation factor λ = 0.5. Therefore, the actual blending weight w = 0.704 × (1 - 0.5 × 0.8) = 0.704 × 0.6 = 0.4224; the final output color... The red channel value is: 0.4224×255+(1-0.4224)×100≈165.5. The other channels are calculated in the same way to obtain the mixed color.

[0186] Step 1068: Perform smoothing filtering on the mixed output image to ensure that the fusion effect of the virtual information in consecutive video frames maintains visual continuity.

[0187] Among them, smoothing filtering refers to applying spatial and temporal filtering operations to the target area in the merged image, aiming to suppress image flickering, jittering or edge jumping caused by small fluctuations in parameters between frames or computational noise.

[0188] In step 1068, the smoothing filtering process focuses on improving the temporal stability of the fusion effect. The process includes two dimensions: in the spatial dimension, a Gaussian blur filter with a small radius is applied to the edge of the fusion region, for example, with a radius of 1 pixel, which can soften the fine jagged edges or irregular edges that may remain from the previous step; in the more critical temporal dimension, a short-term historical cache of the fusion region image is maintained, for example, storing the fusion results of the previous N frames.

[0189] Then, the fusion result of the current frame is weighted and mixed with the latest frame in the historical buffer to generate the final output. During the mixing, the current frame is given a higher weight α and the historical frames are given a lower weight 1-α. This temporal mixing operation constitutes a low-pass filter, which can effectively smooth out high-frequency changes between frames and suppress flicker. Finally, the historical buffer is updated with the final output of the current frame for use in the next frame.

[0190] For example, if the temporal blending weight α is set to 0.7, and the historical cache retains the result of the previous frame, the current frame's fused region image is... The previous frame in the cache is ; then on Apply a slight Gaussian blur to the spatial domain, then use the formula... Perform time-domain mixing, and As the final output, and update the history cache, The next frame equals the current frame. .

[0191] This application achieves accurate, natural and adaptive fusion of virtual information in digital twin video footage through a complete technology chain from dynamic perception, intelligent decision-making, local optimization to post-rendering processing. It not only achieves high-precision registration in spatial location, but also ensures the harmonious unity of the fusion effect with the dynamic environment at the visual perception level, thereby improving the practicality and immersion of augmented reality applications in complex dynamic scenes.

[0192] Figure 3 This application provides a schematic diagram of a specific implementation of a system for registering and fusing video footage and virtual information projections based on a digital twin scene, as shown in the following embodiment. Figure 3 The system may include:

[0193] The first acquisition module 31 is used to acquire the virtual three-dimensional coordinates of virtual information in the digital twin scenario, as well as the time-series video stream and corresponding actual three-dimensional coordinate sequence collected by multiple sensors, wherein the multiple sensors include at least two cameras with different perspectives.

[0194] The second acquisition module 32 is used to acquire or synthesize real-time video images at corresponding moments from the time-series video stream based on the system processing clock, and to use the three-dimensional coordinates in the actual three-dimensional coordinate sequence that are spatiotemporally synchronized with the real-time video images as the corresponding actual three-dimensional coordinates.

[0195] The solution module 33 is used to perform spatial pose calculation based on the actual three-dimensional coordinates and the virtual three-dimensional coordinates using the Perspective-n-Point algorithm to obtain the initial mapping relationship.

[0196] The registration module 34 is used to perform bias registration on the initial mapping relationship using a time-aware support vector machine regression model to obtain the target mapping relationship.

[0197] The extraction module 35 is used to determine the target fusion region of the virtual information in the real-time video frame according to the target mapping relationship, and extract the visual features of the target fusion region.

[0198] The fusion module 36 is used to input the visual features and the image features corresponding to the virtual information into a random forest model for decision analysis, combine the dynamic change index of the digital twin scene to generate fusion control parameters, and dynamically project and fuse the virtual information into the target fusion region according to the fusion control parameters.

[0199] The video image and virtual information projection registration and fusion system based on digital twin scene in this application embodiment is used to implement the aforementioned video image and virtual information projection registration and fusion method based on digital twin scene. Therefore, the specific implementation of the video image and virtual information projection registration and fusion system based on digital twin scene can be found in the embodiment section of the video image and virtual information projection registration and fusion method based on digital twin scene above. The specific implementation can be referred to the description of the corresponding embodiments, and will not be repeated here.

[0200] like Figure 4 The present application also provides an electronic device, including: a memory 41 for storing a computer program; and a processor 42 for executing the computer program to implement the steps of the above-described method for video image and virtual information projection registration and fusion based on a digital twin scene.

[0201] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described methods for video image and virtual information projection registration and fusion based on a digital twin scene.

[0202] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0203] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the embodiments of the video image and virtual information projection registration and fusion method based on a digital twin scene.

[0204] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0205] The foregoing has provided a detailed description of a method and system for video image and virtual information projection registration and fusion based on a digital twin scene, as provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for registration and fusion of video footage and virtual information projection based on a digital twin scene, characterized in that, include: The virtual three-dimensional coordinates of virtual information in a digital twin scenario are obtained, as well as the time-series video stream and corresponding actual three-dimensional coordinate sequence collected by multiple sensors, wherein the multiple sensors include at least two cameras with different perspectives. Based on the system processing clock, real-time video images at corresponding moments are obtained or synthesized from the time-series video stream, and the three-dimensional coordinates in the actual three-dimensional coordinate sequence that are spatiotemporally synchronized with the real-time video images are used as the corresponding actual three-dimensional coordinates. Based on the actual 3D coordinates and the virtual 3D coordinates, the spatial pose is calculated using the Perspective-n-Point algorithm to obtain the initial mapping relationship; A time-aware support vector machine regression model is used to perform bias registration on the initial mapping relationship to obtain the target mapping relationship; Based on the target mapping relationship, the target fusion region of the virtual information in the real-time video frame is determined, and the visual features of the target fusion region are extracted; The visual features and the image features corresponding to the virtual information are input into a random forest model for decision analysis. Combined with the dynamic change index of the digital twin scene, fusion control parameters are generated. Based on the fusion control parameters, the virtual information is dynamically projected and fused into the target fusion region. The time-aware support vector machine regression model is used to perform bias registration on the initial mapping relationship to obtain the target mapping relationship, including: Based on the initial mapping relationship, the actual three-dimensional coordinates are projected onto the real-time video frame to obtain the projection position of the current frame; at the same time, based on the historical mapping relationship of at least the previous frame obtained from the time-series video stream, the corresponding historical actual three-dimensional coordinates are projected onto the corresponding historical video frame to obtain the historical projection position. Based on the projection position of the current frame and the historical projection position, and combined with the correspondence of the actual three-dimensional coordinates between consecutive frames, a deviation set is calculated, which includes the projection deviation of the same three-dimensional coordinate point between the current frame and the historical frame. Based on the spatial distribution of the actual three-dimensional coordinates and the spatiotemporal evolution law of the deviation concentration, spatial correlation features are constructed, wherein the spatiotemporal evolution law includes the deviation amplitude and the deviation change trend; The spatial correlation features are input into a time-aware support vector machine regression model for error prediction to obtain an error registration field. The time-aware support vector machine regression model embeds a time-series smoothing constraint during prediction. Based on the error registration field, the initial mapping relationship is compensated to generate the target mapping relationship.

2. The method according to claim 1, characterized in that, The construction of spatial correlation features based on the spatial distribution of the actual three-dimensional coordinates and the spatiotemporal evolution of the deviation concentration deviation includes: Based on the spatial distance between each coordinate point in the actual three-dimensional coordinates, an adjacency graph structure is constructed, and each graph node in the adjacency graph structure corresponds to an actual three-dimensional coordinate point. Based on the deviation magnitude and deviation change trend corresponding to each coordinate point in the deviation set, the association strength of the corresponding edges in the adjacency graph structure is calculated, and the edge weights of the adjacency graph structure are dynamically adjusted according to the association strength to obtain a weighted adjacency graph. At least one layer of graph convolution operation is performed on the weighted adjacency graph, and the deviation association features of each graph node are extracted and updated through information propagation and aggregation between nodes. The spatial correlation features are obtained by globally aggregating the deviation correlation features of all graph nodes.

3. The method according to claim 1, characterized in that, The step involves inputting the visual features and the image features corresponding to the virtual information into a random forest model for decision analysis, and combining this with the dynamic change index of the digital twin scene to generate fusion control parameters, including: Based on the time-series video stream, the change in scene motion features of the real-time video frame relative to the previous frame is calculated to obtain a first dynamic index; simultaneously, based on the state update information of the objects associated with the virtual information in the digital twin scene, a second dynamic index is obtained. The first dynamic index and the second dynamic index are fused to generate the dynamic change index of the digital twin scenario; The visual features, the image features corresponding to the virtual information, and the dynamic change index are input together into a pre-trained random forest decision model. The random forest decision model outputs an initial parameter set corresponding to the current frame based on the input features. The initial parameter set includes at least fusion transparency, size scaling factor, and position adjustment vector. Based on the visual saliency value of the target fusion region in the real-time video frame, the parameters in the initial parameter set are fine-tuned to generate fusion control parameters.

4. The method according to claim 3, characterized in that, The step of dynamically projecting and fusing the virtual information into the target fusion region according to the fusion control parameters includes: Based on the position adjustment vector in the fusion control parameters, the target mapping relationship is dynamically adjusted to generate a real-time projection matrix; Based on the real-time projection matrix and the size scaling factor in the fusion control parameters, the virtual information is rendered to a size and position that matches the target fusion region, generating a virtual layer to be fused; Based on the fusion transparency in the fusion control parameters, an adaptive weighted fusion algorithm is used to blend the virtual layer to be fused with the target fusion area in the real-time video frame pixel by pixel. The mixed output image is smoothed and filtered to ensure that the fusion effect of the virtual information in consecutive video frames maintains visual continuity.

5. The method according to claim 1, characterized in that, The initial mapping relationship is obtained by calculating the spatial pose using the Perspective-n-Point algorithm based on the actual 3D coordinates and the virtual 3D coordinates, including: Two-dimensional pixel coordinates of multiple feature points are extracted from the real-time video frame, and three-dimensional spatial coordinates corresponding to the feature points are obtained from the digital twin scene. Based on the two-dimensional pixel coordinates and the three-dimensional spatial coordinates, an initial coordinate set is constructed, and outlier points are filtered from the initial coordinate set to obtain the target coordinate set; The spatial position and orientation parameters of the projection device are obtained by solving the target coordinate set using Perspective-n-Point. Using the spatial location parameters and the orientation parameters, the initial mapping relationship between the virtual information and the real-time video image is calculated.

6. The method according to claim 1, characterized in that, The step of determining the target fusion region of the virtual information in the real-time video frame according to the target mapping relationship, and extracting the visual features of the target fusion region, includes: Based on the target mapping relationship, the projection range of the virtual information in the real-time video frame is calculated to obtain the region mask; The real-time video frame is cropped using the region mask to obtain a preliminary frame, and the preliminary frame is perceptually segmented to obtain the target frame; The target image is divided into regions using a superpixel segmentation method to obtain multiple image sub-blocks, and the color histogram and texture descriptor of the image sub-blocks are extracted. All the color histograms and texture descriptors are aggregated and sorted to obtain visual features.

7. A system for registering and fusing video footage and virtual information projection based on a digital twin scene, characterized in that, A method for implementing the video image and virtual information projection registration and fusion method based on a digital twin scene as described in claim 1 includes: The virtual three-dimensional coordinates of virtual information in a digital twin scenario are obtained, as well as the time-series video stream and corresponding actual three-dimensional coordinate sequence collected by multiple sensors, wherein the multiple sensors include at least two cameras with different perspectives. Based on the system processing clock, real-time video images at corresponding moments are obtained or synthesized from the time-series video stream, and the three-dimensional coordinates in the actual three-dimensional coordinate sequence that are spatiotemporally synchronized with the real-time video images are used as the corresponding actual three-dimensional coordinates. Based on the actual 3D coordinates and the virtual 3D coordinates, the spatial pose is calculated using the Perspective-n-Point algorithm to obtain the initial mapping relationship; A time-aware support vector machine regression model is used to perform bias registration on the initial mapping relationship to obtain the target mapping relationship; Based on the target mapping relationship, the target fusion region of the virtual information in the real-time video frame is determined, and the visual features of the target fusion region are extracted; The visual features and the image features corresponding to the virtual information are input into a random forest model for decision analysis. Combined with the dynamic change index of the digital twin scene, fusion control parameters are generated. Based on the fusion control parameters, the virtual information is dynamically projected and fused into the target fusion region.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the video image and virtual information projection registration and fusion method based on a digital twin scene as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the video image and virtual information projection registration and fusion method based on a digital twin scene as described in any one of claims 1 to 6.

Images