Video texture mapping and real-time rendering method and system based on digital twin scene

Abstract

The application provides a video texture mapping and real-time rendering method and system based on a digital twin scene, relates to the technical field of digital twin rendering, and comprises the following steps: acquiring a real scene monitoring video stream and performing real-time image content analysis; dividing the texture details of different regions of the video stream according to the features and numerical values of the image analysis and generating region detail identifiers associated with the video content; identifying the dynamic features of the video picture based on the identifiers and generating scene state codes representing the comprehensive picture conditions; dynamically selecting a target engine and corresponding rendering quality parameters from preset heterogeneous rendering engines in combination with the current load state of a graphics processor; and mapping the video stream image data to the surface of a corresponding model of the digital twin scene as dynamic texture to realize efficient rendering of the picture, so that dynamic texture mapping and adaptive real-time rendering based on video content analysis and hardware load in the digital twin scene can be realized.

Description

Technical Field

[0001] This application relates to the field of digital twin rendering technology, and in particular to a method and system for video texture mapping and real-time rendering based on digital twin scenes. Background Technology

[0002] Video texture mapping and real-time rendering are the core supporting means of digital twin technology. They can dynamically replicate real scenes into virtual space and are widely used in fields such as industrial monitoring, smart cities, and intelligent manufacturing, providing intuitive visualization support for remote control and decision analysis.

[0003] Currently, most existing methods based on digital twin scenarios directly map real-world video streams to virtual models, or combine some environmental data with a fixed rendering strategy, generating images using a single rendering engine. These methods do not fully consider the impact of dynamic scene changes and hardware operating conditions. In other words, existing digital twin scenarios can only statically reflect the application status of a scene at a specific point in the past and cannot endow digital twins with real-time dynamic real-world features.

[0004] However, such methods are prone to mismatches between rendering quality and scene requirements. They may cause stuttering in complex scenes due to excessive load, or waste hardware resources in simple scenes, and struggle to accurately adapt to the visual presentation demands brought about by dynamic environmental changes. Therefore, existing technologies suffer from insufficient rendering adaptability and efficiency in digital twin scenarios. Summary of the Invention

[0005] The purpose of this application is to provide a video texture mapping and real-time rendering method and system based on digital twin scenarios, so as to solve the problems of insufficient rendering adaptability and efficiency in digital twin scenarios in the existing technology.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a video texture mapping and real-time rendering method based on a digital twin scene, comprising:

[0007] Simultaneously acquire video streams from surveillance cameras in real-world scenarios and perform content analysis on the video streams;

[0008] Based on the results of the content analysis, the texture detail levels of different regions in the video stream are dynamically divided, and regional detail identifiers associated with the content features of the video frames are generated.

[0009] Based on video data carrying the region detail identifiers, the dynamic environmental features of the real scene are identified, and a scene status code is generated according to the dynamic environmental features. The scene status code is used to characterize the overall environmental condition.

[0010] Based on the scene status code and the current load status of the graphics processor, a target rendering engine is dynamically selected from multiple preset heterogeneous rendering engines, and rendering quality parameters corresponding to the target rendering engine and the scene status code are determined simultaneously.

[0011] Based on the rendering quality parameters, the image data in the video stream is used as a dynamic texture and mapped onto the corresponding model surface of the digital twin scene. The scene lighting and shadow parameters are adjusted in real time according to the scene status code to drive the target rendering engine to complete the image rendering.

[0012] Optionally, based on the scene status code and in conjunction with the current load status of the graphics processor, a target rendering engine is dynamically selected from a plurality of preset heterogeneous rendering engines, and rendering quality parameters corresponding to the target rendering engine and the scene status code are determined simultaneously, including:

[0013] The utilization and occupancy of the graphics processor are acquired in real time to calculate the current load coefficient. At the same time, the bits of the scene status code are parsed to extract the environmental complexity flag.

[0014] The current load coefficient is compared with a preset load threshold, and a load-adaptive rendering engine group is selected from a set of multiple heterogeneous rendering engines.

[0015] Based on the environment complexity flag, candidate engines are selected from the rendering engine group;

[0016] Based on the bits representing lighting and motion in the scene state code, a rendering requirement score is calculated, and the texture level and sampling frequency are determined in combination with the performance parameters of the candidate engine.

[0017] The texture level and sampling frequency are weighted and calculated based on the rendering demand score and the current load coefficient to determine the rendering quality parameters to be used by the target rendering engine.

[0018] Optionally, based on video data carrying the region detail identifiers, dynamic environmental features of the real scene are identified, and a scene status code is generated according to the dynamic environmental features. The scene status code is used to characterize the overall environmental condition, including:

[0019] By parsing the metadata channel of the video data, the level of detail code of each grid region is extracted, and based on the level of detail code, grid regions marked as high detail are selected.

[0020] Based on the region detail identifier, the motion amplitude and direction consistency of pixels within the high-detail grid region are statistically analyzed to form a region dynamic intensity index, while the frequency of color texture changes within the region is detected.

[0021] By combining the regional dynamic intensity index and the change frequency of the color texture, a feature vector is generated. The feature vector describes the activity level of the environment and is then input into a state encoder for processing.

[0022] Based on the processed feature vector, a state pattern is matched from a preset state code table to output a binary scene state code, where each bit of the scene state code represents the state of an environmental attribute.

[0023] Optionally, a rendering requirement score is calculated based on the bits representing lighting and motion in the scene state code, including:

[0024] Based on the binary bit sequence structure of the scene status code, locate the preset bit group representing illumination attributes and the bit group representing motion attributes, and read the current logical state of each bit.

[0025] The current logical state of each bit is mapped to a numerical factor, wherein a logical true state is mapped to a first numerical factor and a logical false state is mapped to a second numerical factor, and an independent weight coefficient is assigned to each numerical factor based on a preset factor weight table.

[0026] Perform multiplication operations on all numerical factors and corresponding weight coefficients in the bit group to generate a weighted factor set, and sum the weighted factor set to obtain the initial rendering requirement score;

[0027] Based on the status patterns of other bits in the scene status code, and by querying the preset adjustment coefficient table, the score adjustment amount is obtained, and the initial rendering requirement score is added to the score adjustment amount to obtain the rendering requirement score.

[0028] Optionally, based on the results of the content analysis, the texture detail levels of different regions in the video stream are dynamically divided, and region detail identifiers associated with the content features of the video frames are generated, including:

[0029] By performing image analysis on the video stream, features such as texture complexity, motion vectors, and color changes in the image are identified. Combined with the grid division of the video stream, the feature parameters of each grid region are compared frame by frame with a preset threshold range.

[0030] Analyze the changing trend of the feature parameters within consecutive time frames, and dynamically adjust the boundary values ​​of the preset threshold range based on the changing trend;

[0031] Based on the comparison results, mesh regions with parameters exceeding the first threshold are marked as high-detail regions, mesh regions with parameters at the second threshold are marked as medium-detail regions, and mesh regions with parameters below the third threshold are marked as low-detail regions. A detail level code is assigned to each region.

[0032] Based on the spatial position of features in the video frame, each feature is bound to the corresponding grid region in the video stream to generate a region detail identifier, which includes spatial coordinates and detail level code.

[0033] Optionally, analyzing the changing trend of the feature parameters within consecutive time frames and dynamically adjusting the boundary values ​​of the preset threshold range based on the changing trend includes:

[0034] Select the most recent consecutive time frames as the analysis window, read the feature parameters corresponding to each time frame in the analysis window, and form a time series of the feature parameters.

[0035] The difference between parameters of adjacent time frames in the time series is calculated to obtain a set of difference sequences. The difference sequences are then accumulated to obtain the cumulative change of the feature parameter within the analysis window.

[0036] Identify the overall direction of parameter changes in the time series, determine an upward trend when multiple consecutive differences are positive, and determine a downward trend when multiple consecutive differences are negative, and compare the cumulative change with a preset change threshold;

[0037] Based on the overall direction and the comparison result between the cumulative change and the change threshold, the boundary values ​​of the first threshold, the second threshold and the third threshold are increased or decreased in the same direction by a fixed step size;

[0038] After adjusting the boundary value, the adjusted threshold range is used to mark the grid region details in the next time frame, and the preceding steps are repeated to ensure the continuous dynamic adjustment of the preset threshold range.

[0039] Optionally, based on the rendering quality parameters, the image data in the video stream is used as a dynamic texture and mapped onto the corresponding model surface of the digital twin scene. The scene lighting and shadow parameters are adjusted in real time according to the scene state code to drive the target rendering engine to complete the image rendering, including:

[0040] Based on the texture level in the rendering quality parameters, video frames are sampled to generate dynamic texture data, and the dynamic texture data is loaded into the graphics processor's video memory.

[0041] Using the mapping unit of the target rendering engine, the dynamic texture data is bound to the coordinates of the digital twin model surface, and the texture of the model surface is filled by an interpolation algorithm to render the dynamic detail texture in the video stream in real time.

[0042] The lighting control bits in the scene status code are analyzed to adjust the incident angle and intensity of the light source. At the same time, the range and softness of the shadows are modified according to the shadow-related bits in the scene status code.

[0043] Based on the adjusted light source and shadow parameters, the target rendering engine is driven to execute the shading procedure and output the rendered image to the display device, thus completing the rendering process.

[0044] Secondly, this application provides a video texture mapping and real-time rendering system based on a digital twin scene, including:

[0045] The acquisition module is used to synchronously acquire video streams from surveillance cameras in real-world scenarios and perform content analysis on the video streams.

[0046] The segmentation module is used to dynamically segment the texture detail levels of different regions in the video stream based on the results of the content analysis, and generate regional detail identifiers associated with the content features of the video frames.

[0047] The recognition module is used to identify the dynamic environmental features of the real scene based on video data carrying the detailed identifiers of the area, and generate a scene status code according to the dynamic environmental features. The scene status code is used to characterize the overall environmental condition.

[0048] The selection module is used to dynamically select a target rendering engine from a set of multiple heterogeneous rendering engines based on the scene status code and the current load status of the graphics processor, and simultaneously determine the rendering quality parameters corresponding to the target rendering engine and the scene status code.

[0049] The rendering module is used to map the image data in the video stream as a dynamic texture onto the corresponding model surface of the digital twin scene according to the rendering quality parameters, and to adjust the scene lighting and shadow parameters in real time according to the scene status code to drive the target rendering engine to complete the image rendering.

[0050] Thirdly, this application provides an electronic device, comprising:

[0051] Memory, used to store computer programs;

[0052] A processor, configured to execute the computer program to implement the steps of the video texture mapping and real-time rendering method based on a digital twin scene as described in the first aspect above.

[0053] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the video texture mapping and real-time rendering method based on a digital twin scene as described in the first aspect above.

[0054] The video texture mapping and real-time rendering method based on digital twin scenes provided in this application can provide comprehensive data support for subsequent rendering adaptation by synchronously acquiring and analyzing the monitoring video stream of the real scene in real time; by dynamically dividing the texture detail level of different areas and generating associated identifiers based on the analysis results of the video content, the detail presentation requirements of key areas in the picture can be clearly defined; by identifying the dynamic features of the picture based on the detail level identifiers and generating scene status codes, the overall picture status can be accurately represented; by combining scene status codes and graphics processor load status to dynamically select the target rendering engine and corresponding rendering quality parameters, the reasonable matching of rendering resources can be achieved; by mapping dynamic textures according to rendering quality parameters and adjusting lighting and shadow parameters, the detail realism and adaptability of the rendered picture can be guaranteed.

[0055] Furthermore, by acquiring the graphics processor utilization and occupancy rate in real time to calculate the load coefficient, parsing the scene state code to extract the image complexity flag and lighting and motion-related information, the rendering engine group is screened through load adaptation, and candidate engines are determined by combining the image complexity. Then, the texture level and sampling frequency are determined by calculating the rendering demand score and combining the performance parameters of the candidate engines. Finally, the load coefficient and rendering demand score are weighted to obtain the rendering quality parameters. This achieves fine-grained adaptation between the rendering engine and the rendering quality parameters, which not only ensures the smoothness of rendering in complex texture scenes, but also avoids the waste of hardware resources in simple texture scenes, improving the accuracy of rendering adaptation and the overall rendering efficiency. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 A flowchart illustrating a video texture mapping and real-time rendering method based on a digital twin scene, provided in an embodiment of this application;

[0058] Figure 2 A flowchart illustrating a specific implementation of a video texture mapping and real-time rendering method based on a digital twin scene, provided in this application embodiment;

[0059] Figure 3 This is a schematic diagram of the structure of a video texture mapping and real-time rendering system based on a digital twin scene, provided in an embodiment of this application. Detailed Implementation

[0060] In video texture mapping and rendering in digital twin scenarios, conventional texture mapping can only statically reflect the application status of a scene at a certain point in the past, and cannot endow the digital twin with real-time dynamics. Moreover, existing methods often use indiscriminate mapping of the entire image or fixed rendering strategies, relying on a single rendering engine to generate the image, failing to fully consider the complexity differences of the video content itself and the operating status of the graphics processor. This often leads to rendering effects that are out of touch with actual needs. In complex scenes with rich texture details, the high hardware load causes screen stuttering, while in simple scenes with simple textures, it results in wasted hardware resources, making it difficult to balance the realism of the rendering with resource utilization efficiency.

[0061] To address the aforementioned issues, this application proposes a video texture mapping and real-time rendering method based on digital twin scenes. The core of this method is adaptive rendering by analyzing the real-time content of the video stream and combining it with the hardware's operating status. Specifically, this method synchronously acquires the video stream and analyzes its image features in real time. First, it determines the overall environmental conditions based on the richness of texture details in different areas of the image. Then, based on the image complexity and graphics processor load, it flexibly selects a suitable rendering engine and matches the corresponding rendering quality parameters, while simultaneously adjusting scene lighting and shadow effects. This method breaks through the limitations of traditional fixed rendering modes. By dynamically adapting to video content and hardware status, it avoids rendering stuttering in complex texture scenes and reduces resource waste in simple texture scenes. It fundamentally solves the problems of insufficient rendering adaptability and efficiency in existing technologies, achieving smoother, more detailed, and more efficient visualization in digital twin scenes.

[0062] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] The core of this application is to provide a video texture mapping and real-time rendering method based on digital twin scenes. A flowchart of one specific implementation is shown below. Figure 1 As shown, the method includes:

[0064] S101. Synchronously acquire video streams from surveillance cameras in real-world scenarios and perform real-time image analysis on the video stream content.

[0065] Real-time image analysis includes extracting and quantifying features in video frames that reflect the richness of image details, such as texture complexity, color distribution, and motion vectors.

[0066] In one specific implementation, based on the spatial layout of the real scene, surveillance cameras are deployed at key monitoring locations to continuously capture dynamic images within the scene and form a continuous video stream. Simultaneously, an image analysis module processes each frame or key frame of the video stream to extract data for subsequent judgment.

[0067] S102. Based on the feature data of the image analysis, dynamically divide the texture detail level of different regions in the video stream, and generate regional detail identifiers associated with the content features of the video frames.

[0068] Among them, texture detail level is a standard used to distinguish the texture rendering fineness of different areas in the video stream. Areas with higher levels will receive better resource allocation in subsequent rendering processes to display more details. Area detail identifier is a special mark that integrates area spatial location information and detail level information. Its core function is to enable the subsequent rendering process to quickly identify the location of the area and the appropriate rendering fineness.

[0069] Optionally, step S102 may specifically include the following steps:

[0070] S1021. Based on the feature type extracted by the image analysis, quantify the value of each feature, and in conjunction with the grid division of the video stream, compare the feature value of each grid region with a preset threshold range frame by frame.

[0071] Among them, grid division is a processing method that divides a continuous video stream into multiple regular small blocks of a fixed size; grid region is a regular small block formed after grid division, and each small block corresponds to a specific spatial range in the scene; preset threshold range is a numerical judgment standard composed of a first threshold, a second threshold, and a third threshold, used to divide regions with different levels of detail.

[0072] In one specific implementation, the type of image features to be analyzed is first determined, such as texture entropy, edge density, and motion vector amplitude. Next, image processing algorithms calculate the specific values ​​of these features within each grid region. Then, for each frame of video footage, the feature values ​​corresponding to each grid region are compared with a preset threshold range.

[0073] For example, when rendering a park scene, the park video stream is divided into a 10×10 grid. The image analysis module calculates the texture complexity of each grid region (e.g., using a gray-level co-occurrence matrix). A first threshold is preset to 0.8, a second threshold to 0.5-0.8, and a third threshold to below 0.5. The texture complexity value of each grid is compared frame by frame with this threshold range. (For example, water ripples and swaying leaves have high complexity, while sky and grass areas have low complexity).

[0074] S1022. Analyze the changing trend of the feature value within a continuous time frame, and dynamically adjust the boundary value of the preset threshold range based on the changing trend.

[0075] Specifically, step S1022 may include the following processes: selecting multiple recent consecutive time frames as analysis windows, reading the feature values ​​corresponding to each time frame within the analysis window to form a time series of the feature values; calculating the differences between the values ​​of adjacent time frames in the time series to obtain a set of difference sequences, and accumulating the difference sequences to obtain the cumulative change of the feature values ​​within the analysis window; identifying the overall direction of the value changes in the time series, determining an upward trend when multiple consecutive differences are positive, and a downward trend when multiple consecutive differences are negative, and comparing the cumulative change with a preset change threshold; based on the overall direction and the comparison result of the cumulative change with the change threshold, increasing or decreasing the boundary values ​​of the first threshold, the second threshold, and the third threshold in the same direction by a fixed step size; after adjusting the boundary values, using the adjusted threshold range to mark the grid region detail level of the next time frame, and repeating the preceding steps to ensure the continuous dynamic adjustment of the preset threshold range.

[0076] In the above steps, a continuous time frame is a video frame arranged in chronological order; the trend of change is the upward or downward trend of the feature value within the continuous time frame; and the boundary value is the specific numerical limit of the first threshold, the second threshold, and the third threshold within the preset threshold range.

[0077] In one specific implementation, the most recent 20 consecutive video frames are selected as the analysis window. This window is a segment of consecutive frames used to centrally analyze the changing patterns of characteristic values. The characteristic values ​​of the corresponding grid region in each video frame are read, forming a numerical sequence arranged in chronological order. The numerical differences between adjacent frames are calculated using a formula:

[0078] ;

[0079] in, For the i-th frame and the i-th frame The numerical difference between frames, Let i be the feature value of the i-th frame. For the first After obtaining a set of difference sequences from the feature values ​​of the frames, the cumulative change is calculated using a formula:

[0080] ;

[0081] in, For cumulative changes, To analyze the number of frames within the analysis window, the cumulative change is the sum of the changes in the feature values ​​within the analysis window frame by frame. For example, if n=20 within the analysis window, and the differences between adjacent frames are 0.03, 0.05, 0.04, ..., a total of 19 differences, the cumulative change can be obtained by summing these differences sequentially.

[0082] Next, observe the positive and negative values ​​of the difference sequence. If 10 consecutive differences are positive, the value is considered to be on an upward trend; if 10 consecutive differences are negative, the value is considered to be on a downward trend. The cumulative change is then compared with a change threshold, which is a reference value used to determine whether the cumulative change of the feature value needs to be adjusted. For example, it can be set to 0.5. If the value is on an upward trend and the cumulative change exceeds the change threshold, the boundary values ​​of the first, second, and third thresholds are increased in the same direction by a fixed step size. If the value is on a downward trend and the cumulative change is below the negative change threshold, the boundary values ​​of each threshold are decreased by the same step size. The fixed step size is the fixed value that is increased or decreased each time the threshold boundary is adjusted, and can be set to 0.05.

[0083] After the adjustment is completed, the new threshold range is used to mark the grid area of ​​the next time frame with the level of detail, and the above process is repeated to achieve continuous dynamic adjustment of the threshold range.

[0084] For example, in the aforementioned park scene, the texture complexity values ​​of a certain water ripple mesh area are mostly positive over 20 consecutive frames, with a cumulative change reaching 0.6, exceeding the change threshold of 0.5. Therefore, the first threshold is adjusted to 0.85, the second threshold to 0.55-0.85, and the third threshold to below 0.55. The above example is merely one illustration of this application. In practical applications, the number of analysis window frames, change threshold, and step size can be adjusted according to scene requirements, and this application does not impose any limitations on this.

[0085] In another specific implementation, the number of frames in the analysis window can be adjusted according to the dynamic change frequency of the scene. For example, 30 frames can be selected as the analysis window in a scene with drastic changes, and 10 frames can be selected in a scene with gentle changes, so as to capture the trend of numerical changes more accurately and improve the adaptability of threshold adjustment.

[0086] S1023. Based on the comparison results, grid regions with values ​​exceeding the first threshold are marked as high detail regions, grid regions with values ​​at the second threshold are marked as medium detail regions, and grid regions with values ​​below the third threshold are marked as low detail regions. A detail level code is assigned to each region.

[0087] Among them, high detail regions are grid regions where the feature values ​​exceed the first threshold; medium detail regions are grid regions where the feature values ​​are in the second threshold range; low detail regions are grid regions where the feature values ​​are below the third threshold; and detail level codes are dedicated codes used to characterize the detail category of a region, facilitating rapid identification of detail levels in subsequent processing.

[0088] In one specific implementation, based on the frame-by-frame comparison results of S1021 and the adjusted threshold range of S1022, the level of detail is determined for each grid region. Grid regions with feature values ​​exceeding the first threshold are marked as high-detail regions and assigned a corresponding level of detail code; grid regions with feature values ​​within the second threshold range are marked as medium-detail regions and assigned a corresponding level of detail code; grid regions with feature values ​​below the third threshold are marked as low-detail regions and assigned a corresponding level of detail code.

[0089] For example, in the park scene described above, the texture complexity of a certain water ripple mesh area is 0.9, exceeding the first threshold of 0.85 after adjustment, and is therefore marked as a high-detail area and assigned a corresponding level of detail code; the texture complexity of a certain grass mesh area is 0.6, falling within the second threshold range of 0.55-0.85, and is therefore marked as a medium-detail area and assigned a corresponding level of detail code. The above example is merely one example of this application; in practical applications, different forms of level of detail codes can be designed, and this application does not limit this.

[0090] S1024. Based on the spatial location of the features obtained from image analysis, bind each feature to the corresponding grid region in the video stream to generate a region detail identifier, which includes spatial coordinates and detail level code.

[0091] Coordinate binding is a process that associates the coordinates of a specific grid region in a video frame with the image feature analysis results of that region.

[0092] In one specific implementation, after the video frames are divided into grids and their features are calculated, each grid region naturally possesses its own coordinates. These spatial coordinates are then combined with the level of detail code of the grid region to generate a complete region detail identifier, ensuring that subsequent processing can quickly locate the region and identify its level of detail using this identifier.

[0093] For example, in the park scene above, the grid area at coordinates (10,8) in the video stream corresponds to a swaying leaf, and its level of detail code is the corresponding high level of detail code. When combined, they form a complete area detail identifier.

[0094] This application, through the aforementioned steps, dynamically divides the texture detail levels of different regions by analyzing the image content of the video stream in real time. This achieves accurate identification and marking of areas rich in detail within the image, allowing subsequent rendering resources to be allocated to these key areas, avoiding resource waste and detail loss caused by indiscriminate processing. Simultaneously, by dynamically adjusting the threshold range, it ensures that the detail level division adapts to real-time changes in the video content. Compared to fixed division methods, this approach is more flexible and adaptable, significantly improving the accuracy of region differentiation and providing a scientific basis for achieving high-fidelity, high-efficiency detailed texture rendering.

[0095] S103. Based on the video data carrying the regional detail identifier, identify the dynamic features of the video frame, and generate a scene state code according to the dynamic frame features.

[0096] The scene status code is used to characterize the overall screen status.

[0097] Among them, dynamic image features are image attributes in the video that change over time, including features that reflect the dynamic complexity of the image, such as pixel movement and color and texture changes; scene status codes are codes that represent the overall state of the image in binary form, with each bit corresponding to a specific state of an image attribute, which facilitates rapid subsequent reading and processing.

[0098] Optionally, step S103 may specifically include the following steps:

[0099] S1031. By parsing the metadata channel of the video data, the detail level code of each grid area is extracted, and based on the detail level code, the grid areas marked as high detail are selected.

[0100] The metadata channel is a channel in video data that carries additional information and is used to store related data such as the level of detail code of the grid area.

[0101] In this embodiment, the video data carrying regional detail identifiers is first parsed, focusing on reading the information stored in the metadata channel to extract the detail level code corresponding to each grid region. Then, based on the detail level code, data marked as high-detail regions is retained, while data from medium-to-low-detail regions is excluded, focusing on the core regions rich in detail for subsequent analysis.

[0102] S1032. Based on the region detail identifier, the motion amplitude and direction consistency of pixels within the high-detail grid region are statistically analyzed to form a region dynamic detail index, while the frequency of color texture changes within the region is detected.

[0103] Among them, the regional dynamic detail index is a parameter that comprehensively measures the consistency of the movement amplitude and direction of pixels within a high-detail grid area, and quantifies the richness of dynamic details in the image within that area; the color texture change frequency is the number of times the color texture features in a high-detail grid area change per unit time, reflecting the visual changes in the image within that area.

[0104] In one specific implementation, a selected high-detail mesh region is located based on regional detail markers. Optical flow is then used to calculate the displacement of each pixel within this region across consecutive frames, yielding the pixel's motion amplitude. The average motion amplitude of the region is then calculated using the following formula:

[0105] ;

[0106] in, The average range of motion in the region. This represents the total number of pixels in the high detail area. Let x be the displacement of the k-th pixel in the x-direction. Let be the displacement of the k-th pixel in the y-direction.

[0107] Simultaneously, the consistency of pixel motion direction is statistically analyzed. If the motion direction deviation of most pixels is within 30°, the direction consistency is considered high. This consistency is combined with the average motion amplitude and weighted calculation to form the regional dynamic detail index. For example, if a high detail region contains 1000 pixels, the sum of the square roots of the squares of the motion displacements of each pixel is 5000, the average motion amplitude is 5, and 80% of the pixels have a motion direction deviation within 30°, the corresponding regional dynamic detail index is obtained through weighted calculation.

[0108] Furthermore, by extracting color histogram features from high-detail regions in consecutive frames, the similarity between histograms of adjacent frames is calculated. When the similarity is below a preset threshold, a change in color texture is determined, and the frequency of color texture changes is obtained by counting the number of changes per unit time. For example, when rendering a park scene, the high-detail region is the shimmering water surface on the lake. The above method is used to count the motion amplitude and directional consistency of pixels in this region to form a dynamic detail index for the region. At the same time, the frequency of color texture changes caused by lighting and ripples on the water surface is detected to determine whether the region requires higher-precision dynamic texture rendering.

[0109] The above example is only one example of this application. In practical applications, parameters such as the motion direction deviation threshold and the color histogram similarity threshold can also be adjusted. This application does not limit this.

[0110] S1033. Combining the regional dynamic detail index and the color texture change frequency, a feature vector is generated. The feature vector describes the dynamic complexity of the image and is input into the state encoder for processing.

[0111] Among them, the feature vector is a vector data formed by combining the regional dynamic detail indicators and the change frequency of color texture according to a preset format, which carries the key information of the dynamic complexity of the picture; the state encoder is a processing module used to transform the feature vector and output state information that conforms to the preset format.

[0112] The training process of the state encoder is as follows: collect regional dynamic detail indicators and color and texture change frequency data under different scenarios, divide them into training set and test set according to the proportion, construct an encoder model based on a fully connected neural network, take feature vector as input and preset standard state information as output, adjust the model parameters through backpropagation algorithm until the accuracy of the model on the test set reaches the preset requirements, and complete the training.

[0113] In this embodiment, the regional dynamic detail index and color texture change frequency obtained in S1032 are used as two-dimensional feature values. They are combined in the order of "regional dynamic detail index - color texture change frequency" to generate a two-dimensional feature vector. This vector directly reflects the dynamic complexity of the image. The generated feature vector is then input into a trained state encoder. The encoder performs normalization, dimension transformation, and other processing on the feature vector, and outputs standardized state information.

[0114] For example, in the park scene described above, the dynamic detail index of a certain high-detail area (such as the water surface) is 8, and the color and texture change frequency is 2 times / second. A feature vector [8,2] is generated by combining these values ​​and input into the state encoder to obtain the corresponding standardized state information. The above example is merely one illustration of this application. In practical applications, more feature dimensions can be added to construct feature vectors, and this application does not limit this.

[0115] In another specific implementation, feature dimensions can be increased according to the complexity of the scene, such as adding features like the intensity of changes in screen lighting, to generate a multi-dimensional feature vector, which is then input into the state encoder for processing, in order to more comprehensively reflect the screen condition.

[0116] S1034. Based on the processed feature vector, match the state pattern from the preset state code table to output a binary scene state code.

[0117] Each bit of the scene status code represents the state of a screen attribute.

[0118] The preset status code table is a pre-defined mapping table between status patterns and binary scene status codes, containing status patterns and codes corresponding to various screen conditions; the status pattern is the screen condition type obtained after classifying the processed feature vectors.

[0119] In one specific implementation, a pre-built state code table is constructed, storing state modes and corresponding binary scene state codes for different scene conditions. For example, high dynamic detail corresponds to a state mode and binary code 1010, while medium dynamic detail corresponds to a state mode and binary code 0110, and so on. The feature vector processed by the state encoder is compared with various state modes in the pre-built state code table to find the best matching state mode. The binary code corresponding to this mode is then extracted and output as the scene state code.

[0120] For example, in the park scene mentioned above, the processed feature vector matches the state pattern of "high dynamic and high frequency texture change" (corresponding to the shimmering water surface), and the corresponding binary scene state code is 1001. Each bit of this code represents the scene attributes such as the motion state and texture change state.

[0121] This application, through the aforementioned steps, accurately identifies the dynamic characteristics of video footage and quantifies them into standardized scene state codes. This makes abstract information such as texture complexity and dynamic details of the image intuitive and quantifiable, solving the problem that traditional methods struggle to accurately represent rendering requirements based on the video content itself. This code provides a clear and reliable decision-making basis for the subsequent dynamic adjustment of rendering strategies, ensuring that rendering adaptation is no longer blind but closely matches the actual situation of the video footage. This lays the core foundation for achieving adaptive rendering and improves the matching degree between rendering strategies and image rendering requirements.

[0122] S104. Based on the scene status code and combined with the current load status of the graphics processor, dynamically select the target rendering engine from a plurality of preset heterogeneous rendering engines, and simultaneously determine the rendering quality parameters corresponding to the target rendering engine and the scene status code.

[0123] In this step, heterogeneous rendering engines refer to multiple rendering engines with different functional characteristics and performance focuses, which can adapt to different image complexity and hardware load requirements; the target rendering engine is the rendering engine selected from multiple heterogeneous rendering engines that is most suitable for the current image and hardware status; rendering quality parameters are key parameters that control the rendering effect, including texture level, sampling frequency, etc., which directly affect the image presentation effect and rendering efficiency.

[0124] Optionally, such as Figure 2 As shown, step S104 may specifically include the following steps:

[0125] S1041. Real-time acquisition of the utilization and occupancy rate of the graphics processor to calculate the current load coefficient, and at the same time parsing the bit bits of the scene status code to extract the screen complexity flag.

[0126] Among them, the utilization rate of the graphics processor refers to the percentage of workload of the graphics processor per unit time, the occupancy rate refers to the percentage of resources of the graphics processor occupied by the current task; the current load coefficient is calculated by combining the utilization rate and the occupancy rate, and is a quantitative indicator that characterizes the current operating pressure of the graphics processor; the screen complexity flag is extracted from the scene status code and is an identification information that reflects the complexity of the video screen content.

[0127] In one specific implementation, the utilization and occupancy data of the graphics processor are collected in real time through the system interface, and the current load factor is calculated using a formula.

[0128] ;

[0129] in, This represents the current load factor. To maximize utilization, For occupancy rate, The weighting coefficient for utilization rate (range of values) ).For example, The collection utilization rate was occupancy rate The current load factor .

[0130] Simultaneously, the binary bits of the scene status code are parsed, and according to the preset bit definition, the combination of bits representing the complexity of the scene is extracted as a scene complexity indicator. For example, in rendering a park scene, the scene status code is 1011, where the first two bits are used to represent the scene complexity. After parsing, the scene complexity indicator is extracted as 10, corresponding to the presence of dynamic water surfaces and swaying leaves in the scene, which belongs to the medium complexity level.

[0131] S1042. Compare the current load coefficient with the preset load threshold, and select a load-adaptive rendering engine group from a plurality of preset heterogeneous rendering engines.

[0132] Among them, the preset load threshold is a pre-set reference value used to classify the load level of the graphics processor, which usually includes low load threshold, medium load threshold and high load threshold; the rendering engine group is a collection of engines divided from multiple heterogeneous rendering engines according to load adaptability, and the engines in the same group are suitable for the same load level of the scene.

[0133] In one specific implementation, a low load threshold of 0.3 and a medium load threshold of 0.7 are preset. The current load coefficient is compared with these thresholds: if the current load coefficient is lower than the low load threshold, it is determined to be a low load state; if it is between the low load threshold and the medium load threshold, it is determined to be a medium load state; if it is higher than the medium load threshold, it is determined to be a high load state. Based on the determined load state, a rendering engine adapted to that load state is selected from a preset set of multiple heterogeneous rendering engines to form a load-adaptive rendering engine group.

[0134] For example, in rendering a park scene, the calculated current load factor is 0.68, which is between 0.3 and 0.7, and is determined to be a medium load state. Three engines suitable for medium load are selected from the preset heterogeneous rendering engines to form a rendering engine group.

[0135] S1043. Based on the image complexity flag, select candidate engines from the rendering engine group.

[0136] Among them, the candidate engine is a rendering engine that is further selected from the rendering engine group and is adapted to the current screen complexity. It is a candidate for the target rendering engine.

[0137] In one specific implementation, each heterogeneous rendering engine is pre-labeled with its applicable image complexity range. For example, some engines are adapted to static images with simple textures, while others are adapted to dynamic images with rich textures and medium to high complexity. Based on the image complexity flag extracted in S1041, the complexity level of the current image is determined. From the rendering engine group obtained in S1042, rendering engines that are adapted to the complexity level are selected as candidate engines.

[0138] For example, in rendering a park scene, the complexity level corresponding to the image complexity flag is medium. The rendering engine group includes three engines suitable for medium loads, two of which are labeled as suitable for medium-complexity images; these two engines are the candidate engines. The above example is only one example of this application. In practical applications, the image complexity level classification can be refined according to scene requirements, and this application does not limit this.

[0139] S1044. Based on the bits representing illumination and motion in the scene status code, calculate the rendering requirement score, and combine it with the performance parameters of the candidate engine to determine the texture level and sampling frequency.

[0140] Specifically, step S1044, "calculating the rendering requirement score based on the bits representing lighting and motion in the scene state code," can include the following process:

[0141] Based on the binary bit sequence structure of the scene status code, the preset bit groups representing lighting attributes and motion attributes are located, and the current logical state of each bit is read. The current logical state of each bit is mapped to a numerical factor, where a logically true state is mapped to a first numerical factor and a logically false state is mapped to a second numerical factor. An independent weight coefficient is assigned to each numerical factor based on a preset factor weight table. Multiplication is performed on all numerical factors in the bit group with their corresponding weight coefficients to generate a weighted factor set. The weighted factor set is then summed to obtain the initial rendering requirement score. Based on the state patterns of other bits in the scene status code and by querying a preset adjustment coefficient table, the score adjustment amount is obtained. The initial rendering requirement score is then added to the score adjustment amount to obtain the rendering requirement score.

[0142] In the above steps, texture level is a standard that characterizes the richness of texture details; the higher the level, the clearer the texture details. Sampling frequency is the frequency at which the texture is sampled during the rendering process; the higher the frequency, the more delicate the image. Factor weight table is a pre-defined table that assigns weight coefficients to the numerical factors corresponding to each bit representing lighting and motion. Adjustment coefficient table is a table that stores the correspondence between the state mode of other bits of the scene state code and the score adjustment amount.

[0143] In one specific implementation, the preset lighting attribute bit group and motion attribute bit group in the scene state code are first located, and the logical state of each bit is read. The logical true state is mapped to a first numerical factor of 1.0, and the logical false state is mapped to a second numerical factor of 0.5. Then, a weight coefficient is assigned to each numerical factor through a preset factor weight table. For example, the weight coefficients of the three bits of the lighting attribute are 0.3, 0.2, and 0.2, respectively, and the weight coefficients of the two bits of the motion attribute are 0.2 and 0.1, respectively.

[0144] The weighting factor for each bit is calculated using the following formula:

[0145] ;

[0146] in, Let be the weighting factor for the i-th bit. For numerical factors, These are the weighting coefficients. The initial rendering requirement score is then calculated using the formula:

[0147] ;

[0148] in, The initial rendering requirement score. This represents the total number of bits involved in the calculation. For example, the lighting attribute has 3 bits with logical states of True, True, and False, and the motion attribute has 2 bits with logical states of True and False. The corresponding weighting factors are respectively... Initial rendering requirement score .

[0149] Query the preset adjustment coefficient table, obtain the score adjustment amount of 0.1 based on the state mode of other bits of the scene state code, and calculate the rendering requirement score using the formula:

[0150] ;

[0151] in, To render the required score, For fractional adjustment, i.e. .

[0152] Finally, the performance parameters of the candidate engines are obtained, including processing speed and texture processing capabilities. The corresponding texture level and sampling frequency are matched based on the rendering requirement score; for example, a rendering requirement score of 0.95 corresponds to a high texture level and a high sampling frequency. For instance, in rendering a park scene, the lighting and motion-related bits in the scene state code reflect significant changes in light and shadow, and active dynamics in the water surface and leaves. A rendering requirement score of 0.95 is calculated. Combined with the high-performance parameters of the candidate engines, a high texture level and a high sampling frequency are determined. The above example is merely one illustration of this application. In practical applications, the numerical factors, weighting coefficients, and the contents of the adjustment coefficient table can also be adjusted, and this application does not limit these adjustments.

[0153] S1045. Based on the rendering requirement score and the current load coefficient, the texture level and sampling frequency are weighted and calculated to determine the rendering quality parameters to be used by the target rendering engine.

[0154] In one specific implementation, the weight of the rendering requirement score is set to 0.6, the weight of the current load coefficient is set to 0.4, and the texture level and sampling frequency determined in S1044 are adjusted by weighting.

[0155] For example, if the initial texture level is determined to be 4, the sampling frequency is 60Hz, the current load factor is 0.68, and the rendering requirement score is 0.95, the texture level is adjusted through weighted calculation to... Rounded to level 4; sampling frequency is The frequency was rounded down to 53Hz. Finally, these two parameters were determined as the rendering quality parameters for the target rendering engine.

[0156] Simultaneously, the engine whose performance best matches the adjusted rendering quality parameters is selected from the candidate engines as the target rendering engine. For example, in rendering a park scene, after weighted calculation, the texture level is determined to be 4 and the sampling frequency to be 53Hz. The engine that is more efficient at handling high texture levels and the sampling frequency is selected from the two candidate engines as the target rendering engine. The above example is only one example of this application. In practical applications, the weight ratio and calculation method can be adjusted, and this application does not limit this.

[0157] In another specific implementation, the weight ratio can be adjusted according to the scene's emphasis on rendering effect and efficiency. If more emphasis is placed on rendering effect, the weight of the rendering requirement score can be increased; if more emphasis is placed on running smoothness, the weight of the current load coefficient can be increased.

[0158] This application, through the aforementioned steps, dynamically selects the rendering engine and determines rendering quality parameters based on scene status codes and graphics processor load status, breaking the limitations of traditional single rendering engines and fixed rendering parameters. It avoids rendering stuttering caused by excessive hardware load when rendering complex textures, and eliminates the waste of idle hardware resources when rendering simple textures, achieving optimal configuration of rendering resources. Simultaneously, by finely calculating rendering demand scores and weighted adjusting parameters, the rendering effect satisfies both visual requirements and adapts to hardware operating conditions, significantly improving rendering smoothness, efficiency, and adaptability.

[0159] S105. Based on the rendering quality parameters, the image data in the video stream is used as a dynamic texture and mapped onto the corresponding model surface of the digital twin scene. The scene lighting and shadow parameters are adjusted in real time according to the scene status code to drive the target rendering engine to complete the image rendering.

[0160] Among them, dynamic texture is texture data that uses continuous image data in the video stream as the texture source and updates in real time with the video content; digital twin model is a digital replica of the real scene in virtual space, and its surface is used to carry dynamic texture; lighting and shadow parameters are parameters that control the characteristics of light sources and the shadow rendering effect in the digital twin scene, including incident angle, intensity, shadow range, etc.

[0161] Optionally, step S105 may specifically include the following steps:

[0162] S1051. Based on the texture level in the rendering quality parameters, the video frames are sampled to generate dynamic texture data, and the dynamic texture data is loaded into the video memory area of ​​the graphics processor.

[0163] The video memory area is a high-speed storage area in the graphics processor specifically used to store texture data, rendering instructions, etc., which can provide fast data access support for the rendering process.

[0164] In one specific implementation, based on the texture level determined in S104, each frame of the video stream is sampled using a sampling method with corresponding precision. A higher texture level results in higher sampling precision and retains richer image details. Texture information is extracted from the frame images through sampling and combined to form continuously updated dynamic texture data. The generated dynamic texture data is then encoded and converted according to a format supported by the graphics processor and loaded into the video memory area, preparing data for subsequent texture mapping.

[0165] For example, in rendering a park scene, if the determined texture level is 4 (high level), then a sampling precision of 1920×1080 resolution is used to sample video frames of the park's lake surface to capture rich details of the water ripples, generate dynamic texture data, and load it into the graphics processor's video memory. The above example is merely one example of this application; in practical applications, the sampling precision can be adjusted according to the texture level, and this application does not limit this.

[0166] In another specific implementation, the sampling interval can be dynamically adjusted according to the frame rate of the video stream. If the video frame rate is high, the sampling interval can be appropriately increased to reduce the amount of data; if the frame rate is low, frame-by-frame sampling is used to ensure the smoothness of dynamic textures.

[0167] S1052. Using the mapping unit of the target rendering engine, the dynamic texture data is bound to the coordinates of the digital twin model surface, and the texture of the model surface is filled by an interpolation algorithm.

[0168] Among them, the mapping unit is a functional module in the target rendering engine specifically used to associate the coordinates of the texture with the model surface; the interpolation algorithm is used to fill the gaps between the pixels of the texture on the model surface, making the texture appear smoother and more natural. The commonly used one is the bilinear interpolation algorithm.

[0169] In one specific implementation, the mapping unit of the target rendering engine is invoked to read dynamic texture data from the video memory area, while simultaneously acquiring the three-dimensional coordinate information of the digital twin model's surface. The two-dimensional pixel coordinates of the dynamic texture are bound one-to-one with the three-dimensional coordinates of the model surface according to a preset mapping rule, ensuring the correct positioning of the texture on the model surface. For the texture pixel gaps corresponding to the model surface coordinates, a bilinear interpolation algorithm is used to fill the gaps, and the pixel values ​​at the gaps are calculated using the following formula:

[0170] ;

[0171] in, The coordinates of the gap are... The coefficients calculated for four adjacent pixels ensure that the texture is evenly covered on the model surface and the transition is natural.

[0172] For example, in rendering a park scene, the dynamic texture data of the sampled lake surface ripples is bound to the surface coordinates of the corresponding lake surface in the digital twin model. The texture gaps on the surface of the lake model are filled by an interpolation algorithm, so that the water surface texture is clear and smooth.

[0173] S1053. Analyze the lighting control bits in the scene status code to adjust the incident angle and intensity of the light source, and modify the range and softness of the shadow according to the shadow-related bits in the scene status code.

[0174] In one specific implementation, the meanings of the bits controlling lighting and shadows in the scene status code are predefined. For example, the first two bits control the incident angle of the light, the middle two bits control the light intensity, and the last two bits control the shadow range and softness. The corresponding bits in the scene status code are parsed, and the desired lighting and shadow effects are determined based on the bit combinations. If the lighting-related bit combination is 11, it corresponds to a larger incident angle and high light intensity; if the shadow-related bit combination is 10, it corresponds to a medium shadow range and relatively high softness.

[0175] Based on these analysis results, the light source and shadow parameters in the digital twin scene are adjusted in real time to match the scene lighting and shadows with the environmental conditions presented in the video.

[0176] For example, if the scene status code parsing shows that the current lighting in the video is sufficient and the tree shadows are clear, then the incident angle of the light source in the digital twin scene is adjusted to 60°, the intensity is adjusted to a high level, the shadow range is adjusted to 1.2 times the size of the tree, and the softness is adjusted to medium.

[0177] S1054. Based on the adjusted light source and shadow parameters, drive the target rendering engine to execute the shading program and output the rendered image to the display device to complete the rendering process.

[0178] Among them, the shading program is a program in the target rendering engine used to calculate the surface color and lighting reflection effect of the model, and can generate the final pixel color based on parameters such as light source and texture; the display device is the terminal device used to present the rendered screen, such as monitors and monitoring screens.

[0179] In one specific implementation, the adjusted light source parameters, shadow parameters, and texture-bound digital twin model data are input into the target rendering engine. The engine calls its built-in shading program, combining the texture data and lighting and shadow parameters, to calculate the final color value of each pixel on the model surface, completing the rendering calculation of the scene. After the rendering calculation is complete, the generated rendered image is converted according to the resolution and format supported by the display device and output to the display device for presentation, thus completing the entire rendering process.

[0180] For example, the target rendering engine combines the dynamic texture of the park's lake surface with adjusted lighting and shadow parameters, executes a shading procedure to calculate the color of each pixel, and finally outputs a clear and realistic digital twin rendering of the park on the monitoring screen.

[0181] This application, through the aforementioned steps, maps video stream images as dynamic textures to a digital twin model, achieving visual synchronization between the real scene and the virtual model, making the digital twin scene more realistic and dynamic. By combining scene state codes with real-time adjustments to lighting and shadow parameters, the visual effects of the rendered image highly match the real environment presented in the video, improving the accuracy of the visualization. Optimizing texture sampling, coordinate binding, and interpolation filling processes ensures a clear and smooth presentation of dynamic textures on the model surface, while the efficient execution of the target rendering engine guarantees smooth output of the rendered image. Overall, this achieves a high-fidelity, high-smoothness visualization effect for the digital twin scene, providing users with an intuitive and accurate scene observation experience.

[0182] The method proposed in this application can be illustrated through an example of a "park digital twin," the core objective of which is to render the dynamic ripples on the surface of a park lake in real time and with high fidelity. The process begins with video capture and analysis of the real-world scene, followed by dynamic allocation of rendering resources through intelligent decision-making, ultimately mapping the vivid scene onto a virtual model and completing efficient rendering, thereby achieving visual synchronization between the digital twin scene and the real world.

[0183] First, cameras deployed along the park's lake capture real-time video streams. Upon receiving the video, the system immediately performs image analysis, dividing the image into a grid and calculating the "texture complexity" of each region. The shimmering lake water area, due to its dramatic changes in pixel color and brightness, is assigned a "high level of detail," while the nearby static stone path is marked as "low level of detail." Through this step, the system accurately identifies the dynamic detail areas that require priority rendering, laying the foundation for subsequent resource optimization and allocation (S102).

[0184] Next, the system focuses on the "high-detail" lake area, analyzing its motion and texture changes using optical flow and color histograms to generate a binary "scene state code" (S103) that quantifies the "high dynamic range, strong lighting" scene. This code, along with the real-time monitored GPU load (e.g., 65%), is input into the decision system. Based on these two dimensions, the system intelligently selects the most suitable one from multiple heterogeneous rendering engines and calculates the optimal "rendering quality parameters" that balance rendering quality and hardware performance, such as determining to use a high texture level and an optimized sampling frequency.

[0185] During the rendering execution phase, the selected target rendering engine, based on the high-quality parameters determined in the previous step, performs high-precision sampling of the lake surface image from the camera video, generating a continuously updated "dynamic texture." This dynamic texture is mapped in real-time onto a pre-built 3D model of the lake surface in the digital twin scene. Simultaneously, the system adjusts the angle and intensity of light sources in the virtual scene based on the lighting information parsed from the scene state codes, ensuring that the lighting effects are consistent with the real world. Finally, the engine executes the shading procedure, blending all elements to output the final rendered image.

[0186] In summary, through a complete process from video analysis and intelligent decision-making to dynamic rendering, this method successfully presented a digital twin of a shimmering park lake on a display device, synchronized with the real scene. This example demonstrates that this application, through in-depth analysis of the video content itself, adaptively concentrates rendering resources on areas requiring the most detailed representation, achieving not only high-fidelity visual effects but also ensuring a smooth and efficient rendering process. It perfectly solves the challenge of efficiently and realistically presenting dynamic and complex textures in digital twin scenes.

[0187] Figure 3 This is a schematic diagram illustrating a specific implementation of a video texture mapping and real-time rendering system based on a digital twin scene, as provided in this application. (Refer to...) Figure 3 The system may include:

[0188] The acquisition module 31 is used to synchronously acquire video streams from surveillance cameras in real-world scenes and perform content analysis on the video streams.

[0189] The segmentation module 32 is used to dynamically segment the texture detail levels of different regions in the video stream based on the results of the content analysis, and generate regional detail identifiers associated with the content features of the video frames;

[0190] The identification module 33 is used to identify the dynamic environmental features of the real scene based on video data carrying the regional detail identifier, and generate a scene status code according to the dynamic environmental features. The scene status code is used to characterize the overall environmental condition.

[0191] Selection module 34 is used to dynamically select a target rendering engine from a plurality of preset heterogeneous rendering engines based on the scene status code and in combination with the current load status of the graphics processor, and simultaneously determine the rendering quality parameters corresponding to the target rendering engine and the scene status code.

[0192] The rendering module 35 is used to map the image data in the video stream as a dynamic texture onto the corresponding model surface of the digital twin scene according to the rendering quality parameters, and to adjust the scene lighting and shadow parameters in real time according to the scene status code, so as to drive the target rendering engine to complete the image rendering.

[0193] The video texture mapping and real-time rendering system based on digital twin scenes in this application is used to implement the aforementioned video texture mapping and real-time rendering method based on digital twin scenes. Therefore, the specific implementation of the video texture mapping and real-time rendering system based on digital twin scenes can be found in the embodiment section of the video texture mapping and real-time rendering method based on digital twin scenes above. The specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0194] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described video texture mapping and real-time rendering methods based on digital twin scenes.

[0195] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described video texture mapping and real-time rendering methods based on digital twin scenes.

[0196] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0197] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the video texture mapping and real-time rendering method based on a digital twin scene.

[0198] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0199] The foregoing has provided a detailed description of a video texture mapping and real-time rendering method and system based on a digital twin scene, as provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for video texture mapping and real-time rendering based on digital twin scene, characterized in that, The method comprises the following steps: Synchronously acquiring a video stream of a monitoring camera in a real scene and performing content analysis on the video stream; According to the result of the content analysis, dynamically dividing the texture detail level of different regions in the video stream, and generating a region detail identifier associated with the content features of the video frame; Based on the video data carrying the region detail identifier, identifying the dynamic environment features of the real scene, and generating a scene state code according to the dynamic environment features, wherein the scene state code is used to represent the comprehensive environment condition; Based on the scene state code and in combination with the current load state of the graphics processor, dynamically selecting a target rendering engine from a plurality of preset heterogeneous rendering engines, and synchronously determining a rendering quality parameter corresponding to the target rendering engine and the scene state code; According to the rendering quality parameter, mapping the image data in the video stream as a dynamic texture to the corresponding model surface of the digital twin scene, and adjusting the scene lighting and shadow parameters in real time according to the scene state code, so as to drive the target rendering engine to complete picture rendering.

2. The method of claim 1, wherein, Based on the scene state code and in combination with the current load state of the graphics processor, dynamically selecting a target rendering engine from a plurality of preset heterogeneous rendering engines, and synchronously determining a rendering quality parameter corresponding to the target rendering engine and the scene state code, comprising: Real-time acquisition of the utilization and occupancy of the graphics processor to calculate the current load coefficient, and analysis of the bit of the scene state code to extract the environment complexity flag; Comparing the current load coefficient with a preset load threshold to select a rendering engine group that is suitable for load from a plurality of preset heterogeneous rendering engines; According to the environment complexity flag, screening a candidate engine from the rendering engine group; Based on the bit representing lighting and motion in the scene state code, calculating a rendering requirement score, and in combination with the performance parameter of the candidate engine, determining the texture level and sampling frequency; According to the rendering requirement score and the current load coefficient, performing weighted calculation on the texture level and sampling frequency to determine the rendering quality parameter for the target rendering engine.

3. The method of claim 1, wherein, Based on the video data carrying the region detail identifier, identifying the dynamic environment features of the real scene, and generating a scene state code according to the dynamic environment features, wherein the scene state code is used to represent the comprehensive environment condition, comprising: By analyzing the metadata channel of the video data, extracting the detail level code of each grid region, and based on the detail level code, screening out the grid region marked as high detail; According to the region detail identifier, the motion amplitude and direction consistency of the pixel points in the high-detail grid region are counted to form a region dynamic intensity index, and the change frequency of the color texture in the region is detected; In combination with the region dynamic intensity index and the change frequency of the color texture, a feature vector is generated, which describes the environment activity level, and the feature vector is input into a state encoder for processing; Based on the processed feature vector, a state mode is matched from a preset state code table to output a binary scene state code, each bit of the scene state code representing a state of an environmental attribute.

4. The method of claim 2, wherein, Based on the bits in the scene state code representing illumination and motion, a rendering requirement score is calculated, including: According to the binary bit sequence structure of the scene state code, a preset bit group representing illumination attributes and a bit group representing motion attributes are located, and the current logic state of each bit is read; The current logic state of each bit is mapped to a numerical factor, where a logic true state is mapped to a first numerical factor and a logic false state is mapped to a second numerical factor, and each numerical factor is assigned an independent weight coefficient based on a preset factor weight table; A multiplication operation is performed on all numerical factors and corresponding weight coefficients in the bit group to generate a set of weighted factors, and the set of weighted factors is accumulated to obtain an initial rendering requirement score; According to the state mode of other bits in the scene state code, an adjustment coefficient table is queried to obtain a score adjustment amount, and the initial rendering requirement score is added to the score adjustment amount to obtain a rendering requirement score.

5. The method of claim 1, wherein, According to the results of the content analysis, the texture detail levels of different regions in the video stream are dynamically divided, and region detail identifiers associated with the content features of the video frames are generated, including: By performing image analysis on the video stream, the texture complexity, motion vector, and color change features in the image are identified, and each grid region feature parameter is compared with a preset threshold range frame by frame in combination with the grid division of the video stream; The trend of the feature parameters in consecutive time frames is analyzed, and the boundary values of the preset threshold range are dynamically adjusted according to the trend; Based on the comparison results, grid regions with parameters exceeding a first threshold are marked as high-detail regions, grid regions with parameters within a second threshold are marked as medium-detail regions, and grid regions with parameters below a third threshold are marked as low-detail regions, and a detail level code is assigned to each region; According to the positions of the features in the space in the video frames, each feature is coordinate-bound to the corresponding grid region in the video stream to generate a region detail identifier, which includes a spatial coordinate and a detail level code.

6. The method of claim 5, wherein, The trend of the feature parameters in consecutive time frames is analyzed, and the boundary values of the preset threshold range are dynamically adjusted according to the trend, including: A plurality of consecutive time frames are selected as an analysis window, and the feature parameters corresponding to each time frame in the analysis window are read to form a time sequence of the feature parameters; The difference values of adjacent time frame parameters in the time sequence are calculated to obtain a set of difference value sequences, and the difference value sequences are accumulated to obtain the cumulative change amount of the feature parameters in the analysis window; The overall direction of parameter change in the time sequence is identified, and when a plurality of consecutive difference values are positive, an upward trend is determined, and when a plurality of consecutive difference values are negative, a downward trend is determined, and the cumulative change amount is compared with a preset change amount threshold; According to the overall direction and the comparison result of the accumulated change amount and the change amount threshold, the boundary values of the first threshold, the second threshold and the third threshold are increased or decreased by a fixed step in the same direction; After adjusting the boundary values, the adjusted threshold range is used for grid region detail marking of the next time frame, and the previous steps are repeatedly executed to ensure continuous dynamic adjustment of the preset threshold range.

7. The method of claim 1, wherein, According to the rendering quality parameters, the image data in the video stream is mapped to the corresponding model surface of the digital twin scene as dynamic texture, and the scene lighting and shadow parameters are adjusted in real time according to the scene state code to drive the target rendering engine to complete picture rendering, including: According to the texture level in the rendering quality parameters, the video frame is sampled to generate dynamic texture data, and the dynamic texture data is loaded into the video memory area of the graphics processor; The mapping unit of the target rendering engine binds the dynamic texture data with the coordinates of the digital twin model surface, and fills the texture of the model surface through an interpolation algorithm to render the dynamic detail texture in the video stream in real time; The bits controlling lighting in the scene state code are parsed to adjust the incident angle and intensity of the light source, and the range and softness of the shadow are modified according to the shadow-related bits in the scene state code; Based on the adjusted light source and shadow parameters, the target rendering engine is driven to execute the shading program, and the rendered picture is output to the display device to complete the rendering process. 8.A system for video texture mapping and real-time rendering based on digital twin scene, characterized in that, Comprising: The acquisition module is used for synchronously acquiring the video stream of the monitoring camera in the real scene, and performing content analysis on the video stream; The division module is used for dynamically dividing the texture detail levels of different regions in the video stream according to the results of the content analysis, and generating region detail identifiers associated with the content features of the video frames; The identification module is used for identifying the dynamic environment features of the real scene based on the video data carrying the region detail identifiers, and generating a scene state code according to the dynamic environment features, wherein the scene state code is used for representing the comprehensive environment status; The selection module is used for dynamically selecting a target rendering engine from a plurality of preset heterogeneous rendering engines based on the scene state code and in combination with the current load state of the graphics processor, and synchronously determining the rendering quality parameters corresponding to the target rendering engine and the scene state code; The rendering module is used for mapping the image data in the video stream as dynamic texture to the corresponding model surface of the digital twin scene according to the rendering quality parameters, and adjusting the scene lighting and shadow parameters in real time according to the scene state code to drive the target rendering engine to complete picture rendering.

9. An electronic device, comprising: Comprising: The memory is used for storing a computer program; The processor is used for executing the computer program to implement the steps of the video texture mapping and real-time rendering method based on the digital twin scene according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method for video texture mapping and real-time rendering based on a digital twin scene according to any one of claims 1 to 7.

Images