Real-time high-resolution virtual simulation cloud plug flow method and system

Abstract

The invention belongs to the field of artificial intelligence, computer vision and computer graphics, and discloses a real-time high-resolution virtual simulation cloud plug flow method and system, and the method comprises the steps: obtaining a key frame of an original rendering frame image or video; carrying out block processing and screening on the input key frame or rendering frame image; using a pre-trained ResNet18 model to extract the feature embedding of each screened block; based on feature embedding of each block, obtaining a matched neural network model for each block by using a clustering algorithm; and generating a high-resolution video frame based on the neural network model. The image effect is good, and the video resolution can be improved by at least four times; most importantly, the real-time requirement can be met, and the real-time requirement can be met by setting a model library, predicting a model used by a next frame, partitioning and matching strategies and the like through the cloud and the client respectively.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence, computer vision and computer graphics, and particularly relates to a real-time high-resolution virtual simulation cloud push streaming method and system. BACKGROUND

[0002] As one of the core technologies of modern multimedia processing and transmission, cloud push streaming technology is playing an important role in many fields such as video playback, live streaming, cloud application and cloud gaming. Especially in the field of virtual simulation, the importance of this technology is more prominent. The application scenarios of virtual simulation are very wide, including but not limited to military simulation application, virtual museum tour, virtual reality education, industrial design simulation, etc. In these scenarios, users not only expect realistic visual experience, but also require picture quality to meet high-resolution and high-definition standards to ensure the realism and immersion of simulation effect.

[0003] However, under the existing technical framework, there are many challenges in realizing real-time transmission of high-quality video images. Traditional methods usually fine-tune the neural network model for each video segment independently, and then send the fine-tuned model together with the low-resolution video to the client, and the client executes the high-resolution neural network model to generate high-resolution video frames. Although this method can improve the quality of video images to a certain extent, it has very high requirements for real-time performance, and needs to complete real-time fine-tuning of the model within tens of milliseconds of delay. In addition, since each user needs to be fine-tuned individually, it is necessary to allocate cloud GPUs for each user, which not only increases the computational overhead and cost, but also limits the number of simultaneous online users that the system can support, making it difficult to meet the real-time requirements of large-scale virtual simulation. SUMMARY

[0004] To solve the problems in the prior art, the present application provides a real-time high-resolution virtual simulation cloud push streaming method and system, which has good picture effect and can improve the video resolution by at least 4 times. Most importantly, it can meet the real-time requirements by setting up a model library on the cloud and the client, predicting the model used for the next frame, and using block matching strategies to meet real-time performance.

[0005] To achieve the above-mentioned purposes, the present application provides the following solutions:

[0006] A real-time high-resolution virtual simulation cloud push streaming method, the method comprising:

[0007] obtaining key frames of original rendered frame images or videos;

[0008] performing block processing and screening on the input key frames or rendered frame images;

[0009] extracting feature embeddings of each block after screening using a pre-trained ResNet18 model;

[0010] obtaining a matched neural network model for each block based on the feature embeddings of each block using a clustering algorithm;

[0011] generating high-resolution video frames based on the neural network model.

[0012] Preferably, the method of dividing the input key frame or rendered frame image into blocks comprises:

[0013] identifying key regions using edge detection and texture analysis, i.e., content-based division;

[0014] using overlapping division within the key regions to ensure continuity of information, i.e., overlapping division; wherein the method of overlapping division comprises: determining block size and overlap ratio; calculating block index; extracting each block according to the calculated index;

[0015] dynamically adjusting block size according to the complexity of the region to optimize the use of computing resources, i.e., adaptive division.

[0016] Preferably, the method of adaptive division comprises:

[0017] focusing on texture regions using spatial attention;

[0018] suppressing transient noise using temporal attention;

[0019] introducing an optical flow method to calculate temporal attention, capturing the motion correlation between video frames, and obtaining a spatio-temporal attention score;

[0020] dynamically dividing block size according to the spatio-temporal attention score;

[0021] extracting each block according to the adjusted block size;

[0022] wherein the method of focusing on texture regions using spatial attention comprises:

[0023]

[0024] wherein Q(x,y) and K(x,y) represent the query vector and key vector of pixel position (x,y) respectively, d is the feature dimension of the query and key vector, Softmax is a normalization operation that maps the attention score to the interval [0,1], reflecting the importance of the pixel, Att s (x,y) value indicates that the position contains rich texture or edge information;

[0025] The method of suppressing transient noise using temporal attention comprises:

[0026] Att t(x, y) = a t • Att t-1 (x, y) + (1 - a t ) · ||OpticalFlow(F t , F t-1 ) ||2

[0027] wherein OpticalFlow(F t , F t-1 ) represents a running vector field of adjacent frames F t and F t-1 calculated by optical flow algorithm, ||·||2 is L2 norm for quantifying motion intensity, a t is a decay factor for balancing the weight of historical attention and current motion intensity, and suppressing transient noise;

[0028] The method for introducing optical flow algorithm to calculate temporal attention, capturing motion correlation between video frames, and obtaining spatio-temporal attention score includes:

[0029] Att(x, y) = Att s (x, y) · Att t (x, y)

[0030] wherein Att(x, y) satisfies both high spatial complexity, i.e. texture complexity, and high temporal continuity, i.e. sustained motion, and is assigned the highest attention score;

[0031] The method for dynamically dividing block size according to spatio-temporal attention score includes:

[0032]

[0033] wherein τ1 and τ2 are attention threshold values for controlling the switching boundary of block granularity.

[0034] Preferably, the input of the neural network model is the feature vector of the current video frame, including but not limited to the image features of the frame, edge detection score, and category of the video segment; and the output is the index or identification of the neural network model required for predicting the next video frame;

[0035] Each node in the figure represents a neural network model; the features of the node include the parameter summary of the model, the feature representation learned in the training process, and the performance indicators of the model on historical video segments;

[0036] The edge connects two neural network model nodes, representing the similarity or transition probability between the two neural network model nodes; the weight of the edge is determined by calculating the similarity in the feature space or the transition frequency in the historical usage data of the two models;

[0037] The graph structure needs to be dynamically updated as new models are added or old models are eliminated; the new model is added to the graph as a new node, and an edge is connected according to the similarity or transition probability of the new model and the existing model; the eliminated model removes the corresponding node and related edges from the graph;

[0038] By setting a threshold for similarity or transition probability, only edges with significant relationships that meet the preset requirements are retained.

[0039] The application also provides a real-time high-resolution virtual simulation cloud push system, which is used to implement any of the methods described above, and the system comprises an acquisition module, a blocking module, an extraction module, a clustering module, and a generation module.

[0040] The acquisition module is configured to acquire key frames of original rendering frame images or videos.

[0041] The blocking module is configured to perform blocking processing and screening on the input key frames or rendering frame images.

[0042] The extraction module is configured to extract feature embeddings of each block after screening using a pre-trained ResNet18 model.

[0043] The clustering module is configured to obtain a matching neural network model for each block based on the feature embeddings of each block using a clustering algorithm.

[0044] The generation module is configured to generate high-resolution video frames based on the neural network model.

[0045] Preferably, the blocking module comprises a content blocking unit, an overlapping blocking unit, and an adaptive blocking unit.

[0046] The content blocking unit is configured to identify key regions, i.e., content-based blocking, using edge detection and texture analysis.

[0047] The overlapping blocking unit is configured to use overlapping blocking within the key regions to ensure the continuity of information, i.e., overlapping blocking; the method of overlapping blocking comprises determining block size and overlapping ratio, calculating block index, and extracting each block according to the calculated index.

[0048] The adaptive blocking unit is configured to dynamically adjust the block size according to the complexity of the region to optimize the utilization of computing resources, i.e., adaptive blocking.

[0049] Preferably, the adaptive blocking unit comprises a focusing subunit, an inhibition subunit, a capture subunit, a division subunit, and an adjustment subunit.

[0050] The focusing subunit is configured to focus on texture regions using spatial attention.

[0051] The inhibition subunit is configured to use temporal attention to inhibit transient noise.

[0052] The capture subunit is configured to introduce optical flow to calculate temporal attention, capture motion correlation between video frames, and obtain a spatio-temporal attention score.

[0053] The division subunit is configured to dynamically divide the block size according to the spatio-temporal attention score.

[0054] The adjustment subunit is configured to extract each block according to the adjusted block size.

[0055] The process of focusing on the texture region using spatial attention includes:

[0056]

[0057] Wherein, Q(x,y) and K(x,y) represent the query vector and the key vector of the pixel position (x,y) respectively, d is the feature dimension of the query and key vector, Softmax is a normalization operation, which maps the attention score to the interval [0,1], reflecting the importance of the pixel, Att s (x,y) value indicates that the position contains rich texture or edge information.

[0058] The process of using temporal attention to inhibit transient noise includes:

[0059] Att t (x,y) = α t · Att t-1 (x,y) + (1-α t )·‖OpticalFlow(F t ,F t-1 )‖2

[0060] Wherein, OpticalFlow(F t ,F t-1 ) represents the running vector field of adjacent frames F t and F t-1 calculated by the optical flow algorithm, ‖·‖2 is the L2 norm, which is used to quantify the motion intensity, α t is a decay factor, which is used to balance the weight of historical attention and current motion intensity, and inhibit transient noise.

[0061] The process of introducing optical flow to calculate temporal attention, capturing motion correlation between video frames, and obtaining a spatio-temporal attention score includes:

[0062] Att(x,y) = Att s (x,y)·Att t (x,y)

[0063] Among them, Att(x, y) satisfies the region of high spatial complexity, that is, texture complexity and high time continuity, that is, sustained motion, and the highest attention score is allocated;

[0064] The process of dynamically dividing the block size according to the spatio-temporal attention score includes:

[0065]

[0066] Among them, τ1 and τ2 are attention thresholds for controlling the switching boundary of the block granularity.

[0067] Preferably, the input of the neural network model is the feature vector of the current video frame, including but not limited to the image features of the frame, the edge detection score, and the category of the video segment; and the output is the index or identification of the neural network model required for predicting the next video frame.

[0068] Each node in the figure represents a neural network model; the features of the node include the parameter summary of the model, the feature representation learned in the training process, and the performance indicators of the model on historical video segments;

[0069] The edge connects two neural network model nodes, representing the similarity or transition probability between the two neural network model nodes; the weight of the edge is determined by calculating the similarity of the two models in the feature space or the transition frequency in the historical use data;

[0070] As new models are added or old models are eliminated, the graph structure needs to be dynamically updated; the new model is added to the graph as a new node, and the edge is connected according to the similarity or transition probability between the new model and the existing model; the eliminated model is removed from the graph corresponding to the node and the related edge;

[0071] By setting the threshold of similarity or transition probability, only the edges that meet the preset requirements of significant relationships are retained.

[0072] The application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of any one of the embodiments when executing the program.

[0073] The application also provides a computer readable storage medium, which stores a computer program, and when the computer program is executed, the method of any one of the embodiments is implemented.

[0074] Compared with the prior art, the application has the following beneficial effects:

[0075] The present application proposes an innovative technical solution aimed at reducing redundant video segments in terms of time and space, and particularly fine-tuning key frames, and reusing the fine-tuned neural network model. In this way, not only can the overhead and cost required for fine-tuning be significantly reduced, but also the demand for high frame rate, high resolution real-time rendering on mobile devices can be better met. Specifically, the method first identifies key frames that contain important information, which are often the key factors that determine the quality of video content and user experience. Then, only the key frames are fine-tuned for the neural network model, while other non-key frames can be processed by interpolation or reuse of existing models, thereby greatly reducing the amount of data to be processed and computing resources. The present solution utilizes advanced compression algorithms and optimization strategies (H.265) to enable smooth transmission of high-quality video even under limited bandwidth conditions. At the same time, through careful design and optimization of the neural network architecture, the model complexity can be reduced while maintaining the quality of the output video, and the inference speed can be improved. The benefits of this are obvious: on the one hand, it reduces the dependence on cloud resources and reduces hardware costs; on the other hand, it also improves the response speed and service capacity of the system, so that more users can enjoy high-quality virtual simulation experience at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0076] In order to more clearly illustrate the technical solutions of the present application, the following briefly introduces the drawings needed to be used in the embodiments of the present application. Obviously, the drawings described in the following embodiments are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0077] Figure 1 A real-time high-resolution virtual simulation cloud push flow method process schematic diagram for an embodiment of the present application;

[0078] Figure 2 A structure schematic diagram of an electronic device for an embodiment of the present application.

[0079] 1010, processor; 1020, memory; 1030, input / output interface; 1040, communication interface; 1050, bus. DETAILED DESCRIPTION

[0080] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0081] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0082] Embodiment one

[0083] The present application provides a real-time high-resolution virtual simulation cloud push stream method, the method comprising:

[0084] Obtaining the key frame of the original rendering frame image or video;

[0085] The input key frame or rendering frame image is processed and screened by block;

[0086] Using a pre-trained ResNet18 model to extract the feature embedding of each block after screening;

[0087] Based on the feature embedding of each block, a matching neural network model is obtained for each block using a clustering algorithm;

[0088] Based on the neural network model, a high-resolution video frame is generated.

[0089] In this embodiment, as shown in Figure 1 The method architecture is divided into two parts: "cloud" and "client".

[0090] 1, "Cloud" is used to generate a neural network model used by "client", and to predict the index of the next frame model, to encode the low-resolution video using H.265, and then to transmit the low-resolution video, the generated "high-resolution model", and the index of the next frame model or the predicted model of the next frame to the "client" through WebRTC.

[0091] 2, "Client" inputs the transmitted "low-resolution frame video" into "high-resolution model" to generate "high-resolution frame video", and updates "client model library".

[0092] 3, "Cloud" and "client" are connected through WebRTC, and low-resolution video is encoded through H.265.

[0093] In this embodiment, the VGG convolutional neural network model:

[0094] 1, "VGG convolutional neural network model" is a pre-trained model, the purpose is to automatically screen the key frame of the original "rendering frame image" through the difference of adjacent features, so as to reduce the amount of repeated and redundant data input.

[0095] 2. A skip connection is added to the "VGG Convolutional Neural Network Model". For some "rendered frame images" that change more frequently, they no longer pass through the "VGG Convolutional Neural Network Model" because most of the rendered frames at this time are key frames.

[0096] 3. For real-time virtual simulation, the input of the "VGG Convolutional Neural Network Model" is "rendered frame image"; for non-real-time applications, the input can be video.

[0097] 4. Unsupervised loss based on feature difference: No manual labeling is required, and the loss function is to maximize the feature difference between keyframes and non-keyframes.

[0098] (1) Inter-frame similarity comparison loss: Make the feature differences between key frames as large as possible, and the feature differences between non-key frames as small as possible.

[0099]

[0100] Among them, f(F i ) is the feature of the i-th frame extracted by VGG16, K is the set of keyframes, and λ1 is the weight coefficient. The first term of the formula constrains the smoothness of features between adjacent frames, and the second term forces the keyframe to be significantly different from the features of subsequent frames. N is the total number of frames in the video.

[0101] (2) Clustering loss: cluster keyframe features into different category centers, and non-keyframes are close to the existing category centers.

[0102]

[0103] Among them, {c k} is the cluster center of the key frame initialized by k-means, f(F i ) is the feature of the i-th frame extracted by VGG16, K is the set of key frames. α is the interval threshold, which ensures that non-key frames are far away from the cluster center.

[0104] 5. Loss function based on supervised learning: For labeled data with keyframe positions, a discriminative loss function can be designed.

[0105] (1) Binary cross entropy loss: The recognition of key frames is regarded as a binary classification problem (key frame / non-key frame). VGG16 should be followed by a fully connected layer and a Sigmoid output probability.

[0106]

[0107] Among them, y i ∈{0,1} is the label, p i is the predicted probability. N is the total number of video frames.

[0108] (2) Weighted loss of temporal continuity constraint: combined with temporal smoothness, avoid too dense or sparse keyframes.

[0109]

[0110] where, is the cross-entropy loss, is used to constrain the smoothness of the prediction probability of adjacent frames, β is the weight coefficient, N is the total number of video frames, p i is the probability of the i-th frame prediction, is the true probability of the i-th frame, p i+1 is the probability of the i+1-th frame prediction.

[0111] 6. Total loss function This method combines supervised and unsupervised methods to improve robustness.

[0112]

[0113] where, λ Sim , λ Clu , λ Cross and λ Tem are weight coefficients, used to balance the weights of different loss functions in training the "VGG convolutional neural network model".

[0114] In this embodiment, the block algorithm:

[0115] The input key frame or "rendered frame image" is processed by block, the same frame has the same block size, different frames need to be adaptively blocked according to the complexity of the video frame, the key area in the same frame is selected, and there is a certain overlap area between the two blocks to avoid information loss or discontinuity at the block boundary, and improve the understanding of the overall structure of the image by the model.

[0116] The video frame is blocked and a matching neural network model is found for each small block, the main purposes are: ① improve the retrieval accuracy: different small blocks may have different features and contents, by blocking the model can be matched more carefully, improve the accuracy of retrieval. ② Reduce the computational complexity: after the large frame is decomposed into small blocks, each small block can be processed in parallel, thereby reducing the computational complexity and improving the processing efficiency. ③ Adapt to complex scenes: in complex video scenes, different areas may need different neural network models for processing, block processing can better adapt to this diversity. ④ Improve the utilization rate of the model: different small blocks may match different models, so that various models in the model library can be fully utilized, and the utilization rate of the model can be improved.

[0117] Specifically, first, key regions are identified using edge detection and texture analysis (content-based partitioning), within which overlapping partitions are used to ensure continuity of information (overlapping partitions), and block size is dynamically adjusted according to the complexity of the region to optimize the use of computing resources (adaptive partitioning).

[0118] 1. Content-based partitioning: The content-based partitioning method identifies key regions in the image by analyzing the content features of the video frame, such as edges, textures, etc., and processes these regions as independent blocks, which can better preserve important information of the image and improve the accuracy of model retrieval. The specific steps are as follows:

[0119] (1) Edge detection: Use edge detection algorithms (such as Sobe, Canny, etc.) to calculate the edge intensity of each pixel.

[0120]

[0121] where, and are the horizontal and vertical gradients, respectively.

[0122] (2) Texture analysis: Use texture analysis methods (such as LBP, GLCM, etc.) to calculate the texture features T(x,y) of each region.

[0123] (3) Region segmentation: According to the edge intensity and texture features, the video frame is segmented into multiple regions. Clustering algorithms (such as K-means) or segmentation algorithms (such as Watershed) can be used to achieve this.

[0124] (4) Block extraction: Extract the segmented regions as independent blocks.

[0125] 2. Overlapping partitioning: The overlapping partitioning method allows a certain overlap between adjacent blocks during segmentation to avoid information loss or discontinuity at the block boundary, which can improve the model's understanding of the overall structure of the image. The specific steps are as follows:

[0126] (1) Determine block size and overlap ratio: Choose appropriate block size (such as 64x64 pixels) and overlap ratio (such as 25%).

[0127] (2) Block index calculation: Calculate the starting and ending pixel coordinates of each block, taking into account the overlapping region.

[0128] S x =i×(B size -Z over )

[0129] E x =S x +B size

[0130] where S x represents the pixel coordinate where the i-th block starts, E x represents the pixel coordinate where the i-th block ends, B size represents the size of the i-th block, Z over represents the size of the overlapping region. Similarly, S y and E y can be calculated.

[0131] (3) Block extraction: Extract each block according to the calculated index.

[0132] 3. Adaptive block partitioning: The adaptive block partitioning method dynamically adjusts the size and shape of the block according to the complexity or feature distribution of the video frame. For example, smaller blocks are used in complex scenes or areas with rich details, and larger blocks are used in simple scenes or uniform areas. This method can optimize the utilization of computing resources and improve the efficiency and effectiveness of model retrieval. The specific steps are as follows:

[0133] (1) Spatial attention: Spatial attention focuses on texture complex areas (such as edges, high-frequency details).

[0134]

[0135] where Q(x,y) and K(x,y) represent the query vector and key vector of pixel position (x,y) respectively, extracted by lightweight Vision Transformer (ViT-Tiny). d is the feature dimension of the query and key vectors (e.g. 256 dimensions), used to scale the dot product result to prevent gradient explosion. Softmax is a normalization operation that maps attention scores to the [0,1] interval, reflecting the importance of pixels. Att s (x,y) value indicates that this position contains rich texture or edge information and needs to be processed in detail.

[0136] (2) Temporal attention: Temporal attention suppresses transient noise (such as sudden changes in light, temporary occlusion).

[0137] Att t (x,y) = a t ·Att t-1 (x,y) + (1-a t )·‖OpticalFlow(F t ,F t-1 )‖2

[0138] where OpticalFlow(F t ,F t-1 ) represents the optical flow algorithm (such as Farneback or RAFT) calculated between adjacent frames F t and Ft-1 the running vector field. ||·||2 is the L2 norm to quantify the motion intensity. a t is the decay factor (e.g., a t = 0.9) to balance the weight of historical attention and current motion intensity, suppressing transient noise (e.g., camera shake). Higher Att t (x, y) values indicate that the region has persistent motion (e.g., a walking person) and needs to maintain temporal continuity; low values are likely to be static backgrounds or noise.

[0139] (3) Spatio-temporal attention mechanism: Introduce optical flow to calculate temporal attention, capturing the motion correlation between video frames, avoiding the block boundary discontinuity problem caused by violent motion.

[0140] Att(x, y) = Att s (x, y) · Att t (x, y)

[0141] where Att(x, y) satisfies both high spatial complexity (complex texture) and high temporal continuity (persistent motion) regions, and assigns the highest attention score.

[0142] (4) Dynamic multi-granularity partitioning: dynamically partition block size according to spatio-temporal attention scores: high attention regions (complex texture + violent motion) use small-granularity partitioning (e.g., 32x32), and low attention regions (flat background + static) use large-granularity partitioning (e.g., 128x128). Through priority scheduling, high-attention blocks are processed first, reducing end-to-end delay and improving user experience.

[0143]

[0144] where τ1 and τ2 are attention thresholds (e.g., τ2 = 0.5, τ1 = 0.8) to control the switching boundary of block granularity. 32x32 blocks are used for high-frequency detail regions such as faces and text to ensure clear edges after super-resolution. 128x128 blocks are used for flat regions such as sky and walls to reduce computational load.

[0145] (5) Block priority scheduling: high-weight blocks are prioritized for transmission to the GPU processing queue to ensure low-latency rendering of critical regions.

[0146]

[0147] where Att i is the spatio-temporal attention score of the i-th block. N is the total number of blocks in the current frame.

[0148] (6) Block extraction: extract each block according to the adjusted block size.

[0149] Technical advantages of adaptive patching:

[0150] (1) Efficiency improvement: Through dynamic patching, the calculation amount of complex regions increases limitedly, and the calculation amount of flat regions is greatly reduced, the overall throughput is improved by more than 30%.

[0151] (2) Quality assurance: The space-time attention mechanism avoids missing key areas, and the PSNR index is improved by an average of 1.5dB.

[0152] (3) Real-time optimization: Priority scheduling reduces end-to-end delay, suitable for VR / AR and other scenes with strict real-time requirements.

[0153] In this embodiment, the residual network:

[0154] Use the pre-trained ResNet18 model to extract the feature embedding of each patch.

[0155] 1、Basic steps

[0156] (1) Data preparation: Collect and preprocess video patch data, including labeling similar patch pairs and triplets.

[0157] (2) Model construction: Use the pre-trained ResNet18 model as a feature extractor.

[0158] (3) Loss function definition: Select contrastive loss and triplet loss according to task requirements, and define regularization terms.

[0159] (4) Model training: Use optimization algorithms (such as Adam) to train the model, minimizing the loss function.

[0160] (5) Model evaluation: Evaluate the performance of the model on the validation set, adjust the hyperparameters to optimize the performance.

[0161] 2、Contrastive loss: Contrastive loss is used to learn feature embeddings, so that similar patches are closer in embedding space, and dissimilar patches are farther apart.

[0162]

[0163] Where f i and f j represent the embedding features of a pair of patches x i and x j , y ij is a label indicating whether the embedding features are similar (1 means similar, 0 means not similar). m is the first pre-defined range parameter, used to control the minimum distance between dissimilar pairs.

[0164] 3、Triplet loss: Triplet loss learns the embedding space by considering anchor patches, positive examples and negative examples at the same time.

[0165]

[0166] Among them, f i 、f j and f k Represents a triplet block x i 、x j and x k Embedded features, x i is the anchor block, x j is a positive block (similar to the anchor block), x k is a negative block (i.e., not similar to the anchor block). m is the first predefined range parameter.

[0167] 4. Regularization term: To prevent overfitting, a regularization term, such as L2 regularization, can be added to the loss function. The regularization term can be expressed as the Frobenius norm of the weight matrix:

[0168]

[0169] Where λ2 is the regularization coefficient and W represents the weight matrix (Note: The weight matrix exists between the neurons in each layer and is used to transmit and transform input data. Each weight matrix corresponds to a connection layer in the network, and its size is determined by the number of input and output neurons in the layer).

[0170] 5. Total loss function

[0171]

[0172] Among them, λ contrastive and λ triplet Represents the weight coefficient.

[0173] In this embodiment, the screening algorithm:

[0174] Because the number of blocks may be large, which increases the computational overhead, in order to reduce this computational overhead, it is necessary to calculate the edge score of each block through edge detection algorithms (such as Sobel, Canny, etc.) to measure the significance of the edge in the block and filter out blocks with low significance, thereby reducing the number of blocks participating in the neural network model. Specifically, for each block P i , if P i The edge score edge(P i )<∈, then no search processing is performed on this block. Only edge(P i )>∈, then the search operation is performed on this block.

[0175] In this embodiment, the clustering algorithm:

[0176] 1, input video I, "cloud model library", edge threshold ∈, similarity threshold β, voting threshold ψ, output is the final selected neural network model M final The data stored in the "cloud model library" is:

[0177] F i ←<C i,0 ,C i,1 ,…,C i,k-1 ,M i >

[0178] Where F i represents an item in the "cloud model library", corresponding to a set of cluster centers and a specific super-resolution neural network model. C i,0 ,C i,1 ,…,C i,k-1 is the cluster center obtained by k-means clustering algorithm, which represents the feature vector of the video content adapted by the neural network model, and k represents the total number of categories. M i is the actual neural network model, which is used for super-resolution processing of specific types of video content.

[0179] The neural network models stored in the "cloud model library" and the "client model library" of this method are super-resolution neural network models, which are used to expand the video resolution to 4 times or more, in order to obtain high-quality images.

[0180] 2, input the block P i that has passed the "screening algorithm" into the "residual network", and output the embedding feature f i

[0181] 3, for each neural network model M j in the "cloud model library" (if the "cloud model library" is empty, update this neural network model and cluster center to the "cloud model library"), calculate the cosine similarity between f i and the cluster center C j of M j , that is:

[0182]

[0183] Where j ∈ (0, 1, …, R-1), S c represents the cosine similarity, f i is the corresponding feature vector of P i , and C j is the cluster center of the neural network model M j .

[0184] 4, if the similarity Then M j Record it as a candidate neural network model and increase its vote count.

[0185] 5. Because each block may hit a different neural network model, a voting mechanism is used to select the neural network model selected by the majority of blocks, and the current frame is forced to use the same neural network model, thereby avoiding the high cost of switching neural network models on the client.

[0186] 6. If the maximum number of votes is less than ψ×N total , then the new neural network model fine-tuning is triggered and the "cloud model library" is updated. Otherwise, the neural network model with the most votes is selected as the final model M final .

[0187] In this embodiment, the graph neural network model:

[0188] Since the client's download of the neural network model will compete with the video stream for network bandwidth, the graph neural network reduces the impact of model download on video stream latency by predicting the neural network model that is most likely to be used and downloading it to the client cache in advance.

[0189] 1. Input and output:

[0190] (1) Input: The feature vector of the current video frame, including but not limited to the image features of the frame, edge detection scores, categories of video clips, etc. These features can be obtained through a pre-trained feature extractor (such as ResNet18).

[0191] (2) Output: The index or identifier of the neural network model most likely to be needed to predict the next video frame.

[0192] 2. Graph Structure

[0193] (1) Node: Each node in the graph represents a neural network model. The features of the node can include a summary of the model’s parameters, the feature representation learned during training, and the performance metrics of the model on historical video clips.

[0194] (2) Edges: Edges connect two neural network model nodes and represent the similarity or transition probability between them. The weight of an edge can be determined by calculating the similarity between the two models in the feature space or the transition frequency in historical usage data.

[0195] (3) Dynamic graph construction: As new models are added or old models are eliminated, the graph structure needs to be dynamically updated. New models are added to the graph as new nodes, and edge connections are established based on their similarity or transition probability with existing models; eliminated models have their corresponding nodes and related edges removed from the graph.

[0196] (4) Graph sparsity: In practical applications, the number of models may be large, resulting in a very large and complex graph. To improve computational efficiency and model interpretability, the sparsity of the graph needs to be considered. By setting a threshold for similarity or transition probability, only edges with significant relationships are retained to avoid overly dense graphs.

[0197] 3. Edge weight

[0198] (1) Model similarity: Use cosine similarity to calculate the similarity between two model feature vectors.

[0199]

[0200] Among them, C i and C j Represents model M i and Model M j The cluster center obtained by the k-means clustering algorithm, ‖C i ‖ and ‖C j ‖ respectively represent the cluster center C i and C j The mold length.

[0201] (2) Transition probability: The transition probability p is calculated based on the frequency of model i being used followed by model j in historical data. ij

[0202]

[0203] Among them, N i Represents model M i The number of times it is used, of which N ij Next, we used Model M j .

[0204] (3) Variable weight calculation: Where λ1 and λ2 are the weight coefficients of model similarity and transition probability respectively.

[0205] 4. Node update: The features of each node can be updated by aggregating the features of its neighboring nodes. The formula is:

[0206]

[0207] in, Represents the set of neighbor nodes of node i. ij is the normalization coefficient, It is The weight matrix of the layer is represented by w ij Composition. σ is the activation function.

[0208] 5. Loss function: The cross entropy loss between the prediction model and the actual demand, while adding a regularization term to prevent overfitting.

[0209]

[0210] Among them, λ reg is the regularization coefficient, ‖W‖ 2 is the Frobenius norm of the weight matrix. The predicted probability distribution is y pred , the true label is y true .

[0211] In this embodiment, the model library update strategy is:

[0212] Cache Management: A model cache queue, the "Client Model Library," is maintained on the client side. Least Recently Used (LRU) is used to manage the cache, ensuring that the most frequently used neural network models are always available. If a pre-fetched model already exists in the cache, it does not need to be downloaded again, thus reducing network bandwidth usage.

[0213] 2. Bandwidth allocation: Dynamically allocate bandwidth for downloading neural network models based on the difference in bit rate before and after video compression. While ensuring video stream quality, select as many neural network models as possible from the "Client Model Library."

[0214] The specific process of dynamic bandwidth allocation is as follows:

[0215] Not all models are worth downloading, especially for models already cached on the client. Therefore, the model's value needs to be evaluated based on the GNN's predicted probability of use and the model's inherent image quality improvement capabilities. The compression gain factor is used to determine the compression ratio. When the video stream compression ratio is high (such as in static scenes), more bandwidth can be allocated to model downloads.

[0216] 1. Core parameter definition and formula calculation

[0217] (1) Compression gain factor α t

[0218]

[0219] in, Indicates the bit rate of the original video stream of the tth frame, Indicates the actual bit rate of the t-th frame after H.265 compression. t →0 means no compression gain, α t →1 indicates extremely high compression efficiency.

[0220] (2) Model value score V M :Comprehensive consideration of model Mi probability of next frame being used, model M i PSNR improvement value of current client content, model transmission overhead and client computing load, final model value score V M .

[0221]

[0222] where P GNN (M i ) represents the usage probability, which is predicted by a graph neural network model (GNN) for model M i probability of next frame being used. is a transmission penalty term, which is used to penalize large models and high latency, λ represents a latency sensitive coefficient (increase the penalty when the network is poor). CLF(M i ) is a computing load term, which prevents low-end devices from being overloaded. ΔPSNR(M i ) is a picture quality gain term, which is the PSNR improvement value of the current client content by model M i .

[0223]

[0224] where N is the number of frame data groups, k is the index of frame data, F k is the historical key frame of the kth group, is the low resolution frame corresponding to the historical key frame F k of the kth group, M i is an image processing model, and PSNR(X) is a function for calculating the peak signal-to-noise ratio of image X.

[0225]

[0226] where d(M i ) represents the model download transmission delay, Size(M i ) represents the bandwidth size required by M i , B historical represents historical bandwidth information, where the sliding window W = 5 frames, represents the model download bandwidth at time t. RTT smoth is the smoothed round-trip delay ( available RTT smoth = 0.8·RTT history + 0.2·RTT current , i.e., the historical average smoothing value is weighted with the current value), which represents the additional delay in the network communication process.

[0227]

[0228] Among them, CLF(M i ) is the computational load factor, which usually ranges from [0,1] and is used to reflect the relationship between the model's occupancy of GPU computing resources and the GPU's maximum computing power. i ) represents the model M i The number of floating-point operations required, which measures the number of basic computational operations required by the model during execution. GPU max It is the maximum floating-point computing capability of the GPU, indicating the maximum number of floating-point operations that the GPU can process per unit time under ideal conditions. It is an important indicator for measuring GPU computing performance.

[0229] (3) Bandwidth allocation weight β t

[0230]

[0231] Where τ is the value threshold (default τ = 0.2), used to filter low-value models. U is the client missing and V M >τ model set. eligible Is the client missing, V M >τ and Size(M i ) max Model collection (S max Represents 80% of the remaining storage space of the current device, which is used to limit the size of the model to ensure that the model does not occupy too much storage space, thereby affecting the normal operation of the system. The denominator is normalized to prevent over-allocation of bandwidth when τ is high. t By α t With V M Multiplying together ensures that bandwidth is allocated only when compression efficiency is high and the model value is large.

[0232] Frame-level jumps cause bandwidth oscillations, so β t Perform smoothing filtering:

[0233] β t ←μβ t-1 +(1-μ)β t

[0234] Where μ is the smoothing factor (default μ=0.7).

[0235] 2. Dynamic bandwidth allocation strategy

[0236] (1) Step 1: Real-time bandwidth pool construction

[0237]

[0238] Among them, B total ​denotes the total available bandwidth of the client, denotes the re-allocatable bandwidth pool released by compression gain. M t is the compression gain factor.

[0239] (2) Step 2: Model download priority queue

[0240] 1) Calculate the candidate model value score V M

[0241] 2) Generate the download queue Q, Arrange the queue in descending order.

[0242] (3) Step 3: Bandwidth allocation and scheduling

[0243]

[0244] where, denotes the video stream guarantee bandwidth at time t, B total denotes the total available bandwidth of the client, denotes the re-allocatable bandwidth pool released by compression gain. M i is the image processing model, Size(M i ) denotes M i is the bandwidth size required by the model. Q active is the set of active models currently being downloaded. To prevent the possibility of stalls in extreme scenarios, define the minimum guarantee bandwidth for video streaming, where γ is the basic guarantee ratio (default γ = 0.3), δ is the burst redundancy coefficient (default δ = 1.2), is the real-time video stream demand (obtained from the encoder), B min is the minimum guarantee value of the video bandwidth (e.g., B min = 1 Mbps).

[0245]

[0246] where, denotes the model download bandwidth at time t, β t is the bandwidth allocation weight factor, B total denotes the total available bandwidth of the client. β t × B total is the theoretical value of the model download bandwidth . is a constraint condition that ensures that the model download bandwidth cannot exceed the bandwidth saved by compression, and cannot exceed the bandwidth demand rate of model download, Δt is the maximum allowed download time (default 2 seconds).

[0247] (4) Step 4: Adaptive function interruption mechanism: if the video stream burst code rate rises, i.e.t ↓ If V M <0.3, pause the download queue Q immediately M <0.5, reduce the bandwidth to 10%, and for V M ≥0.5, keep the bandwidth, record the interrupted position, continue downloading after the network is restored (the download can be restored in the order of the download progress), and release the bandwidth to the video stream basic bandwidth, that is

[0248] 3. Advantages compared with existing methods

[0249] (1) The allocation ratio is determined by the compression efficiency α t and the model value V M in real time, which is more flexible than fixed ratio allocation.

[0250] (2) The video content features (compression gain) and model utility (model utility) are fused, avoiding considering only the network congestion state, making the ratio allocation more reasonable.

[0251] 3. Maintenance of "cloud model library" and "client model library"

[0252] (1) Local model fine-tuning and privacy protection: input cloud pre-trained model parameters and client local video stream data D k , output local fine-tuned parameter increment ΔW k , and upload to the cloud after privacy protection.

[0253] 1) Local fine-tuning: the client local video stream data D k lightly trains the cloud model (such as a small number of epoch SGD optimization), and obtains the parameter increment.

[0254]

[0255] where ΔW k is the local fine-tuned parameter increment (reflecting the difference between the local data distribution and the global model, used to optimize the global model), is the global model parameter after the tthround of federal aggregation (the benchmark model in the cloud model library, which is downloaded by all clients), is the tthlocal model parameter.

[0256] 2) Differential privacy protection: add Gaussian noise to the parameter increment to prevent original data information leakage.

[0257]

[0258] where, is a Gaussian noise with mean 0 and variance σ 2 , the noise intensity σ is used to control the degree of privacy protection (the larger σ, the stronger privacy, but the model performance may decrease), σ needs to be adjusted by the privacy budget.

[0259] (2) Cloud Federated Aggregation: Input the privacy protection parameter increment of multiple clients, output the updated global model parameters

[0260]

[0261] where n k is the local data volume (e.g. frame number) of client k. n is the total data volume of all participating clients, is the data volume weight of client k, the larger the data volume, the greater the influence on the global model, reflecting the fairness of federated learning. K is the number of clients participating in federated learning.

[0262] (3) Model performance evaluation and monitoring

[0263] 1) Regular evaluation: Set a fixed period (such as every week or every month) to comprehensively evaluate the performance of all neural network models in the model library. Evaluation indicators include but are not limited to image quality evaluation indicators such as PSNR, SSIM, as well as the adaptability and robustness of the model in different scenarios.

[0264] 2) Real-time monitoring: In the actual cloud application video transmission process, continuously monitor the usage and performance of each neural network model. Record the frequency of model invocation, feedback information of enhancement effect, etc. to timely discover models with performance degradation or not suitable for the current scenario.

[0265] (4) Model elimination mechanism

[0266] 1) Performance-based elimination: According to the results of regular evaluation and real-time monitoring, set a performance threshold. For models whose performance indicators are consistently below the threshold, or models that perform poorly in multiple scenarios (such as weak generalization ability, perform well in a specific scenario but poorly in most other scenarios (such as PSNR, SSIM performance indicators below the preset threshold)), eliminate them from the model library.

[0267] 2) Elimination based on usage frequency: Consider the usage frequency of the model, eliminate models that have not been invoked for a long time or have very low usage frequency. These models may no longer be suitable for the current cloud application video content or have been replaced by better models.

[0268] (5) Model updating and optimization

[0269] 1) Regular updates: Based on the performance evaluation results of the models and the changing trends of cloud application video content, the models in the model library are regularly updated. This includes local fine-tuning, retraining of existing models (the training method is the same as the method of this patent, which is to retrain in the cloud and then update to the local. The reason for retraining may be the enhancement of cloud scene quality or the increase of cloud scene, so the model needs to be retrained to adapt to the new changes) or the introduction of new more advanced neural network models (as research and development continue, there will be more advanced network models, the specific content depends on future research and development direction and progress (not yet present), such as real-time performance direction, or high-quality picture effect direction, etc.).

[0270] 2) Dynamic optimization: During the operation of the model library, the parameters and structure of the models are dynamically optimized according to real-time feedback and new data. For example, when it is found that some models perform poorly in certain scenarios, they are adjusted and optimized in a targeted manner.

[0271] (6) Model similarity analysis and integration

[0272] 1) Similarity analysis: Similarity analysis is performed on the models in the model library to find models with similar functions and performance. Similarity can be judged by comparing the structure, parameters, output results, etc. of the models. The specific process of similarity analysis is as follows: first, construct a comparable feature vector for each model (such as structure features, parameter features, function features, and output features), then calculate the similarity (such as structure similarity, cosine similarity of parameter weight matrix, matching degree of cluster center, output similarity, etc.), weight and fuse the obtained similarity scores in each dimension, cluster grouping, scene test (key scene test, user feedback evaluation, resource consumption comparison, etc.) for models in the same group to further determine the replaceability, and finally retain the optimal model and eliminate redundant models according to the analysis results.

[0273] 2) Model integration: For models with high similarity, consider integrating them (integration means either selecting the best model or fusing multiple models into a general model (such as weighted average method, stacking method, knowledge distillation, Bagging, Boosting, feature fusion, etc.)). Select the model with the best performance and stability as the representative, or fuse multiple similar models to form a more general and efficient model to reduce the redundancy of the model library.

[0274] (7) Expansion and update of the model library

[0275] 1) Introducing new models: Introducing new, potentially (such as model structure innovation, such as loss function innovation, so that new network models have been developed, which can be measured by PSNR, SSIM and other indicators to determine whether the network model has potential, or from the performance point of view) neural network model into the model library. This can increase the diversity of the model library and improve its adaptability to different cloud application video scenarios.

[0276] 2) Adapt to new scenarios: With the development of cloud applications and the continuous change of video content, timely adjust (such as emptying the local model library and re-downloading, fine-tuning the local model, updating the cloud retrained model, and timely cleaning up the unsuitable model and updating the model suitable for the current cloud application) The composition of the model library can adapt to new application types, quality requirements and network environments, etc.

[0277] Embodiment two

[0278] The present application provides a real-time high-resolution virtual simulation cloud push streaming system, which is used to implement the method of embodiment one, and the system comprises an acquisition module, a blocking module, an extraction module, a clustering module and a generation module.

[0279] The acquisition module is used to acquire key frames of original rendering frame images or videos.

[0280] The blocking module is used to block and filter the input key frames or rendering frame images.

[0281] The extraction module is used to extract the feature embedding of each block after filtering using a pre-trained ResNet18 model.

[0282] The clustering module is used to obtain a matching neural network model for each block based on the feature embedding of each block using a clustering algorithm.

[0283] The generation module is used to generate high-resolution video frames based on the neural network model.

[0284] In this embodiment, the blocking module comprises a content blocking unit, an overlapping blocking unit and an adaptive blocking unit.

[0285] The content blocking unit is used to identify key areas based on content blocking using edge detection and texture analysis.

[0286] The overlapping blocking unit is used to use overlapping blocking inside the key area to ensure the continuity of the information, that is, overlapping blocking; wherein the method of overlapping blocking comprises: determining the block size and the overlapping ratio; calculating the block index; extracting each block according to the calculated index.

[0287] An adaptive partitioning unit is used to dynamically adjust the block size according to the complexity of the region, optimize the utilization of computing resources, that is, adaptive partitioning.

[0288] In the embodiment, the adaptive partitioning unit comprises a focusing subunit, an inhibition subunit, a capturing subunit, a division subunit, and an adjustment subunit.

[0289] The focusing subunit is used to focus on the texture region using spatial attention.

[0290] The inhibition subunit is used to suppress transient noise using temporal attention.

[0291] The capturing subunit is used to introduce an optical flow method to calculate temporal attention, capture the motion correlation between video frames, and obtain a spatio-temporal attention score.

[0292] The division subunit is used to dynamically divide the block size according to the spatio-temporal attention score.

[0293] The adjustment subunit is used to extract each block according to the adjusted block size.

[0294] The process of focusing on the texture region using spatial attention comprises:

[0295]

[0296] wherein Q(x, y) and K(x, y) represent the query vector and the key vector of the pixel position (x, y) respectively, d is the feature dimension of the query and key vectors, Softmax is a normalization operation, which maps the attention score to the interval [0, 1] to reflect the importance of the pixel, and Att s (x, y) indicates that the position contains rich texture or edge information.

[0297] The process of suppressing transient noise using temporal attention comprises:

[0298] Att t (x, y) = α t · Att t-1 (x, y) + (1 - α t )·‖OpticalFlow(F t , F t-1 )‖2

[0299] wherein OpticalFlow(F t , F t-1 ) represents the running vector field of the adjacent frames F t and F t-1 calculated by the optical flow algorithm, ‖·‖2 is the L2 norm for quantifying the motion intensity, and α tis a decay factor to balance the weight of historical attention and current motion intensity, to suppress transient noise;

[0300] The process of introducing optical flow to calculate temporal attention, capturing the motion correlation between video frames, and obtaining the spatio-temporal attention score includes:

[0301] Att(x,y)=Att s (x,y)·Att t (x,y)

[0302] Wherein, Att(x,y) satisfies the region of high spatial complexity, i.e. texture complexity and high temporal continuity, i.e. sustained motion, at the same time, and the highest attention score is allocated;

[0303] The process of dynamically dividing the block size according to the spatio-temporal attention score includes:

[0304]

[0305] Wherein, τ1 and τ2 are attention threshold values, used to control the switching boundary of block granularity.

[0306] In this embodiment, the input of the neural network model is the feature vector of the current video frame, including but not limited to the image features of the frame, the edge detection score, and the category of the video segment; the output is the predicted next video frame required by the neural network model index or identification;

[0307] Each node in the figure represents a neural network model; the features of the node include the parameter summary of the model, the feature representation learned in the training process, and the performance indicators of the model on historical video segments;

[0308] The edge connects two neural network model nodes, representing the similarity or transition probability between the two neural network model nodes; the weight of the edge is determined by calculating the similarity of the two models in the feature space or the transition frequency in the historical use data;

[0309] As new models are added or old models are eliminated, the graph structure needs to be dynamically updated; the new model is added to the graph as a new node, and the edge is connected according to the similarity or transition probability between the new model and the existing model; the eliminated model is removed from the graph node and the related edge;

[0310] By setting the threshold of similarity or transition probability, only the edges that meet the preset requirements of significant relationship are retained.

[0311] Embodiment three

[0312] Corresponding to the method of any of the above embodiments based on the same inventive concept, the disclosure also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the method of any of the above embodiments.

[0313] Figure 2 A more specific hardware structure of an electronic device is shown in this embodiment. The device can include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 for internal communication within the device.

[0314] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present disclosure.

[0315] The memory 1020 can be implemented by a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present disclosure are implemented by software or firmware, the related program codes are stored in the memory 1020 and executed by the processor 1010.

[0316] The input / output interface 1030 is used to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0317] The communication interface 1040 is configured to connect a communication module (not shown in the figure) to realize the communication interaction between the device and other devices. The communication module can realize communication through wired mode (such as USB (Universal Serial Bus, Universal Serial Bus), network cable, etc.), or can realize communication through wireless mode (such as mobile network, WIFI (Wireless Fidelity, Wireless Network Communication Technology), Bluetooth, etc.).

[0318] The bus 1050 includes a path for transmitting information between various components (such as the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040) of the device.

[0319] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only contain the components necessary for the implementation of the embodiments of the present specification, and does not have to contain all the components shown in the figure.

[0320] The system of the above embodiment is used to implement the three-dimensional structure recovery method of the high-quality urban renewal landscape building in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0321] Embodiment Four

[0322] Based on the same inventive concept, the disclosure also provides a non-transitory computer readable storage medium, which stores computer instructions for causing the computer to execute the method of any of the above embodiments.

[0323] The computer readable media of the embodiments can include permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible to a computing device.

[0324] The storage medium of the above embodiments stores computer instructions for causing the computer to execute the method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here.

[0325] Those skilled in the art should understand that the discussion of any of the above embodiments is only exemplary, and is not intended to imply that the scope of the present disclosure (including claims) is limited to these examples; the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of brevity.

[0326] In addition, in order to simplify the description and discussion, and so as not to make the embodiments of the present disclosure difficult to understand, the well-known power / ground connections of integrated circuit (IC) chips and other components can or can not be shown in the provided drawings. In addition, the devices can be shown in the form of block diagrams in order to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented to implement the embodiments of the present disclosure (i.e. these details should be fully within the understanding of those skilled in the art). Where specific details (e.g. circuits) are set forth in order to describe an exemplary embodiment of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with variations on these specific details. Therefore, these descriptions should be considered illustrative rather than limiting.

[0327] Although the present disclosure has been described in connection with certain embodiments, numerous modifications and alterations mentioned above are well within the skills of those in the art. For example, other memory architectures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed.

[0328] Therefore, the units of each example described in the embodiments of the present application can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0329] The above-described embodiments are merely intended to describe the preferred modes of the present application, and are not intended to limit the scope of the present application. Various modifications and improvements made by those skilled in the art to the technical solutions of the present application without departing from the design spirit of the present application shall fall within the protection scope of the present application as defined by the claims.

Claims

1. A real-time high-resolution virtual simulation cloud streaming method, characterized in that: The method comprises: Get the original rendered frame image or key frame of the video; Divide the input key frame or rendered frame image into blocks and filter them; Use the pre-trained ResNet18 model to extract the feature embedding of each filtered block; Based on the feature embedding of each block, a clustering algorithm is used to obtain a matching neural network model for each block; Generate high-resolution video frames based on a neural network model.

2. The method according to claim 1, characterized in that Methods for processing input keyframes or rendered frame images into blocks include: Identify key areas using edge detection and texture analysis, i.e. content-based segmentation; Overlapping blocks are used within the key area to ensure the continuity of information, namely overlapping blocks. The overlapping block method includes: determining block size and overlap ratio; calculating block index; extracting each block according to the calculated index; Dynamically adjust the block size according to the complexity of the area to optimize the utilization of computing resources, which is adaptive blocking.

3. The method according to claim 2, characterized in that Adaptive blocking methods include: Use spatial attention to focus on texture areas; Using temporal attention to suppress transient noise; The optical flow method is introduced to calculate temporal attention, capture the motion correlation between video frames, and obtain spatiotemporal attention scores; Dynamically divide the block size according to the spatiotemporal attention score; Extract each block according to the adjusted block size; Among them, the methods of using spatial attention to focus on texture areas include: Among them, Q(x,y) and K(x,y) represent the query vector and key vector of the pixel position (x,y), d is the feature dimension of the query and key vectors, Softmax is a normalization operation that maps the attention score to the [0,1] interval to reflect the pixel importance, Att s The (x,y) value indicates that the location contains rich texture or edge information; Methods for suppressing transient noise using temporal attention include: Att t (x,y)=α t ·Att t-1 (x,y)+(1-α t )·||OpticalFlow(F t ,F t-1 )||2 Among them, OpticalFlow(F t ,F t-1 ) indicates that the adjacent frames F are calculated by the optical flow algorithm t and F t-1 The running vector field, ||·||2 is the L2 norm, used to quantify the motion intensity, α t is a decay factor used to balance the weight of historical attention and current motion intensity and suppress instantaneous noise; The optical flow method is introduced to calculate temporal attention and capture the motion correlation between video frames. The methods for obtaining spatiotemporal attention scores include: That(x,y)=That s (x,y)·To t (x,y) Among them, Att(x,y) satisfies both high spatial complexity (i.e., complex texture) and high temporal continuity (i.e., continuous motion) and is assigned the highest attention score. Methods for dynamically dividing block sizes based on spatiotemporal attention scores include: Among them, τ1 and τ2 are attention thresholds, which are used to control the switching boundary of the block granularity.

4. The method according to claim 1, wherein The input of the neural network model is the feature vector of the current video frame, including but not limited to the image features of the frame, edge detection score, and category of the video clip; the output is the index or identifier of the neural network model required for the next video frame to be predicted; Each node in the graph represents a neural network model; the node features include a summary of the model's parameters, the feature representations learned during training, and the model's performance metrics on historical video clips. An edge connects two neural network model nodes, indicating the similarity or transition probability between the two neural network model nodes. The weight of the edge is determined by calculating the similarity between the two models in the feature space or the transition frequency in the historical usage data. As new models are added or old models are eliminated, the graph structure needs to be dynamically updated. New models are added to the graph as new nodes, and edge connections are established based on the similarity or transition probability between the new model and the existing model. The eliminated models have their corresponding nodes and related edges removed from the graph. By setting a threshold for similarity or transition probability, only edges whose significant relationships meet the preset requirements are retained.

5. A real-time high-resolution virtual simulation cloud streaming system, the system being used to implement the method according to any one of claims 1 to 4, characterized in that: The system includes: an acquisition module, a block module, an extraction module, a clustering module and a generation module; The acquisition module is used to acquire the key frame of the original rendered frame image or video; The block module is used to process and filter the input key frame or rendering frame image into blocks; The extraction module is used to extract the feature embedding of each filtered block using a pre-trained ResNet18 model; The clustering module is used to obtain a matching neural network model for each block using a clustering algorithm based on the feature embedding of each block; The generation module is used to generate high-resolution video frames based on a neural network model.

6. The system according to claim 5, characterized in that The block module includes: a content block unit, an overlapping block unit, and an adaptive block unit; The content segmentation unit is configured to identify key areas, i.e., content-based segmentation, using edge detection and texture analysis; The overlapping block unit is used to use overlapping blocks within the key area to ensure the continuity of information, namely overlapping blocks; wherein the overlapping block method includes: determining the block size and the overlap ratio; calculating the block index; extracting each block according to the calculated index; The adaptive blocking unit is used to dynamically adjust the block size according to the complexity of the region to optimize the utilization of computing resources, namely adaptive blocking.

7. The system according to claim 6, characterized in that The adaptive block division unit includes: a focusing subunit, a suppressing subunit, a capturing subunit, a dividing subunit, and an adjusting subunit; The focusing subunit is used to focus on the texture area using spatial attention; The suppression subunit is configured to suppress transient noise using temporal attention; The capture subunit is used to introduce the optical flow method to calculate the temporal attention, capture the motion correlation between video frames, and obtain the spatiotemporal attention score; The division subunit is used to dynamically divide the block size according to the spatiotemporal attention score; The adjusting subunit is configured to extract each block according to the adjusted block size; The process of using spatial attention to focus on texture areas includes: Among them, Q(x,y) and K(x,y) represent the query vector and key vector of the pixel position (x,y), d is the feature dimension of the query and key vectors, Softmax is a normalization operation that maps the attention score to the [0,1] interval to reflect the pixel importance, Att s The (x,y) value indicates that the location contains rich texture or edge information; The process of suppressing transient noise using temporal attention includes: Att t (x,y)=α t ·Att t-1 (x,y)+(1-α t )·||OpticalFlow(F t ,F t-1 )||2 Among them, OpticalFlow(F t ,F t-1 ) indicates that the adjacent frames F are calculated by the optical flow algorithm t and F t-1 The running vector field, ||·||2 is the L2 norm, used to quantify the motion intensity, α t is a decay factor used to balance the weight of historical attention and current motion intensity and suppress instantaneous noise; The optical flow method is introduced to calculate temporal attention and capture the motion correlation between video frames. The process of obtaining spatiotemporal attention scores includes: That(x,y)=That s (x,y)·To t (x,y) Among them, Att(x,y) satisfies both high spatial complexity (i.e., complex texture) and high temporal continuity (i.e., continuous motion) and is assigned the highest attention score. The process of dynamically dividing the block size according to the spatiotemporal attention scores includes: Among them, τ1 and τ2 are attention thresholds, which are used to control the switching boundary of the block granularity.

8. The system according to claim 5, wherein: The input of the neural network model is the feature vector of the current video frame, including but not limited to the image features of the frame, edge detection score, and category of the video clip; the output is the index or identifier of the neural network model required for the next video frame to be predicted; Each node in the graph represents a neural network model; the node features include a summary of the model's parameters, the feature representations learned during training, and the model's performance metrics on historical video clips. An edge connects two neural network model nodes, indicating the similarity or transition probability between the two neural network model nodes. The weight of the edge is determined by calculating the similarity between the two models in the feature space or the transition frequency in the historical usage data. As new models are added or old models are eliminated, the graph structure needs to be dynamically updated. New models are added to the graph as new nodes, and edge connections are established based on the similarity or transition probability between the new model and the existing model. The eliminated models have their corresponding nodes and related edges removed from the graph. By setting a threshold for similarity or transition probability, only edges whose significant relationships meet the preset requirements are retained.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 4 is implemented.

Images