Self-adaptive high-definition rendering plug flow method and system

Abstract

The invention belongs to the field of artificial intelligence, computer vision and computer graphics, and discloses a self-adaptive high-definition rendering plug flow method and system, and the method comprises the steps: obtaining a rendering frame image; transmitting the rendered frame image to an explicit terminal through a network transmission technology; and the explicit terminal amplifies the resolution of each frame of the input video, namely the rendering frame image, through a deep learning model to obtain a final high-resolution frame image. According to the invention, real-time rendering output is supported; the ultra-high definition requirement is supported; the height is self-adaptive, and dynamic adjustment can be performed according to actual conditions.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence, computer vision and computer graphics, and particularly relates to a self-adaptive high-quality rendering streaming method and system. BACKGROUND

[0002] The rendering streaming technology has a wide application prospect in many fields, including but not limited to cloud rendering, extended reality (XR) head-mounted device, industrial digital twin, smart tourism, etc. These applications have high requirements for real-time performance and picture quality, especially when facing different types of terminal devices, how to ensure smooth user experience and high-quality picture effect is a problem to be solved.

[0003] Currently, the traditional rendering streaming technology faces many challenges. First, in terms of performance, existing solutions are usually difficult to meet the efficient data transmission requirements, which not only involves high-speed data exchange between memory and video memory, but also includes transcoding from raw images to a format suitable for network transmission, as well as encoding and transmission efficiency for distributing content to users through the network. In addition, due to the instability and diversity of network conditions, ensuring stable and low-latency transmission is also an important consideration factor.

[0004] Secondly, the limitations of resolution and bit rate also affect user experience. High resolution and high bit rate can provide more realistic visual effects, but at the same time, it also increases the computational burden and bandwidth consumption, which is a big challenge for resource-limited devices or network environments. While low resolution and low bit rate can reduce resource occupation, it sacrifices picture quality to some extent and cannot fully display content details.

[0005] Finally, the quality of the picture directly affects the user's immersion and interactive experience. If the rendered picture has obvious distortion, stuttering or color deviation, etc., it will greatly reduce the user's satisfaction with the service.

[0006] In order to solve the above problems, the present application proposes a self-adaptive ultra-high-quality rendering streaming solution. SUMMARY

[0007] In order to solve the problems existing in the prior art, the present application provides a self-adaptive high-quality rendering streaming method and system, which supports real-time rendering output; supports high-definition requirements; is highly adaptive and can be dynamically adjusted according to actual conditions.

[0008] To achieve the above purpose, the present application provides the following solutions:

[0009] A self-adaptive high-quality rendering streaming method, the method comprising:

[0010] acquire a rendered frame image;

[0011] transmit the rendered frame image to the explicit end through network transmission technology;

[0012] The explicit end enlarges the resolution of each frame of the input video, i.e., the rendered frame image, through a deep learning model to obtain a final high-resolution frame image.

[0013] Preferably, the method for acquiring the rendered frame image comprises:

[0014] perform color space conversion on the original rendered frame, i.e., an image in RGBA format, to obtain a transcoded rendered frame, i.e., an image in YUV format;

[0015] encode the YUV rendered frame through a video encoding standard to obtain an encoded rendered frame.

[0016] Preferably, the deep learning model is composed of a 3x3 convolution layer, two self-attention modules, a 3x3 convolution layer, and a 4-fold post-up sampling at the end;

[0017] The self-attention module is composed of two parts, a cross-attention module and a feed-forward module.

[0018] The cross-attention module: after the input image feature I passes through a "normalization layer", it enters a first convolution layer to obtain Then, it is split into two and where I1 is regarded as a local feature and is input into a first residual, and I2 is regarded as a global feature and is input into a global variance and feature multiplication operation.

[0019] After I1 extracts a local image feature through the first residual, it outputs The channel number remains unchanged, but the resolution is reduced by 8 times, and I'1 is input into a second convolution and feature addition.

[0020] I'1 is input into the second convolution to perform local detail coding and feature extraction to generate I'1', Then, it is input into the first convolution to generate enhanced local features

[0021] I2 is subjected to a "global variance" operation, i.e., where N is the total number of pixels, x i represents the RGB value of each pixel, μ represents the average value of the RGB of all pixels, is the variance of I2, and then I'2 and I'1 are added in feature, i.e., I'2 is first copied in pixels to make Then the pixel-by-pixel addition is performed, and then the added feature map is input into the second convolution to fuse the local feature and the global feature and find the relationship between the two, to generate I'2', and Finally, through an upsampling operation, I'2" is generated, and

[0022] The I'1" output by the local detail extraction branch, the I'2" output by the global feature extraction branch, and the global feature I2 are all input into the feature multiplication and addition operation, and the multiplication result Then the is pixel-by-pixel added with I'1", to obtain

[0023] Regarding the feedforward module: the feature map I 12 output by the cross attention module is input into the second convolution in the feedforward module, to obtain Then it is split into two and Then the is input into the "second residual", and the output is After that, the and are added in feature, that is, first, through an upsampling operation, the is pixel-by-pixel added, and finally the obtained result is input into the first convolution for feature fusion, and finally output.

[0024] Preferably, the loss function of the deep learning model is:

[0025]

[0026] Wherein, η is the weight of the semantic loss, is the pixel-level L1 loss and the frequency domain loss, is the multi-scale frequency domain constraint, is the semantic consistency constraint.

[0027] The application also provides an adaptive high-quality rendering streaming system, which is used to realize the method described above, and the system comprises an acquisition module, a transmission module, and an amplification module.

[0028] The acquisition module is used to acquire a rendered frame image.

[0029] The transmission module is used to transmit the rendered frame image to an explicit end through a network transmission technology.

[0030] The amplification module is used to amplify the resolution of each frame of the input video, i.e., the rendered frame image, by using a deep learning model, to obtain a final high-resolution frame image.

[0031] Preferably, the acquisition module includes: a conversion unit and an encoding unit;

[0032] The conversion unit is used to convert the color space of the original rendering frame, i.e., the RGBA format image, to obtain the transcoded rendering frame, i.e., the YUV format image.

[0033] The encoding unit is used to encode the YUV rendering frame using a video encoding standard to obtain an encoded rendering frame.

[0034] Preferably, the deep learning model consists of a 3x3 convolutional layer, two self-attention modules, another 3x3 convolutional layer, and a final upsampling by a factor of 4.

[0035] The self-attention module consists of two parts: a cross-attention module and a forward feedback module.

[0036] Regarding the cross-attention module: After the input image feature I passes through the "normalization layer", it enters the first convolutional layer to obtain... Then split it into two according to the number of channels C. and I1 is considered a local feature and is input into the first residual, while I2 is considered a global feature and is input into the global variance and feature multiplication and addition operation.

[0037] After I1 extracts local image features through the first residual, it outputs... That is, the number of channels remains the same, but the resolution is reduced by a factor of 8, and I'1 is input into the second convolution and added to the features;

[0038] I'1 is input into the second convolution to perform local detail encoding and feature extraction, generating I'1'. Then, enhanced local features are generated through the first convolution.

[0039] I2 undergoes a global variance operation, i.e. Where N is the total number of pixels, x i This represents the RGB value of each pixel, and μ represents the average RGB value of all pixels. It is the variance of I2, then the features of I'2 and I'1 are added together, that is, I'2 is first copied to pixels, so that Then, the features are added pixel by pixel, and the resulting feature map is input into the second convolution to fuse local and global features, identify the relationship between them, and generate I'2'. Finally, an upsampling operation is performed to generate I'2", and

[0040] The I'1" output by the local detail extraction branch, the I'2" output by the global feature extraction branch, and the global feature I2 are all input into a feature multiplication and addition operation, and the multiplication result is Then the is pixel-by-pixel added with the I'1", to obtain

[0041] Regarding the front feedback module: the feature map I 12 output by the cross attention module is input into a second convolution in the front feedback module, to obtain Then, according to the channel number C, the I and the I are split into two and the I are input into a second residual, to output and the I are feature-added, that is, the I is first subjected to an upsampling operation, and then pixel-by-pixel addition is performed, and finally the obtained result is input into a first convolution for feature fusion, and finally output.

[0042] Preferably, the loss function of the deep learning model is:

[0043]

[0044] wherein η is the weight of the semantic loss, is the pixel-level L1 loss and the frequency domain loss, is the multi-scale frequency domain constraint, is the semantic consistency constraint.

[0045] The application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described above when executing the program.

[0046] The application also provides a computer readable storage medium, which stores a computer program, and when the computer program is executed, the method described above is implemented.

[0047] Compared with the prior art, the application has the following beneficial effects:

[0048] The application provides a self-adaptive high-quality rendering streaming method and system, which aims to dynamically adjust the output resolution and strategy according to the performance difference between the rendering end and the display end device in different application scenarios, the dynamic change of the real-time rendering frame rate, the change of the triangular face number of the camera view pyramid, the complexity of the virtual scene material and light, and the device heating, so as to realize the high adaptability of the whole scene. The self-adaptive rendering streaming method not only solves the problems of low performance, insufficient resolution / code rate and poor picture effect in the prior art, but also realizes the dynamic balance between high performance and high-quality picture through intelligent management and personalized customization, thereby providing strong technical support for various applications. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the present application, the following briefly introduces the drawings needed to be used in the embodiments. Obviously, the drawings described in the following embodiments are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0050] Figure 1 A self-adaptive high-quality rendering streaming method flowchart for an embodiment of the present application;

[0051] Figure 2 A self-attention module schematic diagram for an embodiment of the present application;

[0052] Figure 3 An inverted residual block schematic diagram for an embodiment of the present application;

[0053] Figure 4 A sub-pixel convolution schematic diagram for an embodiment of the present application;

[0054] Figure 5 A structure schematic diagram of an electronic device for an embodiment of the present application.

[0056] 1010, processor; 1020, memory; 1030, input / output interface; 1040, communication interface; 1050, bus. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0058] In order to make the above objectives, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0059] Embodiment one

[0060] The present application provides a self-adaptive high-quality rendering streaming method, which comprises:

[0061] Obtaining a rendering frame image;

[0062] Transmitting the rendering frame image to the display end through network transmission technology;

[0063] The display end enlarges the resolution of each frame of the input video, i.e. the rendering frame image, through a deep learning model to obtain the final high-resolution frame image.

[0064] In this embodiment, as shown in Figure 1 the whole process is composed of three parts: "rendering end", "transmission end" and "display end".

[0065] 2、Firstly, "original rendering frame" is rendered in the "rendering end", which is an image in RGBA format. RGBA is a common image data format that includes four color channels: red (R), green (G), blue (A), and transparency (A).

[0066] 3、Secondly, the "original rendering frame" is converted in the "rendering end" to change the color space, converting the image in RGBA format to YUV format (i.e. "transcoded rendering frame"). YUV is a widely used video encoding format that separates color information into luminance (Y) and two chrominance (U, V) channels. This format is more suitable for video transmission and storage, helping to reduce the amount of data required during encoding and transmission. This part provides two transcoding schemes, one of which processes the transcoding process in the CPU, and the other of which processes the transcoding process in the GPU. The advantage of using GPU for transcoding is that it can take advantage of the powerful parallel processing capabilities of GPU to more effectively process large amounts of pixel data, improving the speed of the transcoding process and reducing the dependence on CPU and overall system load. In addition, the bandwidth between memory and video memory is limited, and the data volume of YUV format is smaller, which can significantly reduce the data exchange bandwidth between memory and video memory, thereby reducing the delay and bandwidth demand of data transmission. However, if the "rendering end" GPU power is insufficient or the GPU performance is poor, the transcoding process needs to be processed in the CPU.

[0067] 4、Then the YUV rendering frame is encoded by a video encoding standard (i.e. "encoded rendering frame", which does not limit specific encoding techniques, and can be H.264, H.265, AV1, etc.), in order to reduce the amount of data transmitted.

[0068] 5, "Transmission end" transmits video stream to "Display end" through network transmission technology (not limited to specific technical implementation, such as RTMP (Real Time Messaging Protocol), HLS (HTTP Live Streaming), RTSP (Real Time Streaming Protocol) and WebRTC (Web Real-Time Communication) and so on. It should be noted that we actually use WebRTC technology scheme).

[0069] 6, "Display end" enlarges the resolution of each frame of the input video by 4 times through a deep learning model (for low-end devices, this operation is not performed). In order to improve the overall performance and enable real-time operation, the model structure is simplified, a 3x3 convolutional layer, 2 self-attention modules (not detailed in the architecture diagram, discussed in detail below), then a 3x3 convolutional layer, and finally a 4 times "post-up sampling" model (the model of "post-up sampling" is not limited, detailed below), to get the final high-resolution frame image.

[0070] 7, Loss function of "Display end" neural network model:

[0071] (1) Pixel-level L1 loss and frequency domain loss: Where I out is the final output frame, is the ground truth of the data set during training, ‖·‖1 is the L1 norm, representing the sum of absolute differences, used to calculate the pixel-level difference between two images. Lambda is the weight parameter (set to 0.05 in practice), used to balance the contribution between the two loss terms. FFT is the fast Fourier transform, which can convert images in the spatial domain to the frequency domain, thus capturing information of different frequency components. FFT(I out ) and are the results of the fast Fourier transform on the predicted high-resolution image and the real high-resolution image, respectively. The design of this loss function aims to ensure that the model is not only accurate in the spatial domain (i.e. pixel values), but also matches the real high-resolution image in the frequency domain (i.e. different frequency components of the image). In this way, the model can better learn the detailed information needed to reconstruct high-resolution images.

[0072] (2) Multi-scale frequency domain constraint: calculate the difference between the FFT results by frequency band, low-frequency difference constraint ensures the correct main structure, medium-frequency difference constraint preserves edge sharpness, high-frequency difference controls detail restoration and noise suppression, so multi-scale frequency domain constraint can avoid the blurring problem caused by global loss function, while refining the reconstruction of each frequency band feature.

[0073]

[0074] wherein, is a band-pass filter set, which is a set of filters that each allows signals of a specific frequency range to pass through, used to divide the frequency domain components of a signal into different frequency bands, k = 1 is low frequency, k = 2 is medium frequency, and k = 3 is high frequency.γ k is a frequency band weight (e.g., take γ1 = 0.3, γ2 = 0.5, γ3 = 0.2).I out is the final output frame map, is the true value annotated by the data set during training, and ‖·‖1 is the L1 norm.

[0075] (3) Semantic consistency constraint: a pre-trained ViT (i.e., Vision Transformer) model is used to extract deep features, aiming to align high-level semantic features to ensure that the generated high-resolution image not only approaches the real image at the pixel level but also maintains consistency at the semantic level (such as object structure, texture, class, etc.), thereby improving visual quality and applicability to downstream tasks.

[0076]

[0077] wherein, φ ViT represents the feature of the second-to-last layer of the ViT network, I out is the final output frame map, is the true value annotated by the data set during training, is the L2 norm.

[0078] (4) Total loss function

[0079]

[0080] wherein, η is the weight of the semantic loss (e.g., η = 0.1).

[0081] In this embodiment, as shown in Figure 2 , 1, the "self-attention module" is a network module of a self-attention mechanism, which has a similar function to the Transformer, but the principle is different, but the efficiency will be much higher. The "self-attention module" is composed of two parts, one is a "cross-attention module", and the other is a "forward feedback module".

[0082] 2, the input image feature is (i.e., I is the input image, its resolution is H x W, i.e., height x width, and C represents the number of channels), which is first passed through a "normalization layer", without limiting the specific normalization method, such as batch normalization, group normalization, and layer normalization.

[0083] 3. Explain the relationship and difference between "first convolution" and "second convolution", "first residual" and "second residual"

[0084] (1) "First convolution" and "second convolution" are both 1x1 convolution layers, the only difference is that "second convolution" is followed by a nonlinear activation layer, which does not limit the specific normalization method, which can be ReLU, LeakyReLU, SELU, Softplus, etc.

[0085] (2) Similarly, "first residual" and "second residual" are both 5x5 "inverted residual layers", the only difference is that "second residual" is followed by a nonlinear activation layer, which does not limit the specific normalization method, which can be ReLU, LeakyReLU, SELU, Softplus, etc.

[0086] (3) The purpose of "inverted residual layer" is to run on low-end devices, which can greatly reduce the parameter quantity compared with convolution block, which is conducive to model lightweight. "Inverted residual layer" has multiple "inverted residual blocks" (this model has 3), each inverted residual block needs: ① Dimensionality: The inverted residual block first uses 1x1 convolution to increase the channel number of the input feature map. Although this step seems to increase the parameters, it actually provides more information and representation ability in the subsequent depth separable convolution, thereby improving the performance of the model at relatively low computational cost; ② Depth separable convolution: On the dimensionality-increased feature map, the inverted residual block applies depth separable convolution for feature extraction. Depth separable convolution is an efficient convolution method that decomposes standard convolution into depth convolution and pointwise convolution (1x1 convolution), thereby significantly reducing the number of parameters and computation; ③ Dimensionality reduction: Finally, the inverted residual block uses another 1x1 convolution to reduce the dimensionality of the feature map, i.e. reduce the number of channels, to restore or adjust the dimensionality of the feature map. This step also helps to reduce the number of parameters. The architecture diagram of "inverted residual block" is shown below Figure 3

[0087] 4. "Cross attention module"

[0088] (1) The role of "cross attention module" is similar to that of Transformer, which calculates the association between local features and global features to improve the fineness of local details.

[0089] (2) After the input image feature I passes through the "normalization layer", it enters the "first convolution" layer, which aims to duplicate the channel to twice the original, i.e. Then it is split into two and ​where I1 is regarded as local feature and input into the first residual, and I2 is regarded as global feature and input into the global variance and feature multiplication and addition.

[0090] (3) I1 is output after extracting local image features through the first residual. That is, the number of channels does not change, but the resolution is reduced by 8 times in order to make the model pay more attention to local details. And I'1 is input into the second convolution and feature addition.

[0091] (4) Local detail extraction branch: I'1 is input into the second convolution to perform local detail encoding and feature extraction, generating I'1', And then through the first convolution to generate enhanced local features

[0092] (5) Global feature extraction branch: I2 is operated by the global variance, that is, Where N is the total number of pixels, x i represents the RGB value of each pixel, μ represents the average value of all pixel RGB, is the variance of I2. Then I'2 and I'1 are added, that is, I'2 is copied first, and Then add pixel by pixel. Then input the added feature map into the second convolution to fuse local features and global features and find the relationship between them to generate I"2, and Finally, through an upsampling operation (this upsampling operation does not present a specific method of use, such as bilinear interpolation, bicubic interpolation, Lanczos interpolation, nearest neighbor upsampling, fast bilinear interpolation, etc.), I'2" is generated, and

[0093] (6) Then input I'1", I'2" output by the local detail extraction branch, and global feature I2 into the feature multiplication and addition operation. So-called "feature multiplication and addition" is that I2 and I'2" are first multiplied by feature (i.e. pixel and pixel matrix multiplication, the purpose is to further highlight the global feature), and the result of multiplication Then add I'1" pixel by pixel to get

[0094] 5. The purpose of the second residual connection is to prevent the above-mentioned cross-attention module from having zero gradient.

[0095] 6. "Front feedback module"

[0096] ​(1) The "feed-forward module" here is similar in function to the feed-forward network in the Transformer, which extracts and fuses feature information from different positions, enhances the non-linear capability of the model, and improves the expression capability of the model, thereby further improving the performance of the model in processing sequence data.

[0097] (2) The feature map I 12 output by the "cross attention module" is input into the "second convolution" in the "feed-forward module", which has a similar purpose, i.e., duplicating the channels to twice the original number, i.e., C x 2. Then it is split into two and according to the number of channels C. Then is input into the "second residual", and the output is obtained. and are added in terms of features, i.e., first through an upsampling operation (this upsampling operation does not present a specific method of use, such as bilinear interpolation, bicubic interpolation, Lanczos interpolation, nearest neighbor upsampling, fast bilinear interpolation, etc.) to make Then the pixel is added. Finally, the obtained result is input into the "first convolution" for feature fusion, and the final output is obtained.

[0098] 7. The purpose of the "first residual connection" is to prevent the problem of zero gradient occurring simultaneously in the "cross attention module" and the "feed-forward module".

[0099] In this embodiment, the post- upsampling model is:

[0100] 1. The purpose of the "post- upsampling" is to avoid the impact of high-dimensional space calculation and improve network calculation efficiency, so the upsampling layer is placed at the end of the network.

[0101] 2. The "post- upsampling" model usually uses a learning-based upsampling method, such as deconvolution (e.g., FSRCNN, SRDenseNet, SRFBN, etc.), sub-pixel convolution (e.g., S-CNN, EDSR, SRGAN, M SRN, RCAN, SAN, DAN, HDRN, CRN, SwinIR, etc.), nearest neighbor interpolation (e.g., EnhanceNet, ESRGAN, RealSR, BSRGAN, Real-ESRGAN, etc.), meta- upsampling (e.g., Meta-SR), etc., all of which can realize end-to-end automatic learning.

[0102] 3. Introduction to the principle of sub-pixel convolution

[0103] (1) The principle of sub-pixel convolution is as shown above Figure 4 First, increase the number of channels, and then reorder.

[0104] (2) Increase the number of channels:

[0105] 1) The size of the input feature map is HxWxC, where H is the height, W is the width, and C is the number of channels.

[0106] 2) Through convolution operation, the number of channels of the output feature map is increased to rC, where r is the magnification. 2

[0107] 3) The size, step and padding of the convolution kernel can be adjusted according to the specific task.

[0108] (3) Reorder:

[0109] 1) Reorganize the increased channels after convolution into spatial dimensions.

[0110] 2) Specifically, it rearranges the rC channel values in each r x r block into an r x r spatial block, thereby realizing image magnification. 2

[0111] 3) In this way, the resolution of the output image is rH x rW, that is, image magnification and resolution improvement are realized.

[0112] 4) The "post-up-sampling" model does not limit the above methods.

[0113] In this embodiment, adaptive discussion (note: "adaptivity" means automatic adaptation to multiple working conditions):

[0114] 1, "explicit end" working condition

[0115] (1) "Explicit end" can be a personal computer (local or other machine), mobile phone, tablet, XR head-mounted device (such as VR, AR, MR, etc.), large screen and other devices.

[0116] (2) When the "explicit end" is a low-end device without GPU chip or with poor GPU performance: At this time, the "input frame graph" is not processed, and is directly given to the explicit device for output.

[0117] ​​(3) When the "display end" is a middle-end device and the GPU performance is not high: At this time, the "rendering end" needs to cooperate, and a resolution threshold (such as 1080P) can be set. If the video resolution output from the "rendering end" is greater than or equal to the set resolution threshold, the "display end" does not perform any processing and directly outputs to the display device. If the video resolution output from the "rendering end" is less than the set resolution threshold, the "display end" performs resolution processing through a network model composed of two "self-attention modules". The "rendering end" will adaptively adjust the rendering frame resolution and H.264 or H.265 code rate based on its own rendering frame rate, scene complexity (including the number of triangular faces, lighting conditions, etc.), device heating conditions, and other factors.

[0118] 1) For example, when the "rendering end" outputs rendering frames above the upper frame rate threshold (such as 60FPS), it directly outputs resolutions and code rates above the set resolution threshold (such as 2K). Conversely, when the "rendering end" outputs rendering frames below the lower frame rate threshold (such as 30FPS), it directly outputs resolutions and code rates below the set resolution threshold (such as 250P).

[0119] 2) For example, when the "rendering end" outputs rendering frames with a total number of triangular faces in the view frustum (excluding triangular faces outside the view frustum and those removed by occlusion culling) below the triangular face number threshold (such as 100 million faces), it directly outputs resolutions and code rates above the set resolution threshold (such as 2K). Conversely, when the "rendering end" outputs rendering frames with a total number of triangular faces in the view frustum above the triangular face number threshold (such as 100 million faces), it directly outputs resolutions and code rates below the set resolution threshold (such as 250P).

[0120] 3) Other conditions are similar to the above.

[0121] (4) When the "display end" is a high-end device with high GPU performance: At this time, the "display end" should always generate high-resolution (generally 4 times the input resolution, such as input 1080P, output 4K) through a network model composed of two "self-attention modules".

[0122] 2, "Rendering end" working conditions

[0123] (1) The "rendering end" must have a GPU chip or integrated graphics card (i.e., it cannot be a low-end device), otherwise it cannot complete the output task of rendering frames. Generally, the "rendering end" requires high GPU performance, such as NVIDIA RTX 4090.

[0124] (2) When the "rendering end" is a middle-end device, the GPU performance is not high (or integrated graphics card): At this time, the resolution of the rendered frame should be controlled below the upper limit resolution threshold (such as 1080P), and the output resolution and code rate should be adaptively adjusted according to the situation of the "display end".

[0125] 1) When the "display end" is a high-end device, the "rendering end" can further reduce the output resolution and code rate (such as 250P).

[0126] 2) If the "display end" is a low-end device, the "rendering end" should output a resolution and code rate close to the upper limit resolution threshold (such as 1080P).

[0127] 3) If the "display end" is a middle-end device, the output resolution needs to be adaptively and dynamically adjusted according to the specific situation (such as rendering frame rate, scene complexity, heat condition, etc.).

[0128] (3) When the "rendering end" is a high-end device, the output resolution also needs to be adaptively and dynamically adjusted according to the specific situation (such as rendering frame rate, scene complexity, heat condition, etc.).

[0129] 3, "rendering end" adaptive code rate allocation strategy and encoding strategy - a space-time aware dynamic encoding parameter optimization algorithm, combined with the space-time features of video content and real-time network status, dynamically adjusts the encoding parameters (such as QP value, GOP length, code rate allocation), realizes the balance of bandwidth utilization and picture quality. The problems solved are: (1) Network fluctuation: fixed code rate allocation is easy to cause stuttering or bandwidth waste when the network bandwidth fluctuates. (2) Device heterogeneity: low-end devices have limited decoding capability, high-end devices need high code rate to avoid block effects, and traditional encoding cannot be adaptively adapted. (3) Video content diversity: complex scenes (such as fast motion, high texture) need more fine code rate allocation, and simple scenes have higher compressibility.

[0130] (1) Space-time feature extraction: by quantifying the texture complexity (spatial dimension) and motion intensity (temporal dimension) of the video frame, the basic index for encoding decision is constructed.

[0131] 1) Calculate the texture complexity

[0132]

[0133] wherein, represents the pixel value matrix of the luminance channel (Y channel) of the t-th frame YUV image, wherein H represents the height of the YUV image, W represents the width of the YUV image, represents the luminance value of the i-th row, j-th column pixel of the luminance channel of the t-th frame YUV image. μ is the mean value of the luminance channel, is the variance of the luminance channel, which measures the complexity of the texture within a frame.

[0134] 2) Calculate the motion intensity

[0135]

[0136] where the horizontal and vertical gradients are calculated by Sobel operators, and the gradient norm is larger, the more intense the motion.

[0137] 3) Calculate the spatio-temporal complexity

[0138]

[0139] where, is the variance of the luminance channel, which measures the complexity of the texture within a frame. is the gradient L1 norm of the luminance channel, which measures the intensity of the motion. α var and are weight coefficients (such as α var = 0.7, ), which prioritize the impact of texture complexity on subsequent code rate allocation.

[0140] (2) Deep Reinforcement Learning Network (DQN): Model the encoding parameter optimization as a Markov decision process, use DQN to learn the optimal policy online, output adaptive QP values and code rate allocation.

[0141] 1) DQN network design

[0142] Input layer: state parameters s t (3D vector), s t = [S t , B t , D t ] T , where S t is the spatio-temporal complexity calculated by "spatio-temporal feature extraction", B t is the real-time network bandwidth (which can be estimated by TCP congestion control, or by RTCP feedback), and D t is the target device decoding capability (which can be represented by available quantization levels, such as 0 for low-end devices, 1 for mid-end devices, and 2 for high-end devices).

[0143] Hidden layer: 2 layers of full connection (256 neurons, ReLU activation).

[0144] Output layer: Q matrix (20 QP levels, and 5-level code rate allocation strategy, a total of 25 parameters, forming a 5x5 matrix).

[0145] 2) Reward function: DQN learns the optimal policy by maximizing the cumulative reward. The design of the reward function needs to balance the picture quality (PSNR) and bandwidth utilization. Higher PSNR and lower code rate / bandwidth ratio will get higher reward, driving the network to choose better encoding parameters.

[0146]

[0147] where, MSE is the mean square error, MAX I MAX represents the maximum value of pixels (MAX I is the reference value for PSNR calculation for normalization), and PSNR is used to measure the quality of the coded image, with a higher value indicating less distortion. B t is the real-time network bandwidth. a PSNR is the weight coefficient of the PSNR term (such as a PSNR = 1.2), which is used to control the importance of picture quality in the reward, with a larger value tending to higher picture quality. b B is the weight coefficient of the code rate penalty term (such as b B = 0.8), which is used to control the penalty strength of the code rate / bandwidth ratio, with a larger value suppressing higher code rate.

[0148] 3) Adaptive adjustment strategy

[0149] “QP value” is used to control the quantization step size during encoding. The smaller the QP, the lower the quantization error and the higher the picture quality, but the larger the code rate; conversely, the compression rate is higher and the picture quality is lower. Specifically, according to the QP level output by DQN (such as 20 discrete QP levels), it is mapped to the actual QP value (range 18-42), such as level 0 corresponding to QP = 18, level 19 corresponding to QP = 42, and then linear interpolation is used to generate intermediate mapping values. Dynamically set the QP parameter in the encoder to adjust the quantization intensity in real time.

[0150] “Code rate allocation strategy” determines the code rate weight of different regions (such as foreground and background), and prioritizes the picture quality of key regions (such as moving objects and faces). Specifically, use a semantic segmentation model (such as DeepLabV3, etc.) to divide the image into foreground and background, and according to the 5-level code rate allocation strategy output by DQN (such as 0.2, 0.4, 0.6, 0.8, 1.0), higher weight is allocated to the foreground region, for example, select the strategy level 3 (5-level strategy, 0, 1, 2, 3, 4, so level 3 corresponds to 0.8), then the foreground code rate weight is 0.8 and the background is 0.2. In the encoder, more bits are allocated to high-weight regions through ROI (Region of Interest) encoding technology to improve local picture quality.

[0151] Each frame adaptive adjustment process: first, the current state s tThe input is fed into the DQN network, which outputs 20 QP levels and 5 bitrate allocation strategies. Then, the QP registration is converted into actual QP values, and encoder parameters are set. After that, the bit allocation weights of the ROI region are adjusted according to the bitrate allocation strategy. Finally, the encoder generates the compressed bitstream according to the new parameters and transmits it to the receiving end.

[0152] (3) Adaptive Group of Pictures (GOP) structure: The GOP length is dynamically adjusted according to the spatiotemporal complexity to balance coding efficiency and random access capability. Bitrate represents the temporal decoding bitrate of the current frame, reflecting the amount of data after encoding. The higher the bitrate, the more bandwidth is used.

[0153] 1) Definition and Function of GOP: A GOP (Group of Pictures) is a collection of consecutive video frames, containing one I-frame (Intra-coded frame, keyframe) and several P-frames (Predicted frames) or B-frames (Bidirectional predicted frames). Its functions include:

[0154] Compression efficiency: Reduce redundant data and improve compression ratio by inter-frame prediction (P / B frames).

[0155] Fault tolerance and random access: I-frames serve as independent encoding reference points, supporting random positioning playback (such as dragging the progress bar).

[0156] Error control: Periodic refresh of I-frames can interrupt the cumulative propagation of prediction errors.

[0157] 2) Adaptive GOP length adjustment strategy: based on time and space complexity S t This allows for dynamic adjustment of the GOP length and the number of B-frames.

[0158]

[0159]

[0160] For highly complex scenarios (S) t >0.8): Using L GOT =8, meaning the GOP length is 8, indicating that an I-frame is inserted every 8 frames. By frequently inserting I-frames, the refresh rate of highly complex scenes (such as fast motion and complex textures) is ensured, the risk of prediction error propagation is reduced, and fault tolerance is improved. N B-frame =20% means reducing the number of B frames (B frames account for 20%), reducing the risk of prediction error accumulation caused by violent motion, and reducing the decoding pressure on low-end devices.

[0161] For low-complexity scenarios (S) t <0.3): Using LGOT = 16, i.e. GOP length is 16, means inserting an I-frame every 16 frames, by reducing the proportion of I-frames, improving compression efficiency, significantly reducing the code rate, suitable for static or simple scene. N B-frame = 60% is to increase the number of B-frames (B-frame ratio 60%), use bidirectional prediction to improve the compression rate of static or simple motion scene, and does not affect the decoding performance.

[0162] For other scenes (0.3≤S t ≤0.8): L GOT = 12, i.e. GOP length is 12, means inserting an I-frame every 12 frames, N B-frame = 40% means B-frame ratio 40%, by balancing strategy, compromise between compression efficiency and fault tolerance, adapt to general scene demand.

[0163] 4、When "rendering end" and "explicit end" are installed on different devices (this is the most common case), network transmission module is required at this time.

[0164] 5、Under certain requirements, "rendering end" and "explicit end" can also be on the same device.

[0165] (1) If on the same device, no network transmission module is required at this time, nor does the "rendering frame" need to be converted from RGBA format to YUV, nor does the video need to be encoded, but the "rendering frame" is directly input into the neural network model composed of 2 "self-attention modules".

[0166] (2) In order to balance performance and rendering effect, the rendering output is generally 1080p, and then the neural network model outputs real-time pictures of 4K resolution, that is, it meets the demand of high resolution and ensures the rendering performance. And can be adjusted according to the specific circumstances (such as rendering frame rate, scene complexity, heating condition, etc.), such as:

[0167] 1) If the device heats to the temperature threshold (such as 90 degrees), the rendering output resolution is reduced (such as 250P).

[0168] 2) If the rendering frame rate is lower than a certain frame rate threshold (such as 60FPS), the rendering output resolution is reduced (such as 250P).

[0169] 6、Special working conditions

[0170] (1) If there is a special high-definition (8K, 16K) requirement, the "rendering end" needs to provide a 2k or 4k resolution video stream, and then through the neural network model composed of 2 "self-attention modules", the resolution is raised to 8k and 16k, meeting the demand of special high-definition.

[0171] If there is a requirement for transmission delay, it is necessary to reduce the resolution of the rendering frame and encode it with more advanced encoding technology (such as H.265) to reduce the amount of data transmitted over the network.

[0172] Embodiment two

[0173] The application also provides a self-adaptive high-quality rendering streaming system, which is used to implement the method of embodiment one, and the system comprises an acquisition module, a transmission module and an amplification module.

[0174] The acquisition module is used to acquire a rendering frame image.

[0175] The transmission module is used to transmit the rendering frame image to a display end through a network transmission technology.

[0176] The amplification module is used to amplify the resolution of each frame of the input video, i.e. the rendering frame image, by a deep learning model to obtain a final high-resolution frame image.

[0177] In this embodiment, the acquisition module comprises a conversion unit and an encoding unit.

[0178] The conversion unit is used to convert the color space of an original rendering frame, i.e. an image in RGBA format, to obtain a converted rendering frame, i.e. an image in YUV format.

[0179] The encoding unit is used to encode the YUV rendering frame through a video encoding standard to obtain an encoded rendering frame.

[0180] In this embodiment, the deep learning model is composed of a 3x3 convolution layer, two self-attention modules, a 3x3 convolution layer, and a 4-fold post-up sampling.

[0181] The self-attention module is composed of two parts, a cross-attention module and a feed-forward module.

[0182] The cross-attention module: after the input image feature I passes through a "normalization layer", it enters a first convolution layer to obtain Then, according to the channel number C, it is split into two and where I1 is regarded as a local feature and is input into a first residual, and I2 is regarded as a global feature and is input into a global variance and feature multiplication operation.

[0183] After I1 extracts the local image feature through the first residual, the output is The channel number remains unchanged, but the resolution is reduced by 8 times, and I'1 is input into a second convolution and feature addition.

[0184] I'1 is input into the second convolution to perform local detail encoding and feature extraction, generating I″1. Then, enhanced local features are generated through the first convolution.

[0185] I2 undergoes a global variance operation, i.e. Where N is the total number of pixels, x i This represents the RGB value of each pixel, and μ represents the average RGB value of all pixels. It is the variance of I2, then the features of I'2 and I'1 are added together, that is, I'2 is first copied to pixels, so that Then, the features are added pixel by pixel, and the resulting feature map is input into the second convolution to fuse local and global features, find the relationship between them, and generate I″2. Finally, through an upsampling operation, I″'2 is generated, and

[0186] The outputs I″'1 from the local detail extraction branch, I″'2 from the global feature extraction branch, and the global feature I2 are all input into the feature multiplication-addition operation. The result of the multiplication is... Then Adding I″'1 pixel by pixel, we get

[0187] Regarding the pre-feedback module: It takes the feature map I output by the cross-attention module... 12 The input is fed into the second convolution in the prefeedback module to obtain... Then split it into two according to the number of channels C. and Then Input into "Second Residual", output After that and Feature addition is performed first by an upsampling operation. Then, the pixels are added together, and the result is input into the first convolution for feature fusion, and finally output.

[0188] In this embodiment, the loss function of the deep learning model is:

[0189]

[0190] Where η is the weight of the semantic loss, It consists of pixel-level L1 loss and frequency domain loss. It is a multi-scale frequency domain constraint. It is a semantic consistency constraint.

[0191] Example 3

[0192] Based on the same inventive concept, the disclosure also provides an electronic device corresponding to the method of any of the above embodiments, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method of any of the above embodiments.

[0193] Figure 5 A more specific hardware structure of an electronic device is shown in this embodiment. The device can include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 for internal communication within the device.

[0194] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present disclosure.

[0195] The memory 1020 can be implemented by a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present disclosure are implemented by software or firmware, the related program codes are stored in the memory 1020 and executed by the processor 1010.

[0196] The input / output interface 1030 is used to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0197] The communication interface 1040 is configured to connect a communication module (not shown in the figure) to realize the communication interaction between the device and other devices. The communication module can realize communication through wired mode (such as USB (Universal Serial Bus), network cable, etc.), or can realize communication through wireless mode (such as mobile network, WIFI (Wireless Fidelity), Bluetooth, etc.).

[0198] The bus 1050 includes a path for transmitting information between various components (such as the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040) of the device.

[0199] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only contain the components necessary for the implementation of the embodiments of the present specification, and does not have to contain all the components shown in the figure.

[0200] The system of the above embodiment is used to realize the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here.

[0201] Embodiment four

[0202] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present disclosure also provides a non-transitory computer readable storage medium storing computer instructions for causing the computer to execute the method of any of the above embodiments.

[0203] The computer readable medium of the present embodiment includes permanent and non-permanent, removable and non-removable media, which can be realized by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0204] The storage medium of the above-mentioned embodiments stores computer instructions for causing the computer to execute the method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not repeated here.

[0205] It should be understood by those of ordinary skill in the art that the discussion of any of the above embodiments is merely exemplary and is not intended to suggest that the scope of the disclosure (including the claims) is limited to these examples; the above embodiments or technical features among different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of the disclosure as described above, which are not provided in detail for the sake of brevity. It is intended that each of the foregoing further patent application embodiments is encompassed by this disclosure, be incorporated into this disclosure by reference as if individually incorporated, and be deemed a separate and distinct embodiment of the disclosure.

[0206] In addition, in order to simplify the description and discussion, and so as not to make the embodiments of the disclosure difficult to understand, the well-known power / ground connections of integrated circuit (IC) chips and other components can or can not be shown in the provided drawings. In addition, devices can be shown in the form of block diagrams in order to avoid making the embodiments of the disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented (i.e., these details should be fully within the understanding of those skilled in the art). Where specific details (e.g., circuitry) are set forth in order to describe an illustrative embodiment of the disclosure, it will be apparent to those skilled in the art that the embodiments of the disclosure can be practiced without these specific details or with variations on these specific details. Therefore, these descriptions should be considered as illustrative rather than limiting.

[0207] Although the disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications and variations will be apparent to those skilled in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed.

[0208] Therefore, the units of each example described in the embodiments of the present application can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0209] The above described embodiments are only to illustrate the preferred modes of the present application, and are not intended to limit the scope of the present application. Any modification and improvement made by those skilled in the art to the technical solutions of the present application without departing from the design spirit of the present application shall fall within the protection scope of the present application as defined by the claims.

Claims

1. An adaptive high-quality rendering push method, characterized in that, The method comprises: acquiring a rendering frame image; transmitting the rendering frame image to an explicit end through network transmission technology; the explicit end enlarges the resolution of each frame of the input video, i.e., the rendering frame image, through a deep learning model to obtain a final high-resolution frame image.

2. The method of claim 1, wherein, The method for acquiring the rendering frame image comprises: performing color space conversion on an original rendering frame, i.e., an image in RGBA format, to obtain a transcoded rendering frame, i.e., an image in YUV format; encoding the YUV rendering frame through a video encoding standard to obtain an encoded rendering frame.

3. The method of claim 1, wherein, The deep learning model is composed of a 3x3 convolution layer, two self-attention modules, a 3x3 convolution layer connected in sequence, and a 4-fold post-up sampling at the end; The self-attention module is composed of two parts, one is a cross-attention module, and the other is a feed-forward module. Wherein, about cross attention module: the input image feature I enters the first convolutional layer after the "normalization layer", obtains Then split into two according to the number of channels C And Wherein I1 is regarded as a local feature and input into the first residual, and I2 is regarded as a global feature and input into the global variance and feature multiplication operation. I1 outputs the local image features after extracting by the first residual That is, the number of channels does not change, but the resolution is reduced by 8 times, and I'1 is input to the second convolution and feature addition; I'1 is input into a second convolution to perform local detail encoding and feature extraction to generate I"1, and then a first convolution is performed to generate enhanced local features I2 is operated by "global variance", that is Where N is the total number of pixels, x i represents the RGB value of each pixel, μ represents the average value of all pixel RGB, is the variance of I2, then I'2 and I'1 features are added, that is, I'2 is copied to the pixel first, and then then add pixel by pixel, and then input the added feature map to the second convolution to fuse the local features and the global features and find out the relationship between the two, generate I"2, and Finally, through an upsampling operation, I'2" is generated, and The I'1", output by the local detail extraction branch, the I'2", output by the global feature extraction branch, and the global feature I2 are all input into a feature multiplication-addition operation, and the multiplication result is Then the is pixel-wise added with the I'1" to obtain Regarding the front feedback module: the feature map I 12 output by the cross attention module is input into the second convolution in the front feedback module to obtain Then it is split into two and Then the is input into the "second residual", and the output is After that, the and are added in terms of features, that is, the is first upsampled, and then added in terms of pixels, and finally the obtained result is input into the first convolution for feature fusion, and finally output.

4. The method of claim 1, wherein, The loss function of the deep learning model is: wherein η is a weight of semantic loss, is a pixel-level LI loss and a frequency domain loss, is a multi-scale frequency domain constraint, is a semantic consistency constraint.

5. A self-adaptive high-quality rendering streaming system, the system is used to implement the method of any one of claims 1-4, characterized in that, The system comprises an acquisition module, a transmission module, and an enlargement module. The acquisition module is configured to acquire a rendering frame image. The transmission module is configured to transmit the rendering frame image to an explicit end through network transmission technology. The enlargement module is configured to enlarge the resolution of each frame of the input video, i.e., the rendering frame image, through a deep learning model to obtain a final high-resolution frame image.

6. The system of claim 5, wherein, The acquisition module comprises a conversion unit and an encoding unit. The conversion unit is configured to perform color space conversion on an original rendering frame, i.e., an image in RGBA format, to obtain a transcoded rendering frame, i.e., an image in YUV format. The encoding unit is configured to encode the YUV rendering frame through a video encoding standard to obtain an encoded rendering frame.

7. The system of claim 5, wherein, The deep learning model is composed of a 3x3 convolution layer, two self-attention modules, a 3x3 convolution layer connected in sequence, and a 4-fold post-up sampling at the end; The self-attention module is composed of two parts, one is a cross-attention module, and the other is a feed-forward module. Wherein, about cross attention module: the input image feature I enters the first convolutional layer after the "normalization layer", obtains Then split into two according to the number of channels C And Wherein I1 is regarded as a local feature and input into the first residual, and I2 is regarded as a global feature and input into the global variance and feature multiplication operation. I1 outputs the local image features after extracting by the first residual That is, the number of channels does not change, but the resolution is reduced by 8 times, and I'1 is input to the second convolution and feature addition; I'1 is input into the second convolution to generate I'1', which is a local detail encoding and feature extraction result, Then, the first convolution is performed to generate enhanced local features I2 is operated by global variance, that is Where N is the total number of pixels, x i represents the RGB value of each pixel, μ represents the average value of all pixel RGB, is the variance of I2, then the features of I'2 and I'1 are added, that is, first copy the pixels of I'2, make Then add pixel by pixel, then input the added feature map into the second convolution to fuse the local features and the global features and find out the relationship between the two, generate I'2', and Finally, through an upsampling operation, generate I'2", and The I'1", output by the local detail extraction branch, the I'2", output by the global feature extraction branch, and the global feature I2are all input into a feature multiplication-addition operation, and the multiplication result is Then the is pixel-by-pixel added with the I'1" to obtain Regarding the front feedback module: the feature map I 12 output by the cross attention module is input into the second convolution in the front feedback module to obtain Then it is split into two and Then is input into the "second residual", and the output is After that and are added, that is, first, the two are made to have the same size through an upsampling operation Then they are added pixel by pixel, and finally the obtained result is input into the first convolution for feature fusion, and finally output.

8. The system of claim 5, wherein, The loss function of the deep learning model is: wherein η is a weight of semantic loss, is a pixel-level L1 loss and a frequency domain loss, is a multi-scale frequency domain constraint, is a semantic consistency constraint.

9. An electronic device, comprising: The computer readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 4 is implemented.

Images