Method, device and equipment for virtual scene video transmission and computer storage medium

Abstract

The application discloses a virtual scene video transmission method and device, equipment and computer storage medium, relates to the virtual reality technical field, and the method comprises the following steps: a user equipment sends a video transmission request to a server, wherein the video transmission request comprises a video identifier; the server receives the video transmission request sent by the user equipment, extracts features of a target video image corresponding to the video identifier, and obtains multimodal features; the server determines that the complexity corresponding to the multimodal features is a target complexity of the target video image; the server performs down-sampling processing on the target video image according to the target complexity, and obtains a down-sampling video; the server sends the down-sampling video to the user equipment; and the user equipment receives the down-sampling video sent by the server, restores the down-sampling video, and obtains a target video. According to the application, the image is subjected to down-sampling processing by judging the complexity of the video image, the bandwidth occupation in the transmission process is reduced, and the transmission delay is reduced.

Description

Technical Field

[0001] This application belongs to the field of virtual reality technology, and in particular relates to a method, apparatus, device and computer storage medium for virtual scene video transmission. Background Technology

[0002] With the development of virtual reality technology, users can watch videos by wearing head-mounted displays and other user devices, and switch their gaze range by turning their heads to change the viewing direction. However, transmitting high-resolution full-view videos requires a lot of network bandwidth, resulting in significant bandwidth pressure.

[0003] To balance transmission bandwidth pressure and video resolution, existing technologies divide panoramic video into multiple video regions and transmit different regions based on the user's field of view. High-resolution images are transmitted for regions within the user's field of view, while low-resolution images are transmitted for regions outside the user's field of view, thus reducing bandwidth consumption. However, when the user rapidly switches perspectives, the user device will send multiple high-resolution video requests to the server within a short period. The server's transmission of high-resolution video from multiple regions to the user device still consumes significant bandwidth resources, causing latency and stuttering during video playback. Summary of the Invention

[0004] This application provides a method, apparatus, device, and computer storage medium for virtual scene video transmission to solve the problem that in the prior art, video transmission consumes a large amount of bandwidth resources when the user quickly switches perspectives, resulting in latency and stuttering during video playback.

[0005] In a first aspect, embodiments of this application disclose a method for transmitting virtual scene video, applied to a server, the method comprising:

[0006] Receive a video transmission request sent by a user equipment, the video transmission request including a video identifier;

[0007] Feature extraction is performed on the target video image corresponding to the video identifier to obtain multimodal features, which include at least two of the following: texture features, contour features, and color features.

[0008] The complexity corresponding to the multimodal features is determined as the target complexity of the target video image;

[0009] The target video image is downsampled based on the target complexity to obtain a downsampled video.

[0010] Send downsampled video to user equipment so that user equipment can restore the target video based on the downsampled video.

[0011] Secondly, embodiments of this application disclose a method for transmitting virtual scene video, applied to a user equipment, the method comprising:

[0012] A video transmission request is sent to the server. The video transmission request includes a video identifier, which is used by the server to extract features from the target video image corresponding to the video identifier to obtain multimodal features. The complexity corresponding to the multimodal features is determined as the target complexity of the target video image. The target video image is then downsampled according to the target complexity to obtain a downsampled video. The multimodal features include at least two of the following: texture features, contour features, and color features.

[0013] Receive downsampled video sent by the server;

[0014] The downsampled video is then restored to obtain the target video.

[0015] Thirdly, embodiments of this application disclose a device for transmitting virtual scene video, applied to a server, the device comprising:

[0016] The receiving module is used to receive video transmission requests sent by user equipment. The video transmission request includes a video identifier.

[0017] The extraction module is used to extract features from the target video image corresponding to the video identifier to obtain multimodal features, which include at least two of the following: texture features, contour features, and color features.

[0018] The determination module is used to determine the complexity corresponding to the multimodal features as the target complexity of the target video image;

[0019] The sampling module is used to downsample the target video image according to the target complexity to obtain a downsampled video.

[0020] The sending module is used to send downsampled video to the user equipment so that the user equipment can restore the target video based on the downsampled video.

[0021] Fourthly, embodiments of this application disclose a device for transmitting virtual scene video, applied to a user equipment, the device comprising:

[0022] The sending module is used to send a video transmission request to the server. The video transmission request includes a video identifier, which is used by the server to extract features from the target video image corresponding to the video identifier to obtain multimodal features. The complexity corresponding to the multimodal features is determined as the target complexity of the target video image. The target video image is then downsampled according to the target complexity to obtain a downsampled video. The multimodal features include at least two of the following: texture features, contour features, and color features.

[0023] The receiving module is used to receive downsampled video sent by the server;

[0024] The restoration module is used to restore the downsampled video to obtain the target video.

[0025] Fifthly, embodiments of this application provide a terminal device, the device comprising: a processor and a memory storing computer program instructions; the processor executing the computer program instructions to implement a method for virtual scene video transmission as described in the first or second aspect.

[0026] Sixthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the virtual scene video transmission method as described in the first or second aspect.

[0027] In a seventh aspect, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform a virtual scene video transmission method as described in the first or second aspect.

[0028] This application provides a method for virtual scene video transmission. The method involves a user device sending a video transmission request to a server, the video transmission request including a video identifier. The server receives the video transmission request from the user device and extracts features from the target video image corresponding to the video identifier to obtain multimodal features. The multimodal features include at least two of texture features, contour features, and color features. The server determines the complexity corresponding to the multimodal features as the target complexity of the target video image. The server downsamples the target video image based on the target complexity to obtain a downsampled video. The server sends the downsampled video to the user device. The user device receives the downsampled video from the server and restores the downsampled video to obtain the target video. This application's server performs complexity analysis on the video scene using multimodal features, providing a multi-dimensional basis for subsequent complexity assessment, avoiding the one-sidedness that may result from single feature extraction, and improving the accuracy and reliability of complexity assessment. By calculating the feature distance between the features and the cluster centers, the target complexity of the target video image can be quickly and accurately determined. The server downsamples the target video according to a preset downsampling factor corresponding to the target complexity and then transmits it. The user device receives the downsampled video and restores it, thereby minimizing the use of bandwidth resources while ensuring video quality. This allows the user device to receive video content quickly and stably under limited bandwidth and resources, thus solving the problems of latency and stuttering in video playback. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a video block diagram illustrating the field-of-view transmission method provided in the embodiments of this application.

[0031] Figure 2 This is a flowchart illustrating the method for virtual scene video transmission provided in an embodiment of this application;

[0032] Figure 3 This is a flowchart illustrating one implementation method for determining the target complexity provided in the embodiments of this application;

[0033] Figure 4 This is a flowchart illustrating one implementation method of training the complexity evaluation model provided in the embodiments of this application;

[0034] Figure 5 This is a schematic diagram of the result of the device for virtual scene video transmission applied to a server provided in the embodiments of this application;

[0035] Figure 6 This is a schematic diagram of the result of the device for virtual scene video transmission applied to user equipment provided in the embodiments of this application;

[0036] Figure 7 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0037] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0038] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0039] Existing technologies for transmitting virtual scene video often employ field of view (FOV) transmission methods. Among FOV-based transmission methods, the primary approach is the wireless transmission scheme based on video chunking (TWS). This scheme utilizes the chunking characteristics of H.265 / HEVC, such as... Figure 1 As shown in Figure a, during video encoding, a video frame is divided into multiple rectangular regions. Since these regions are independent of each other, encoding and transmission can proceed without relying on other regions. During transmission, the encoding / transmission process switches based on network conditions and the user's field of view. Figure 1 As shown in Figure b, when transmitting panoramic video content, it first determines which blocks overlap with the user's field of view. Then, it transmits a high-bitrate, high-quality version for the blocks that overlap with the field of view, and a low-quality version for the blocks outside the user's field of view. The client receives and decodes the bitstream data of each block, and renders the blocks to the corresponding positions in the original video frames based on spatial relationship descriptions.

[0040] However, in practice, to achieve a viewing quality similar to that of regular videos, panoramic videos should have a resolution of 8K or 16K or higher. Therefore, transmitting high-resolution or even ultra-high-resolution videos still requires significant bandwidth and a good network environment. Furthermore, in situations with poor or unstable networks, during frequent FOV (Field of View) switching, there exists an intermediate scene between different FOVs. The distance of this intermediate scene depends on the rotation angle. If the user rotates their head rapidly, the client needs to send requests frequently as the gaze point changes. The server then receives these requests and transmits a high-quality stream for the current gaze point. Rapid rotation places high demands on real-time processing performance, leading to frequent network requests and unavoidable video interruptions and lag.

[0041] To address the problems of existing technologies, this application provides a method for virtual scene video transmission. The method involves a user device sending a video transmission request to a server, the request including a video identifier. The server receives the video transmission request from the user device and extracts features from the target video image corresponding to the video identifier, obtaining multimodal features. These multimodal features include at least two of texture features, contour features, and color features. The server determines the complexity corresponding to the multimodal features as the target complexity of the target video image. The server downsamples the target video image based on the target complexity to obtain a downsampled video. The server sends the downsampled video to the user device. The user device receives the downsampled video from the server and restores it to obtain the target video. This application's server performs complexity analysis on the video scene using multimodal features, providing a multi-dimensional basis for subsequent complexity assessment, avoiding the bias that may result from single feature extraction, and improving the accuracy and reliability of complexity assessment. By calculating the feature distance between features and cluster centers, the target complexity of the target video image can be quickly and accurately determined. The server downsamples the target video according to a preset downsampling factor corresponding to the target complexity and then transmits it. The user device receives the downsampled video and restores it, thereby minimizing the use of bandwidth resources while ensuring video quality. This allows the user device to receive video content quickly and stably under limited bandwidth and resources, thus solving the problems of latency and stuttering in video playback.

[0042] The method for virtual scene video transmission provided in the embodiments of this application is described below with reference to the accompanying drawings.

[0043] Figure 2 A flowchart illustrating a method for virtual scene video transmission according to an embodiment of this application is shown. Figure 2 As shown, the method may include the following steps: S201 to S206.

[0044] S201, the user equipment sends a video transmission request to the server. The video transmission request includes a video identifier.

[0045] Among them, user equipment refers to the terminal equipment used by the user, such as mobile phones, virtual reality VR headsets, etc., which is used to send requests to the server and receive data. Video transmission request is a request initiated by the user equipment to request specific video content from the server. Video identifier is a code or number used to uniquely identify the target video, which can be a video file name or ID.

[0046] The user equipment sends a video transmission request, including a video identifier, to the server. The server can quickly locate the target video based on the video identifier, thus improving response efficiency.

[0047] S202, the server receives a video transmission request sent by the user equipment and extracts features from the target video image corresponding to the video identifier to obtain multimodal features, which include at least two of the following: texture features, contour features, and color features.

[0048] Among them, multimodal features include various aspects of video image features, which can describe the characteristics of video image from multiple dimensions. Texture features are the texture details and pattern features of the image, contour features are the edge and shape features of objects in the image, and color features are the color distribution and statistical information of the image.

[0049] In some embodiments, the server can process the target video image using specific image feature extraction algorithms to extract features from multiple dimensions and combine them to form multimodal features. For example, texture features can be extracted using gray-level co-occurrence matrix, contour features can be obtained using edge detection algorithms, and color features can be statistically analyzed using color histograms.

[0050] Multimodal features can describe the content of video images more comprehensively and accurately. Compared with single features, they can provide richer information, which helps in the subsequent analysis and processing of video images. For example, they can more accurately assess the complexity of video images and provide more precise basis for operations such as downsampling.

[0051] S203, the server determines the complexity corresponding to the multimodal features as the target complexity of the target video image.

[0052] Among them, target complexity is a quantitative representation of the complexity of the target video image, which is determined by multimodal features and reflects the richness of details, texture complexity, etc. of the video image.

[0053] Quantifying the complexity of video images provides a basis for subsequent downsampling processing based on complexity. It enables dynamic adjustment according to the actual complexity of the video image, improving processing flexibility and avoiding over- or under-downsampling. This ensures video quality while improving transmission efficiency and saving bandwidth.

[0054] S204, the server downsamples the target video image based on the target complexity to obtain a downsampled video.

[0055] Downsampling reduces the number of pixels in an image, thereby reducing the amount of data and lowering the video resolution. Downsampled video is the video obtained after downsampling, and its resolution is lower than that of the original target video image, resulting in a corresponding reduction in data volume.

[0056] In some embodiments, the server determines a corresponding downsampling strategy based on the target complexity, obtains a preset downsampling factor corresponding to the target complexity, and then processes the target video image according to the downsampling factor to reduce the number of pixels in the image, thus obtaining a downsampled video. For example, if the downsampling factor is 2, the number of pixels in both the horizontal and vertical directions of the image is halved.

[0057] Downsampling reduces the amount of video data, enabling the video to be sent to the user's device faster during network transmission. This reduces transmission time and bandwidth usage, improving system performance and user experience. In particular, it effectively avoids video stuttering and other problems when network bandwidth is limited.

[0058] S205, the server sends downsampled video to the user equipment.

[0059] Sending downsampled video results in a smaller data size, reducing transmission time and improving user experience.

[0060] S206, the user equipment receives the downsampled video sent by the server and restores the downsampled video to obtain the target video.

[0061] In some embodiments, the user equipment can use a corresponding image restoration algorithm to process the downsampled video and restore the target video for the user to watch. For example, it can use interpolation algorithms to process the downsampled pixels and restore an effect close to the original image.

[0062] This ensures that users can view the restored target video on their devices, guaranteeing that users can normally obtain and watch the video content they need, and ensuring the quality of the video.

[0063] Multimodal feature analysis of video scenes provides a multi-dimensional basis for subsequent complexity assessment, avoiding the bias that may result from single feature extraction and improving the accuracy and reliability of complexity assessment. By calculating the feature distance between features and cluster centers, the target complexity of the target video image can be quickly and accurately determined. The server downsamples the target video according to a preset downsampling factor corresponding to the target complexity and transmits it. The user device receives the downsampled video and restores it, thereby minimizing bandwidth consumption while ensuring video quality. This allows user devices to receive video content quickly and stably under limited bandwidth and resources, thus solving the latency and stuttering problems in video playback.

[0064] In some embodiments, texture features may include the second moment of the gray-level co-occurrence matrix, entropy, contrast, inverse difference matrix, and correlation statistics.

[0065] The formula for calculating the second angular moment (ASM) is:

[0066]

[0067] The formula for calculating entropy ENT is:

[0068]

[0069] The formula for calculating the contrast ratio (ACR) is:

[0070]

[0071] The formula for calculating the inverse difference matrix IDM is:

[0072]

[0073] The formula for calculating the correlation statistic is:

[0074]

[0075] Where P(i,j) is the probability that pixels with gray values ​​i and j appear simultaneously in the same direction, and μ x μ y δ represents the mean of the grayscale values ​​in both the row and column directions. x δ y The variance of grayscale values ​​in the row and column directions is represented by X1, where X1 = [ASM, ENT, ACR, IDM, Correlation].

[0076] In some embodiments, the contour features are determined by using the Sobal operator to calculate the image gradient through OpenCV to determine the edge. The edge length L_edge, edge density D_edge, and edge pixel count N_edge are selected as features. The contour is represented by X2, X2 = [L_edge, D_edge, N_edge].

[0077] In some embodiments, the color features use the average pixel complexity h_mean, standard deviation h_var, maximum value h_max, and minimum value h_min as features, denoted by X3, where X3 = [h_mean, h_var, h_max, h_min].

[0078] The formula for calculating color characteristics is:

[0079]

[0080] Where h(x,y) is the color complexity at pixel (x,y), c(i,j) is the pixel value at (i,j), E(c(i,j),c(x,y)) is the Euclidean distance of the pixel in the LAB color space, and gw is the Gaussian weight, and V is the normalization factor.

[0081] In some embodiments, before performing feature extraction on the target video image corresponding to the video identifier to obtain multimodal features, the method may further include:

[0082] Get the number of times the user device sends video transmission requests within a preset time period;

[0083] If the number of requests exceeds a set threshold, feature extraction is performed on the target video image corresponding to the video identifier to obtain multimodal features.

[0084] When a large number of transmission requests are received within a preset time period, both the server and client need to send video at a high frequency, placing high demands on real-time processing performance and inevitably causing video interruptions and lag. Therefore, a pre-set threshold for the number of requests is used to determine whether feature extraction of the target video image is necessary. When the number of requests exceeds this threshold, it is considered that the user has a high demand for video or the system is under high load, thus initiating further processing to ensure the quality and smoothness of the video during transmission and improve the user's viewing experience.

[0085] In some embodiments, the field of view (FOV) transformation rate can also be set, and the calculation formula is as follows:

[0086] r = Δω / Δt

[0087] Where Δω is the difference in rotational radians, Δt is the time interval, and r is the FOV transformation rate.

[0088] In some embodiments, the multimodal feature extraction formula is:

[0089] feature = F-feature(fov) i ),r>δ

[0090] Where feature refers to multimodal features, fov i This represents the current field of view, F-feature(fov) i ) is the current fov i Multimodal feature processing, where δ is the transformation rate threshold.

[0091] In some embodiments, determining the complexity corresponding to the multimodal features as the target complexity of the target video image may include:

[0092] The multimodal features are input into the complexity evaluation model. The feature distances between the multimodal features and multiple pre-trained cluster centers in the complexity evaluation model are calculated respectively. The complexity of the cluster to which the cluster center with the minimum feature distance belongs is determined as the target complexity of the target video image.

[0093] By using pre-trained models, image complexity can be quickly assessed, reducing manual intervention. By calculating feature distances, multimodal features can be compared with pre-trained cluster centers to quantify image complexity.

[0094] In some embodiments, the formula for calculating the target complexity is:

[0095] k label =f k-means (feature)

[0096] Where, k label For the target complexity, f k-means For the pre-trained scene complexity evaluation clustering model, the number of clusters is set to K=3, representing high, medium, and low complexity levels respectively. feature=CONCAT[X1, X2, X3], and k_label represents the current FOV scene as defined by f. k-means The cluster with the smallest Euclidean distance between the current feature and the three clusters is used as the discrimination type.

[0097] In some embodiments, such as Figure 3 As shown, the feature distances between the multimodal features and multiple pre-trained cluster centers in the complexity evaluation model are calculated respectively, and the complexity of the cluster to which the cluster center corresponding to the minimum feature distance belongs is determined as the target complexity of the target video image. This can include:

[0098] S301, determine the feature vector value of each pixel in the target video image of each frame based on multimodal features.

[0099] The feature vector value is the representation of each pixel in the multimodal feature space. It is usually a numerical vector, and the feature vector value is composed of feature vectors.

[0100] In some embodiments, the features of each dimension of the extracted multimodal features can be represented as numerical vectors and combined to form the feature vector of each pixel.

[0101] Multimodal features can more comprehensively describe the attributes of each pixel, providing richer information for subsequent processing and thus more accurately reflecting the complexity of the image, which helps to improve the accuracy of subsequent complexity assessment.

[0102] S302, each frame of image is divided into multiple regions based on the feature vector value and the region segmentation threshold, and the multimodal features of each region are obtained.

[0103] The region segmentation threshold is a threshold used to divide an image into multiple regions, and it is set according to the distribution of feature vector values.

[0104] In some embodiments, one or more thresholds can be set according to the distribution of feature vector values ​​to divide the image into multiple regions. For example, clustering algorithms (such as K-means) can be used to divide pixels into different regions based on feature vector values.

[0105] By dividing an image into multiple regions, the complexity of each region can be evaluated separately, improving the accuracy of complexity assessment.

[0106] S303, calculate the feature distance between the multimodal features of each region and the multiple cluster centers pre-trained in the complexity evaluation model.

[0107] In some embodiments, the characteristic distance can be Euclidean distance, Manhattan distance, and cosine distance, etc.

[0108] S304, determine the complexity of the cluster to which the cluster center corresponding to the minimum feature distance belongs as the target complexity of each region in each frame of the target video image.

[0109] By determining the complexity of different regions in each image, the complexity of the image can be described more comprehensively. In subsequent downsampling processing, different complexity regions can be downsampled separately according to the downsampling factor corresponding to different complexity regions, which can further avoid image quality loss caused by over-downsampling.

[0110] In some embodiments, such as Figure 4 As shown, before inputting the multimodal features into the complexity evaluation model, the method may further include:

[0111] S401, Obtain the training multimodal features of the training video images.

[0112] Training video images are the set of video images used to train the model. Training multimodal features are features extracted from the training video images that contain information about multiple modalities.

[0113] Multimodal features provide a data foundation for subsequent model training, as they can more comprehensively describe the characteristics of video images and help improve the accuracy of the model's assessment of video image complexity.

[0114] S402, input the trained multimodal features into the preset clustering model, and randomly select a preset number of trained multimodal features as cluster centers.

[0115] Randomly selecting cluster centers can give the model a certain degree of randomness and diversity in the initial stage, avoiding the model from getting trapped in local optima, and thus providing a more reasonable starting point for the subsequent clustering process.

[0116] S403, calculate the feature distance between the remaining training multimodal features and each cluster center, and assign the remaining training multimodal features to the cluster center with the smallest feature distance.

[0117] By calculating feature distance to divide into clusters, the training multimodal features can be reasonably grouped according to feature similarity, so that features within the same cluster have high similarity and features between different clusters have large differences, thus providing a basis for subsequent determination of the complexity classification of video images.

[0118] S404, calculate the feature mean of the training multimodal features for each cluster and use it as the new cluster center.

[0119] In some embodiments, for each cluster, the summation of each dimension of all trained multimodal features within the cluster is performed, and then divided by the number of features within the cluster to obtain the average value of each dimension. The vector composed of these average values ​​is the feature mean of the cluster, which is used as the new cluster center. For example, for a cluster containing N trained multimodal features, each feature vector has M dimensions. The average value of the N feature values ​​in each dimension is calculated to obtain a new M-dimensional vector, which is used as the new cluster center.

[0120] Using the feature mean as the new cluster center can more accurately represent the feature distribution of each cluster, making the clustering results more stable and accurate. As the number of iterations increases, the cluster centers will gradually converge to more reasonable positions, improving the accuracy of the model.

[0121] S405: Given that the cluster centers no longer change, obtain the pre-trained cluster centers, and update the preset clustering model based on the pre-trained cluster centers to obtain the complexity evaluation model.

[0122] By obtaining stable pre-trained cluster centers through iterative training, the complexity assessment model can better adapt to the distribution characteristics of the training data, improve the model's accuracy and generalization ability in assessing the complexity of video images, and thus provide strong support for subsequent accurate assessment of the complexity of target video images based on multimodal features.

[0123] In some embodiments, downsampling the target video image according to the target complexity to obtain a downsampled video may include:

[0124] Obtain the preset downsampling factor corresponding to the target complexity of the target video image;

[0125] The target video images are downsampled according to the preset downsampling factor to obtain downsampled videos.

[0126] The preset downsampling factor is a sampling ratio determined in advance based on experience or experiments, corresponding to different target complexities. High-resolution transmission is used for images with complex textures and rich information, while low-resolution transmission is used for static, simple scenes.

[0127] The downsampling factor can be flexibly selected according to the complexity of the video image. A smaller downsampling factor is used for images with high complexity, and a larger downsampling factor is used for images with low complexity. This can avoid image quality loss caused by excessive downsampling and effectively reduce the amount of data and improve transmission efficiency while ensuring a certain image quality.

[0128] In some embodiments, the downsampling processing formula is:

[0129] resolution = F(k_label)

[0130] Where F represents adaptive downsampling transmission processing, which transmits images with different downsampling factors from the server according to the complexity level of the discrimination type (high, medium, or low), and resolution represents the actual transmission resolution.

[0131] In some embodiments, before restoring the downsampled video to obtain the target video, the method may further include:

[0132] Receive the preset downsampling factor sent by the server;

[0133] The target video is obtained by restoring the corresponding target video images according to the preset downsampling factor.

[0134] By receiving a preset downsampling factor sent by the server, the resolution of the video image is restored. The user equipment can determine the restoration factor for the downsampled video, ensuring the accuracy and consistency of the restoration process.

[0135] In some embodiments, when restoring a target video image, bilinear interpolation, bicubic interpolation, etc., can be used to calculate the pixel values ​​to be inserted based on the existing pixel values ​​in the surrounding area, thereby increasing the number of pixels in the image and improving the image resolution.

[0136] In some embodiments, the restoration formula is:

[0137] fov = f DLSS (fov i )

[0138] Among them, f DLSSThis is for neural oversampling restoration processing; fov represents the image after oversampling restoration.

[0139] Figure 5 This application illustrates an apparatus 500 for virtual scene video transmission applied to a server, which may include:

[0140] The receiving module 501 is used to receive a video transmission request sent by the user equipment, the video transmission request including a video identifier;

[0141] The extraction module 502 is used to extract features from the target video image corresponding to the video identifier to obtain multimodal features, which include at least two of texture features, contour features and color features;

[0142] The determination module 503 is used to determine the complexity corresponding to the multimodal features as the target complexity of the target video image;

[0143] The sampling module 504 is used to downsample the target video image according to the target complexity to obtain a downsampled video.

[0144] The sending module 505 is used to send downsampled video to the user equipment so that the user equipment can restore the target video based on the downsampled video.

[0145] In some embodiments, the virtual scene video transmission apparatus 500 may further include:

[0146] The acquisition module is used to acquire the number of times the user device sends video transmission requests within a preset time period;

[0147] The extraction module 502 is also used to extract features from the target video image corresponding to the video identifier when the number of requests exceeds a set threshold, so as to obtain multimodal features.

[0148] In some embodiments, the determining module 503 is further configured to input multimodal features into the complexity evaluation model, calculate the feature distance between the multimodal features and multiple pre-trained cluster centers in the complexity evaluation model, and determine the complexity of the cluster to which the cluster center corresponding to the minimum feature distance belongs as the target complexity of the target video image.

[0149] In some embodiments, the virtual scene video transmission apparatus 500 may further include:

[0150] The determination module 503 is also used to determine the feature vector value of each frame of target video image pixel based on multimodal features;

[0151] The segmentation module is used to divide each frame of image into multiple regions based on feature vector values ​​and region segmentation thresholds, and obtain the multimodal features of each region;

[0152] The computation module is used to calculate the feature distance between the multimodal features of each region and the multiple cluster centers pre-trained in the complexity evaluation model;

[0153] The determination module 503 is also used to determine the complexity of the cluster to which the cluster center corresponding to the minimum feature distance belongs as the target complexity of each region in each frame of the target video image.

[0154] In some embodiments, the virtual scene video transmission apparatus 500 may further include:

[0155] The acquisition module is also used to acquire the training multimodal features of the training video images;

[0156] The selection module is used to input the trained multimodal features into the preset clustering model and randomly select a preset number of trained multimodal features as cluster centers.

[0157] The calculation module is also used to calculate the feature distance between the remaining training multimodal features and each cluster center, and to assign the remaining training multimodal features to the cluster center with the smallest feature distance;

[0158] The computation module is also used to calculate the feature mean of the training multimodal features for each cluster and use it as the new cluster center;

[0159] The update module is used to obtain the pre-trained cluster centers when the cluster centers no longer change, and to update the preset clustering model based on the pre-trained cluster centers to obtain the complexity evaluation model.

[0160] In some embodiments, the acquisition module is used to acquire a preset downsampling factor corresponding to the target complexity of the target video image;

[0161] The sampling module 504 is also used to perform downsampling processing on the corresponding target video images according to the preset downsampling factor to obtain downsampled videos.

[0162] Figure 5 The various modules in the illustrated device can achieve Figure 1 The various steps involved, and the corresponding technical effects achieved, will not be elaborated upon here for the sake of brevity.

[0163] Figure 6 This application illustrates an apparatus for virtual scene video transmission applied to a user equipment, which may include:

[0164] The sending module 601 is used to send a video transmission request to the server. The video transmission request includes a video identifier, which is used by the server to extract features from the target video image corresponding to the video identifier to obtain multimodal features, determine the complexity corresponding to the multimodal features as the target complexity of the target video image, and perform downsampling processing on the target video image according to the target complexity to obtain a downsampled video. The multimodal features include at least two of texture features, contour features and color features.

[0165] The receiving module 602 is used to receive downsampled video sent by the server;

[0166] The restoration module 603 is used to restore the downsampled video to obtain the target video.

[0167] In some embodiments, the receiving module 602 is further configured to receive a preset downsampling factor sent by the server;

[0168] The restoration module 603 is also used to restore the corresponding target video images according to the preset downsampling factor to obtain the target video.

[0169] Figure 6 The various modules in the illustrated device can achieve Figure 1 The various steps involved, and the corresponding technical effects achieved, will not be elaborated upon here for the sake of brevity.

[0170] Figure 7 A schematic diagram of the hardware structure of the terminal device provided in an embodiment of this application is shown.

[0171] The terminal device may include a processor 701 and a memory 702 storing computer program instructions.

[0172] Specifically, the processor 701 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0173] Memory 702 may include mass storage for data or instructions. For example, and not limitingly, memory 702 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one instance, memory 702 may include removable or non-removable (or fixed) media, or memory 702 may be non-volatile solid-state memory. Memory 702 may be internal or external to the integrated gateway disaster recovery device.

[0174] In one example, memory 702 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method of virtual scene video transmission according to this disclosure.

[0175] The processor 701 reads and executes computer program instructions stored in the memory 702 to achieve... Figure 1 The method for transmitting virtual scene video in the illustrated embodiment.

[0176] In one example, the terminal device may also include a communication interface 703 and a bus 704. Wherein, for example... Figure 7 As shown, the processor 701, memory 702, and communication interface 703 are connected through bus 704 and complete communication with each other.

[0177] The communication interface 703 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0178] Bus 704 includes hardware, software, or both, that couples components of an end device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 704 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0179] Furthermore, in conjunction with the virtual scene video transmission method in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the virtual scene video transmission methods in the above embodiments.

[0180] This application also provides a computer program product, including a computer program, which, when executed, implements any of the virtual scene video transmission methods described in the above embodiments.

[0181] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0182] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or text segments used to perform the required tasks. Programs or text segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Text segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0183] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0184] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0185] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for transmitting video in a virtual scene, characterized in that, Applied to a server, the method includes: Receive a video transmission request sent by a user equipment, wherein the video transmission request includes a video identifier; Feature extraction is performed on the target video image corresponding to the video identifier to obtain multimodal features, wherein the multimodal features include at least two of texture features, contour features and color features; The complexity corresponding to the multimodal features is determined as the target complexity of the target video image; The target video image is downsampled based on the target complexity to obtain a downsampled video. The downsampled video is sent to the user equipment so that the user equipment can restore the target video based on the downsampled video.

2. The method for virtual scene video transmission according to claim 1, characterized in that, Before extracting features from the target video image corresponding to the video identifier to obtain multimodal features, the method further includes: Get the number of times the user device sends video transmission requests within a preset time period; If the number of requests exceeds a set threshold, feature extraction is performed on the target video image corresponding to the video identifier to obtain multimodal features.

3. The method for virtual scene video transmission according to claim 1, characterized in that, Determining the complexity corresponding to the multimodal features as the target complexity of the target video image includes: The multimodal features are input into the complexity evaluation model, and the feature distances between the multimodal features and multiple pre-trained cluster centers in the complexity evaluation model are calculated respectively. The complexity of the cluster to which the cluster center corresponding to the minimum feature distance belongs is determined as the target complexity of the target video image.

4. The method for virtual scene video transmission according to claim 3, characterized in that, The step of calculating the feature distances between the multimodal features and multiple pre-trained cluster centers in the complexity evaluation model, and determining the complexity of the cluster to which the cluster center corresponding to the minimum feature distance belongs as the target complexity of the target video image, includes: The feature vector value of each pixel in the target video image is determined based on multimodal features; Each frame of image is divided into multiple regions based on the feature vector values ​​and the region segmentation threshold, and the multimodal features of each region are obtained. Calculate the feature distance between the multimodal features of each region and the multiple cluster centers pre-trained in the complexity evaluation model; The complexity of determining the cluster to which the cluster center corresponding to the minimum feature distance belongs is the target complexity of each region in each frame of the target video image.

5. The method for virtual scene video transmission according to claim 3, characterized in that, Before inputting the multimodal features into the complexity evaluation model, the method further includes: Obtain the training multimodal features of the training video images; The trained multimodal features are input into a preset clustering model, and a preset number of trained multimodal features are randomly selected as cluster centers. Calculate the feature distance between the remaining training multimodal features and each cluster center, and assign the remaining training multimodal features to the cluster center with the smallest feature distance; Calculate the feature mean of the training multimodal features for each cluster and use it as the new cluster center; If the cluster centers remain unchanged, pre-trained cluster centers are obtained, and the preset clustering model is updated based on the pre-trained cluster centers to obtain the complexity evaluation model.

6. The method for virtual scene video transmission according to claim 1, characterized in that, The step of downsampling the target video image based on the target complexity to obtain a downsampled video includes: Obtain the preset downsampling factor corresponding to the target complexity of the target video image; The target video images are downsampled according to the preset downsampling factor to obtain downsampled videos.

7. A method for transmitting video in a virtual scene, characterized in that, Applied to user equipment, the method includes: A video transmission request is sent to the server. The video transmission request includes a video identifier, which is used by the server to extract features from the target video image corresponding to the video identifier to obtain multimodal features. The complexity corresponding to the multimodal features is determined as the target complexity of the target video image. The target video image is then downsampled according to the target complexity to obtain a downsampled video. The multimodal features include at least two of texture features, contour features, and color features. Receive the downsampled video sent by the server; The downsampled video is then restored to obtain the target video.

8. The method for virtual scene video transmission according to claim 7, characterized in that, Before restoring the downsampled video to obtain the target video, the method further includes: Receive the preset downsampling factor sent by the server; The target video is obtained by restoring the corresponding target video image according to the preset downsampling factor.

9. A device for transmitting virtual scene video, characterized in that, Applied to a server, the device includes: A receiving module is used to receive a video transmission request sent by a user equipment, wherein the video transmission request includes a video identifier; The extraction module is used to extract features from the target video image corresponding to the video identifier to obtain multimodal features, wherein the multimodal features include at least two of texture features, contour features and color features; The determination module is used to determine the complexity corresponding to the multimodal features as the target complexity of the target video image; A sampling module is used to downsample the target video image according to the target complexity to obtain a downsampled video; The sending module is used to send the downsampled video to the user equipment so that the user equipment can restore the target video based on the downsampled video.

10. A device for transmitting virtual scene video, characterized in that, Applied to user equipment, the device includes: A sending module is used to send a video transmission request to a server. The video transmission request includes a video identifier, which is used by the server to extract features from the target video image corresponding to the video identifier to obtain multimodal features, determine the complexity corresponding to the multimodal features as the target complexity of the target video image, and perform downsampling processing on the target video image according to the target complexity to obtain a downsampled video. The multimodal features include at least two of texture features, contour features, and color features. A receiving module is used to receive the downsampled video sent by the server; The restoration module is used to restore the downsampled video to obtain the target video.

11. A terminal device, characterized in that, The device includes: a processor and a memory storing computer program instructions; the processor executes the computer program instructions to implement the virtual scene video transmission method as described in any one of claims 1-8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the method for virtual scene video transmission as described in any one of claims 1-8.

13. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the virtual scene video transmission method as described in any one of claims 1-8.

Images