Camera redirection methods, devices, equipment, storage media, and software products

Abstract

This invention discloses a camera redirection method, apparatus, device, storage medium, and program product. The method includes: creating a corresponding virtual camera for a camera peripheral on a cloud desktop terminal; displaying video data collected by the camera peripheral through the virtual camera; and dynamically adjusting the video capture parameters of the camera peripheral based on the real-time monitored network packet loss rate and the video capture parameter gradient table of the camera peripheral. This invention dynamically adjusts the video capture parameters of the camera peripheral on the cloud desktop terminal by using the network packet loss rate and video capture parameter gradient table during video data transmission, thereby achieving redirection of camera peripherals for cloud desktops and reducing the amount of transmitted data and network bandwidth usage.

Description

Technical Field

[0001] This invention relates to the field of video data processing technology, and in particular to a camera redirection method, apparatus, device, storage medium, and program product. Background Technology

[0002] Cloud computing is a new computing model based on cloud computing technology. It migrates the "hardware resources" and "computing power" of traditional computers to cloud data centers. Users can remotely access and use these "virtual computers" in the cloud through ordinary terminals (such as mobile phones, tablets, low-end computers, and TVs) without relying on local hardware performance. This allows idle computing resources in the cloud to be available to users. Cloud computing is widely used due to its advantages such as ease of management, high security, and low cost. Daily use of cloud computing requires support for common peripherals. USB cameras, as a video streaming peripheral, can be used in cloud computing through port redirection and device redirection techniques.

[0003] Currently, USB cameras can be used in cloud computers using port redirection technology. By listening to the cloud desktop's operations on the USB camera, the cloud desktop protocol sends server information to the cloud desktop terminal. After receiving the information, the terminal then operates the USB camera. However, this redirection method sends the data collected by the USB camera directly to the cloud desktop without compression, resulting in a large amount of data and high network bandwidth consumption during transmission. Summary of the Invention

[0004] To address the problems existing in the prior art, embodiments of the present invention provide a camera redirection method, apparatus, device, storage medium, and program product, which can effectively reduce the amount of data transmitted and network bandwidth usage.

[0005] In a first aspect, embodiments of the present invention provide a camera redirection method, comprising:

[0006] Create corresponding virtual cameras for the camera peripherals of the cloud desktop terminal;

[0007] The video data collected by the camera peripheral is displayed through the virtual camera;

[0008] The video capture parameters of the camera peripheral are dynamically adjusted based on the real-time monitored network packet loss rate and the gradient table of the video capture parameters of the camera peripheral.

[0009] As an improvement to the above solution, the step of displaying the video data collected by the camera peripheral through the virtual camera includes:

[0010] Construct a mapping relationship between each virtual camera and its corresponding camera peripheral buffer group; wherein, the buffer group includes a video data buffer and a frame loss recovery buffer;

[0011] Frame drop detection is performed on the video data frames currently acquired by the camera peripheral;

[0012] When no frame loss is detected, the currently acquired video data frame is saved to the video data buffer, and the currently acquired video data frame is read from the video data buffer and displayed through the virtual camera;

[0013] When frame loss is detected, the lost video data frame is predicted based on the latest video data frame in the frame loss recovery buffer, and the predicted video data frame is displayed through the virtual camera.

[0014] As an improvement to the above solution, the step of predicting the lost video data frames based on the latest video data frame in the frame loss recovery buffer, and displaying the predicted video data frames through the virtual camera, includes:

[0015] Based on the latest video data frame in the frame loss recovery buffer, the lost video data frame is predicted using a pre-trained video prediction module;

[0016] The predicted video data frames are updated to the frame loss recovery buffer to replace the latest video data frames in the frame loss recovery buffer.

[0017] The predicted video data frames are read from the frame loss recovery buffer using the virtual camera and then displayed.

[0018] As an improvement to the above solution, before creating a corresponding virtual camera for the camera peripheral of the cloud desktop terminal, the method further includes:

[0019] After logging into the cloud desktop on the cloud desktop terminal, the device receives video peripheral information sent by the cloud desktop terminal; wherein, the video peripheral information includes the device name, video capture parameters, and first identification information of at least one of the camera peripherals;

[0020] Based on the video capture parameters of each of the camera peripherals, a video capture parameter gradient table and a first mapping table between the first identification information of the corresponding camera peripheral and the video capture parameter gradient table are constructed.

[0021] As an improvement to the above scheme, the video capture parameters include resolution and sampling frame rate; the video capture parameter gradient table includes: a resolution gradient table and a sampling frame rate gradient table;

[0022] The first identification information includes a manufacturer identification code and a product identification code; the first mapping table is used to indicate the mapping relationship between the manufacturer identification code, the product identification code and the resolution gradient table and the sampling frame rate gradient table.

[0023] As an improvement to the above solution, the method further includes:

[0024] Query the resolution and sampling frame rate of each of the aforementioned camera peripherals;

[0025] If any of the camera peripherals does not support the preset default resolution and default sampling frame rate, the resolution gradient table and sampling frame rate gradient table of the corresponding camera peripheral are queried according to the first identification information of the corresponding camera peripheral.

[0026] Based on the retrieved resolution gradient table and sampling frame rate gradient table, calculate the amount of data that can be transmitted under different combinations of resolution and sampling frame rate;

[0027] Select the resolution and sampling frame rate combination that is closest to the preset default data volume as the current configuration resolution and sampling frame rate of the corresponding camera peripheral;

[0028] If any of the camera peripherals supports the default resolution and the default sampling frame rate, the default resolution and the default sampling frame rate shall be used as the resolution and sampling frame rate currently configured for the corresponding camera peripheral.

[0029] As an improvement to the above solution, the creation of a corresponding virtual camera for the camera peripheral of the cloud desktop terminal includes:

[0030] A virtual camera with the same resolution is generated based on the camera peripheral selected by the user on the cloud desktop terminal;

[0031] Construct a second mapping table between the first identifier information of the user-selected camera peripheral and the second identifier information of the corresponding virtual camera.

[0032] As an improvement to the above solution, the step of dynamically adjusting the video capture parameters of the camera peripheral based on the real-time monitored network packet loss rate and the gradient table of the video capture parameters of the camera peripheral includes:

[0033] Real-time monitoring of network packet loss rate, and comparison of the network packet loss rate of multiple transmissions with a preset packet loss rate threshold;

[0034] When the network packet loss rate is greater than or equal to the packet loss rate threshold, the resolution gradient table and sampling frame rate gradient table of the corresponding camera peripheral are queried according to the first identification information carried by the currently received video data frame.

[0035] Based on the retrieved resolution gradient table and sampling frame rate gradient table, calculate the amount of data that can be transmitted under different combinations of resolution and sampling frame rate;

[0036] Different combinations of resolution and sampling frame rate are sorted from highest to lowest according to the amount of data;

[0037] Select the resolution and sampling frame rate combination that is one bit lower than the current resolution and sampling frame rate combination of the corresponding camera peripheral, and update the current resolution and sampling frame rate of the corresponding camera peripheral.

[0038] As an improvement to the above solution, the method further includes:

[0039] Feature extraction and image reconstruction are performed on the currently acquired video data; wherein, the currently acquired video data is obtained by the cloud desktop terminal using a pre-trained deep learning module to perform image enhancement processing on the video data currently acquired by the camera peripheral.

[0040] Secondly, embodiments of the present invention provide a camera redirection device, comprising:

[0041] The virtual camera creation module is used to create corresponding virtual cameras for the camera peripherals of the cloud desktop terminal;

[0042] A video data display module is used to display video data collected by the camera peripheral through the virtual camera;

[0043] The parameter dynamic adjustment module is used to dynamically adjust the video capture parameters of the camera peripheral based on the real-time monitored network packet loss rate and the video capture parameter gradient table of the camera peripheral.

[0044] Thirdly, embodiments of the present invention provide a camera redirection device, comprising: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the camera redirection method as described in any one of the first aspects.

[0045] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the camera redirection method as described in any one of the first aspects.

[0046] Fifthly, embodiments of the present invention provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the camera redirection method as described in any one of the first aspects.

[0047] Compared to existing technologies, this invention provides a camera redirection method, apparatus, device, storage medium, and program product. The cloud desktop creates a corresponding virtual camera for the camera peripheral of the cloud desktop terminal; then displays the video data collected by the camera peripheral through the virtual camera; subsequently, it dynamically adjusts the video capture parameters of the camera peripheral based on the real-time monitored network packet loss rate and the video capture parameter gradient table of the camera peripheral. By dynamically adjusting the video capture parameters of the camera peripheral of the cloud desktop terminal through the network packet loss rate during video data transmission and the preset video capture parameter gradient table, the cloud desktop can achieve redirection of the camera peripheral for the cloud desktop, reducing the amount of data transmitted and network bandwidth usage. Attached Figure Description

[0048] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of a camera redirection method provided in an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the overall process of camera redirection based on cloud desktop provided in an embodiment of the present invention;

[0051] Figure 3 This is a structural block diagram of a camera redirection device provided in an embodiment of the present invention;

[0052] Figure 4 This is a structural block diagram of a camera redirection device provided in an embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] It is understood that the various numerical designations used in the embodiments of this invention are merely for descriptive convenience and are not intended to limit the scope of this application. The order of the process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.

[0055] In embodiments of the invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0056] Please see Figure 1 , Figure 1 This is a flowchart of a camera redirection method provided in an embodiment of the present invention. The camera redirection method, applied to a cloud computer, specifically includes:

[0057] S11: Create a corresponding virtual camera for the camera peripheral of the cloud desktop terminal;

[0058] Among them, cloud desktop terminals include, but are not limited to, terminal devices such as mobile phones, tablets, computers, and TVs; camera peripherals include, but are not limited to, external camera hardware independent of terminal devices such as USB cameras, HDMI cables, and wireless cameras; cloud desktop is one of the functional forms of cloud computers.

[0059] S12: Display the video data collected by the camera peripheral through the virtual camera;

[0060] S13: Based on the real-time monitored network packet loss rate and the gradient table of the video capture parameters of the camera peripheral, dynamically adjust the video capture parameters of the camera peripheral.

[0061] In this embodiment of the invention, after a user logs into the cloud desktop via a cloud desktop terminal, the cloud desktop creates a virtual camera for the cloud desktop terminal's camera peripheral, such as a USB camera. Subsequently, the cloud desktop terminal transmits the video data captured in real time by the camera peripheral to the cloud desktop. The cloud desktop caches the video data captured in real time by the camera peripheral locally and displays it through the virtual camera, thereby achieving external camera redirection. During the video data transmission process, the cloud desktop dynamically adjusts the video capture parameters of the camera peripheral based on the real-time monitored network packet loss rate and a preset video capture parameter gradient table, enabling the camera peripheral to adapt its video capture parameters (such as resolution and sampling frame rate) during video data transmission, thereby reducing the amount of data transmitted and network bandwidth usage.

[0062] Furthermore, the method also includes:

[0063] Before the cloud computer runs, the cloud desktop terminal pulls down the peripheral permission policy from the cloud computer management platform and enables peripheral information reporting. The peripheral permission policy is used to indicate whether the peripheral devices of the cloud desktop terminal, such as camera peripherals, can be connected to the cloud desktop, and what operation permissions they have after being connected to the cloud desktop. Peripheral information reporting is used to instruct the cloud desktop terminal to report the peripheral information of peripheral devices, such as the video peripheral information of camera peripherals, to the cloud desktop.

[0064] The cloud desktop terminal queries all local and external peripheral devices. When any peripheral device malfunctions (such as the peripheral device being unrecognized, an enumeration error, or a running error), the abnormal peripheral device information is uploaded to the cloud desktop via the network. The abnormal peripheral device information includes, but is not limited to, the first identification information of the abnormal peripheral device, the peripheral device type, and the cause of the abnormality.

[0065] The cloud desktop uploads information related to abnormal peripheral devices to the cloud computer management platform.

[0066] This invention utilizes a cloud computer management platform to periodically output collected anomaly-related information from peripheral devices (such as camera peripherals). This facilitates subsequent analysis and optimization of the reasons for peripheral device unavailability, allowing users to quickly locate unavailable peripheral devices and the causes of their unavailability. If peripheral device unavailability is due to peripheral permission policy configuration issues, the default peripheral permission policy configuration is optimized, granting the necessary redirected peripheral devices access to the cloud desktop and the operational permissions they will have after accessing the cloud desktop, thus making the peripheral devices available and improving the user experience.

[0067] In an optional embodiment, before creating a corresponding virtual camera for the camera peripheral of the cloud desktop terminal, the method further includes:

[0068] After logging into the cloud desktop on the cloud desktop terminal, the device receives video peripheral information sent by the cloud desktop terminal; wherein, the video peripheral information includes the device name, video capture parameters, and first identification information of at least one of the camera peripherals;

[0069] Based on the video capture parameters of each of the camera peripherals, a video capture parameter gradient table and a first mapping table between the first identification information of the corresponding camera peripheral and the video capture parameter gradient table are constructed.

[0070] The video capture parameters include resolution and sampling frame rate; the video capture parameter gradient table includes a resolution gradient table and a sampling frame rate gradient table.

[0071] The first identification information includes a vendor ID (VID) and a product ID (PID); the first mapping table is used to indicate the mapping relationship between the vendor ID, the product ID, and the resolution gradient table and the sampling frame rate gradient table.

[0072] In this embodiment of the invention, after the cloud desktop terminal successfully logs into the cloud desktop, the cloud desktop terminal queries the video peripheral information of all local and external camera peripherals, including device name, supported resolution, supported sampling frame rate, VID, PID and other information, and sends the enumerated video peripheral information to the cloud desktop.

[0073] After receiving video peripheral information from the cloud desktop terminal, the cloud desktop enumerates all retrieved video peripheral information in the cloud desktop camera peripheral information bar, displaying information such as the device name, VID, and PID of the camera peripheral for the user to select. The user can then select the desired camera peripheral for redirection from the cloud desktop camera peripheral information bar.

[0074] The cloud desktop constructs a resolution gradient table and a sampling frame rate gradient table based on the multiple resolutions and sampling frame rates supported by each camera peripheral on the cloud desktop terminal. Simultaneously, it constructs a first mapping table from the VID and PID fields in the camera peripheral information to these two gradient tables for subsequent resolution and sampling frame rate adaptive lookups. The resolution gradient table indicates the multiple resolutions supported by the camera peripheral, and the resolutions recorded in the resolution gradient table are sorted in a predetermined order (e.g., from high to low). The sampling frame rate gradient table indicates the multiple sampling frame rates supported by the camera peripheral, and the sampling frame rates recorded in the sampling frame rate gradient table are sorted in a predetermined order (e.g., from high to low). Specifically, based on one or both of the VID and PID fields, the resolution gradient table and sampling frame rate gradient table for the corresponding camera peripheral can be found by querying the first mapping table. Then, by using the retrieved resolution gradient table and sampling frame rate gradient table, the supported resolutions and sampling frame rates of the corresponding camera peripheral can be determined.

[0075] In this embodiment of the invention, during the device enumeration stage, a resolution gradient table, a sampling frame rate gradient table, and a first mapping table from VID and PID to the two gradient tables are constructed for each camera peripheral. This can transform the hardware capabilities of the camera peripheral into manageable, schedulable, and adaptable digital assets, enabling accurate identification of camera peripheral parameters and reducing the management complexity of camera peripherals.

[0076] Furthermore, the method also includes:

[0077] Query the resolution and sampling frame rate of each of the aforementioned camera peripherals;

[0078] If any of the camera peripherals does not support the preset default resolution and default sampling frame rate, the resolution gradient table and sampling frame rate gradient table of the corresponding camera peripheral are queried according to the first identification information of the corresponding camera peripheral.

[0079] Based on the retrieved resolution gradient table and sampling frame rate gradient table, calculate the amount of data that can be transmitted under different combinations of resolution and sampling frame rate;

[0080] Select the resolution and sampling frame rate combination that is closest to the preset default data volume as the current configuration resolution and sampling frame rate of the corresponding camera peripheral;

[0081] If any of the camera peripherals supports the default resolution and the default sampling frame rate, the default resolution and the default sampling frame rate shall be used as the resolution and sampling frame rate currently configured for the corresponding camera peripheral.

[0082] For example, assume that the default resolution of the camera peripheral is 640x480 and the default sampling frame rate is 30FPS.

[0083] If the camera peripheral does not support the set default resolution and default sampling frame rate, the cloud desktop queries the VID and PID corresponding to the camera peripheral, obtains the resolution gradient table and sampling frame rate gradient table corresponding to the VID and PID based on the first mapping table mentioned above, and reads the table entries of the resolution gradient table and sampling frame rate gradient table corresponding to the VID and PID, calculates the product of resolution and sampling frame rate under different combinations of resolution and sampling frame rate, and uses it as the amount of data transmitted per second under the corresponding combination of resolution and sampling frame rate.

[0084] For different combinations of resolution and sampling frame rate, the resolution and sampling frame rate combination that is closest to the default data volume is taken as the new default resolution and default sampling frame rate, and then updated to the camera peripheral to update the current default resolution and default sampling frame rate of the camera peripheral to the new default resolution and default sampling frame rate.

[0085] If the camera peripheral supports the default resolution and default sampling frame rate settings, then maintain the current configuration default resolution (e.g., 640x480) and default sampling frame rate (e.g., 30FPS) of the camera peripheral.

[0086] The table below shows the resolution and sampling frame rate of a certain camera peripheral. It provides the data volume under five resolutions (640x480, 352x288, 320x240, 176x144, and 160x120) and two sampling frame rates (30FPS and 15FPS). The camera peripheral supports the default resolution of 640x480 and the default sampling frame rate of 30FPS. If not supported, the data volume under the following resolution and frame rate combinations are calculated sequentially: 640x480x15, 352x288x30, 320x240x30, 352x288x15, and 320x240x15. The closest resolution and sampling frame rate combination to the default data volume is then taken. For example, 640x480x15 is used as the resolution and frame rate configuration for sampling when the camera peripheral is running.

[0087]

[0088] This invention allows for the pre-configuration of the resolution and sampling frame rate of camera peripherals, maximizing their hardware capabilities. In cases where the default configuration is not supported, the configuration can be updated downwards to avoid resource waste while ensuring video quality and improving the user experience.

[0089] In one optional embodiment, creating a corresponding virtual camera for the camera peripheral of the cloud desktop terminal includes:

[0090] A virtual camera with the same resolution is generated based on the camera peripheral selected by the user on the cloud desktop terminal;

[0091] Construct a second mapping table between the first identifier information of the user-selected camera peripheral and the second identifier information of the corresponding virtual camera.

[0092] For example, after the camera peripheral is configured, the cloud desktop will generate a virtual camera with the same resolution as the camera peripheral selected by the user, and construct a second mapping table from the VID and PID of the camera peripheral to the second identification information of the virtual camera (i.e., the virtual camera ID) for subsequent video data forwarding.

[0093] The cloud desktop allocates two data buffers for each camera peripheral selected by the user, forming a buffer group. A third identifier is assigned to each buffer group. The buffer group includes a video data buffer and a frame loss recovery buffer. Each camera peripheral corresponds to one video data buffer and one frame loss recovery buffer. The video data buffer stores the video data (including image data sequences) captured by the camera peripheral, while the frame loss recovery buffer stores the latest video data frame (i.e., the latest frame of image data).

[0094] After a virtual camera is successfully created, a third mapping table is established between the second identification information of each virtual camera and the third identification information of a buffer group including a video data buffer and a frame loss recovery buffer. The virtual camera directly extracts and displays the video data collected by the camera peripheral from the video data buffer. If network lag causes frame loss and the data in the video data buffer is not updated, the latest video data frame from the frame loss recovery buffer is read as the video data to be processed.

[0095] This invention avoids data confusion among multiple devices by creating a video data buffer and a frame loss recovery buffer for each camera peripheral. Simultaneously, by directly associating the buffer group with the virtual camera's identifier, the complexity and latency of cloud desktop data scheduling are reduced. The virtual camera directly retrieves and displays the video data collected by the camera peripheral from the video data buffer. If network lag causes frame loss and the data in the video data buffer is not updated, the latest video data frame from the frame loss recovery buffer is read as the video data to be processed. The latest frame from the frame loss recovery buffer temporarily substitutes for the data, preventing black screens, freezes, or jumps, thus reducing the sense of lag and ensuring video continuity.

[0096] In one optional embodiment, displaying the video data collected by the camera peripheral through the virtual camera includes:

[0097] Construct a mapping relationship between each virtual camera and its corresponding camera peripheral buffer group; wherein, the buffer group includes a video data buffer and a frame loss recovery buffer;

[0098] Frame drop detection is performed on the video data frames currently acquired by the camera peripheral;

[0099] When no frame loss is detected, the currently acquired video data frame is saved to the video data buffer, and the currently acquired video data frame is read from the video data buffer and displayed through the virtual camera;

[0100] When frame loss is detected, the lost video data frame is predicted based on the latest video data frame in the frame loss recovery buffer, and the predicted video data frame is displayed through the virtual camera.

[0101] Specifically, the step of predicting the lost video data frames based on the latest video data frame in the frame loss recovery buffer, and displaying the predicted video data frames through the virtual camera, includes:

[0102] Based on the latest video data frame in the frame loss recovery buffer, the lost video data frame is predicted using a pre-trained video prediction module;

[0103] The predicted video data frames are updated to the frame loss recovery buffer to replace the latest video data frames in the frame loss recovery buffer.

[0104] The predicted video data frames are read from the frame loss recovery buffer using the virtual camera and then displayed.

[0105] In this embodiment of the invention, the cloud desktop terminal encodes the video data currently collected by the camera peripheral to obtain corresponding data packets and sends them to the cloud desktop. It is understood that in this embodiment of the invention, the cloud desktop terminal encodes and packages each video data frame (i.e., one frame of image data) collected by the camera peripheral and sends it to the cloud desktop.

[0106] After receiving the data packet, the cloud desktop starts the video frame drop detection process, which is as follows:

[0107] Each video data frame (which can also be described as an image data frame) sent by the cloud desktop terminal records the sequence number of the data packet in the sequence number of the data packet header. The video frame loss detection process decodes and parses the received data packets, reads the field content of the data packet header, that is, the sequence number of the data packet, and combines it with the sequence number of the data packet of the previously sent video data frame to determine whether the currently sent video data frame has arrived in order.

[0108] If the currently sent video data frames arrive in order, the video data frame is stored in the video data buffer of the corresponding camera peripheral, and a copy is made to update the frame loss recovery buffer, so that the frame loss recovery buffer caches the latest video data frame. At this time, the virtual camera driver directly reads the video data from the video data buffer for display;

[0109] If the currently sent video data frame times out or arrives out of order, the cloud desktop will use the video data frame in the frame loss recovery buffer as input to the pre-trained video prediction model based on a convolutional neural network. It will then use the lost video data frames in the video prediction model to make predictions, output the predicted video data frame, and update the video data buffer and frame loss recovery buffer with the predicted video data frame as the latest video data frame for the virtual camera to read and display.

[0110] It should be noted that the network architecture and training methods of video prediction models built on convolutional neural networks are existing technologies. For example, the video prediction model adopts the network architecture of the SimVP (Simpler yet Better Video Prediction) model and uses the time series of historical video data frames as training samples for model training, which will not be elaborated here.

[0111] In this embodiment of the invention, the pre-trained video prediction module generates predicted video data frames that conform to the scene logic based on historical frames (the latest video data frames in the frame loss recovery buffer). This avoids repeated frame drops and makes the display more realistic and dynamic, improving visual smoothness. Simultaneously, the predicted video data frames update the frame loss recovery buffer, ensuring that the buffer always contains valid frames close to the current time. If frame drops continue, new frames can be generated based on the updated video data frames, preventing the scene from becoming disconnected from the actual scene due to repeated use of the same old frames. Furthermore, when the network is continuously lagging, the predicted video data frames continuously supplement the video stream, preventing the screen from freezing or going black for extended periods. Even if the network does not recover, basic dynamic display effects can be maintained, adapting to more complex weak network environments.

[0112] In one optional embodiment, the step of dynamically adjusting the video capture parameters of the camera peripheral based on the real-time monitored network packet loss rate and the video capture parameter gradient table of the camera peripheral includes:

[0113] Real-time monitoring of network packet loss rate, and comparison of the network packet loss rate of multiple transmissions with a preset packet loss rate threshold;

[0114] When the network packet loss rate is greater than or equal to the packet loss rate threshold, the resolution gradient table and sampling frame rate gradient table of the corresponding camera peripheral are queried according to the first identification information carried by the currently received video data frame.

[0115] Based on the retrieved resolution gradient table and sampling frame rate gradient table, calculate the amount of data that can be transmitted under different combinations of resolution and sampling frame rate;

[0116] Different combinations of resolution and sampling frame rate are sorted from highest to lowest according to the amount of data;

[0117] Select the resolution and sampling frame rate combination that is one bit lower than the current resolution and sampling frame rate combination of the corresponding camera peripheral, and update the current resolution and sampling frame rate of the corresponding camera peripheral.

[0118] For example, during video data transmission, the cloud desktop initiates a resolution and sampling frame rate adaptive process, as follows:

[0119] The network packet loss rate is monitored in real time. The network packet loss rate of multiple transmissions is selected as the current network packet loss rate metric. For example, the network packet loss rate of the 50 transmissions before the current time is selected, that is, the percentage of data packets that were not successfully received in the 50 transmissions out of the total number of data packets sent, and compared with the preset packet loss rate threshold (e.g., 5%). If the network packet loss rate is greater than or equal to the preset packet loss rate threshold, the VID and PID fields in the header of the latest received video data frame are read. The corresponding resolution gradient table and sampling frame rate gradient table are then queried by referring to the first mapping table of the two gradient tables (VID and PID fields). Next, the entries in the resolution gradient table and sampling frame rate gradient table are queried to calculate the data volume under various resolution and frame rate combinations. These combinations are then sorted in descending order of data volume. The resolution and sampling frame rate combination with the lowest data volume (ranked one position lower than the currently configured combination) is selected as the adaptive sampling resolution and sampling frame rate obtained by the cloud desktop. This adaptive sampling resolution and sampling frame rate are then fed back to the cloud desktop terminal in real time via the network. The cloud desktop terminal updates the configuration of the corresponding camera peripherals using the new sampling resolution and sampling frame rate, enabling the camera device to acquire images according to the adaptive sampling resolution and sampling frame rate obtained by the cloud desktop. This reduces network resource consumption and achieves dynamic adjustment of the terminal device's sampling resolution and sampling frame rate under different network conditions.

[0120] Furthermore, the method also includes:

[0121] Feature extraction and image reconstruction are performed on the currently acquired video data; wherein, the currently acquired video data is obtained by the cloud desktop terminal using a pre-trained deep learning module to perform image enhancement processing on the video data currently acquired by the camera peripheral.

[0122] In this embodiment of the invention, after the cloud desktop terminal collects video data from the camera peripheral, it uses a deep learning model deployed on the cloud desktop terminal to perform image enhancement processing on the currently collected video data frames, and then encodes and packages the image-enhanced video data frames and sends them to the cloud desktop.

[0123] For example, in the image enhancement process, video data frames can undergo data preprocessing such as standardization, color space calibration, and frame size adaptation to the model's input size. The preprocessed video data frames are then input into a deep learning model (such as ResNet, EDSR, RCAN, etc.). By extracting low-level features (edges, textures) and high-level semantic features (object structure, scene context), specific optimizations are performed based on these features to address image defects (such as low light, blur, and noise), such as enhancing the luminance channel, suppressing noise, improving sharpness, and calibrating the color gamut. The output is then an enhanced video data frame. The network structure and training process of the deep learning model are existing technologies and will not be detailed here. For instance, the original image data can be used as samples, and the enhanced image data as labels, to train the deep learning model.

[0124] After receiving the data packet sent by the cloud desktop, the cloud desktop can obtain the video data frame after encoding and parsing. Then, through efficient feature extraction and image reconstruction technology, the video data frame is repaired in detail. It can process multiple video data frames at the same time, achieving a significant improvement in picture quality and overcoming the problem of dim and unclear images from camera peripherals in low-light environments.

[0125] For example, in the process of feature extraction and image reconstruction, low-level features and high-level semantic features of video data frames can be extracted through feature engineering. Then, based on the extracted low-level features and high-level semantic features, a generator (such as a GAN generator) is used to supplement high-frequency information of blurred details of the video data frames through texture transfer (based on low-level features). Local packet loss areas are filled in with content through context matching (based on high-level semantic features), resulting in reconstructed and repaired video data. The reconstructed and repaired video data is then combined with the original video data frames for boundary fusion and sharpening post-processing to improve detail contrast, resulting in the final video data frame with complete details.

[0126] In this embodiment of the invention, the VID and PID of the camera peripheral and the sequence number of the data packet are written into the header of the video data frame sent from the cloud desktop terminal to the server-side cloud desktop. The video data frame after image enhancement processing is then encoded. The encoded data is directly sent to the cloud desktop through the peripheral redirection channel. The cloud desktop receives the transmitted data packet, decompresses it, reads the VID and PID header fields, and obtains the two gradient tables and the third identification information of the corresponding buffer groups based on the mapping of VID and PID to the two gradient tables and the second identification information of the virtual camera. Then, the video frame loss detection process performs packet loss detection and buffer distribution on the data packet. Based on the real-time network packet loss rate of the cloud desktop, the optimal resolution and sampling frame rate parameters are updated through the resolution and sampling frame rate adaptive process. This information is fed back to the cloud desktop terminal through the peripheral redirection channel, enabling the cloud desktop terminal to dynamically adjust the resolution and sampling frame rate parameters of the camera peripheral. Figure 2 As shown, the camera's resolution and sampling frame rate can be intelligently adjusted in weak network environments to reduce interface lag and screen tearing caused by data packet loss, thereby improving the user experience.

[0127] See Figure 3 , Figure 3 This is a structural block diagram of a camera redirection device provided in an embodiment of the present invention. The camera redirection device includes:

[0128] The virtual camera creation module 11 is used to create corresponding virtual cameras for the camera peripherals of the cloud desktop terminal;

[0129] The video data display module 12 is used to display the video data collected by the camera peripheral through the virtual camera;

[0130] The parameter dynamic adjustment module 13 is used to dynamically adjust the video capture parameters of the camera peripheral based on the real-time monitored network packet loss rate and the video capture parameter gradient table of the camera peripheral.

[0131] In one optional embodiment, the video data display module 12 includes:

[0132] A mapping relationship construction unit is used to construct a mapping relationship between each virtual camera and the corresponding camera peripheral buffer group; wherein, the buffer group includes a video data buffer and a frame loss recovery buffer;

[0133] The frame loss detection unit is used to detect frame loss in the video data frames currently acquired by the camera peripheral.

[0134] The first reading and display unit is used to save the currently acquired video data frame to the video data buffer when no frame loss is detected, and to read the currently acquired video data frame from the video data buffer through the virtual camera for display.

[0135] The second reading and display unit is used to predict the lost video data frame based on the latest video data frame in the frame loss recovery buffer when frame loss is detected, and to display the predicted video data frame through the virtual camera.

[0136] In one optional embodiment, the second reading display unit includes:

[0137] The frame loss prediction subunit is used to predict the lost video data frames based on the latest video data frame in the frame loss recovery buffer using a pre-trained video prediction module.

[0138] The frame loss update subunit is used to update the predicted video data frame to the frame loss recovery buffer, so as to replace the latest video data frame in the frame loss recovery buffer.

[0139] The frame loss reading and display subunit is used to read the predicted video data frames from the frame loss recovery buffer through the virtual camera and display them.

[0140] In an optional embodiment, the device further includes:

[0141] The video peripheral information receiving module is used to receive video peripheral information sent by the cloud desktop terminal after the cloud desktop terminal logs in to the cloud desktop before creating a corresponding virtual camera for the camera peripheral of the cloud desktop terminal; wherein, the video peripheral information includes at least one device name, video capture parameters, and first identification information of the camera peripheral;

[0142] The mapping table construction module is used to construct a video capture parameter gradient table and a first mapping table between the first identification information of the corresponding camera peripheral and the video capture parameter gradient table according to the video capture parameters of each of the camera peripherals.

[0143] In one optional embodiment, the video capture parameters include resolution and sampling frame rate; the video capture parameter gradient table includes: a resolution gradient table and a sampling frame rate gradient table;

[0144] The first identification information includes a manufacturer identification code and a product identification code; the first mapping table is used to indicate the mapping relationship between the manufacturer identification code, the product identification code and the resolution gradient table and the sampling frame rate gradient table.

[0145] In an optional embodiment, the device further includes:

[0146] The first query module is used to query the resolution and sampling frame rate of each of the aforementioned camera peripherals;

[0147] The second query module is used to query the resolution gradient table and sampling frame rate gradient table of the corresponding camera peripheral according to the first identification information of the corresponding camera peripheral when any of the camera peripherals does not support the preset default resolution and default sampling frame rate.

[0148] The data volume calculation module is used to calculate the amount of data that can be transmitted under different combinations of resolution and sampling frame rate based on the queried resolution gradient table and sampling frame rate gradient table.

[0149] The first configuration module is used to select the resolution and sampling frame rate combination corresponding to the data volume that is closest to the preset default data volume, as the current configuration resolution and sampling frame rate of the corresponding camera peripheral.

[0150] The second configuration module is used to, when any of the camera peripherals supports the default resolution and the default sampling frame rate, use the default resolution and the default sampling frame rate as the currently configured resolution and sampling frame rate of the corresponding camera peripheral.

[0151] In an optional embodiment, the virtual camera creation module 11 includes:

[0152] The virtual camera generation unit is used to generate a virtual camera with the same resolution based on the camera peripheral selected by the user of the cloud desktop terminal.

[0153] The mapping table creation unit is used to construct a second mapping table between the first identification information of the user-selected camera peripheral and the second identification information of the corresponding virtual camera.

[0154] In one optional embodiment, the parameter dynamic adjustment module includes:

[0155] The packet loss rate monitoring unit is used to monitor the network packet loss rate in real time and compare the network packet loss rate of multiple transmissions with a preset packet loss rate threshold.

[0156] The gradient table lookup unit is used to look up the resolution gradient table and sampling frame rate gradient table of the corresponding camera peripheral based on the first identification information carried by the currently received video data frame when the network packet loss rate is greater than or equal to the packet loss rate threshold.

[0157] The calculation unit is used to calculate the amount of data that can be transmitted under different combinations of resolution and sampling frame rate based on the retrieved resolution gradient table and sampling frame rate gradient table.

[0158] The sorting unit is used to sort different combinations of resolutions and sampling frame rates according to the amount of data from high to low.

[0159] The resolution and sampling frame rate combination selection unit is used to select the resolution and sampling frame rate combination that is one bit lower than the resolution and sampling frame rate combination currently configured for the corresponding camera peripheral, and update the resolution and sampling frame rate currently configured for the corresponding camera peripheral.

[0160] In an optional embodiment, the device further includes:

[0161] The video data processing module is used to perform feature extraction and image reconstruction processing on the currently acquired video data; wherein, the currently acquired video data is obtained by the cloud desktop terminal using a pre-trained deep learning module to perform image enhancement processing on the video data currently acquired by the camera peripheral.

[0162] It should be noted that the working process of each module in the camera redirection device described in the embodiments of the present invention can refer to the working process of the camera redirection method described in the above embodiments, and the technical effect achieved is the same as that of the camera redirection method described in the above embodiments, so it will not be repeated here.

[0163] See Figure 4 , Figure 4 This is a structural block diagram of a camera redirection device provided in an embodiment of the present invention. The camera redirection device includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the various camera redirection method embodiments described above, such as steps S11 to S13.

[0164] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the camera redirection device.

[0165] The camera redirection device may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of a camera redirection device and does not constitute a limitation on the device. It may include more or fewer components than illustrated, or combine certain components, or use different components. For example, the camera redirection device may also include input / output devices, network access devices, buses, etc.

[0166] The processor 21 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the camera redirection device, connecting various parts of the entire camera redirection device via various interfaces and lines.

[0167] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the camera redirection device by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0168] If the integrated modules / units of the camera redirection device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 21, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0169] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0170] The above description represents the preferred embodiments of the present invention. It should be noted that, for those skilled in the art, various improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A camera redirection method, characterized in that, include: Create corresponding virtual cameras for the camera peripherals of the cloud desktop terminal; The video data collected by the camera peripheral is displayed through the virtual camera; Based on the real-time monitored network packet loss rate and the gradient table of the video capture parameters of the camera peripheral, the video capture parameters of the camera peripheral are dynamically adjusted; the video capture parameters include resolution and sampling frame rate. The video capture parameter gradient table includes: a resolution gradient table and a sampling frame rate gradient table; The step of dynamically adjusting the video capture parameters of the camera peripheral based on the real-time monitored network packet loss rate and the gradient table of the camera peripheral's video capture parameters includes: Real-time monitoring of network packet loss rate, and comparison of the network packet loss rate of multiple transmissions with a preset packet loss rate threshold; When the network packet loss rate is greater than or equal to the packet loss rate threshold, the resolution gradient table and sampling frame rate gradient table of the corresponding camera peripheral are queried according to the first identification information carried by the currently received video data frame. Based on the retrieved resolution gradient table and sampling frame rate gradient table, calculate the amount of data that can be transmitted under different combinations of resolution and sampling frame rate; Different combinations of resolution and sampling frame rate are sorted from highest to lowest according to the amount of data; Select the resolution and sampling frame rate combination that is one bit lower than the current resolution and sampling frame rate combination of the corresponding camera peripheral, and update the current resolution and sampling frame rate of the corresponding camera peripheral.

2. The camera redirection method as described in claim 1, characterized in that, The step of displaying video data collected by the camera peripheral through the virtual camera includes: Construct a mapping relationship between each virtual camera and its corresponding camera peripheral buffer group; wherein, the buffer group includes a video data buffer and a frame loss recovery buffer; Frame drop detection is performed on the video data frames currently acquired by the camera peripheral; When no frame loss is detected, the currently acquired video data frame is saved to the video data buffer, and the currently acquired video data frame is read from the video data buffer and displayed through the virtual camera; When frame loss is detected, the lost video data frame is predicted based on the latest video data frame in the frame loss recovery buffer, and the predicted video data frame is displayed through the virtual camera.

3. The camera redirection method as described in claim 2, characterized in that, The step of predicting lost video data frames based on the latest video data frame in the frame loss recovery buffer, and displaying the predicted video data frames through the virtual camera, includes: Based on the latest video data frame in the frame loss recovery buffer, the lost video data frame is predicted using a pre-trained video prediction module; The predicted video data frames are updated to the frame loss recovery buffer to replace the latest video data frames in the frame loss recovery buffer. The predicted video data frames are read from the frame loss recovery buffer using the virtual camera and then displayed.

4. The camera redirection method as described in claim 1, characterized in that, Before creating a corresponding virtual camera for the camera peripheral on the cloud desktop terminal, the method further includes: After logging into the cloud desktop on the cloud desktop terminal, the device receives video peripheral information sent by the cloud desktop terminal; wherein, the video peripheral information includes the device name, video capture parameters, and first identification information of at least one of the camera peripherals; Based on the video capture parameters of each of the camera peripherals, a video capture parameter gradient table and a first mapping table between the first identification information of the corresponding camera peripheral and the video capture parameter gradient table are constructed.

5. The camera redirection method as described in claim 4, characterized in that, The first identification information includes a manufacturer identification code and a product identification code; the first mapping table is used to indicate the mapping relationship between the manufacturer identification code, the product identification code and the resolution gradient table and the sampling frame rate gradient table.

6. The camera redirection method as described in claim 5, characterized in that, The method further includes: Query the resolution and sampling frame rate of each of the aforementioned camera peripherals; If any of the camera peripherals does not support the preset default resolution and default sampling frame rate, the resolution gradient table and sampling frame rate gradient table of the corresponding camera peripheral are queried according to the first identification information of the corresponding camera peripheral. Based on the retrieved resolution gradient table and sampling frame rate gradient table, calculate the amount of data that can be transmitted under different combinations of resolution and sampling frame rate; Select the resolution and sampling frame rate combination that is closest to the preset default data volume as the current configuration resolution and sampling frame rate of the corresponding camera peripheral; If any of the camera peripherals supports the default resolution and the default sampling frame rate, the default resolution and the default sampling frame rate shall be used as the resolution and sampling frame rate currently configured for the corresponding camera peripheral.

7. The camera redirection method as described in claim 1, characterized in that, The creation of a corresponding virtual camera for the camera peripheral of the cloud desktop terminal includes: A virtual camera with the same resolution is generated based on the camera peripheral selected by the user on the cloud desktop terminal; Construct a second mapping table between the first identifier information of the user-selected camera peripheral and the second identifier information of the corresponding virtual camera.

8. The camera redirection method as described in claim 1, characterized in that, The method further includes: Feature extraction and image reconstruction are performed on the currently acquired video data; wherein, the currently acquired video data is obtained by the cloud desktop terminal using a pre-trained deep learning module to perform image enhancement processing on the video data currently acquired by the camera peripheral.

9. A camera redirection device, characterized in that, include: The virtual camera creation module is used to create corresponding virtual cameras for the camera peripherals of the cloud desktop terminal; A video data display module is used to display video data collected by the camera peripheral through the virtual camera; The parameter dynamic adjustment module is used to dynamically adjust the video capture parameters of the camera peripheral based on the real-time monitored network packet loss rate and the video capture parameter gradient table of the camera peripheral; the video capture parameters include resolution and sampling frame rate; The video capture parameter gradient table includes: a resolution gradient table and a sampling frame rate gradient table; The parameter dynamic adjustment module includes: The packet loss rate monitoring unit is used to monitor the network packet loss rate in real time and compare the network packet loss rate of multiple transmissions with a preset packet loss rate threshold. The gradient table lookup unit is used to look up the resolution gradient table and sampling frame rate gradient table of the corresponding camera peripheral based on the first identification information carried by the currently received video data frame when the network packet loss rate is greater than or equal to the packet loss rate threshold. The calculation unit is used to calculate the amount of data that can be transmitted under different combinations of resolution and sampling frame rate based on the retrieved resolution gradient table and sampling frame rate gradient table. The sorting unit is used to sort different combinations of resolutions and sampling frame rates according to the amount of data from high to low. The resolution and sampling frame rate combination selection unit is used to select the resolution and sampling frame rate combination that is one bit lower than the resolution and sampling frame rate combination currently configured for the corresponding camera peripheral, and update the resolution and sampling frame rate currently configured for the corresponding camera peripheral.

10. A camera redirection device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the camera redirection method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the camera redirection method as described in any one of claims 1 to 8.

12. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the camera redirection method according to any one of claims 1 to 8.

Images