Video data transmission method and apparatus

Abstract

The present application provides a video data transmission method and an apparatus. The video data transmission method of the present application comprises: a main control device sends first information to a USB video capturing device; the USB video capturing device switches the resolution of a first switching frame to a target resolution on the basis of the first information; the USB video capturing device encodes the first switching frame on the basis of the target resolution, so as to obtain a code stream of the first switching frame; the USB video capturing device sends second information to the main control device; the main control device determines the first switching frame on the basis of the second information; and the main control device sends the code stream of the first switching frame to a network. The present application enables the transmission code rate to adapt to changes of network conditions, and transmits videos with the highest possible quality under the support of the network conditions, thus improving user experience.

Description

Video data transmission method and device Technical Field

[0001] The present application relates to Universal Serial Bus (USB) technology, and in particular to a method and apparatus for transmitting video data. Background Art

[0002] USB video capture devices (e.g., USB cameras) can be used in video call scenarios such as personal, home, and office use. The device collects video data and then transmits it via USB to a host device such as a mobile phone, personal computer (PC), or conference terminal. The host device then sends the video stream to the remote video call device over the network.

[0003] To adapt to network bandwidth, the USB video capture device can transmit video streams corresponding to multiple resolutions to the host device based on the USB Video Device Class (UVC) protocol. The host device then determines the transmission stream to be sent to the remote end based on the current network bandwidth.

[0004] However, the above method has high requirements on the performance of the USB video capture device, which will increase hardware cost and power consumption. In addition, the transmission of multiple video streams is limited by the USB bandwidth, and additional control of the bit rate of the transmitted video streams is required. Summary of the Invention

[0005] The present application provides a method and device for transmitting video data, which can not only adapt the transmission bit rate to changes in network conditions, but also transmit video with the highest possible quality when supported by the network conditions, thereby improving the user experience.

[0006] In a first aspect, the present application provides a method for transmitting video data, comprising: a main control device sends first information to a USB video capture device; the USB video capture device switches the resolution of a first switching frame to a target resolution based on the first information; the USB video capture device encodes the first switching frame according to the target resolution to obtain a code stream of the first switching frame; the USB video capture device sends second information to the main control device; the main control device determines the first switching frame based on the second information; and the main control device sends the code stream of the first switching frame to a network.

[0007] In this embodiment, by transmitting indication information for switching resolution, the video resolution is switched as the network conditions change. This allows the transmission bit rate to adapt to changes in network conditions and allows the highest possible quality video to be transmitted when the network conditions support it, thereby improving the user experience.

[0008] The main control device can instruct the USB video capture device to switch the resolution of the video frame to the target resolution through the first information, wherein the target resolution can be determined by the main control device based on network transmission status information.

[0009] In an embodiment of the present application, the main control device can determine whether it is necessary to switch the resolution of the transmitted video based on the transmission status of the video stream in the network. For example, if the current network transmission status improves, you can consider increasing the resolution of the video, or if the current network transmission status deteriorates, you can consider lowering the resolution of the video. In this way, as the network transmission status changes, the video resolution is adjusted in real time, which can not only allow the transmission bit rate to adapt to changes in the network status, but also transmit the highest possible quality video when the network status supports it, so as to improve the user experience.

[0010] In one possible implementation, the network transmission status information includes network bandwidth and transmission bit rate. The master control device can periodically query the network bandwidth and transmission bit rate and determine whether to switch the resolution of the transmitted video based on the real-time values ​​of the two. For example, if the network bandwidth is 30 MB and the transmission bit rate is 40 MB / s, the transmission bit rate is greater than the network bandwidth, and the video resolution can be reduced. Alternatively, if the network bandwidth is 50 MB and the transmission bit rate is 40 MB / s, the transmission bit rate is less than the network bandwidth, and the video resolution can be increased. Alternatively, if the network bandwidth is 30 MB and the transmission bit rate is 40 MB / s, the transmission bit rate is greater than the network bandwidth, and the difference between the transmission bit rate and the network bandwidth is 10 MB, which is greater than a preset threshold (e.g., 5 MB, not specifically limited), the video resolution can be reduced. Alternatively, if the network bandwidth is 30 MB and the transmission bit rate is 33 MB / s, the transmission bit rate is greater than the network bandwidth, and the difference between the transmission bit rate and the network bandwidth is 3 MB, which is less than a preset threshold (e.g., 5 MB, not specifically limited), the video resolution can be left unchanged. Alternatively, if the network bandwidth is 30 MB and the transmission bit rate is 20 MB / s, the transmission bit rate is less than the network bandwidth, and the difference between the transmission bit rate and the network bandwidth is 10 MB, which is greater than a preset threshold (e.g., 5 MB, not specifically limited), the video resolution can be increased. In addition, the master control device may also set other judgment conditions, which are not specifically limited in the embodiments of the present application.

[0011] In one possible implementation, the network transmission status information includes network bandwidth, transmission bit rate and stream cache information. In addition to the conditions described above, the master control device can also add stream cache information to the judgment conditions. For example, the network bandwidth is 30MB, the transmission bit rate is 40MB / s, there is no video stream or only a small amount of video stream in the cache (for example, the data volume of the video stream is less than the threshold). At this time, the cache can alleviate the network fluctuations that may be caused by the transmission stream being greater than the network bandwidth. Therefore, the video resolution can be unchanged, thereby avoiding the freeze caused by the resolution switching of the video. In addition, the master control device can also set other judgment conditions, which are not specifically limited in the embodiments of the present application.

[0012] In an embodiment of the present application, the master control device can determine the target resolution based on the above-mentioned network transmission status information. Through the interaction between the USB video capture device and the master control device, the master control device can learn the processing capabilities of the USB video capture device. For example, the highest video quality that the USB video capture device can capture is 4K video, and it can also capture 1080P, 720P, or 480P video, or the USB video capture device can downsample the captured 4K video to 1080P, 720P, or 480P video. In this case, the master control device can determine that the video resolution can be switched between 4K, 1080P, 720P, and 480P.

[0013] Alternatively, the master device can use a step-by-step resolution switching method. This method involves first determining whether to lower or increase the resolution, and then setting the target resolution to one level lower or higher than the current video resolution. For example, if the current video resolution is 720P, the target resolution would be 1080P if the resolution is lowered, and 480P if the resolution is increased. This method is slower but more cautious in switching video resolutions, and can avoid video freezes and other issues caused by resolution switching.

[0014] Optionally, the master device can implement a skip-level switching method based on network transmission status information. That is, based on the network transmission status information, the master device determines whether to reduce or increase the resolution while also determining which resolution the current network conditions are suitable for transmitting. For example, if the current video resolution is 720P, the master device can determine to increase the resolution from 720P to 4K (the target resolution). This method makes the video resolution switching process more direct, can improve video transmission efficiency, and can adapt to network conditions more quickly.

[0015] In one possible implementation, the host device can send the first information to the USB video capture device via a socket channel. This is a new channel (socket channel) established between the host device and the USB video capture device, in addition to transmitting the video stream over the USB based on the UVC protocol. This allows the solution of the embodiment of the present application to be implemented without changing the UVC protocol.

[0016] The master control device may directly write the target resolution into the first information, or may write the identifier or index of the target resolution into the first information, which is not specifically limited in the embodiment of the present application.

[0017] In one possible implementation, a USB video capture device captures multiple video frames at a first resolution that is higher than a target resolution; and downsamples a resolution of a first switching frame from the first resolution to the target resolution, wherein the multiple video frames include the first switching frame.

[0018] The USB video capture device can always capture video at the highest resolution. When it receives the first information from the main control device, it downsamples the captured video frames according to the target resolution indicated by the first information to obtain video frames of the target resolution. Thereafter, before receiving the next first information, the USB video capture device can downsample the captured video frames to the target resolution. In the aforementioned process, the first video frame in which the resolution is switched can be called a switching frame (corresponding to the above-mentioned first switching frame). This places lower hardware requirements on the USB video capture device, and it does not need to have multi-resolution acquisition capabilities. The downsampling operation has lower computing power requirements, will not cause excessive consumption of the USB video capture device, and will not be limited by the USB bandwidth.

[0019] In a possible implementation, the USB video capture device captures one or more video frames at a target resolution starting from the first switching frame.

[0020] USB video capture devices can have multi-resolution acquisition capabilities, so that the quality of the video they capture can be adjusted according to the target resolution. In addition, USB video capture devices adjust the video resolution in real time, without the need to transmit multiple (corresponding to multiple resolutions) video streams to the host device via USB, and are not limited by USB bandwidth.

[0021] After switching the resolution of the first switching frame, the USB video capture device may encode the first switching frame according to the new resolution (target resolution) and transmit the code stream of the first switching frame at a matching code rate.

[0022] In the embodiment of the present application, the second information includes the following two situations:

[0023] (1) The second information includes the identifier of the first switching frame

[0024] The USB video capture device switches the resolution of the video frame in the first switching frame. To allow the host device to correctly decode the first switching frame, the host device may be informed of the identifier of the first switching frame via the second information. Optionally, the identifier of the first switching frame includes one or more of the following information: a frame sequence number of the first switching frame or a presentation time stamp (PTS) corresponding to the first switching frame. Thus, when the host device processes the video frame corresponding to the identifier of the first switching frame, it can determine that the video frame is the first switching frame and then process the frame according to the new resolution (target resolution).

[0025] In one possible implementation, the USB video capture device can send the second information to the host device via a socket channel. This also establishes a new channel (socket channel) between the host device and the USB video capture device, in addition to transmitting the video stream over USB based on the UVC protocol. This allows the solution of the embodiment of the present application to be implemented without changing the UVC protocol.

[0026] (2) The second information includes a video stream

[0027] The USB video capture device switches the resolution of the video frame in the first switching frame, and may not directly notify the host device through a message. In this case, the second information only includes the video stream, and the host device can determine the first switching frame based on the syntax elements in the video stream. The second information including only the video stream can be sent from the USB video capture device to the host device based on the UVC protocol. Optionally, the aforementioned syntax elements include one or more of the following information: Sequence Parameter Set (SPS) or Picture Parameter Set (PPS).

[0028] The code stream of the first switching frame is obtained by encoding based on the resolution of the first switching frame. After receiving the first switching frame, the master control device can encode it according to the resolution of the first switching frame and transmit the code stream of the first switching frame to the network at a matching code rate.

[0029] In one possible implementation, the main control device can transmit the code stream of the second frame according to the resolution of the first switching frame, where the second frame is a video frame after the first switching frame and before the second switching frame, and the resolution of the second frame is the same as the resolution of the first switching frame. The second switching frame is after the first switching frame and the resolution of the second switching frame is different from the resolution of the first switching frame.

[0030] As described above, the main control device can periodically determine whether the video resolution needs to be switched. Therefore, between two cycles or before processing the next switching frame (the second switching frame), the USB video capture device can encode and send the video frame at the target resolution. Correspondingly, the main control device will also encode the received video frame at the target resolution and transmit it to the network.

[0031] In a second aspect, the present application provides a main control device, comprising: a sending module for sending first information to a USB video capture device, wherein the first information is used to instruct the USB video capture device to switch the resolution of the video frame to a target resolution; a receiving module for receiving second information sent by the USB video capture device; a processing module for determining a first switching frame based on the second information, wherein the resolution of the first switching frame is the target resolution and is different from the resolution of the first frame, and the first frame is the previous frame of the first switching frame; the sending module is also used to send a code stream of the first switching frame to a network, wherein the code stream of the first switching frame is encoded based on the resolution of the first switching frame.

[0032] In a possible implementation manner, the processing module is specifically configured to determine the first switching frame according to the identifier of the first switching frame when the second information includes the identifier of the first switching frame.

[0033] In a possible implementation, the identifier of the first switching frame includes one or more of the following information: a frame sequence number of the first switching frame or a display timestamp PTS corresponding to the first switching frame.

[0034] In a possible implementation, the processing module is specifically configured to, when the second information includes a video stream, determine the first switching frame according to syntax elements in the video stream.

[0035] In a possible implementation manner, the syntax element includes one or more of the following information: SPS or PPS.

[0036] In a possible implementation manner, the first information is sent through a socket channel.

[0037] In a possible implementation manner, the second information is received through a socket channel.

[0038] In a possible implementation, the second information is received based on a UVC protocol.

[0039] In a possible implementation, the processing module is further configured to obtain network transmission status information; and determine the target resolution according to the network transmission status information.

[0040] In a possible implementation, the network transmission status information includes network bandwidth and transmission bit rate.

[0041] In a possible implementation, the network transmission status information further includes stream buffer information.

[0042] In one possible implementation, the sending module is also used to transmit the code stream of the second frame according to the resolution of the first switching frame, where the second frame is a video frame after the first switching frame and before the second switching frame, and the resolution of the second frame is the same as the resolution of the first switching frame. The second switching frame is after the first switching frame, and the resolution of the second switching frame is different from the resolution of the first switching frame.

[0043] In a third aspect, the present application provides a USB video capture device, comprising: a receiving module for receiving first information sent by a main control device, the first information being used to indicate that the resolution of a video frame is switched to a target resolution; a processing module for switching the resolution of a first switching frame to the target resolution according to the first information; encoding the first switching frame according to the target resolution to obtain a code stream of the first switching frame; and a sending module for sending second information to the main control device, the second information including the code stream of the first switching frame.

[0044] In a possible implementation, the sending module is further configured to send third information to the master control device, where the third information includes an identifier of the first switching frame.

[0045] In a possible implementation, the identifier of the first switching frame includes one or more of the following information: a frame sequence number of the first switching frame or a display timestamp PTS corresponding to the first switching frame.

[0046] In a possible implementation, the third information is sent through a socket channel.

[0047] In a possible implementation, the second information is sent based on a UVC protocol.

[0048] In one possible implementation, the processing module is specifically used to capture multiple video frames at a first resolution, where the first resolution is higher than the target resolution; and downsample the resolution of the first switching frame from the first resolution to the target resolution, where the multiple video frames include the first switching frame.

[0049] In a possible implementation, the processing module is specifically configured to capture one or more video frames at the target resolution starting from the first switching frame.

[0050] In a fourth aspect, the present application provides an electronic device comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement a method as described in any one of the above-mentioned first aspects.

[0051] In a fifth aspect, the present application provides a computer-readable storage medium comprising a computer program, wherein when the computer program is executed on a computer, the computer executes the method according to any one of the above-mentioned first aspects.

[0052] In a sixth aspect, the present application provides a computer program product, which includes a computer program code. When the computer program code is run on a computer, the computer executes any one of the methods in the first aspect.

[0053] In a seventh aspect, the present application provides a chip comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute a method as described in any one of the above-mentioned first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] FIG1 is a schematic diagram of an application scenario of an embodiment of the present application;

[0055] FIG2 is a flow chart of a process 200 of a video transmission method provided in an embodiment of the present application;

[0056] FIG3 is a flow chart of a method for transmitting video data according to an embodiment of the present application;

[0057] FIG4 is a flow chart of a method for transmitting video data according to an embodiment of the present application;

[0058] FIG5 is a schematic structural diagram of an electronic device 500 of the present application. DETAILED DESCRIPTION

[0059] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0060] The terms "first," "second," and the like in the description, embodiments, claims, and drawings of this application are used solely for descriptive purposes and are not to be construed as indicating or implying relative importance or order. Furthermore, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions, such as, for example, inclusion of a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0061] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0062] Figure 1 is a schematic diagram of the application scenario of an embodiment of the present application. As shown in Figure 1, in scenarios such as personal, home, and office video calls, live broadcasts, and cloud conferences, a USB video capture device (for example, a USB camera) can be connected to a main control device (for example, a mobile phone, a personal computer (PC) or a conference terminal, etc.) via USB, and the collected video data is transmitted to the main control device via USB; the main control device is connected to the network and sends the video code stream to the remote main control device through the network.

[0063] Between the USB video capture device and the host device, a video stream (also referred to as video data) can be transmitted based on the USB Video Device Class (UVC) protocol.

[0064] Video streams can be transmitted between master control devices on different ends based on network protocols.

[0065] The cloud conferencing application scenario shown in Figure 1 includes venues 1 and 2. Venue 1 is a large conference room. The main control device on the end side is a split-type conference terminal. The high-definition conference camera and omnidirectional microphone are connected to the split-type conference terminal via USB. The video stream captured by the high-definition conference camera is transmitted to the split-type conference terminal based on the UVC protocol. The audio stream captured by the omnidirectional microphone is transmitted to the split-type conference terminal based on the UVC protocol. The split-type conference terminal encodes the video and audio streams and uploads them to the cloud (network), which is then transmitted to venue 2 via the cloud. Venue 2 is a personal conference. The main control device on the end side is a PC. The camera or integrated conference peripheral is connected to the PC via USB. The video stream captured by the camera or integrated conference peripheral is transmitted to the PC based on the UVC protocol. The PC encodes the video stream and uploads it to the cloud (network), which is then transmitted to venue 1 via the cloud.

[0066] It should be noted that Figure 1 only illustrates an application scenario of an embodiment of the present application as an example, but this does not constitute a limitation on the application scenario. The embodiment of the present application can also be applied to other scenarios, and no specific limitation is made to this.

[0067] FIG2 is a flow chart of process 200 of the video transmission method provided in an embodiment of the present application. Process 200 can be performed jointly by the USB video capture device and the host control device described above. Process 200 is described as a series of steps or operations. It should be understood that process 200 can be performed in various orders and / or occur simultaneously, and is not limited to the execution order shown in FIG2. Process 200 may include:

[0068] Step 201: The main control device sends first information to the USB video capture device.

[0069] The main control device can instruct the USB video capture device to switch the resolution of the video frame to the target resolution through the first information, wherein the target resolution can be determined by the main control device based on network transmission status information.

[0070] In an embodiment of the present application, the main control device can determine whether it is necessary to switch the resolution of the transmitted video based on the transmission status of the video stream in the network. For example, if the current network transmission status improves, you can consider increasing the resolution of the video, or if the current network transmission status deteriorates, you can consider lowering the resolution of the video. In this way, as the network transmission status changes, the video resolution is adjusted in real time, which can not only allow the transmission bit rate to adapt to changes in the network status, but also transmit the highest possible quality video when the network status supports it, so as to improve the user experience.

[0071] In one possible implementation, the network transmission status information includes network bandwidth and transmission bit rate. The master control device can periodically query the network bandwidth and transmission bit rate and determine whether to switch the resolution of the transmitted video based on the real-time values ​​of the two. For example, if the network bandwidth is 30 MB and the transmission bit rate is 40 MB / s, the transmission bit rate is greater than the network bandwidth, and the video resolution can be reduced. Alternatively, if the network bandwidth is 50 MB and the transmission bit rate is 40 MB / s, the transmission bit rate is less than the network bandwidth, and the video resolution can be increased. Alternatively, if the network bandwidth is 30 MB and the transmission bit rate is 40 MB / s, the transmission bit rate is greater than the network bandwidth, and the difference between the transmission bit rate and the network bandwidth is 10 MB, which is greater than a preset threshold (e.g., 5 MB, not specifically limited), the video resolution can be reduced. Alternatively, if the network bandwidth is 30 MB and the transmission bit rate is 33 MB / s, the transmission bit rate is greater than the network bandwidth, and the difference between the transmission bit rate and the network bandwidth is 3 MB, which is less than a preset threshold (e.g., 5 MB, not specifically limited), the video resolution can be left unchanged. Alternatively, if the network bandwidth is 30 MB and the transmission bit rate is 20 MB / s, the transmission bit rate is less than the network bandwidth, and the difference between the transmission bit rate and the network bandwidth is 10 MB, which is greater than a preset threshold (e.g., 5 MB, not specifically limited), the video resolution can be increased. In addition, the master control device may also set other judgment conditions, which are not specifically limited in the embodiments of the present application.

[0072] In one possible implementation, the network transmission status information includes network bandwidth, transmission bit rate and stream cache information. In addition to the conditions described above, the master control device can also add stream cache information to the judgment conditions. For example, the network bandwidth is 30MB, the transmission bit rate is 40MB / s, there is no video stream or only a small amount of video stream in the cache (for example, the data volume of the video stream is less than the threshold). At this time, the cache can alleviate the network fluctuations that may be caused by the transmission stream being greater than the network bandwidth. Therefore, the video resolution can be unchanged, thereby avoiding the freeze caused by the resolution switching of the video. In addition, the master control device can also set other judgment conditions, which are not specifically limited in the embodiments of the present application.

[0073] In an embodiment of the present application, the master control device can determine the target resolution based on the above-mentioned network transmission status information. Through the interaction between the USB video capture device and the master control device, the master control device can learn the processing capabilities of the USB video capture device. For example, the highest video quality that the USB video capture device can capture is 4K video, and it can also capture 1080P, 720P, or 480P video, or the USB video capture device can downsample the captured 4K video to 1080P, 720P, or 480P video. In this case, the master control device can determine that the video resolution can be switched between 4K, 1080P, 720P, and 480P.

[0074] Alternatively, the master device can use a step-by-step resolution switching method. This method involves first determining whether to lower or increase the resolution, and then setting the target resolution to one level lower or higher than the current video resolution. For example, if the current video resolution is 720P, the target resolution would be 1080P if the resolution is lowered, and 480P if the resolution is increased. This method is slower but more cautious in switching video resolutions, and can avoid video freezes and other issues caused by resolution switching.

[0075] Optionally, the master device can implement a skip-level switching method based on network transmission status information. That is, based on the network transmission status information, the master device determines whether to reduce or increase the resolution while also determining which resolution the current network conditions are suitable for transmitting. For example, if the current video resolution is 720P, the master device can determine to increase the resolution from 720P to 4K (the target resolution). This method makes the video resolution switching process more direct, can improve video transmission efficiency, and can adapt to network conditions more quickly.

[0076] In one possible implementation, the host device can send the first information to the USB video capture device via a socket channel. This is a new channel (socket channel) established between the host device and the USB video capture device, in addition to transmitting the video stream over the USB based on the UVC protocol. This allows the solution of the embodiment of the present application to be implemented without changing the UVC protocol.

[0077] The master control device may directly write the target resolution into the first information, or may write the identifier or index of the target resolution into the first information, which is not specifically limited in the embodiment of the present application.

[0078] Step 202: The USB video capture device switches the resolution of the first switching frame to the target resolution according to the first information.

[0079] In one possible implementation, a USB video capture device captures multiple video frames at a first resolution that is higher than a target resolution; and downsamples a resolution of a first switching frame from the first resolution to the target resolution, wherein the multiple video frames include the first switching frame.

[0080] The USB video capture device can always capture video at the highest resolution. When it receives the first information from the main control device, it downsamples the captured video frames according to the target resolution indicated by the first information to obtain video frames of the target resolution. Thereafter, before receiving the next first information, the USB video capture device can downsample the captured video frames to the target resolution. In the aforementioned process, the first video frame in which the resolution is switched can be called a switching frame (corresponding to the above-mentioned first switching frame). This places lower hardware requirements on the USB video capture device, and it does not need to have multi-resolution acquisition capabilities. The downsampling operation has lower computing power requirements, will not cause excessive consumption of the USB video capture device, and will not be limited by the USB bandwidth.

[0081] In a possible implementation, the USB video capture device captures one or more video frames at a target resolution starting from the first switching frame.

[0082] USB video capture devices can have multi-resolution acquisition capabilities, so that the quality of the video they capture can be adjusted according to the target resolution. In addition, USB video capture devices adjust the video resolution in real time, without the need to transmit multiple (corresponding to multiple resolutions) video streams to the host device via USB, and are not limited by USB bandwidth.

[0083] Step 203: The USB video capture device encodes the first switching frame according to the target resolution to obtain a code stream of the first switching frame.

[0084] After switching the resolution of the first switching frame, the USB video capture device may encode the first switching frame according to the new resolution (target resolution) and transmit the code stream of the first switching frame at a matching code rate.

[0085] Step 204: The USB video capture device sends second information to the main control device.

[0086] Step 205: The master control device determines a first switching frame according to the second information.

[0087] In the embodiment of the present application, the second information includes the following two situations:

[0088] (1) The second information includes the identifier of the first switching frame

[0089] The USB video capture device switches the resolution of the video frame in the first switching frame. To allow the host device to correctly decode the first switching frame, the host device may be informed of the identifier of the first switching frame via the second information. Optionally, the identifier of the first switching frame includes one or more of the following information: a frame sequence number of the first switching frame or a presentation time stamp (PTS) corresponding to the first switching frame. Thus, when the host device processes the video frame corresponding to the identifier of the first switching frame, it can determine that the video frame is the first switching frame and then process the frame according to the new resolution (target resolution).

[0090] In one possible implementation, the USB video capture device can send the second information to the host device via a socket channel. This also establishes a new channel (socket channel) between the host device and the USB video capture device, in addition to transmitting the video stream over USB based on the UVC protocol. This allows the solution of the embodiment of the present application to be implemented without changing the UVC protocol.

[0091] (2) The second information includes a video stream

[0092] When the USB video capture device switches the video frame resolution in the first switching frame, it may not directly notify the host device via a message. In this case, the second information only includes the video stream, and the host device can determine the first switching frame based on syntax elements in the video stream. The second information including only the video stream can be sent from the USB video capture device to the host device based on the UVC protocol. Optionally, the syntax elements include one or more of the following: SPS or PPS.

[0093] Step 206: The master control device sends the code stream of the first switching frame to the network.

[0094] The code stream of the first switching frame is obtained by encoding based on the resolution of the first switching frame. After receiving the first switching frame, the master control device can encode it according to the resolution of the first switching frame and transmit the code stream of the first switching frame to the network at a matching code rate.

[0095] In one possible implementation, the main control device can transmit the code stream of the second frame according to the resolution of the first switching frame, where the second frame is a video frame after the first switching frame and before the second switching frame, and the resolution of the second frame is the same as the resolution of the first switching frame. The second switching frame is after the first switching frame and the resolution of the second switching frame is different from the resolution of the first switching frame.

[0096] As described above, the main control device can periodically determine whether the video resolution needs to be switched. Therefore, between two cycles or before processing the next switching frame (the second switching frame), the USB video capture device can encode and send the video frame at the target resolution. Correspondingly, the main control device will also encode the received video frame at the target resolution and transmit it to the network.

[0097] In this embodiment, by transmitting indication information for switching resolution, the video resolution is switched as the network conditions change. This allows the transmission bit rate to adapt to changes in network conditions and allows the highest possible quality video to be transmitted when the network conditions support it, thereby improving the user experience.

[0098] The following uses several specific embodiments to explain in detail the technical solution of the method embodiment shown in FIG2 .

[0099] Figure 3 is a flowchart of the video data transmission method of an embodiment of the present application. As shown in Figure 3, the USB camera (USB Device) is connected to the PC (USB Host) via USB. The USB camera includes socket communication, video encoder, and UVC data transmission. The PC includes socket communication, UVC data transmission, and call services.

[0100] The PC determines the resolution change strategy based on the upstream network status and cached data. If the network is good, the resolution is increased; if the network is poor, the resolution is reduced. For example, if the network is poor, the resolution is reduced from 1080P to 720P. This resolution change information is sent to the USB camera via socket communication.

[0101] In a USB camera, the video encoder encodes the captured video based on the frame resolution to generate a video stream (frame data). When a resolution-switching frame is encountered, the frame number of the switching frame is recorded. In one possible implementation, the USB camera sends the video stream to the PC using the UVC protocol, and includes the resolution switching information in the syntax elements of the video stream. In another possible implementation, the USB camera sends the frame number of the switching frame to the PC via socket communication.

[0102] In order to implement the call service in the PC, the socket information is parsed to obtain the frame number of the switching frame. When the switching frame is processed, the switching frame is encoded according to the new resolution and sent to the remote network.

[0103] In this embodiment, by transmitting indication information for switching resolution, the video resolution is switched as the network conditions change. This allows the transmission bit rate to adapt to changes in network conditions and allows the highest possible quality video to be transmitted when the network conditions support it, thereby improving the user experience.

[0104] FIG4 is a flow chart of a method for transmitting video data according to an embodiment of the present application. As shown in FIG4 , a USB camera (USB device) is connected to a PC (USB host) via a USB, and the method includes the following steps:

[0105] S1. The master device controls the USB camera through the USB connection and establishes a socket channel;

[0106] Compared with the related art, in addition to transmitting video data based on the UVC protocol between the USB camera and the main control device, the embodiment of the present application also creates a socket channel to transmit information related to resolution switching, so that the solution of the embodiment of the present application can be implemented without changing the UVC protocol.

[0107] S2. Configure the UVC transmission properties of the USB camera according to the high-resolution requirements and configure the adaptive resolution properties;

[0108] In response to control commands from a host device, the USB camera can configure UVC transmission properties according to high-resolution requirements, enabling high-quality video transmission to the remote end and providing a high-quality video playback experience. Furthermore, the USB camera also has adaptive resolution properties, responding to resolution switching commands from the host device and changing the video frame resolution in real time to adapt to network transmission conditions.

[0109] S3. The USB camera configures video parameters and collects code, which is then transmitted via the UVC protocol.

[0110] S4. The master device determines the uplink network status and stream buffering status and decides on video attribute switching, such as resolution, based on the network status and stream buffering. For example, if the network improves, the resolution will be increased from 720P to 1080P.

[0111] This step can refer to step 201 in the above embodiment and will not be described again here.

[0112] S5. The master device transmits the video quality attribute change information through the socket channel;

[0113] The main control device notifies the USB camera to switch the resolution through the socket channel, so that the solution of the embodiment of the present application can be implemented without changing the UVC protocol.

[0114] S6.USB camera analyzes video quality attribute information, for example: resolution is increased from 720P to 1080P;

[0115] S7.USB camera timing remains unchanged, and resolution switching is achieved through post-scaling;

[0116] In this embodiment, the USB video capture device can always capture video at the highest resolution. When receiving the first information from the main control device, the captured video frame is downsampled according to the target resolution indicated by the first information to obtain a video frame of the target resolution. Thereafter, before receiving the next first information, the USB video capture device can downsample the captured video frame to the target resolution. In the aforementioned process, the first video frame in which the resolution is switched can be called a switching frame (corresponding to the above-mentioned first switching frame). This places lower hardware requirements on the USB video capture device, and it does not need to have the ability to capture multiple resolutions. The downsampling operation has lower computing power requirements, does not cause excessive consumption of the USB video capture device, and is not limited by the USB bandwidth.

[0117] S8. When the USB camera switches frame resolution, stop 720P encoding, start 1080P encoding, and record the frame number or PTS of the switching frame;

[0118] S9. The USB camera notifies the main control device through the socket channel to switch the frame number or PTS;

[0119] S10. The master device parses the socket channel information to obtain the switched video attributes, the switching frame number or PTS;

[0120] S11. The master device receives the switched video stream based on the UVC protocol;

[0121] S12. The master device matches the received video stream according to the frame number or PTS, finds the switching frame and modifies the corresponding video quality attributes and uploads it to the network;

[0122] S13. As the network transmission status of the master device changes, the video quality attributes also change accordingly, and the previous steps are repeated.

[0123] In this embodiment, the main control device and the USB camera can repeatedly execute the process of S1-S12 to adapt to changes in network transmission conditions. In this way, once the network fluctuates and the current network transmission condition improves, you can consider increasing the video resolution, or, if the current network transmission condition deteriorates, you can consider reducing the video resolution. In this way, as the network transmission condition changes, the video resolution is adjusted in real time, which can not only allow the transmission bit rate to adapt to changes in network conditions, but also transmit the highest possible quality video when the network conditions support it, thereby improving the user experience.

[0124] FIG5 is a schematic diagram of the structure of the electronic device 500 of the present application. As shown in FIG5, the electronic device 500 of this embodiment can be applied to the above-mentioned master device or USB video capture device. The electronic device 500 may include: a sending module 501, a processing module 502 and a receiving module 503. When used as a master device,

[0125] A sending module 501 is used to send first information to a USB video capture device, where the first information is used to instruct the USB video capture device to switch the resolution of the video frame to a target resolution; a receiving module 503 is used to receive second information sent by the USB video capture device; a processing module 502 is used to determine a first switching frame based on the second information, where the resolution of the first switching frame is the target resolution and is different from the resolution of the first frame, and the first frame is the previous frame of the first switching frame; the sending module 501 is also used to send a code stream of the first switching frame to a network, where the code stream of the first switching frame is obtained by encoding based on the resolution of the first switching frame.

[0126] In a possible implementation, the processing module 502 is specifically configured to determine the first switching frame according to the identifier of the first switching frame when the second information includes the identifier of the first switching frame.

[0127] In a possible implementation, the identifier of the first switching frame includes one or more of the following information: a frame sequence number of the first switching frame or a display timestamp PTS corresponding to the first switching frame.

[0128] In a possible implementation, the processing module 502 is specifically configured to, when the second information includes a video stream, determine the first switching frame according to syntax elements in the video stream.

[0129] In a possible implementation manner, the syntax element includes one or more of the following information: SPS or PPS.

[0130] In a possible implementation manner, the first information is sent through a socket channel.

[0131] In a possible implementation manner, the second information is received through a socket channel.

[0132] In a possible implementation, the second information is received based on a UVC protocol.

[0133] In a possible implementation, the processing module 502 is further configured to obtain network transmission status information; and determine the target resolution according to the network transmission status information.

[0134] In a possible implementation, the network transmission status information includes network bandwidth and transmission bit rate.

[0135] In a possible implementation, the network transmission status information further includes stream buffer information.

[0136] In one possible implementation, the sending module 501 is also used to transmit the code stream of the second frame according to the resolution of the first switching frame, where the second frame is a video frame after the first switching frame and before the second switching frame, and the resolution of the second frame is the same as the resolution of the first switching frame. The second switching frame is after the first switching frame, and the resolution of the second switching frame is different from the resolution of the first switching frame.

[0137] When used as a USB video capture device, the receiving module 503 is used to receive first information sent by the main control device, where the first information is used to indicate that the resolution of the video frame is switched to the target resolution; the processing module 502 is used to switch the resolution of the first switching frame to the target resolution according to the first information; the first switching frame is encoded according to the target resolution to obtain a code stream of the first switching frame; and the sending module 501 is used to send second information to the main control device, where the second information includes the code stream of the first switching frame.

[0138] In a possible implementation, the sending module 501 is further configured to send third information to the master control device, where the third information includes an identifier of the first switching frame.

[0139] In a possible implementation, the identifier of the first switching frame includes one or more of the following information: a frame sequence number of the first switching frame or a display timestamp PTS corresponding to the first switching frame.

[0140] In a possible implementation, the third information is sent through a socket channel.

[0141] In a possible implementation, the second information is sent based on a UVC protocol.

[0142] In one possible implementation, the processing module 502 is specifically used to capture multiple video frames at a first resolution, where the first resolution is higher than the target resolution; and downsample the resolution of the first switching frame from the first resolution to the target resolution, where the multiple video frames include the first switching frame.

[0143] In a possible implementation, the processing module 502 is specifically configured to capture one or more video frames at the target resolution starting from the first switching frame.

[0144] The device of this embodiment can be used to execute the technical solution of the method embodiment shown in Figure 2. Its implementation principle and technical effects are similar and will not be repeated here.

[0145] During implementation, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware coding processor, or can be executed by a combination of hardware and software modules in the coding processor. The software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0146] The memory mentioned in the above embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0147] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0148] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0149] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0150] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0151] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0152] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0153] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for transmitting video data, characterized in that: include: Sending first information to a universal serial bus (USB) video capture device, where the first information is used to instruct the USB video capture device to switch the resolution of the video frame to a target resolution; receiving second information sent by the USB video capture device; determining a first switching frame according to the second information, wherein a resolution of the first switching frame is the target resolution and is different from a resolution of the first frame, and the first frame is a previous frame of the first switching frame; A code stream of the first switching frame is sent to a network, where the code stream of the first switching frame is obtained by encoding based on the resolution of the first switching frame.

2. The method according to claim 1, characterized in that The determining the first switching frame according to the second information includes: When the second information includes the identifier of the first switching frame, the first switching frame is determined according to the identifier of the first switching frame.

3. The method according to claim 2, characterized in that The identifier of the first switching frame includes one or more of the following information: a frame sequence number of the first switching frame or a display timestamp PTS corresponding to the first switching frame.

4. The method according to claim 1, wherein The determining the first switching frame according to the second information includes: When the second information includes a video stream, the first switching frame is determined according to syntax elements in the video stream.

5. The method according to claim 4, characterized in that The syntax element includes one or more of the following information: a sequence parameter set SPS or a picture parameter set PPS.

6. The method according to any one of claims 1 to 5, characterized in that The first information is sent through a socket channel.

7. The method according to claim 2 or 3, characterized in that The second information is received through a socket channel.

8. The method according to claim 4 or 5, characterized in that The second information is received based on the USB Video Class UVC protocol.

9. The method according to any one of claims 1 to 8, characterized in that Also includes: Get network transmission status information; The target resolution is determined according to the network transmission status information.

10. The method according to claim 9, characterized in that The network transmission status information includes network bandwidth and transmission bit rate.

11. The method according to claim 10, characterized in that The network transmission status information also includes stream buffer information.

12. The method according to any one of claims 1 to 11, characterized in that Also includes: The code stream of the second frame is transmitted according to the resolution of the first switching frame, the second frame is a video frame after the first switching frame and before the second switching frame, the resolution of the second frame is the same as the resolution of the first switching frame, the second switching frame is after the first switching frame, and the resolution of the second switching frame is different from the resolution of the first switching frame.

13. A method for transmitting video data, characterized in that: include: receiving first information sent by a master control device, where the first information is used to instruct to switch the resolution of the video frame to a target resolution; Switching the resolution of the first switching frame to the target resolution according to the first information; Encoding the first switching frame according to the target resolution to obtain a code stream of the first switching frame; Second information is sent to the master control device, where the second information includes a code stream of the first switching frame.

14. The method according to claim 13, wherein: Also includes: Sending third information to the master control device, where the third information includes an identifier of the first switching frame.

15. The method according to claim 14, characterized in that The identifier of the first switching frame includes one or more of the following information: a frame sequence number of the first switching frame or a display timestamp PTS corresponding to the first switching frame.

16. The method according to claim 14 or 15, characterized in that The third information is sent through the socket channel.

17. The method according to any one of claims 13 to 16, characterized in that The second information is sent based on the USB Video Class UVC protocol.

18. The method according to any one of claims 13 to 17, characterized in that The step of switching the resolution of the first switching frame to the target resolution according to the first information includes: capturing a plurality of video frames at a first resolution, the first resolution being higher than the target resolution; A resolution of the first switching frame is downsampled from the first resolution to the target resolution, the plurality of video frames including the first switching frame.

19. The method according to any one of claims 13 to 17, characterized in that The step of switching the resolution of the first switching frame to the target resolution according to the first information includes: One or more video frames are captured at the target resolution starting from the first switching frame.

20. A main control device, characterized in that: include: a sending module, configured to send first information to a universal serial bus (USB) video capture device, wherein the first information is used to instruct the USB video capture device to switch the resolution of the video frame to a target resolution; A receiving module, configured to receive second information sent by the USB video capture device; a processing module, configured to determine a first switching frame according to the second information, wherein a resolution of the first switching frame is the target resolution and is different from a resolution of the first frame, and the first frame is a previous frame of the first switching frame; The sending module is further configured to send a code stream of the first switching frame to a network, where the code stream of the first switching frame is obtained by encoding based on the resolution of the first switching frame.

21. The device according to claim 20, characterized in that The processing module is specifically configured to determine the first switching frame according to the identifier of the first switching frame when the second information includes the identifier of the first switching frame.

22. The device according to claim 21, characterized in that The identifier of the first switching frame includes one or more of the following information: a frame sequence number of the first switching frame or a display timestamp PTS corresponding to the first switching frame.

23. The device according to claim 20, characterized in that The processing module is specifically configured to determine the first switching frame according to syntax elements in the video code stream when the second information includes a video code stream.

24. The device according to claim 23, characterized in that The syntax element includes one or more of the following information: a sequence parameter set SPS or a picture parameter set PPS.

25. The device according to any one of claims 20 to 24, characterized in that The first information is sent through a socket channel.

26. The device according to claim 21 or 22, characterized in that The second information is received through a socket channel.

27. The device according to claim 23 or 24, characterized in that The second information is received based on the USB Video Class UVC protocol.

28. The device according to any one of claims 20 to 27, characterized in that The processing module is further configured to obtain network transmission status information; and determine the target resolution according to the network transmission status information.

29. The device according to claim 28, characterized in that The network transmission status information includes network bandwidth and transmission bit rate.

30. The device according to claim 29, characterized in that The network transmission status information also includes stream buffer information.

31. The device according to any one of claims 20 to 30, characterized in that The sending module is also used to transmit the code stream of the second frame according to the resolution of the first switching frame, where the second frame is a video frame after the first switching frame and before the second switching frame, and the resolution of the second frame is the same as the resolution of the first switching frame. The second switching frame is after the first switching frame, and the resolution of the second switching frame is different from the resolution of the first switching frame.

32. A USB video capture device, characterized in that: include: A receiving module, configured to receive first information sent by a master control device, wherein the first information is used to instruct to switch the resolution of the video frame to a target resolution; a processing module, configured to switch the resolution of the first switching frame to the target resolution according to the first information; Encoding the first switching frame according to the target resolution to obtain a code stream of the first switching frame; A sending module is used to send second information to the main control device, where the second information includes a code stream of the first switching frame.

33. The device according to claim 32, characterized in that The sending module is further configured to send third information to the master control device, where the third information includes an identifier of the first switching frame.

34. The device according to claim 33, characterized in that The identifier of the first switching frame includes one or more of the following information: a frame sequence number of the first switching frame or a display timestamp PTS corresponding to the first switching frame.

35. The device according to claim 33 or 34, characterized in that The third information is sent through the socket channel.

36. The device according to any one of claims 32 to 35, characterized in that The second information is sent based on the USB Video Class UVC protocol.

37. The device according to any one of claims 32 to 36, characterized in that The processing module is specifically configured to capture multiple video frames at a first resolution, where the first resolution is higher than the target resolution; and downsample the resolution of the first switching frame from the first resolution to the target resolution, wherein the multiple video frames include the first switching frame.

38. The device according to any one of claims 32 to 36, characterized in that The processing module is specifically configured to capture one or more video frames at the target resolution starting from the first switching frame.

39. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 19.

40. A computer-readable storage medium, characterized in that The invention comprises a computer program which, when executed on a computer, causes the computer to perform the method according to any one of claims 1 to 19.

41. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on a computer, the computer is caused to perform the method according to any one of claims 1 to 19.

42. A chip, characterized in that: The system comprises a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 19.

Images