Video live broadcast method, system, device and storage medium based on drone

Abstract

The embodiment of the present invention discloses a method, system, device and storage medium for live video broadcast based on drones. The method includes: obtaining original video stream data sent by drones; encoding and decoding the original video stream data to obtain output buffer data; converting the output buffer data into video frames supported by the WebRtc protocol; sending the video frames to a preset server through a WebRtc transmission channel, so that the preset server performs live broadcast display according to the video frames. The embodiment of the present invention uses a terminal to realize data transmission through a WebRtc transmission channel, which reduces the delay of the live broadcast picture, and there is no need to set a WebRtc conversion device and a network communication device on the drone, which reduces the hardware cost of the drone, and the terminal-based data conversion solution has a mature underlying architecture that is easy to implement, low development cost, and can transform the existing drone system with only software modification, and has a wide range of application scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle data transmission, and in particular to a video live broadcast method, system, device and storage medium based on an unmanned aerial vehicle. Background Art

[0002] With the development of drone technology, its application scenarios are becoming more and more extensive, and it has played a huge role in scenarios such as power line inspection, traffic rescue, and customs and border defense. The expansion of application scenarios has put higher requirements on the performance of drones. For example, for tasks that require drones to transmit live images in real time, due to the operating characteristics and operating environment of drones, the live broadcast delay is often very high, thus affecting the completion of the task.

[0003] At present, in the application scenarios where live broadcast is required, there are two main live broadcast methods for drone systems: one is to place the client and drone used to view live broadcast in the same network, but in this case, only the client in the network can watch the live broadcast, and the video transmission method usually adopts HLS, HLV and other protocols, which has poor real-time performance and high latency; the other is to send the original video shot by the drone to the terminal device installed with live broadcast software, and use the terminal device to realize live broadcast, but the live broadcast software usually uses TCP as the transport layer protocol during live broadcast. Once packet loss occurs, all subsequent packets will be buffered, causing live broadcast jams and delays, and accidents are prone to occur when drone flight requires real-time control. Therefore, a new live broadcast method for drones is needed to ensure low latency and low jams in drone live broadcast images. Summary of the invention

[0004] In view of this, the present invention provides a drone-based live video broadcast method, system, device and storage medium, so as to perform drone-based live video broadcast with the help of WebRtc protocol, reduce live broadcast delay and avoid freeze.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a method for live video broadcasting based on a drone, the method comprising:

[0007] Get the original video stream data sent by the drone;

[0008] Encoding and decoding the original video stream data to obtain output buffer data;

[0009] Convert the output buffer data into video frames supported by the WebRtc protocol;

[0010] The video frame is sent to a preset server through a WebRtc transmission channel, so that the preset server performs live display according to the video frame.

[0011] Optionally, in some embodiments, the original video stream data includes H264 data, and encoding and decoding the original video stream data to obtain output buffer data includes:

[0012] Writing the H264 data into the codec input buffer and submitting it to the codec;

[0013] The codec encodes and decodes the H264 data to generate ByteBuffer video stream data and writes the ByteBuffer video stream data into the codec output buffer.

[0014] Optionally, in some embodiments, converting the output buffer data into a video frame supported by the WebRtc protocol includes:

[0015] Read the ByteBuffer video stream data from the codec output buffer;

[0016] The ByteBuffer video stream data is decoded and converted to obtain VideoFrame data.

[0017] Optionally, in some embodiments, before sending the video frame to a preset server through the WebRtc transmission channel, the method further includes:

[0018] Detect network signal strength;

[0019] Determine the current resolution according to the first rule based on the network signal strength;

[0020] The video stream resolution of the video frame is adjusted according to the current resolution.

[0021] Optionally, in some embodiments, determining the current resolution according to the network signal strength according to a first rule includes:

[0022] Determining whether the network signal strength is greater than or equal to a first signal strength;

[0023] If it is greater than or equal to the first signal strength, the current resolution is determined to be 1920x1080p;

[0024] If it is less than the first signal strength, it is determined that the current resolution is 1080x720p.

[0025] Optionally, in some embodiments, before obtaining the original video stream data sent by the drone, the method further includes:

[0026] Detect network signal strength;

[0027] Determine the video stream bit rate according to the network signal strength according to a second rule;

[0028] The video stream bit rate is sent to the drone, so that the drone sends the original video stream data based on the video stream bit rate.

[0029] Optionally, in some embodiments, the WebRtc transmission channel includes an RTCPeerConnection channel and / or an RTCDataChannel channel.

[0030] In a second aspect, the present invention provides a page guiding device based on a QR code, comprising:

[0031] The data receiving module is used to obtain the original video stream data sent by the drone;

[0032] A coding and decoding module, used for coding and decoding the original video stream data to obtain output buffer data;

[0033] A data conversion module, used for converting the output buffer data into video frames supported by the WebRtc protocol;

[0034] The WebRtc transmission module is used to send the video frame to a preset server through a WebRtc transmission channel, so that the preset server performs live display according to the video frame.

[0035] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the computer program, it implements the drone-based video live broadcast method provided in any embodiment of the present invention.

[0036] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and wherein the program instructions, when executed, implement the drone-based live video broadcasting method provided in any embodiment of the present invention.

[0037] Compared with the prior art, the video live broadcast method based on a drone provided in an embodiment of the present invention is applied to a terminal that maintains communication with a drone, firstly obtains the original video stream data sent by the drone, then encodes and decodes the original video stream data to obtain output buffer data, performs data conversion on the output buffer data to obtain video frames supported by the WebRtc protocol, and finally sends the video frames to a preset server through a WebRtc transmission channel, and the preset server performs live broadcast display according to the video frames. During the live broadcast of the drone, the method realizes real-time live broadcast based on the WebRtc protocol with the help of the terminal, which greatly reduces the delay of the live broadcast picture, and there is no need to set a WebRtc conversion device and a network communication device on the drone, which reduces the hardware cost of the drone, and the computing performance of the terminal also far exceeds the computing power requirement of data conversion during the live broadcast of the drone, and the underlying architecture of the data conversion based on the terminal is mature and easy to implement, with low development cost, and the existing drone system can be transformed with the help of software, and the application scenarios are wide. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0039] Figure 1 is a flow chart of a video live broadcast method based on a drone provided in Embodiment 1 of the present invention;

[0040] Figure 2 is a sub-flow chart of the video live broadcast method based on drone provided in Embodiment 1 of the present invention;

[0041] Figure 3 is a sub-flow chart of the video live broadcast method based on drone provided in Embodiment 1 of the present invention;

[0042] Figure 4 is a flow chart of a method for live video broadcasting based on a drone provided in Embodiment 2 of the present invention;

[0043] Figure 5 is a sub-flow chart of a video live broadcast method based on a drone provided in Embodiment 2 of the present invention;

[0044] Figure 6 is a flow chart of a method for live video broadcasting based on a drone provided in Embodiment 2 of the present invention;

[0045] Figure 7 1 is a schematic diagram of the structure of a video live broadcast system based on a drone provided in Embodiment 3 of the present invention;

[0046] Figure 8 It is a structural schematic diagram of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0047] The following is a clear and complete description of the technical solutions in the implementation of this application in conjunction with the drawings in the embodiments of the present application. It is understandable that the specific embodiments described herein are only part of the embodiments of the present application, rather than all of the embodiments, and are only used to explain the present application, rather than to limit the present application. It should also be noted that all other embodiments obtained by ordinary technicians in this field without creative work based on the embodiments in the present application belong to the scope of protection of this application.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0049] In addition, the terms "first", "second", etc. may be used in this article to describe various directions, actions, steps or elements, etc., but these directions, actions, steps or elements are not limited by these terms. These terms are only used to distinguish a first direction, action, step or element from another direction, action, step or element. For example, without departing from the scope of the present invention, the first use case may be referred to as the second use case, and similarly, the second use case may be referred to as the first use case. Both the first use case and the second use case are use cases, but they are not the same use case. The terms "first", "second", etc. cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include a combination of one or more features. In the description of the present invention, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. It should be noted that when a part is referred to as "fixed to" another part, it may be directly on the other part or there may be a central part. When a part is considered to be "connected" to another part, it may be directly connected to the other part or there may be a central part at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only and do not represent the only implementations.

[0050] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0051] Embodiment 1

[0052] See also Figure 1 This embodiment provides a method for live video broadcast based on a drone. The method can be applied to a drone system, which includes a drone, a terminal and a preset server, wherein: the drone is an unmanned aircraft with a mission payload that is controlled by a remote control device or a self-contained program control device; the terminal is an electronic device connected to the remote control device, which communicates with the drone through the remote control device and is used for data transmission relay between the drone and the preset server, usually using mobile phones, tablets and other devices; the preset server maintains a communication connection with the terminal. The method for live video broadcast based on a drone provided in this embodiment can be executed by a drone or a terminal, or it can be completed by the interaction between the terminal and the drone. In this embodiment, the terminal is taken as an example for specific description. Figure 1 As shown, the method comprises the following steps:

[0053] S110: Obtain original video stream data sent by the drone.

[0054] In this embodiment, a flight control module and an image acquisition module are provided on the drone, wherein the flight control module is used to monitor and control the flight and mission of the drone, and the image acquisition module is used to acquire images during the flight of the drone. The original video stream data is the data acquired by the image acquisition module and transmitted to the terminal.

[0055] Specifically, in this embodiment, the image acquisition module is connected to the remote control device through a dedicated communication link (usually a radio communication link established through the antenna of the remote control device and the drone). The data collected by the image acquisition module is first transmitted by the drone to the remote control device, and then transmitted by the remote control device to the terminal. The original video stream data finally obtained by the terminal is usually H264 data. In some alternative embodiments, the original video stream data can also use other data formats, which are not limited here.

[0056] S120: Encode and decode the original video stream data to obtain output buffer data.

[0057] The output buffer data is a special encapsulation of the original video stream data by the multimedia codec, so that it can be further converted into special data that can be transmitted through the WebRtc transmission channel (i.e., the video frames supported by the WebRtc protocol referred to below). Different from the prior art, in this embodiment, when performing drone-based video live broadcast, the WebRtc protocol is used as the data transmission protocol between the terminal and the preset server. It does not require the installation of any plug-ins. Through a simple JavaScript API, real-time communication can become a standard function, and there will be no problem of live broadcast jamming due to a problem with one message causing all messages to go wrong. However, in the prior art, the WebRtc protocol directly processes and transmits the images captured by the camera when applied, while in this embodiment, the images captured by the drone need to be transmitted to the terminal through a dedicated communication link. The terminal obtains the original video stream data, which cannot be directly transmitted through the WebRtc protocol. Therefore, in this embodiment, the data in the original video needs to be encoded and decoded to obtain data that complies with the WebRtc protocol. Specifically, the encoding and decoding of the original video stream data in step S120 is actually to convert it into a highly efficient byte-processing encapsulation class data, that is, output buffer data. This is because when the original video stream data is encoded at the bottom layer to make it conform to the WebRtc protocol, bytes are processed. If the operation is performed simply with the help of byte arrays, first of all, the efficiency is very problematic, and secondly, the biggest problem with the array is that it reports exceptions and other information when crossing the boundary. Therefore, a highly efficient byte-processing encapsulation class is used. And since the data still needs to be further processed later, the output buffer data is placed in the output buffer area after the codec processing is completed for subsequent use.

[0058] Specifically, in some embodiments, when the terminal is an Android mobile phone or tablet device, taking the original video stream data including H264 data as an example, the encoding and decoding process is as follows: Figure 2 As shown, steps S121-122 are included:

[0059] S121, writing the H264 data into a codec input buffer and submitting it to the codec.

[0060] In this embodiment, the codec uses the codec provided by MediaCodec. MediaCodec is part of the Android low-level multimedia architecture and is usually used in conjunction with MediaExtractor, MediaMuxer, and AudioTrack. It can encode and decode common audio and video formats such as H.264, H.265, AAC, and 3gp.

[0061] S122, the codec encodes and decodes the H264 data to generate ByteBuffer video stream data and writes the ByteBuffer video stream data into the codec output buffer area.

[0062] Specifically, MediaCodec uses a set of input / output buffers to process data synchronously or asynchronously during the encoding and decoding process: first, the terminal writes the data to be encoded and decoded (that is, the original video stream data) to the acquired encoding and decoding input buffer and submits it to the codec. After the codec is processed, it generates ByteBuffer video stream data, transfers the ByteBuffer video stream data to the encoding output buffer, and at the same time reclaims the client's ownership of the input buffer; then, the terminal reads the encoded data from the acquired encoding and decoding output buffer for processing (not the content in step S122), and after the processing is completed, the codec reclaims the client's ownership of the output buffer.

[0063] S130: Convert the output buffer data into video frames supported by the WebRtc protocol.

[0064] The video frame is a VideoFrame data object supported by the WebRtc protocol, which can be pushed through the push streaming interface provided by WebRtc. The output buffer data obtained in step S120 can only achieve efficient data processing, and still cannot be directly transmitted through the WebRtc protocol. Therefore, it is further converted through step S130: the terminal continuously reads the output buffer data from the codec output buffer area for conversion to obtain video frames that can be transmitted through the WebRtc protocol.

[0065] Specifically, continuing to use the above example of using an Android mobile phone or tablet device as a terminal, in some embodiments, the conversion process of the output buffer data is also step S130, such as Figure 3 The steps shown include steps S131-132:

[0066] S131, reading the ByteBuffer video stream data from the codec output buffer.

[0067] S132. Decode and convert the ByteBuffer video stream data to obtain VideoFrame data.

[0068] As mentioned above, the ByteBuffer video stream data is directly written into the codec output buffer after being generated. When the UAV system is performing live broadcast display, the terminal continuously reads the ByteBuffer video stream data from the codec output buffer, and sets the width, height, timestamp and other information of each frame image decoded from the video stream, and encapsulates it into VideoFrame data.

[0069] S140: Send the video frame to a preset server through a WebRtc transmission channel, so that the preset server performs live display according to the video frame.

[0070] The WebRtc transmission channel is a data path between two devices connected by the push streaming interface provided by WebRtc. In this implementation, the push streaming interface provided by WebRtc connects the terminal and the preset server. The WebRtc transmission channel is actually implemented with the help of network signals, such as 4 networks. Specifically, after obtaining the video frame supported by the WebRtc protocol, the terminal can transmit the video frame to the preset server through the WebRtc transmission channel. After receiving the video frame, the preset server displays it in the form of live video.

[0071] In some specific embodiments, the WebRtc transmission channel includes an RTCPeerConnection channel and / or an RTCDataChannel channel. The RTCPeerConnection channel is a data path built by the RTCPeerConnetion API, which can achieve efficient and stable data stream communication; the RTCDataChannel is a data path built by the RTCDataChannel API, which can achieve low latency and high throughput point-to-point data exchange.

[0072] The video live broadcast method based on drone provided in this embodiment is applied to a terminal that maintains communication with the drone. The original video stream data sent by the drone is first obtained, and then the original video stream data is encoded and decoded to obtain output buffer data. The output buffer data is converted to obtain video frames supported by the WebRtc protocol. Finally, the video frames are sent to a preset server through a WebRtc transmission channel, and the preset server performs live broadcast display according to the video frames. The method realizes real-time live broadcast based on the WebRtc protocol with the help of the terminal during the drone live broadcast process, which greatly reduces the delay of the live broadcast picture. In practical applications, the video live broadcast method based on drone provided in this embodiment realizes that the live broadcast picture delay is less than 200ms when previewed through the public network, while the drone live broadcast delay of conventional technology is more than 1s. In this embodiment, there is no need to set a WebRtc conversion device and a network communication device on the drone, which reduces the hardware cost of the drone. The computing performance of the terminal also far exceeds the computing power requirement of data conversion during the drone live broadcast process. In addition, the underlying architecture of the data conversion based on the terminal is mature and easy to implement, with low development cost. The existing drone system can be transformed with the help of software only, and the application scenarios are wide.

[0073] Embodiment 2

[0074] Embodiment 2 provides a method for live video broadcast based on a drone, which can be implemented on the basis of Embodiment 1, and specifically supplements or exemplifies some of the contents in Embodiment 1, for example, provides a process for adjusting the resolution according to the network signal strength, specifically including:

[0075] like Figure 4 As shown, the video live broadcast method based on drone provided in this embodiment includes:

[0076] S210: Obtain original video stream data sent by the drone.

[0077] S220: Encode and decode the original video stream data to obtain output buffer data.

[0078] S230: Convert the output buffer data into video frames supported by the WebRtc protocol.

[0079] S240: Detect network signal strength.

[0080] The network signal strength indicates the network signal strength currently used by the terminal to connect to the preset server. For example, the terminal can currently use 4G network and 5G network, but the terminal and the preset server are connected through the 4G network. Then the network signal strength indicates the current 4G signal strength of the terminal.

[0081] S250: Determine a current resolution according to the network signal strength in accordance with a first rule.

[0082] The current resolution represents the resolution of the live image in the video frame. It is a resolution value that is adaptively adjusted according to the network signal strength, and is used to ensure that the data transmission speed does not affect the live broadcast delay. It is understandable that when the network signal strength is low, the data transmission speed is slow, and the data volume of the live image with high resolution is necessarily larger than the data volume of the live image with low resolution. If the data transmission is not timely, it will inevitably lead to a higher delay in the live broadcast. Therefore, in this embodiment, a first rule for adjusting the current resolution according to the network signal strength is set, which can automatically select a resolution value adapted to the network signal strength as the current resolution.

[0083] In a specific embodiment, Figure 5 The step S250 shown specifically includes steps S251-253:

[0084] S251. Determine whether the network signal strength is greater than or equal to a first signal strength.

[0085] S252: If the signal strength is greater than or equal to the first signal strength, determine that the current resolution is 1920x1080p.

[0086] S253: If it is less than the first signal strength, determine that the current resolution is 1080x720p.

[0087] Steps S251-253 are only a specific example of determining the current resolution. The first signal strength is a specific signal strength value, such as -67dBm, which can be set according to actual needs without limitation. Figure 5 Only the process of adjusting between two resolution values ​​according to the first signal strength is given, but in actual applications, the second signal strength and the third signal strength can also be set. In addition to 1080p (1920x1080p) and 720p (1080x720p), the current resolution can also be set to 360p and 4k. Of course, in some embodiments, in order to ensure display clarity, 720p is usually the lowest clarity.

[0088] S260: Adjust the video stream resolution of the video frame according to the current resolution.

[0089] Video stream resolution is a parameter of a video frame, which is used to determine the resolution of the image that can be generated by the video frame. After determining the current resolution, the terminal adjusts the video stream resolution of the video frame according to the current resolution, which is actually adjusting the amount of data that is ultimately transmitted.

[0090] S270: Send the video frame to a preset server through a WebRtc transmission channel, so that the preset server performs live display according to the video frame.

[0091] More specifically, in some embodiments, Figure 6 As shown, it also includes the process of adjusting the bit rate of the original video stream data according to the network signal strength:

[0092] S201: Detect network signal strength.

[0093] Similar to step S240, the network signal strength indicates the network signal strength currently used by the terminal to connect to the preset server.

[0094] S202: Determine a video stream bit rate according to the network signal strength and a second rule.

[0095] The second rule represents a rule for increasing or decreasing the video stream bit rate according to the increase or decrease of the network signal strength. In this embodiment, the video stream bit rate increases with the increase of the network signal strength. In some specific embodiments, the second rule can be a proportional relationship formula with the network signal strength and the video stream bit rate as the associated quantity, indicating that the two (network signal strength and video stream bit rate) are in a proportional relationship. The video stream bit rate is used to indicate the image clarity when the drone and the terminal are transmitting the image. It changes with the change of the network signal strength and is a constantly fluctuating value, which determines the data volume of the original video stream data. It can be understood that the larger the data volume of the original video stream data, the more time the terminal spends in processing the conversion. When the network signal strength is relatively low, it is also necessary to adjust the resolution when converting it into a video frame, which further increases the processing time. Therefore, in this implementation, the video stream bit rate when the drone and the terminal are transmitting the image is automatically adjusted according to the network signal strength to reduce the overall live broadcast delay.

[0096] S203: Send the video stream bit rate to the drone, so that the drone sends original video stream data based on the video stream bit rate.

[0097] After the terminal determines the current appropriate video stream bit rate based on the network signal strength, it sends this information to the drone, which then transmits the original video stream data to the terminal according to the video stream bit rate.

[0098] S204: Obtain original video stream data sent by the drone.

[0099] S205: Encode and decode the original video stream data to obtain output buffer data.

[0100] S206: Convert the output buffer data into video frames supported by the WebRtc protocol.

[0101] S207: Send the video frame to a preset server through a WebRtc transmission channel, so that the preset server performs live display according to the video frame.

[0102] This embodiment further provides a process for adjusting the video stream resolution of the video frame according to the network signal strength, and a process for adjusting the video stream bit rate of the original video stream data, so that the video stream resolution and the video stream bit rate are adapted to the network signal strength, thereby avoiding live broadcast delays caused by the mismatch between data volume and data transmission speed.

[0103] Embodiment 3

[0104] Figure 7 A schematic diagram of a video live broadcast system 300 based on a drone provided in Embodiment 3 of the present invention is shown in FIG. Figure 7 The system 300 comprises:

[0105] The data receiving module 310 is used to obtain the original video stream data sent by the drone;

[0106] The encoding and decoding module 320 is used to encode and decode the original video stream data to obtain output buffer data;

[0107] A data conversion module 330, configured to convert the output buffer data into video frames supported by the WebRtc protocol;

[0108] The WebRtc transmission module 340 is used to send the video frame to a preset server through a WebRtc transmission channel, so that the preset server performs live display according to the video frame.

[0109] Optionally, in some embodiments, the original video stream data includes H264 data, and the encoding and decoding module 320 is specifically used for:

[0110] Writing the H264 data into the codec input buffer and submitting it to the codec;

[0111] The codec encodes and decodes the H264 data to generate ByteBuffer video stream data and writes the ByteBuffer video stream data into the codec output buffer.

[0112] Optionally, in some embodiments, the data conversion module 330 is specifically used for:

[0113] Read the ByteBuffer video stream data from the codec output buffer;

[0114] The ByteBuffer video stream data is decoded and converted to obtain VideoFrame data.

[0115] Optionally, in some embodiments, the method further includes:

[0116] Network detection module, used to detect network signal strength;

[0117] A resolution confirmation module, configured to determine a current resolution according to a first rule based on the network signal strength;

[0118] The video frame adjustment module is used to adjust the video stream resolution of the video frame according to the current resolution.

[0119] Optionally, in some embodiments, the resolution confirmation module is specifically used to:

[0120] Determining whether the network signal strength is greater than or equal to a first signal strength;

[0121] If it is greater than or equal to the first signal strength, the current resolution is determined to be 1920x1080p;

[0122] If it is less than the first signal strength, it is determined that the current resolution is 1080x720p.

[0123] Optionally, in some embodiments, the method includes:

[0124] Network detection module, used to detect network signal strength;

[0125] A bit rate confirmation module, used to determine the video stream bit rate according to the network signal strength according to a second rule;

[0126] The bit rate adjustment module is used to send the video stream bit rate to the drone, so that the drone sends the original video stream data based on the video stream bit rate.

[0127] Optionally, in some embodiments, the WebRtc transmission channel includes an RTCPeerConnection channel and / or an RTCDataChannel channel.

[0128] The present embodiment provides a video live broadcast system based on a drone, which is applied to a terminal that maintains communication with the drone. The original video stream data sent by the drone is first obtained, and then the original video stream data is encoded and decoded to obtain output buffer data, and the output buffer data is converted to obtain video frames supported by the WebRtc protocol. Finally, the video frames are sent to a preset server through a WebRtc transmission channel, and the preset server performs live display according to the video frames. During the drone live broadcast process, the method realizes real-time live broadcast based on the WebRtc protocol with the help of the terminal, which greatly reduces the delay of the live broadcast picture. In addition, there is no need to set up a WebRtc conversion device and a network communication device on the drone, which reduces the hardware cost of the drone. The computing performance of the terminal also far exceeds the computing power requirement of data conversion during the drone live broadcast process. In addition, the underlying architecture of the data conversion based on the terminal is mature and easy to implement, with low development cost. The existing drone system can be transformed with the help of software only, and the application scenarios are wide.

[0129] Embodiment 4

[0130] Figure 8 A schematic diagram of the structure of an electronic device 400 that can implement a method for live video broadcasting based on a drone is provided in Embodiment 4 of the present invention. Figure 8 As shown, the device includes a memory 410 and a processor 420. The number of processors 420 in the device may be one or more. Figure 8 A processor 420 is taken as an example; the memory 410 and the processor 420 in the device can be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0131] The memory 410, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the drone-based live video broadcasting method in the embodiment of the present invention (for example, the data receiving module 310, the encoding and decoding module 320, the data conversion module 330 and the WebRtc transmission module 340 in the drone-based live video broadcasting system). The processor 420 executes the various functional applications and data processing of the QR code-based page guidance second page guidance module by running the software programs, instructions and modules stored in the memory 410, that is, realizing the above-mentioned drone-based live video broadcasting method.

[0132] Among them, the processor 420 is used to run the computer executable program stored in the memory 410 to implement the following steps: step S110, obtain the original video stream data sent by the drone; step S120, encode and decode the original video stream data to obtain output buffer data; step S130, convert the output buffer data into video frames supported by the WebRtc protocol; step S140, send the video frames to the preset server through the WebRtc transmission channel, so that the preset server performs live display according to the video frames.

[0133] Of course, the drone-based live video broadcast device provided in an embodiment of the present invention is not limited to the operation of the method described above, and can also perform related operations in the drone-based live video broadcast method provided in any embodiment of the present invention.

[0134] The memory 410 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 410 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 410 may further include a memory remotely arranged relative to the processor 420, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0135] Embodiment 5

[0136] Embodiment 5 of the present invention further provides a storage medium including computer executable instructions, wherein the computer executable instructions are used to execute a method for live video broadcasting based on a drone when executed by a computer processor. The method for live video broadcasting based on a drone includes:

[0137] Get the original video stream data sent by the drone;

[0138] Encoding and decoding the original video stream data to obtain output buffer data;

[0139] Convert the output buffer data into video frames supported by the WebRtc protocol;

[0140] The video frame is sent to a preset server through a WebRtc transmission channel, so that the preset server performs live display according to the video frame.

[0141] Of course, the storage medium including computer executable instructions provided in an embodiment of the present invention, whose computer executable instructions are not limited to the method operations described above, can also execute related operations in the drone-based video live broadcast method provided in any embodiment of the present invention.

[0142] Through the above description of the implementation methods, the technicians in the field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a computer floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk or an optical disk, etc., including a number of instructions for enabling an electronic device (which can be a personal computer, a device, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0143] It is worth noting that in the embodiment of the above-mentioned authorization system, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0144] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A video live broadcast method based on drones, applied to a drone system, the system comprising a drone, a terminal and a preset server, characterized in that: include: The drone is provided with a flight control module and an image acquisition module; The flight control module is used to monitor and control the flight and mission of the UAV; The image acquisition module is used to acquire images during the flight of the UAV; The terminal obtains the original video stream data sent by the drone; The terminal encodes and decodes the original video stream data to obtain output buffer data; wherein the output buffer data is the encoding and decoding of the original video stream data to convert it into highly efficient processing byte encapsulation data; The terminal converts the output buffer data into video frames supported by the WebRtc protocol; The terminal sends the video frame to a preset server through a WebRtc transmission channel, so that the preset server performs live display according to the video frame; Before the terminal obtains the original video stream data sent by the drone, it includes: an image acquisition module acquires the original video stream data and transmits it to the terminal; The original video stream data includes H264 data, and the terminal encodes and decodes the original video stream data to obtain output buffer data, including: The H264 data is written into the codec input buffer and submitted to the codec; the codec encodes and decodes the H264 data to generate ByteBuffer video stream data and writes it into the codec output buffer; The terminal converts the output buffer data into a video frame supported by the WebRtc protocol, including: Read the ByteBuffer video stream data from the codec output buffer; decode and convert the ByteBuffer video stream data to obtain VideoFrame data; Before obtaining the original video stream data sent by the drone, the method further includes: Detect network signal strength; Determine the video stream bit rate according to the second rule based on the network signal strength; wherein the second rule is a rule for increasing or decreasing the video stream bit rate according to the increase or decrease of the network signal strength, and the video stream bit rate is used to indicate the image clarity when the drone and the terminal are transmitting the image; The video stream bit rate is sent to the drone, so that the drone sends the original video stream data based on the video stream bit rate.

2. The method for live video broadcasting based on drone according to claim 1, characterized in that: Before sending the video frame to the preset server through the WebRtc transmission channel, the method further includes: Detect network signal strength; Determine the current resolution according to the first rule based on the network signal strength; The video stream resolution of the video frame is adjusted according to the current resolution.

3. The method for live video broadcasting based on drone according to claim 2, characterized in that: Determining the current resolution according to the first rule based on the network signal strength includes: Determining whether the network signal strength is greater than or equal to a first signal strength; If it is greater than or equal to the first signal strength, the current resolution is determined to be 1920x1080p; If it is less than the first signal strength, it is determined that the current resolution is 1080x720p.

4. The method for live video broadcasting based on drone according to claim 1, characterized in that: The WebRtc transmission channel includes an RTCPeerConnection channel and / or an RTCDataChannel channel.

5. A video live broadcast system based on drone, characterized in that: include: The system includes a drone, a terminal and a preset server; wherein the drone is provided with a flight control module and an image acquisition module; the flight control module is used to monitor and control the flight and mission of the drone; the image acquisition module is used to acquire images during the flight of the drone; The image acquisition module acquires and transmits raw video stream data to the terminal; The terminal comprises: The data receiving module is used to obtain the original video stream data sent by the drone; A codec module, used for encoding and decoding the original video stream data to obtain output buffer data; wherein the original video stream data includes H264 data; the output buffer data is the codec of the original video stream data to convert it into highly efficient processing byte encapsulation data; A data conversion module, used for converting the output buffer data into video frames supported by the WebRtc protocol; A WebRtc transmission module, used for sending the video frame to a preset server through a WebRtc transmission channel, so that the preset server performs live display according to the video frame; The codec module is specifically used to: write the H264 data into the codec input buffer and submit it to the codec; the codec encodes and decodes the H264 data to generate ByteBuffer video stream data and writes it into the codec output buffer; The data conversion module is specifically used to: read the ByteBuffer video stream data from the codec output buffer; decode and convert the ByteBuffer video stream data to obtain VideoFrame data; The system further comprises: Network detection module, used to detect network signal strength; A bit rate confirmation module, used to determine the video stream bit rate according to the network signal strength according to a second rule; wherein the second rule is a rule for increasing or decreasing the video stream bit rate according to the increase or decrease of the network signal strength, and the video stream bit rate is used to indicate the image clarity when the drone and the terminal are transmitting the image; The bit rate adjustment module is used to send the video stream bit rate to the drone, so that the drone sends the original video stream data based on the video stream bit rate.

6. An electronic device, characterized in that: It includes a memory and a processor, the memory stores a computer program that can be run on the processor, and the processor implements the drone-based video live broadcast method as described in any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which includes program instructions. When the program instructions are executed, the drone-based video live broadcast method according to any one of claims 1 to 4 is implemented.

Images