Video distribution system, instruction interaction system, method, storage medium, and device

Abstract

The present disclosure relates to a video distribution system, an instruction interaction system, a method, a storage medium and equipment, and relates to the technical field of multimedia information processing. The system comprises a server end, a first device end and a second device end. The server end comprises a first communication component corresponding to the first device end, a second communication component corresponding to the second device end and a data distribution component. The first device end is configured to send an original code stream to the first communication component through a first data channel. The first communication component is configured to send a target video stream corresponding to the original code stream to the data distribution component. The data distribution component is configured to send the target video stream to the second communication component. The second communication component is configured to distribute the target video stream to the second device end through a second data channel. The present disclosure solves the problem that a video collected by an offline device cannot be directly distributed to an online device.

Description

Technical Field

[0001] This disclosure relates to the field of multimedia information processing technology, and more specifically, to a video distribution system, a video distribution method, an instruction interaction system, an instruction interaction method, a computationally readable storage medium, and an electronic device. Background Technology

[0002] In existing audio and video distribution scenarios, specific audio and video distribution scenarios may include, but are not limited to: on the one hand, distributing audio and video collected by online devices to online devices; on the other hand, distributing audio and video collected by online devices to offline devices; and on yet another hand, distributing audio and video collected by offline devices to online devices.

[0003] However, existing video distribution solutions cannot directly distribute videos collected by offline devices to online devices.

[0004] It should be noted that the information in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a video distribution system, video distribution method, command interaction system, command interaction method, computationally readable storage medium, and electronic device, thereby overcoming, to at least a certain extent, the problem that videos captured by offline devices cannot be directly distributed to online devices due to limitations and defects in related technologies.

[0006] According to one aspect of this disclosure, a video distribution system is provided, comprising: Server-side, first device-side, and second device-side; The server side includes a first communication component corresponding to the first device side, a second communication component corresponding to the second device side, and a data distribution component; The first device is connected end-to-end with the first communication component through the first data channel, and the second device is connected end-to-end with the second communication component through the second data channel. The first device is configured to send the raw bitstream to the first communication component through the first data channel; The first communication component is configured to send a target video stream corresponding to the original bitstream to the data distribution component; The data distribution component is configured to send the target video stream to the second communication component; The second communication component is configured to distribute the target video stream to the second device via the second data channel.

[0007] In one exemplary embodiment of this disclosure, the first communication component includes a Web RTC communication component and / or a Web Socket communication component; the first data channel includes a Web RTC data channel and / or a Web Socket data channel.

[0008] In one exemplary embodiment of this disclosure, the second communication component includes a Web RTC communication component; the second data channel includes a Web RTC data channel.

[0009] In one exemplary embodiment of this disclosure, the server side further includes an address identification service; The first device is also configured to establish a first data channel between the first device and the first communication component through the address identification service.

[0010] In one exemplary embodiment of this disclosure, the first device is further configured to: establish a first communication connection between the first device and the address identification service, and send a first address identification request to the address identification service based on the first communication connection; The second device is further configured to: establish a second communication connection between the second device and the address identification service, and send a second address identification request to the address identification service based on the second communication connection.

[0011] In one exemplary embodiment of this disclosure, the first communication connection is a first UDP connection, and the second communication connection is a second UDP connection.

[0012] In one exemplary embodiment of this disclosure, the server side further includes a controller component; The address identification service is configured to: respond to the first address identification request, identify the first virtual communication address included in the first address identification request, obtain the first real communication address, and send the first real communication address to the first device. The first device is further configured to: construct a first session description protocol packet based on a first real communication address; and send the first session description protocol packet to the controller component to request the establishment of the first data channel; The controller component is configured to: enable the first communication component corresponding to the first device; The address identification service is further configured to: identify the second virtual communication address of the first communication component to obtain the second real communication address; The first communication component is configured to: construct a second session description protocol packet based on the second real communication address; and establish a first data channel based on the first session description protocol packet and the second session description protocol packet.

[0013] In one exemplary embodiment of this disclosure, the first real communication address includes a first IP address and a first port address; the second real communication address includes a second IP address and a second port address. The first device is further configured to: construct a first candidate address set based on the first IP address and the first port address, and construct a first session description protocol packet based on the first candidate address set; The first communication component is configured to: construct a second candidate address set based on the second IP address and the second port address, and construct a second session description protocol packet based on the second candidate address set; use the first session description protocol packet as a remote connection object, and perform address matching on the first session description protocol packet and the second session description protocol packet to obtain a first target candidate address pair; and establish the first data channel based on the first target candidate address pair.

[0014] In an exemplary embodiment of this disclosure, the first communication component is further configured to: establish a first node connection interface, and obtain a first time node for establishing the first node connection interface and a second time node for receiving the first session description protocol packet; Calculate the time difference between the first time node and the second time node; if the time difference is greater than a preset time threshold, determine that the first data channel has failed to be established; if the time difference is less than or equal to the preset time threshold, determine that the first data channel has been successfully established. If it is determined that the first data channel has failed to be established, the first communication connection is disconnected.

[0015] In one exemplary embodiment of this disclosure, the first device further includes a data collector and a transcoder; The data acquisition device is configured to acquire raw video streams and / or raw audio streams; The transcoder is configured to: transcode the original video stream and / or the original audio stream to obtain original video data packets and / or original audio data packets, and obtain the original bitstream based on the original video data packets and / or original audio data packets; The first device is also configured to send the raw bitstream to the first communication component via the first data channel.

[0016] In one exemplary embodiment of this disclosure, the server side further includes a first encoder-decoder; The first communication component is further configured to: determine whether the original bitstream needs to be transcoded based on the original parameter information of the original bitstream that the first device can produce and the target parameter information that the first communication component can support, and send the original bitstream to the first encoder-decoder when it is determined that transcoding is required; The first encoder-decoder is further configured to transcode the original bitstream according to the target parameter information to obtain a target video stream corresponding to the original bitstream, and to transmit the target video stream to the first communication component.

[0017] In one exemplary embodiment of this disclosure, the first encoder-decoder is further configured to: invoke the encoding and decoding algorithm included in the first encoder-decoder to resample the original audio RTP packets included in the original bitstream to obtain redundant audio data, and obtain the target video stream based on the transcoded original bitstream and the redundant audio data.

[0018] In one exemplary embodiment of this disclosure, the server side further includes a second encoder-decoder and a neural network model; The first communication component is also configured to send the raw bitstream to the second encoder-decoder; The second encoder-decoder is configured to: determine the original frame rate of the original bitstream based on the original parameter information included in the first session description protocol packet, and obtain the key frame to be processed from the original bitstream based on the original frame rate; and transmit the key frame to be processed to the neural network model. The neural network model is configured to: extract the original feature points included in the keyframe to be processed, and identify the target object included in the keyframe to be processed based on the original feature points; construct a label filter based on the target object and the original parameter information, and transmit the label filter to the second encoder-decoder; The second encoder-decoder is further configured to: mark the keyframes to be processed in the original bitstream according to the marking filter to obtain the target video stream, and send the target video stream to the first communication component.

[0019] In one exemplary embodiment of this disclosure, the second device is further configured to: determine the standard jitter-resistant latency required by the server, and send the standard jitter-resistant latency to the second communication component through the second data channel; The second communication component is further configured to: receive multiple target RTP packets sent by the standard jitter-resistant delay data distribution component at intervals, and send the multiple target RTP packets to the second device end through the second data channel.

[0020] In one exemplary embodiment of this disclosure, the second device is further configured to: determine the target anti-jitter latency of the target video stream based on the maximum video transmission frame corresponding to the target video stream received within a standard time; and determine the standard anti-jitter latency required by the server based on the target anti-jitter latency.

[0021] In one exemplary embodiment of this disclosure, the second device is further configured to: The channel transmission rate of the second data channel when transmitting the current target RTP packet in the target video stream is obtained, as well as the real-time video transmission frame of the current target RTP packet within the standard time. The maximum video transmission frame and the average video transmission frame are determined based on the real-time video transmission frames, and the noise variance of the signal transmission noise of the current target RTP packet during transmission is calculated. The target anti-jitter delay is calculated based on the maximum video transmission frame, the average video transmission frame, the channel transmission rate, and the noise variance.

[0022] In one exemplary embodiment of this disclosure, the real-time video transmission frame includes the current video frame and the previous video frame corresponding to the current video frame; The second device is further configured to: calculate the frame delay of the target video stream based on the current frame time of receiving the current video frame and the previous frame time of receiving the previous video frame, and calculate the video frame difference of the target video stream based on the current video frame of the current video frame and the previous video frame of the previous video frame. Based on the frame delay, channel transmission rate, network queuing delay of the current target RTP packet during transmission, and video frame difference, the frame residual is calculated, and the average noise of signal transmission noise is determined based on the frame residual and a preset probability coefficient. The noise variance is determined based on the frame residual and the average noise.

[0023] In an exemplary embodiment of this disclosure, the second device is further configured to: calculate a first proportional coefficient based on the frame index and total number of frames of the current target RTP packet, and calculate an average video transmission frame based on the first proportional coefficient and the current video frame; The frame variance is calculated based on the average video transmission frame and the current video frame, and the maximum video transmission frame is determined based on the frame variance.

[0024] In one exemplary embodiment of this disclosure, the second device is further configured to: Calculate the first difference between the maximum video transmission frame and the average video transmission frame, and calculate the first product of the first difference and the reciprocal of the channel transmission rate. Calculate the square root of the noise variance, and calculate the second product between the preset noise figure and the square root. The second difference between the result of the second product operation and the preset noise reduction constant is calculated, and the target anti-jitter delay is obtained based on the second difference and the result of the first product operation.

[0025] In one exemplary embodiment of this disclosure, the second device is further configured to: adjust the target anti-jitter delay in real time based on the maximum video transmission frame.

[0026] In one exemplary embodiment of this disclosure, the second device terminal is further configured to: The maximum video transmission frame at the current moment is compared with the maximum video transmission frame at the next moment, and when it is determined that the maximum video transmission frame at the current moment is greater than the maximum video transmission frame at the next moment, the target anti-jitter latency is reduced. When it is determined that the maximum video transmission frame at the current moment is less than the maximum video transmission frame at the next moment, the target anti-jitter latency is increased.

[0027] In one exemplary embodiment of this disclosure, the server side further includes a controller component; The second device is configured to send control commands to the controller component via the second data channel, which are applied to the first device. The controller component is configured to send the control command to the first communication component; The first communication component is configured to send the control command to the first device via the first data channel; The first device is also configured to execute the control command.

[0028] In one exemplary embodiment of this disclosure, the first device is further configured to: parse the control instruction to obtain the instruction type included in the control instruction, and execute the target instruction corresponding to the instruction type.

[0029] In one exemplary embodiment of this disclosure, the second device is further configured to: generate the control command; The controller component is further configured to: determine the target controlled terminal according to the control command, and send the control command to the first communication component corresponding to the target controlled terminal.

[0030] In one exemplary embodiment of this disclosure, the second device is further configured to: Obtain the current user identifier of the current controlling user corresponding to the second device terminal, and determine the first device serial number of the first device terminal that the second device terminal can control based on the current user identifier; The control command is generated based on the first device serial number of the first device terminal; or The control command is generated based on the first device serial number and the instruction data input by the current control user according to the current state of the first device.

[0031] In one exemplary embodiment of this disclosure, the first device is further configured to: send a first device status event to the controller component via the first data channel; The controller component is also configured to send the first device status event to the second communication component; The second communication component is also configured to send the first device status event to the second device via the second data channel; The second device is also configured to determine and display the current state of the first device based on the first device state event, and to receive instruction data input by the current user based on the displayed current state.

[0032] In one exemplary embodiment of this disclosure, the controller component is further configured to: determine a first device serial number of a first device terminal that can be controlled by the second device terminal based on the current user identifier included in the control instruction, and designate the first device terminal corresponding to the first device serial number as the target controlled terminal; or

[0033] The target controlled terminal is determined based on the first device serial number included in the control command.

[0034] In one exemplary embodiment of this disclosure, the first device is the controlled device and the second device is the master device.

[0035] In one exemplary embodiment of this disclosure, the video distribution system further includes a third device, which is connected end-to-end with a third communication component via a third data channel; The third device is configured to: obtain a token identifier of the interactive page, generate a video acquisition request based on the token identifier, and send the video acquisition request to the controller component; The controller component is further configured to: in response to the video acquisition request, determine the target video stream to be sent based on the token identifier, allocate a channel identifier required for transmitting the target video stream to be sent to the third device, and send the channel identifier to the data distribution component; The data distribution component is configured to push the target video stream to be sent to the third device based on the channel identifier.

[0036] In one exemplary embodiment of this disclosure, the server side further includes a relay server; The first communication component is configured to push the target video stream to a relay server; The relay server is also configured to push the target video stream to a display terminal; wherein the display terminal includes one or more of a second device terminal, a third device terminal, and other display terminals with interactive functions.

[0037] According to one aspect of this disclosure, a command interaction system is provided, comprising: The system comprises a server, a first device, and a second device. The server includes a controller component, a first communication component corresponding to the first device, and a second communication component corresponding to the second device. The first device is connected end-to-end with the first communication component through the first data channel, and the second device is connected end-to-end with the second communication component through the second data channel. The second device is configured to send control commands to the controller component via the second data channel, which act on the first device. The controller component is configured to send the control command to the first communication component; The first communication component is configured to send the control command to the first device via the first data channel; The first device is also configured to execute the control command.

[0038] In one exemplary embodiment of this disclosure, the first device is further configured to: parse the control instruction to obtain the instruction type included in the control instruction, and execute the target instruction corresponding to the instruction type.

[0039] In one exemplary embodiment of this disclosure, the second device is further configured to: generate the control command; The controller component is further configured to: determine the target controlled terminal according to the control command, and send the control command to the first communication component corresponding to the target controlled terminal.

[0040] In one exemplary embodiment of this disclosure, the second device is further configured to: Obtain the current user identifier of the current controlling user corresponding to the second device terminal, and determine the first device serial number of the first device terminal that the second device terminal can control based on the current user identifier; The control command is generated based on the first device serial number of the first device terminal; or The control command is generated based on the first device serial number and the instruction data input by the current user based on the current state of the first device.

[0041] In one exemplary embodiment of this disclosure, the first device is further configured to: send a first device status event to the controller component via the first data channel; The controller component is also configured to send the first device status event to the second communication component; The second communication component is also configured to send the first device status event to the second device via the second data channel; The second device is also configured to determine and display the current state of the first device based on the first device state event, and to receive instruction data input by the current user based on the displayed current state.

[0042] In one exemplary embodiment of this disclosure, the controller component is configured to: determine a first device serial number of a first device terminal that can be controlled by the second device terminal based on the current user identifier included in the control instruction, and designate the first device terminal corresponding to the first device serial number as the target controlled terminal; or

[0043] The target controlled terminal is determined based on the first device serial number included in the control command.

[0044] According to one aspect of this disclosure, a video distribution method is provided, configured on a server side. The server side includes a first communication component corresponding to a first device, a second communication component corresponding to a second device, and a data distribution component. The first device is connected end-to-end to the first communication component via a first data channel, and the second device is connected end-to-end to the second communication component via the second data channel. The video distribution method includes: The first communication component receives the raw bitstream sent by the first device through the first data channel and sends the target video stream corresponding to the raw bitstream to the data distribution component; The data distribution component sends the target video stream to the second communication component; The second communication component pushes the target video stream to the second device via the second data channel.

[0045] In one exemplary embodiment of this disclosure, the server side further includes a controller component and an address recognition service, and the video distribution method further includes: The address identification service receives a first address identification request sent by a first device, identifies the first virtual communication address included in the first address identification request, obtains a first real communication address, and sends the first real communication address to the first device. The controller component receives a first session description protocol packet generated based on a first real communication address sent by the first device, and starts the first communication component corresponding to the first device. The first communication component constructs a second session description protocol packet and performs address matching on the first session description protocol packet and the second session description protocol packet to obtain a first target candidate address pair; The first communication component establishes the first data channel based on the first target candidate address pair.

[0046] In one exemplary embodiment of this disclosure, the video distribution method further includes: The first communication component determines whether the original bitstream needs to be transcoded based on the original parameter information of the original bitstream that the first device can produce and the target parameter information that the first communication component can support. When it is determined that transcoding is required, the original bitstream is sent to the first encoder-decoder. The first encoder-decoder transcodes the original bitstream according to the target parameter information to obtain the target video stream.

[0047] In one exemplary embodiment of this disclosure, the video distribution method further includes: The first encoder-decoder calls the encoding and decoding algorithm included in the first encoder-decoder to resample the original audio RTP packets included in the original bitstream to obtain redundant audio data, and obtains the target video stream based on the transcoded original bitstream and the redundant audio data.

[0048] In one exemplary embodiment of this disclosure, the video distribution method further includes: The first communication component sends the raw bitstream to the second encoder / decoder, which determines the raw frame rate of the raw bitstream based on the raw parameter information included in the first session description protocol packet. The second encoder-decoder obtains the key frame to be processed from the original bitstream according to the original frame rate, and transmits the key frame to be processed to the neural network model; The neural network model extracts the original feature points included in the key frame to be processed, and identifies the target object included in the key frame to be processed based on the original feature points; The neural network model constructs a label filter based on the target object and the original parameter information, and transmits the label filter to the second encoder-decoder; The second encoder-decoder marks the keyframes to be processed in the original bitstream according to the marking filter to obtain the target video stream, and sends the target video stream to the first communication component.

[0049] In one exemplary embodiment of this disclosure, the video distribution method further includes: In response to a video acquisition request sent by a third device, the controller component determines the target video stream to be sent based on the token identifier included in the video acquisition request. The controller component allocates a channel identifier to the third device for transmitting the target video stream to be sent, and sends the channel identifier to the data distribution component; The data distribution component pushes the target video stream to be sent to the third device according to the channel identifier.

[0050] In one exemplary embodiment of this disclosure, the video distribution method further includes: The controller component pushes the target video stream to the relay server, so that the relay server pushes the target video stream to the display terminal; wherein, the display terminal includes one or more of the following: a second device terminal, a third device terminal, and other display terminals with interactive functions.

[0051] According to one aspect of this disclosure, a command interaction method is provided, configured on a server side. The server side includes a first communication component corresponding to a first device, a second communication component corresponding to a second device, and a controller component. The first device is connected end-to-end to the first communication component via a first data channel, and the second device is connected end-to-end to the second communication component via the second data channel. The command interaction method includes: The controller component receives control commands sent by the second device terminal to the first device terminal via the second data channel, and sends the control commands to the first communication component; The first communication component sends the control command to the first device through the first data channel, so that the first device executes the control command.

[0052] In one exemplary embodiment of this disclosure, sending the control command to the first communication component includes: The target controlled terminal is determined according to the control command, and the control command is sent to the first communication component corresponding to the target controlled terminal.

[0053] In one exemplary embodiment of this disclosure, the control command is generated by the second device in the following manner: Obtain the current user identifier of the current controlling user corresponding to the second device terminal, and determine the first device serial number of the first device terminal that the second device terminal can control based on the current user identifier; The control command is generated based on the first device serial number of the first device terminal; or The control command is generated based on the first device serial number and the instruction data input by the current user based on the current state of the first device.

[0054] In one exemplary embodiment of this disclosure, the instruction interaction method further includes: The controller component receives a first device status event sent by the first device terminal and sends the first device status event to the second communication component; The second communication component sends the first device status event to the second device through the second data channel, so that the second device can determine and display the current status of the first device based on the first device status event, and receive the instruction data input by the current user based on the displayed current status.

[0055] In one exemplary embodiment of this disclosure, determining the target controlled terminal according to the control command includes: Based on the current user identifier included in the control command, determine the first device serial number of the first device terminal that the second device terminal can control, and designate the first device terminal corresponding to the first device serial number as the target controlled terminal; or The target controlled terminal is determined based on the first device serial number included in the control command.

[0056] According to one aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the video distribution method and the instruction interaction method described in any one of the preceding claims.

[0057] According to one aspect of this disclosure, an electronic device is provided, comprising: Processor; and Memory for storing the executable instructions of the processor; The processor is configured to execute the video distribution method and the instruction interaction method described above by executing the executable instructions.

[0058] This disclosure provides a video distribution system in which, on one hand, a first device is connected end-to-end to a first communication component via a first data channel, and a second device is connected end-to-end to a second communication component via a second data channel. Simultaneously, the first device can send an original bitstream to the first communication component via the first data channel; the first communication component can send a target video stream corresponding to the original bitstream to a data distribution component; furthermore, the data distribution component can send the target video stream to the second communication component, enabling the second communication component to distribute the target video stream to the second device via the second data channel. That is, the first device (offline device) can send the original bitstream to a server, and then the server can distribute the bitstream to be sent to the second device, solving the problem in the prior art where videos captured by offline devices cannot be directly distributed to online devices. On the other hand, this video distribution system realizes the separation of video capture and video playback, meaning that video can be captured by the first device and then played by the second device, thus enabling its application to all scenarios requiring video capture and playback separation.

[0059] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0060] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0061] Figure 1 The diagram schematically illustrates a video distribution system according to an exemplary embodiment of the present disclosure.

[0062] Figure 2 A block diagram illustrating a server-side functional component according to an exemplary embodiment of the present disclosure is shown.

[0063] Figure 3 A block diagram schematically illustrates another server-side functional component according to an exemplary embodiment of this disclosure.

[0064] Figure 4 The illustration shows an example scenario of a first data channel being established between a first device and a server according to an example embodiment of the present disclosure.

[0065] Figure 5 The illustration shows an example scenario of a first device sending a raw bitstream to a server according to an example embodiment of the present disclosure.

[0066] Figure 6 The illustration shows an example scenario of sending a stream of data to be distributed to a relay server or data distributor according to an example embodiment of the present disclosure.

[0067] Figure 7 This diagram illustrates a scenario where a second device and a server establish a second data channel according to an example embodiment of this disclosure.

[0068] Figure 8 The illustration shows an example scenario of establishing a second communication component on the server side according to an example embodiment of the present disclosure.

[0069] Figure 9 The illustration shows an example scenario of sending a bitstream to a second device via a second data channel according to an example embodiment of the present disclosure.

[0070] Figure 10 The illustration shows an example scenario in which a second device sends a control command to a first device according to an example embodiment of the present disclosure.

[0071] Figure 11 The diagram illustrates an example scenario of a third device establishing a third data channel with a server according to an example embodiment of the present disclosure.

[0072] Figure 12 The illustration shows an example scenario of establishing a third communication component on the server side according to an example embodiment of the present disclosure.

[0073] Figure 13 The illustration shows an example scenario of sending a bitstream to a third device via a third data channel according to an example embodiment of the present disclosure.

[0074] Figure 14 The illustration schematically depicts an electronic device for implementing a video distribution method or an instruction interaction method according to an example embodiment of the present disclosure. Detailed Implementation

[0075] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0076] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0077] WebRTC (Web Real-Time Communications) is a peer-to-peer (P2P) technology used in modern browsers. In practical applications, users can interact based on real-time multimedia streams using WebRTC, currently primarily used in private audio and video interactions such as video interviews and video conferencing. Currently, WebRTC technology is closed-loop, meaning encoding, transmission, and decoding are integrated into a single system. This is highly beneficial for business orchestration within the browser. The closed-loop nature described here refers to the following: Traditional WebRTC opens a peer on one terminal, using a local UDP (User Datagram Protocol) port. A cloud-based relay service sends the encoded audio and video information (the sending end), along with the port number, to the peer (the SDP offer). Upon receiving this, the peer responds by also opening a local UDP port and returning the same SDP answer. This constitutes a basic closed-loop connection for network P2P. At the same time, both parties will use cameras / microphones to capture, encode, push, receive and play audio and video, which is a simple audio and video interaction scenario.

[0078] However, precisely because of its closed-loop nature, WebRTC technology has largely been confined to browser-based video conferencing scenarios. Here, browser-based video conferencing refers to applications that use a browser as the carrier, WebRTC technology as the support, and RTC (Real-Time Communication) and RTCP (Real-time Transport Control Protocol) as the transmission protocols, with NAT (Network Address Translation) traversal or TURN (Traversal Using Relays around NAT) relay as the network foundation for acquisition, transmission, and playback. However, real-world audio and video needs are diverse. Audio and video scenarios are not limited to online-to-online (i.e., communication between cloud services), but also include online-to-offline (communication between cloud services and SoC terminals with SIM card communication modules) and offline-to-offline (communication between SoC terminals with SIM card communication modules). Examples include multi-terminal medical consultations and even high-definition surgery, central fire command, and remote control of drones / robotic dogs. Here, "online" generally refers to cloud services, while "offline" generally refers to various SoC (System on Chip) terminals with SIM card communication modules. Further, "online-online" generally refers to an interaction where the acquisition end is a website webpage or mobile application, and the playback end is a related or protocol-based website webpage or mobile application; "online-offline" generally refers to an interaction where an offline device acts as the acquisition end, and the playback end is a related or protocol-based website webpage or mobile application; "offline-offline" generally refers to an interaction where one offline device acts as the acquisition end, and another offline device with a communication module acts as the playback end.

[0079] Based on this, the exemplary embodiments of this disclosure first provide a video distribution system. This video distribution system implements a composite multimedia interactive system based on WebRTC, which can be widely used in terminals that support WebRTC, such as Android, iOS, and modern browsers. Its design supports the needs of various multimedia services and supports Actor plug-and-play (i.e., various different Actors are extensible Actors, which can be set, configured, or extended according to actual needs in practical applications). With the support of Ffmpeg, it forms a flexible multimedia interactive system. Furthermore, the exemplary embodiments of this disclosure will use a SOC with embedded WebRTC Client as a remote controlled terminal as an example to illustrate the design concept of WebRTC-based multimedia interaction. Meanwhile, the video distribution system described in the example embodiments of this disclosure adopts a video distribution system with separate acquisition and playback, that is, video acquisition and video playback are performed on two different devices. Based on the scheme described in the example embodiments of this disclosure, the video acquisition device is a first device end, and the video playback device is a second device end and / or a third device end. At the same time, the video distribution system described in the example embodiments of this disclosure needs to be limited to the scenario of separate acquisition and playback because the first device end described in the example embodiments of this disclosure is a controlled terminal, which does not support video playback or video viewing through online means. It can only perform video acquisition and then upload the acquired video to the server end, and then the server end distributes the video, so that users can watch or play the video through the second device end.

[0080] In one example embodiment, reference is made to... Figure 1 As shown, the video distribution system may include a server (Web RTCall-in-one Server) 110, a first device (Drone) 120, and a second device (Browser) 130. Meanwhile, refer to... Figure 2As shown, the server-side component may include a controller component 211, a first communication component 212 corresponding to the first device, a second communication component 213 corresponding to the second device, and a data processing component 214. Specifically, in practical applications, the first device establishes an end-to-end connection (i.e., a P2P connection) with the first communication component through a first data channel, and the second device establishes an end-to-end connection with the second communication component through a second data channel. Furthermore, the first device is configured to send the raw bitstream to the controller component through the first data channel; the first communication component is configured to send the target video stream corresponding to the raw bitstream to the data distribution component; the data distribution component is configured to send the target video stream to the second communication component; and the second communication component is configured to distribute the target video stream to the second device through the second data channel. The target video stream described herein may include the original bitstream itself, or the target video stream obtained after processing the original bitstream; this example does not impose any special restrictions on this. Meanwhile, the first communication component described herein includes a Web RTC communication component (Web RTC Actor) or a Web Socket communication component (Web SocketActor); the first data channel includes a Web RTC data channel or a Web Socket data channel, etc.; the second communication component includes a Web RTC communication component (Web RTC Actor), and the second data channel includes a Web RTC data channel.

[0081] In one example embodiment, the end-to-end connection described above can also be referred to as a peer-to-peer connection, i.e., Peer-to-Peer, P2P, etc. Peer-to-peer connections can also be referred to as peer-to-peer interconnection technologies, which rely on the computing power and bandwidth of participants in the network, rather than concentrating the reliance on a few servers. Furthermore, WebRTC (Web Real-Time Communications) described above is a real-time communication technology that allows web applications or websites to establish peer-to-peer connections between browsers without the aid of an intermediary, enabling the transmission of video and / or audio streams or other arbitrary data. Simultaneously, the standards included in WebRTC make it possible for users to create peer-to-peer data sharing and teleconferencing without installing any plugins or third-party software. Even further, WebSocket described above is a protocol for full-duplex communication over a single TCP connection; simultaneously, WebSocket simplifies data exchange between clients and servers, allowing the server to proactively push data to the client. In the WebSocket API, the browser and server only need to complete a handshake once to establish a persistent connection and perform bidirectional data transmission.

[0082] In one example embodiment, the Actor model described above can be considered a conceptual model that can be used to handle concurrent computation. Simultaneously, the Actor model defines a series of general rules on how system components should act and interact; the most famous programming language using these rules is Erlang. An Actor refers to a basic unit of computation that can receive a message and perform computation based on it. The Actor model is a lock-free design model designed to solve high-concurrency scenarios (high concurrency typically refers to designing a system to ensure it can handle a large number of requests in parallel). In practical applications, Actors are mostly working modules that consume significant computing power, memory, or bandwidth resources. The Actor model is introduced here primarily to address performance issues such as resource locking (exclusive access to resources or the ability to share writes / exclusive reads) caused by resource contention in low-latency audio and video interaction scenarios due to numerous frequent messages and events. The only drawback of the Actor model might be that it can lead to message blocking when dealing with large tasks. Therefore, some large tasks need to be specially handled during the design to prevent message blocking. The special handling mentioned here means that resources cannot be exclusively used when processing large tasks. For example, when dealing with H.265 video, decoding and encoding without hardware acceleration requires CPU computing power, which is software decoding / encoding. However, software decoding / encoding requires time to wait. During the waiting period, the playback task lock cannot be exclusively held, otherwise the user will not be able to create a new playback task. Therefore, the task lock of the Codec Actor can only be a read lock. The RTC Actor that depends on the codecActor needs to continuously send redundant packets to maintain the connectivity. Furthermore, the Actor has a controller part externally and a database part, a codec part, and an RTC part internally. Except for the controller and RTC parts, the other Actors are pluggable (that is, they can be expanded, settable, or customized).

[0083] In one example embodiment, the database Actor mentioned above is mainly used to extract data from the data storage. Due to the design of the database Actor, there is no need to consider the lock issue, and the Actor consumes messages one by one from its own Mailbox. Each message consumed is followed by a database transaction operation, and the data obtained for the next message is always the latest data. Therefore, data inconsistency caused by data competition can be avoided, thereby achieving atomicity and data consistency under high-concurrency messages. At the same time, common operations of the database Actor include the controller querying terminal details, the controller recording terminal events, and the controller recording control terminal signals. In addition, in order to speed up message query speed, the database Actor will perform some caching internally.

[0084] In one example embodiment, the controller component described above can be used to coordinate the work of other actors (e.g., RTC Actor, Codec Actor, WS Actor, etc.) described in the example embodiments of this disclosure. Since the controller is designed based on actors, message communication is implemented using the Actor Receipient (container) address. This means the controller has the communication addresses of all other actor modules and can issue commands to them. Furthermore, since Web RTC itself has a data channel, it can be used for multi-terminal communication. If the offline terminal does not have a WebRTC Client, WebSocket (WS) can be used instead, thus adding a WS Actor. This example embodiment is exemplified using a Web RTC data channel (a WebRTC channel is a data stream channel based on RTCP). It should be noted that both Web RTC Client and WebSocket can establish bidirectional communication channels. For some terminals, such as RiscV, which lack mature .NET or audio / video libraries, WebRTC is difficult to compile successfully; in such cases, WebSocket can be considered as a substitute. RTC uses UDP, while WebSocket uses TCP; each has its advantages. In practical applications, the choice can be made based on actual needs, and this example does not impose any special restrictions.

[0085] In one example embodiment, a P2P channel is established after the RTC Actor and the control browser (second device) establish a connection. Here, P2P refers to an end-to-end direct connection channel, with one end being the control browser (first device) and the other end being the RTC Actor. Since the RTC Actor is backed by a controller, the controller and control browser can be considered to interact. Furthermore, the first device (i.e., the controlled terminal, which could be an RsicV terminal) also establishes a connection with the RTC module, thus establishing a connection with the controller. Under certain conventions, such as specifying which devices a user can control, the user's control webpage can use the logged-in user's ID as a unique identifier, and the terminal device can use its serial number as a unique identifier. The controller can then establish a mapping between the control terminal (second device) and the controlled terminal (first device) under these conventions, continuously sending control signals from the control browser's data channel through the control-end P2P RTC Actor to the controller to locate the controlled RTC. The Actor injects signals into the data channel of the controlled end, thereby enabling the online command to be sent to the offline process; conversely, the controlled end can continuously report signals such as video, audio, status events and abnormal events to the control end, thereby enabling the online and offline interactive process.

[0086] In one example embodiment, reference is made to... Figure 3 As shown, the server side may also include an address identification service 311, a relay server 312, and a data distributor 313. Unlike the pluggable Actors, the Web RTC functional components (secondary functional components) are typically essential for coordination, including a STUN service (address identification service) for establishing P2P hole punching, potentially needed relay servers such as RTSP services, and SFU (Selective Forwarding Unit) distributors for distributing data, etc.

[0087] In one example embodiment, most current networks are based on IPv4; meanwhile, for the STUN (Session Traversal Utilities for NAT) service described in the example embodiments of this disclosure, IPv4-based networks can support NAT (Network Address Translation), that is, IPv4-based networks can support network address translation. In practical applications, when a computer in a private network needs to access an address on an external network service, that address on the external network service sees a port on a unified external address within the private network accessing it. Based on this, with the help of NAT, when the "internal" network of a private (reserved) address sends data packets through a router, the private address is translated into a legal IP address. A local area network only needs a small number of IP addresses (even just one) to meet the communication needs of all computers within the private address network with the Internet. However, establishing a P2P connection requires both parties to expose their real addresses. Therefore, if one end of an end-to-end connection is within a private network, that end will not know its real address. Thus, in order to obtain the real address, a service needs to be exposed to the public network. Whenever a request is sent to it, the service will return the real address of the requesting end.

[0088] In one example embodiment, if the terminal does not support Web RTC Client, an alternative method to supplement audio and video transmission is to use the alternative Lib RTSP to push the stream to the Relay Server here. Alternatively, if a high-concurrency live streaming service is required, a separate setup with a video source on the same network is needed for the push operation. At the same time, the Relay Server can also act as a Pusher, remuxing the RTP streams collected by the RTC Actor into new RTP packets and sending them to the live streaming service using the RTSP protocol.

[0089] In one example embodiment, WebRTC network topology has three modes: P2P direct connection (decentralized), SFU (centralized service forwarding), and MCU (centralized service convergence). In the scheme described in the example embodiment of this disclosure, the distributor designed based on the SFU mode exists in the form of a cross-thread channel. The specific time limit principle is that the process shares the memory heap, which can include various subscription methods: one-to-many (1:n), many-to-many (n:m), and many-to-one (n:1), etc., so it is not exactly the same as the SFU mode. For example, from the perspective of scenarios, the 1-to-many mode is suitable for one person to operate and many people to watch, such as a medical consultation scenario; the n:m many-to-many mode is suitable for n people to be active and m people to watch, such as an online karaoke scenario; and the many-to-one mode is suitable for multiple terminals and a single control terminal, such as a city traffic command center based on road cameras, etc.

[0090] In one example embodiment, during a specific application process, the first device is further configured to establish a first data channel between the first device and the first communication component through the address identification service. The establishment of the first data channel can be implemented as follows: First, the first device establishes a first communication connection between itself and the address identification service, and sends a first address identification request to the address identification service based on the first communication connection; simultaneously, the second device establishes a second communication connection between itself and the address identification service, and sends a second address identification request to the address identification service based on the second communication connection; wherein, the first communication connection described here is a first UDP connection, and the second communication connection is a second UDP connection. In other words, in a specific application process, firstly, the first or second device can send a first communication connection establishment request or a second communication connection establishment request to the STUN service based on the public IP address of the server. The first communication connection can be a first UDP (User Datagram Protocol) connection, and the second communication connection can also be a second UDP connection. Furthermore, after the first or second UDP connection is established, when the first and / or second device can send a message in STUN binding format based on the first or second UDP connection, the UDP server will return the identified peer address (real communication address) to the first or second device in the form of the mapped-address field in the Bingding Response.

[0091] In one example embodiment, the specific establishment process of the first data channel can be implemented as follows: First, the address identification service responds to the first address identification request, identifies the first virtual communication address included in the first address identification request, obtains the first real communication address, and sends the first real communication address to the first device; second, the first device constructs a first session description protocol packet based on the first real communication address; sends the first session description protocol packet to the controller component to request the establishment of the first data channel; further, the controller component activates the first communication component corresponding to the first device; even further, the address identification service identifies the second virtual communication address of the first communication component to obtain the second real communication address; finally, the first communication component constructs a second session description protocol packet based on the second real communication address; and establishes the first data channel based on the first session description protocol packet and the second session description protocol packet.

[0092] In one possible example embodiment, during the establishment of the first data channel, the first real communication address described above may include a first IP address and a first port address; the second real communication address includes a second IP address and a second port address; wherein, in a specific application process, the first device constructs a first candidate address set based on the first IP address and the first port address, and constructs a first session description protocol packet based on the first candidate address set; the first communication component constructs a second candidate address set based on the second IP address and the second port address, and constructs a second session description protocol packet based on the second candidate address set; the first session description protocol packet is used as a remote connection object, and address matching is performed on the first session description protocol packet and the second session description protocol packet to obtain a first target candidate address pair; the first data channel is established based on the first target candidate address pair.

[0093] In one possible example embodiment, the first IP address described above may include the Internet Protocol address of the first device; the first port address may include the port address on the first device that supports SIM card functionality, such as a mobile port, a China Unicom port, a China Netcom port, or a China Telecom port, etc.; at the same time, the construction process of the first candidate address set can be based on the first IP address and the first port address; at the same time, the first device is a SoC terminal with a SIM card communication module, so the port described here may include multiple ports, thus requiring the establishment of a first candidate address set; furthermore, the first session description protocol packet described above may include the first candidate address set, and may also include the original bitrate and original resolution of the original bitstream collected by the first device, and may also include the device identifier and first device attribute information (such as device model, size, etc.) of the first device, etc., which are not specifically limited in this example.

[0094] In one possible example embodiment, the first communication component described above can communicate with the address identification service. In practical applications, the first communication component can send a corresponding address identification request to the address identification service to obtain a second real communication address. This second real communication address may include a second IP address and a second port address. Further, the second IP address may include the Internet Protocol address of the server; the second port address may include a port address that can communicate with the first device, such as a mobile port, a China Unicom port, a China Netcom port, or a China Telecom port, etc. The construction of the second candidate address set can be based on the second IP address and the second port address. The second IP address included here may include one or more, and the second port address may also include multiple. Furthermore, the second session description protocol packet described above may include the second candidate address set and server-side attribute information, etc., which are not specifically limited in this example.

[0095] In one possible example embodiment, the first data channel described above can be used as a dedicated channel for communication or data transmission between the first device and the server. Meanwhile, since the first device and the server are connected end-to-end, a corresponding data channel needs to be established to facilitate the transmission of the original bitstream.

[0096] In an exemplary embodiment of this disclosure, the first communication component is further configured to: establish a first node connection interface and obtain a first time node for establishing the first node connection interface and a second time node for receiving the first session description protocol packet; calculate the time difference between the first time node and the second time node; determine that the first data channel establishment has failed when the time difference is greater than a preset time threshold, and determine that the first data channel establishment has succeeded when the time difference is less than or equal to the preset time threshold; and disconnect the first communication connection when the first data channel establishment has failed.

[0097] The following will explain and illustrate the specific process of establishing the first communication connection and the specific process of establishing the first data channel.

[0098] First, the first device described herein will be explained and described. Specifically, the first device can be a controlled terminal, which can be a 64-bit x86 / ARM architecture Android application, an iOS application, or a Linux / Unix desktop application, or it can be opened with a modern browser that supports WebRTC; the example embodiment of this disclosure uses a 64-bit x86 Linux SOC that supports WebRTC Client as the controlled terminal for explanation and description. Simultaneously, for video distribution to be realized, a connection between the controlled terminal (the first device) and the All-in-one Server is a prerequisite for all business operations. In practical applications, the first device will support a WebRTC Client based on a Unix-based operating system. Upon startup, it will proactively connect to the server's STUN service to collect local IP addresses (the first IP address) and ports (the first port address) to form a candidate set. Based on the candidate address set, it will construct a first SDP (Session Description Protocol) packet for negotiation preparation. The first SDP packet described here is a string that can be encrypted and sent to the server's controller component via an HTTP (Hypertext Transfer Protocol) interface. Furthermore, when the controller component receives the first SDP packet, it will initiate an RTC... The Actor (first communication component) initializes by creating a PeerConnection (first node connection interface). It also uses the same STUN service to collect local candidates (second candidate address combinations) and generate a local SDP (second SDP packet). Furthermore, it sets the SDP (first session description protocol packet) sent by the controlled end as the remote connection object. The WebRTC built-in library then selects the most suitable candidate pair (first target address pair) based on the two SDPs to establish a UDP (or TCP) connection. The WebRTC built-in library described here can be considered a functional unit or component within WebRTC, capable of performing the corresponding address matching tasks.

[0099] It should be noted that UDP provides an application (the first device) with a way to send encapsulated IP packets without establishing a connection. Compared to TCP, UDP does not consider an acknowledgment mechanism, so it is faster and more efficient, but the quality may be worse. WebRTC's Nack mechanism effectively addresses this shortcoming. Of course, for clients that do not support WebRTC Client, TCP connections can still be selected, and this example does not impose any special restrictions on this.

[0100] In one example embodiment, after the first communication connection is established, the local SDP (Second Session Description Protocol Packet) needs to be returned to the first device based on the first communication connection. After receiving the Second Session Description Protocol Packet, the first device will also try to connect using the Candidate that has not established a connection until the last pairing possibility is exhausted (i.e., the second target candidate address pair). At the same time, unlike the traditional WebRTC which uses PC State events to determine the success of the connection, this example embodiment relies on the data channel, and the connection path is determined by the connection of the data channel. Therefore, if the PeerConnection (first connection node) does not receive the data channel establishment message within 10 seconds (i.e., the preset time threshold mentioned above), it is determined that the connection has failed and it will actively disconnect the connection (first communication connection). In addition, the controller Actor has a self-loop health check. Once it is found that all PeerConnections on one end of a mapping are disconnected, the other end will be automatically disconnected to avoid the existence of zombie connections, release ports and increase concurrency performance. It should be noted that in practical applications, the matching process for the first candidate address pair can be performed on the server side, on the first device side, or simultaneously on both sides. This example does not impose any special restrictions on this. Furthermore, performing the matching of the first candidate address pair twice can avoid the problem of inconsistencies between the communication operators of the server and the first device. This method ensures that a first data channel can be successfully established between the server and the first device, allowing the first device to send the original bitstream to the server through this first data channel.

[0101] It should be further noted that, in practical applications, SFUs can also be configured for the first communication component based on the business requirements. The Distributor's subscription channel; that is, if the first device disconnects for some reason, when the first device needs to reconnect to the server, the thread or process where the first communication component corresponding to the first device is located has already been bound to other first devices or viewers. Therefore, in this scenario, the controller component will re-sort the mapping connection relationship between the first communication component and each other first device or viewer according to the correspondence in the database or business agreement, and add the first device back to the relationship mapping table in the database. At the same time, it is also necessary to send the device connection event of the first device (e.g., controlled terminal A) to the already connected second device and viewer (e.g., viewer D and viewer E, etc.), and then determine whether to start the renegotiation process (this renegotiation process is also the process of re-exchanging the first session description protocol packet and the second session description protocol packet) based on the original parameter information of the reconnected controlled terminal A. If the original parameter information of the reconnected controlled terminal A is consistent with the original parameter information of the controlled terminal A before disconnection, then there is no need to start the renegotiation process; otherwise, the renegotiation process needs to be started.

[0102] The following will explain and illustrate the specific acquisition and generation process of the raw bitstream. Specifically, since the raw bitstream originates from the first device, a data acquisition unit and a transcoder need to be set up on the first device to generate the corresponding raw bitstream. Specifically, the specific generation process of the raw bitstream can be as follows: the data acquisition unit is configured to acquire the raw video stream and / or the raw audio stream; the transcoder is configured to transcode the raw video stream and / or the raw audio stream to obtain raw video data packets and / or raw audio data packets, and obtain the raw bitstream based on the raw video data packets and / or the raw audio data packets; furthermore, the first device is also configured to send the raw bitstream to the first communication component through the first data channel. Specifically, since the video distribution method described in the example embodiments of this disclosure is applicable to scenarios where acquisition and broadcasting are separated, the first device can use a local data collector to collect the raw video stream and / or raw audio stream. The raw video data packet is an H.264 raw stream, and the raw audio data packet is a raw 16-bit PCM (Pulse Code Modulation) audio signal. Furthermore, it needs to undergo transcoding processing via a local transcoder on the first device, thereby obtaining RTP packets of the H.264 raw stream that can be transmitted via WebRTC and audio RTP packets encrypted in Opus format, i.e., the raw bitstream. Further, after obtaining the raw bitstream, the first device is also configured to send the raw bitstream to the controller component through the first data channel.

[0103] The following will explain and illustrate the specific forwarding and processing of the original bitstream on the server side. Specifically, the forwarding and processing of the original bitstream on the server side can be divided into the following two cases: In one scenario, the original parameter information included in the original bitstream is consistent with the target parameter information supported by the first communication component (Web RTC). In this case, after receiving the original bitstream, the first communication component sends it to the data distribution component, which then forwards it directly to the second communication component. The second communication component then distributes the original bitstream to the second device via the second data channel (i.e., the corresponding target video stream is the original bitstream itself, requiring no processing). In another scenario, the original parameter information and the target parameter information are inconsistent, requiring transcoding of the original bitstream on the server side. Specifically, transcoding the original bitstream can be achieved using a first encoder-decoder; this first encoder-decoder can be a codec. In specific applications, the transcoding process of the original bitstream can be implemented as follows: The first communication component determines whether transcoding of the original bitstream is necessary based on the original parameter information of the original bitstream produced by the first device and the target parameter information supported by the first communication component. If transcoding is required, the original bitstream is sent to the first encoder / decoder. The first encoder / decoder transcodes the original bitstream according to the target parameter information to obtain a target video stream corresponding to the original bitstream, and transmits the target video stream to the first communication component. Further, the first encoder / decoder also needs to call the encoding / decoding algorithm included in the first encoder / decoder to resample the original audio RTP packets included in the original bitstream to obtain redundant audio data. Based on the transcoded original bitstream and the redundant audio data, the target video stream is obtained. The original parameter information mentioned above may include the original bitrate, original frame rate, and original resolution of the original bitstream, etc., and the target parameter information may include the target bitrate and target frame rate supported by the first communication component, etc.

[0104] In one example embodiment, the specific processing of the original bitstream by the first codec actor can be as follows: Specifically, the codec actor is a dedicated thread responsible for performing various transformations on the original bitstream; in practical applications, a first device usually has a data acquisition source (data acquisition device), so each first device has a dedicated RTC and encoding / decoding thread (first communication component and first codec actor); at the same time, the function of the first codec actor is to perform various transformations on the original bitstream; for example, transcoding, filtering, frame rate and bitrate adjustment, and audio resampling, etc.

[0105] In one example embodiment, the underlying framework of the first encoder-decoder can use the FFMPEG (Fast Forward Moving Picture Expert Group) application, or of course, the Gstreamer application; this example does not impose any special restrictions. Furthermore, if the codec needs to process the raw bitstream, the controller component creates an actor (encoder-decoder) and sends its address to the first communication component. The first communication component continuously sends its own RTP packets (the raw RTP packets corresponding to the raw bitstream) to the codec. After processing, the codec continuously feeds back the target video stream to the first communication component. It should be noted that, to ensure data integrity, the video data (target video stream) output by the codec (encoder-decoder) typically includes an h264 raw stream, which can be a target video stream with profile 100 level 40 and / or a target audio stream with standard samples at an opus sampling rate of 48000 Hz. Furthermore, after obtaining the target video stream and target audio stream, they can be encapsulated into target RTP packets and injected into the WebRTC audio / video channel (second data channel) for transmission to the second device. It should be noted that WebRTC itself has a Jitter Buffer (actually a buffer, typically with a set maximum jitter time; however, the audio portion is smaller and, due to its linear nature and the lack of correlation between packets, the encoder / decoder can use its own encoding / decoding tools to resample the audio to obtain Redundant Audio Data (RED); this encoder / decoder can include an XOR processor or a Gstreamer). This buffer can reassemble out-of-order RTP packets in weak network environments and report lost network packets to achieve target RTP retransmission, thereby realizing RED audio transmission.

[0106] It's important to note that RED data is restored at the consumer end (second device) to replace audio drops caused by network packet loss. Simultaneously, the XOR redundancy ratio can be set, calculated as: Final packet loss rate = (1 + RED ratio) x Network packet loss rate. In poor network conditions, if the second device detects poor audio quality in the target video stream, it can adjust the above parameters and initiate an update stream parameter command. Furthermore, upon receiving this command, the controller component forwards it to the encoder / decoder and instructs the corresponding RTC to initiate a renegotiation process. Simultaneously, the renegotiationd P2P connection receives new codec output data, thus achieving video adjustment and improving user experience.

[0107] In one example embodiment, special processing of the original bitstream is required in certain special scenarios; the special processing described herein may include, but is not limited to, filtering. Under this premise, the server may further include a neural network model and a second encoder-decoder. Further, the first communication component is configured to send the original bitstream to the second encoder-decoder; specifically, the second encoder-decoder determines the original frame rate of the original bitstream based on the original parameter information included in the first session description protocol packet, and obtains keyframes to be processed from the original bitstream based on the original frame rate; transmits the keyframes to be processed to the neural network model; the neural network model extracts the original feature points included in the keyframes to be processed, and identifies the target objects included in the keyframes to be processed based on the original feature points; constructs a label filter based on the target objects and the original parameter information, and transmits the label filter to the second encoder-decoder; the second encoder-decoder labels the keyframes to be processed in the original bitstream according to the label filter to obtain the target video stream, and sends the target video stream to the first communication component.

[0108] In one example embodiment, the processing of the original bitstream using a neural network model and a second encoder-decoder can be achieved as follows: Specifically, in scenarios such as using aircraft for ground landmark recognition, medical data processing, or KTV scenarios, an OpenCV Actor (which may include a neural network model or other functional models, and can be configured according to actual needs, without special restrictions here) can be added to the server side. When the second device establishes a P2P connection, it can send a trained neural network model as a parameter to the server side, and then the server side can send the trained neural network model to the OpenCV Actor. In other words, the trained neural network model can be integrated into the OpenCV Actor in the form of an SDK. In practical applications, after the controller component receives the raw bitstream sent by the first device, it can be proxied to the second encoder / decoder via the RTC actor (first communication component). The second encoder / decoder, based on the frame rate and bit rate of the raw bitstream as parameters, periodically extracts frame data (i.e., keyframes to be processed) and continuously sends these keyframes to the neural network model in the OpenCV actor for object feature detection. After matching, a draw box filter is applied based on the video stream parameters sent by the aircraft. The OpenCV actor then sends the box coordinates to the codec. The codec performs a draw box filter on the current frame, re-encodes it, and returns it to the RTC actor via the data channel. It should be noted that the neural network model described here can be the FAST R-CNN model or other models; this example does not impose any special restrictions on it. It should be noted that the OpenCV Actor described above plays the role of object feature detection in a specific scenario, which is different from the Codec described above; at the same time, the second encoder-decoder described here can be a Codec, or other components or models with encoding and decoding functions, and this example does not impose any special restrictions on it.

[0109] In one example embodiment, to improve the user experience during video viewing on the second device, the second device is further configured to: determine the standard anti-jitter latency required by the server, and send the standard anti-jitter latency to the second communication component via a second data channel; the second communication component is further configured to: receive multiple target RTP packets sent by the data distribution component at intervals of the standard anti-jitter latency, and send the multiple target RTP packets to the second device via the second data channel. The specific process for determining the standard anti-jitter latency can be as follows: the second device is further configured to: determine the target anti-jitter latency of the target video stream based on the maximum video transmission frame received within a standard time period corresponding to the target video stream; and determine the standard anti-jitter latency required by the server based on the target anti-jitter latency. The standard time mentioned here can be considered as a unit of time, that is, the maximum number of video transmission frames received within a certain time period. This unit of time can be defined according to actual needs; this example does not impose any special restrictions. The maximum number of video transmission frames mentioned here can be understood as the maximum number of video frames included in the target RTP packet received within that unit of time, i.e., the maximum number of frames that can be included. The standard jitter-resistant latency mentioned here refers to the jitter-resistant latency required by the data distributor during the distribution of the target video stream. This standard jitter-resistant latency can be adjusted based on the current network conditions. The system can adaptively adjust based on network conditions. For example, if the network condition is good, the standard's anti-jitter latency can be reduced; if the network condition is poor, the standard's anti-jitter latency can be increased. Alternatively, it can be adjusted in real time based on the maximum video transmission frame. The specific adjustment process can include: comparing the maximum video transmission frame at the current moment with the maximum video transmission frame at the next moment, and if it is determined that the maximum video transmission frame at the current moment is greater than the maximum video transmission frame at the next moment, adjusting the target anti-jitter latency downwards; if it is determined that the maximum video transmission frame at the current moment is less than the maximum video transmission frame at the next moment, adjusting the target anti-jitter latency upwards.

[0110] In one example embodiment, the specific calculation process for the target jitter immunity delay can be implemented as follows: Obtain the channel transmission rate of the second data channel when transmitting the current target RTP packet in the target video stream, and the real-time video transmission frames of the current target RTP packet within a standard time; determine the maximum video transmission frame and the average video transmission frame based on the real-time video transmission frames, and calculate the noise variance of the signal transmission noise of the current target RTP packet during transmission; calculate the target jitter immunity delay based on the maximum video transmission frame, the average video transmission frame, the channel transmission rate, and the noise variance. Here, the channel (Information Channels) transmission rate can be defined as the transmission rate of data (e.g., the target RTP packet) on the transmission medium (channel).

[0111] In one example embodiment, the noise variance described above can be calculated as follows: The second device calculates the frame delay of the target video stream based on the current frame time of receiving the current video frame and the previous frame time of receiving the previous video frame, and calculates the video frame difference of the target video stream based on the current video frame and the previous video frame; calculates the frame residual based on the frame delay, channel transmission rate, network queuing delay of the current target RTP packet during transmission, and the video frame difference, and determines the average noise of the signal transmission noise based on the frame residual and a preset probability coefficient; and determines the noise variance based on the frame residual and the average noise. The real-time video transmission frame described here may also include the next video frame corresponding to the current video frame. In practical applications, this can be defined according to actual needs, and this example does not impose any special restrictions on this.

[0112] In one example embodiment, the specific calculation process for the maximum video transmission frame described above can be implemented as follows: The second device calculates a first proportional coefficient based on the frame index and the total number of frames in the current target RTP packet, and calculates the average video transmission frame based on the first proportional coefficient and the current video frame; it calculates the frame variance based on the average video transmission frame and the current video frame, and determines the maximum video transmission frame based on the frame variance. Here, the frame index, also known as FrameIndex, can be the index value of all received video frames, which can be accumulated starting from zero.

[0113] In one example embodiment, the specific calculation process of the target anti-jitter delay can be implemented as follows: the second device calculates the first difference between the maximum video transmission frame and the average video transmission frame, and calculates the first product operation result between the first difference and the reciprocal of the channel transmission rate; calculates the square root of the noise variance, and calculates the second product operation result between the preset noise coefficient and the square root; calculates the second difference between the second product operation result and the preset noise subtraction constant, and obtains the target anti-jitter delay based on the second difference and the first product operation result.

[0114] The following will further explain and illustrate the specific implementation process of the target anti-jitter delay and the standard anti-jitter delay. Specifically, unlike the controlled end (first device end), the control end (second device end) is a user-initiated behavior. The user can start or not start the second device end, that is, the second device end and the server end are decoupled. At the same time, in the actual application process, the second device end needs to make operation judgments based on audio and video, and send the corresponding instruction data through the second data channel; therefore, for the second device end, audio and video are uplink data. For uplink data sent by UDP, there are packet loss and out-of-order situations; under this premise, the second device end needs to set up anti-jitter buffer; the calculation formula of the buffer is based on the Kaufman filter equation of the difference between the maximum and minimum frames within the standard time, which can be shown in the following formula (1): JitterDelay controller = theta[0] (maxFrameSize–avgFrameSize) +[noiseStdDevs sqrt(varNoise) – noiseStdDevOffset]; Formula (1) Wherein, JitterDelay controller is the target anti-jitter delay, theta[0] is the reciprocal of the channel transmission rate, maxFrameSize is the maximum frame size received since the start of the session, avgFrameSize represents the average frame size, noiseStdDevs represents the noise coefficient, which can be 2.33, varNoise represents the noise variance, and noiseStdDevOffset is the noise subtraction constant, which can be 30. The specific calculation process of noise variance, average frame, and maximum frame can include: First, define the frame delay `frameDelay` and the frame rate difference `deltaFS`. Where: frameDelay = t(i) – t(i-1) – (T(i) – T(i-1)); deltaFS = frameSize – prevFrameSize; Where t(i) represents the current frame time, t(i-1) represents the previous frame time; T(i) represents the Predetermined Time System (PTS) time of the current frame, T(i-1) represents the PTS time of the previous frame; frameSize represents the size of the current frame, and prevFrameSize represents the size of the previous frame. Secondly, let avgFrameSize = phi avgFrameSize + (1-phi) frameSize; where avgFrameSize is the average video transmission frame size; varFrameSize = phi varFrameSize + (1-phi) (frameSize – avgFramesize)^2; where varFrameSize is the frame variance; maxFrameSize = max(maxFrameSize, varFrameSize); where maxFrameSize is the maximum number of video frames to transmit.

[0115] phi = frameIndex / totalFrameNum; Among them, frameIndex (frame index) is accumulated starting from 0, and totalFrameNum is the total number of frames; residual = frameDelay – (theta[0] deltaFS + theta[1]); avgNoise = alpha avgNoise + (1-alpha) residual; varNoise = alpha varNoise + (1 – alpha) (residual – avgNoise)^2; alpha = pow(399 / 400, 30 / fps); Wherein, alpha represents the probability coefficient and is affected by the frame rate fps. That is, when the fps decreases, alpha will decrease, indicating that the current noise increases and the noise variance is more affected by the current noise. In actual application, the closer the real-time frame rate is to 30fps, the better. Furthermore, avgNoise represents the average noise since the beginning, theta[0] represents the reciprocal of the channel transmission rate, and theta[1] represents the network queuing delay, which is the difference between the maximum and minimum video transmission frames within the standard time. In actual application, if theta[1] increases, the second device will have more time and space to require the server to resend lost or out-of-order RTP packets, which will make the picture and sound more stable. Conversely, the quality of the picture and sound may decrease or even cause screen tearing. Of course, the increase of theta[1] means that the time difference between the terminal and the control end will increase. Therefore, when dealing with different audio and video formats in different scenarios, it is often necessary to weigh the anti-jitter delay of the target, while ensuring the stability of the uplink stream of the first device and reducing the delay time of the control end.

[0116] In practical applications, since the human reaction time is 200 milliseconds, if the jitter delay on the control end is jitter_delay_controller (standard anti-jitter delay) and the jitter delay on the server-side RTC Actor is jitter_delay_target (target anti-jitter delay), then as long as jitter_delay_controller + jitter_delay_target + other_delay <= 200ms, the user will not experience any noticeable operational lag. Here, other_delay in the formula represents the necessary delay for stable encoding, decoding, and rendering; it is a relatively stable delay. Different CPU, VPU, or GPU configurations will produce different other_delays, which can be manually adjusted during deployment. Therefore, the specific formula for calculating the target anti-jitter delay is: jitter_delay_target = Max(200ms – jitter_delay_controller – other_delay, jitter_delay_controller ); where the standard anti-jitter delay jitter_delay_controller is sent by the control channel RTCP on the second device side, via the second communication component RTC. The Actor sends the data to the controller, which then forwards it to the first communication component, RTC Actor. The reason for sending the standard anti-jitter delay to the first communication component, RTC Actor, is that the first communication component can send the raw bitstream to the data distribution component based on the standard anti-jitter delay, thus avoiding data congestion in the data distribution component. Sending the target anti-jitter delay to the second communication component is so that the second communication component can send the corresponding target video stream to the second device based on the target anti-jitter delay.

[0117] The following will explain and illustrate the specific implementation process of the second device sending corresponding control commands to the first device.

[0118] Specifically, in one example embodiment, the issuance and execution of specific control commands can be implemented as follows: the second device sends a control command acting on the first device to the controller component through the second data channel; the controller component sends the control command to the first communication component; the first communication component sends the control command to the first device through the first data channel; and the first device executes the control command. During the execution of the control command, the first device can parse the control command to obtain the command type included in the control command and execute the target command corresponding to the command type. The control commands described here may differ in different scenarios; for example, when using a drone for surface landmark detection, the control commands described here may be drone-related commands, such as horizontal calibration, takeoff, forward / backward / left / right movement, ascent / descent, yaw / elevation, and return-to-home maneuvers; or, in a multi-person consultation scenario, the control commands described here may include camera-related commands, such as zoom or rotation; in practical applications, appropriate control commands can be adaptively provided according to the actual scenario, and this example does not impose any special limitations on this.

[0119] In one example embodiment, the control command described above is generated by the second device. Upon receiving the control command, the controller component can determine the target controlled terminal based on the control command and send the control command to the first communication component corresponding to the target controlled terminal. In a feasible example embodiment, the control command can be generated as follows: the second device obtains the current user identifier of the current controlling user corresponding to the second device and determines the first device serial number of the first device that the second device can control based on the current user identifier; the control command is generated based on the first device serial number of the first device; or the control command is generated based on the first device serial number and the instruction data input by the current controlling user according to the current state of the first device. That is, in the process of generating the control command, the corresponding control command can be generated directly based on the first device serial number, or based on the current device state of the first device, or the need to issue a corresponding control command can be determined based on the video stream uploaded by the first device. This example does not impose any special restrictions on this.

[0120] In one possible example embodiment, the process of generating control commands based on the device status of the first device can be implemented as follows: the first device sends a first device status event to the controller component through the first data channel; the controller component is further configured to send the first device status event to a second communication component; the second communication component is further configured to send the first device status event to the second device through a second data channel; the second device is further configured to determine and display the current status of the first device based on the first device status event, and receive command data input by the current user based on the displayed current status. That is, after viewing the corresponding first device status event, the current user can input corresponding commands based on the specific status of the first device, thereby enabling the second device to generate corresponding control commands based on the input commands.

[0121] In one possible example embodiment, when generating control commands, the second device described above may not consider the first device it can control, but instead directly generate control commands and let the server determine which first device to send the control command to. Specifically, in actual use, this can be achieved as follows: the controller component determines the first device serial number of the first device that the second device can control based on the current user identifier included in the control command, and takes the first device corresponding to the first device serial number as the target controlled device; or it determines the target controlled device based on the first device serial number included in the control command. That is, after receiving a control command, the controller component can determine whether it needs to determine which first device to send the control command to based on whether the control command includes the first device serial number of the first device; if it includes the first device serial number, it can forward it directly; if it does not include the first device serial number, it can determine the corresponding first device based on the first device serial number of the first device bound to the current user identifier; in actual application, it can determine this according to actual needs, and this example does not impose any special restrictions on this.

[0122] It should be further explained here that the first or second data channel mentioned above is the channel through which the second device sends instructions to the first device or receives events in return. Although the second device establishes a connection, it does not know the state of the other party. In other words, the second device is stateful. Without establishing a data channel or obtaining the state of the other party, according to the ACID (Atomicity, Consistency, Isolation, Durability) design principles, the second device will not acknowledge that the first device is in a controlled state. Therefore, for both the first and second devices, they are always connected to the first and second data channels, respectively. They are only connected to the data distribution component when the target video stream needs to be distributed.

[0123] The video distribution system described in this exemplary embodiment further includes a third device. Specifically, the third device is connected end-to-end to a third communication component via a third data channel; and the third device is configured to obtain a token identifier of an interactive page, generate a video acquisition request based on the token identifier, and send the video acquisition request to a controller component. Simultaneously, the process of distributing the stream to be distributed to the third device can be implemented as follows: the controller component, in response to the video acquisition request, determines the target video stream to be sent based on the token identifier, allocates a channel identifier to the third device for transmitting the target video stream, and sends the channel identifier to a data distribution component; the data distribution component is configured to push the target video stream to be sent to the third device based on the channel identifier. In determining the channel identifier, it can be determined based on the system category of the third device; that is, different systems (e.g., Android, iOS, or Linux, etc.) can be assigned different channel identifiers.

[0124] In one example embodiment, the third device described above can be a viewer terminal; wherein, the viewer terminal described here is only used to watch the corresponding video and does not need to send control commands based on the video; in actual application, when it is necessary to watch a certain video, when the viewer terminal initiates a connection, it will send a connection request to the controller component on the server side according to the business agreement, such as the controlled end device number and the live broadcast token and other global locators as parameters. The controller component responds and searches for the corresponding available distribution channel in the SFU Distributor (that is, to determine the corresponding channel identifier). If there is one, it can subscribe to and obtain a distribution receiver, and inject the corresponding RTC of the viewer terminal. Within the Actor, the receiver continuously receives packets, obtains the raw stream, segments it into RTP packets, and then sends them to the audio and video channels on the viewer's end, allowing the viewer to watch the audio and video. It's worth noting that since the viewer's end can have a certain latency, the anti-jitter setting can be artificially set to a large value, such as 1 second. Of course, without port multiplexing, the available ports for WebRTC are limited, so RTSP streaming can be used as a proxy to other RTSP streaming servers. Viewers can then connect to these servers for online live streaming. The playback format from RTSP Remux can include M3U8, MPEG DASH, and FLV, etc. This example does not impose any special restrictions on this.

[0125] In one example embodiment, the server side may further include a relay server; specifically, the relay server can be used to push the corresponding video stream to other display terminals. The specific implementation process may include: a first communication component pushing the target video stream to the relay server; the relay server pushing the target video stream to the display terminal; wherein the display terminal includes a second device, a third device, and other display terminals with interactive functions, etc. That is, whenever a device establishes a P2P connection with the controller component, the controller component queries the database component for the device configuration corresponding to that device; if the database component contains a push address corresponding to that device, the RTC instance initialization process will include the initialization of the RTSP client and the push address. When the RTC receives packets, it will perform the above operations and forward them to the target live streaming service.

[0126] It is worth noting that, in order to conserve bandwidth, the control commands and device events described above can be designed as streaming messages, which belong to a proprietary protocol format. Without a lookup table, even if the messages are intercepted and deciphered, it is difficult to understand the meaning of the transmission. For example, in scenarios involving a single control end (single second device) versus a single controlled end (single first device), or in scenarios involving a single control end versus multiple controlled ends, the specific sending process of control commands or events needs to be explained and described. Specifically, control commands are messages sent from the downlink control terminal to the controlled terminal, and device events are messages sent from the controlled terminal or viewer terminal to the control terminal / other viewer terminals via the uplink. These messages are forwarded to the corresponding RTC Actor data channel via the controller. In practical applications, there is a mapping relationship for any scenario (a mapping relationship can be called a task). The mapping relationship can be configured through other backend management pages and then stored in the database component. Furthermore, the controller component will operate according to the configuration in the database when making any distribution. At the same time, the device will be attached to the user's name during the configuration phase. When the user connects to the controller component and the device under the user is connected, the controller will automatically start a mapping relationship, and the user will be the administrator of the task.

[0127] It should be further explained that the audience terminals described here can join tasks at any time, and the global identifier of a task is a token; and after the administrator connects, if a task is created, he will receive the token number of these tasks, and the administrator can send the token to the audience offline; in addition, the administrator has the permission to disconnect the audience, and the audience terminal will be forcibly disconnected if the administrator goes offline; the device can go offline after a task is established, and if it does not reconnect within the configurable RECONNECT_THRESHOLD time, it will be judged as offline by the controller. In this case, both the control terminal and the audience terminal will receive a task Dismiss event and be forcibly disconnected. Furthermore, if the device comes online in time, the task's sequence number will be incremented by 1. The sequence number is the tag for all transmitted data. If the tag does not match, it means that the data is invalid and will be cleared. Thus, the raw stream packets remaining from the previous connection in the SFU Distributor will be judged to be expired due to the sequence number mismatch, the cache will be cleared, and new audio and video packets with new sequence numbers will be accepted again. At the same time, the same applies to uplink events and downlink commands. Since the Actor consumes data one by one, there may be messages remaining in the controller's mailbox. If the controller fails to process the messages from the previous connection in time, a batch of commands / events will be ignored when the device connects again and processes messages due to the sequence number mismatch. The specific process of forced disconnection involves finding the corresponding P2P candidate for pairing and forcibly closing the already connected UDP or TCP connection. Based on this, the basic operation flow design for distribution scenarios with a single control end versus a single controlled end or a single control end versus multiple controlled ends is completed. Of course, developers can perform secondary development for the specific control commands of the controlled end, as long as the control end and the controlled end have mutually recognized messages that do not conflict with the system. This example does not impose any special restrictions on this.

[0128] This disclosure also provides an example embodiment of an instruction interaction system. Specifically, the instruction interaction system may include a server, a first device, and a second device. The server includes a controller component, a first communication component corresponding to the first device, and a second communication component corresponding to the second device. The first device is connected to the first communication component via a first data channel, and the second device is connected to the second communication component via a second data channel. The second device is configured to send a control instruction acting on the first device to the controller component via the second data channel. The controller component is configured to send the control instruction to the first communication component. The first communication component is configured to send the control instruction to the first device via the first data channel. The first device is further configured to execute the control instruction. The specific execution process of the control instruction by the first device may include parsing the control instruction to obtain the instruction type included in the control instruction, and executing the target instruction corresponding to the instruction type.

[0129] In one example embodiment, the control commands described above may differ in different scenarios. For instance, when using a drone for surface landmark detection, the control commands described here may be drone-related commands, such as horizontal calibration, takeoff, forward / backward / left / right movement, ascent / descent, yaw / angle control, and return-to-home avoidance. Similarly, in a multi-person consultation scenario, the control commands described here may include camera-related commands, such as zooming or rotation. In practical applications, appropriate control commands can be adaptively provided according to the actual scenario; this example does not impose any special limitations on this. Furthermore, the control commands described above are generated by the second device. Upon receiving the control commands, the controller component can determine the target controlled terminal based on the control commands and send the control commands to the first communication component corresponding to the target controlled terminal. In a feasible example embodiment, the control command can be generated as follows: the second device obtains the current user identifier of the current controlling user corresponding to the second device, and determines the first device serial number of the first device that the second device can control based on the current user identifier; the control command is generated based on the first device serial number of the first device; or the control command is generated based on the first device serial number and the instruction data input by the current controlling user according to the current state of the first device. That is, in the process of generating the control command, the corresponding control command can be generated directly based on the first device serial number, or the corresponding control command can be generated based on the current device state of the first device, or the need to issue the corresponding control command can be determined based on the video stream uploaded by the first device. This example does not impose any special restrictions on this.

[0130] In one possible example embodiment, the process of generating control commands based on the device status of the first device can be implemented as follows: the first device sends a first device status event to the controller component through the first data channel; the controller component is further configured to send the first device status event to a second communication component; the second communication component is further configured to send the first device status event to the second device through a second data channel; the second device is further configured to determine and display the current status of the first device based on the first device status event, and receive command data input by the current user based on the displayed current status. That is, after viewing the corresponding first device status event, the current user can input corresponding commands based on the specific status of the first device, thereby enabling the second device to generate corresponding control commands based on the input commands.

[0131] In one possible example embodiment, when generating control commands, the second device described above may not consider the first device it can control, but instead directly generate control commands and let the server determine which first device to send the control command to. Specifically, in actual use, this can be achieved as follows: the controller component determines the first device serial number of the first device that the second device can control based on the current user identifier included in the control command, and takes the first device corresponding to the first device serial number as the target controlled device; or it determines the target controlled device based on the first device serial number included in the control command. That is, after receiving a control command, the controller component can determine whether it needs to determine which first device to send the control command to based on whether the control command includes the first device serial number of the first device; if it includes the first device serial number, it can forward it directly; if it does not include the first device serial number, it can determine the corresponding first device based on the first device serial number of the first device bound to the current user identifier; in actual application, it can determine this according to actual needs, and this example does not impose any special restrictions on this.

[0132] The video distribution system described in the exemplary embodiments of this disclosure will be further explained and described below with reference to the accompanying drawings.

[0133] Specifically, firstly, refer to Figure 4 As shown, the controlled end (Drone) initializes, obtains the public IP address and port through the address recognition service WebRTCSTUN, and sends a WebRTC offer to the controller component. The controller component then generates an RTC Actor thread (the first communication component) to respond; secondly, refer to... Figure 5As shown, the first communication component, RTC Actor, determines whether to rely on a Codec Actor (encoder / decoder) based on the original and target parameter information. Simultaneously, it returns the first real communication address and the second session description protocol packet to the controlled end (Drone), thus successfully establishing the connection. Further, if the Codec Actor is relied upon, the first communication component continuously forwards the RTP data from the audio and video channels. The Codec Actor receives this data, integrates it into a raw stream, and sends it to the decoder for decoding / the encoder for encoding. After completing its task, it continuously returns h264 / opus samples to the first communication component. If the Codec Actor is not relied upon, the first communication component automatically collects the RTP data from the audio and video channels and synthesizes it into h264 / opus samples. Then, refer to... Figure 6 As shown, the first communication component sends the h264 / opus samples to the SFU distributor or the RTSP streaming server in the cloud by channel; further, refer to... Figure 7 As shown, the controller attempts to access the controller component in a similar manner, entering a control password (compared in the database) through the ICE (Internet Communications Engine, middleware platform) HTTP interface to gain control of a device; furthermore, refer to... Figure 8 As shown, the controller component will generate a second communication component corresponding to the control terminal; further, refer to... Figure 9 As shown, after the control end establishes a connection, the second communication component retrieves the h264 / opus sample from the SFU repeater based on the controlled end's channel ID (i.e., the rule), decomposes it into RTP packets, and sends them to the control end. At this point, the control end receives the influence of the controlled end. Based on this, refer to... Figure 10 As shown, the control terminal sends control commands with device IDs to the controller component. The controller component identifies the corresponding first communication component based on the device ID and forwards the command. This achieves the goal of sending commands from the control terminal to the controlled terminal. Conversely, the controlled terminal reports events to the controller component. The controller component identifies the corresponding control terminal / viewer terminal based on the SFU consumer and forwards the terminal events according to the rules. Similarly, refer to... Figure 11 As shown, the viewer can retrieve the video feed from the controlled device without entering a password; specifically, first, obtain your own public IP address and port; then, refer to... Figure 12 As shown, the controller component will also establish a third communication component with the viewer; finally, refer to Figure 13 As shown, the third communication component will obtain the h264 / opus sample based on the channel ID, convert it into an RTP packet, and send it to the viewer.

[0134] Thus, the video distribution system and command interaction system described in the exemplary embodiments of this disclosure have been completed. Furthermore, the exemplary embodiments of this disclosure also provide a video distribution method, which can be configured on a server side. The server side includes a first communication component corresponding to a first device, a second communication component corresponding to a second device, and a data distribution component. The first device is connected end-to-end to the first communication component via a first data channel, and the second device is connected end-to-end to the second communication component via the second data channel. Specifically, the video distribution method may include: the first communication component receiving an original bitstream sent by the first device through the first data channel, and sending a target video stream corresponding to the original bitstream to the data distribution component; the data distribution component sending the target video stream to the second communication component; and the second communication component pushing the target video stream to the second device through the second data channel.

[0135] In one exemplary embodiment of this disclosure, the server further includes a controller component and an address recognition service. The video distribution method further includes: the address recognition service receiving a first address recognition request sent by a first device, recognizing a first virtual communication address included in the first address recognition request to obtain a first real communication address, and sending the first real communication address to the first device; the controller component receiving a first session description protocol packet generated based on the first real communication address sent by the first device, and activating a first communication component corresponding to the first device; the first communication component constructing a second session description protocol packet, and performing address matching on the first session description protocol packet and the second session description protocol packet to obtain a first target candidate address pair; and the first communication component establishing the first data channel based on the first target candidate address pair.

[0136] In one exemplary embodiment of this disclosure, the video distribution method further includes: the first communication component determining whether the original bitstream needs to be transcoded based on the original parameter information of the original bitstream that the first device can produce and the target parameter information that the first communication component can support; and when it is determined that transcoding is required, sending the original bitstream to the first encoder-decoder; the first encoder-decoder transcoding the original bitstream according to the target parameter information to obtain the target video stream.

[0137] In one exemplary embodiment of this disclosure, the video distribution method further includes: the first encoder-decoder calling the encoding and decoding algorithm included in the first encoder-decoder to resample the original audio RTP packets included in the original bitstream to obtain redundant audio data, and obtaining the target video stream based on the transcoded original bitstream and the redundant audio data.

[0138] In one exemplary embodiment of this disclosure, the video distribution method further includes: a first communication component sending the original bitstream to a second encoder-decoder; the second encoder-decoder determining the original frame rate of the original bitstream based on the original parameter information included in a first session description protocol packet; the second encoder-decoder obtaining keyframes to be processed from the original bitstream based on the original frame rate and transmitting the keyframes to be processed to a neural network model; the neural network model extracting original feature points included in the keyframes to be processed and identifying target objects included in the keyframes to be processed based on the original feature points; the neural network model constructing a label filter based on the target object and the original parameter information and transmitting the label filter to the second encoder-decoder; the second encoder-decoder labeling the keyframes to be processed in the original bitstream according to the label filter to obtain a target video stream and sending the target video stream to the first communication component.

[0139] In one exemplary embodiment of this disclosure, the video distribution method further includes: a controller component responding to a video acquisition request sent by a third device terminal, determining a target video stream to be sent based on a token identifier included in the video acquisition request; the controller component allocating a channel identifier required for transmitting the target video stream to be sent to the third device terminal, and sending the channel identifier to a data distribution component; and the data distribution component pushing the target video stream to be sent to the third device terminal based on the channel identifier.

[0140] In one exemplary embodiment of this disclosure, the video distribution method further includes: the controller component pushing the original bitstream to a relay server, so that the relay server pushes the original bitstream to a display terminal; wherein the display terminal includes one or more of a second device terminal, a third device terminal, and other display terminals with interactive functions.

[0141] This disclosure also provides an example embodiment of an instruction interaction method configured on a server side. The server side includes a first communication component corresponding to a first device, a second communication component corresponding to a second device, and a controller component. The first device is connected end-to-end to the first communication component via a first data channel, and the second device is connected end-to-end to the second communication component via the second data channel. Specifically, the instruction interaction method may include: the controller component receiving a control instruction from the second device acting on the first device via the second data channel, and sending the control instruction to the first communication component; the first communication component sending the control instruction to the first device via the first data channel, so that the first device executes the control instruction.

[0142] In one exemplary embodiment of this disclosure, sending the control command to the first communication component can be achieved by: determining the target controlled terminal according to the control command, and sending the control command to the first communication component corresponding to the target controlled terminal.

[0143] In one exemplary embodiment of this disclosure, the control instruction is generated by the second device terminal in the following manner: obtaining the current user identifier of the current control user corresponding to the second device terminal, and determining the first device serial number of the first device terminal that the second device terminal can control based on the current user identifier; generating the control instruction based on the first device serial number of the first device terminal; or generating the control instruction based on the first device serial number and the instruction data input by the current user according to the current state of the first device terminal.

[0144] In one exemplary embodiment of this disclosure, the instruction interaction method further includes: the controller component receiving a first device status event sent by a first device terminal and sending the first device status event to a second communication component; the second communication component sending the first device status event to a second device terminal through a second data channel, so that the second device terminal determines and displays the current status of the first device terminal based on the first device status event, and receives instruction data input by the current user based on the displayed current status.

[0145] In one exemplary embodiment of this disclosure, determining the target controlled terminal according to the control instruction can be achieved in the following ways: determining the first device serial number of the first device terminal that can be controlled by the second device terminal according to the current user identifier included in the control instruction, and taking the first device terminal corresponding to the first device serial number as the target controlled terminal; or determining the target controlled terminal according to the first device serial number included in the control instruction.

[0146] The specific details of the aforementioned video distribution method and command interaction method have been described in detail in the corresponding video distribution system and command interaction system, so they will not be repeated here.

[0147] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0148] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0149] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0150] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0151] The following reference Figure 14 To describe an electronic device 1400 according to such an embodiment of the present disclosure. Figure 14 The electronic device 1400 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0152] like Figure 14 As shown, the electronic device 1400 is manifested in the form of a general-purpose computing device. The components of the electronic device 1400 may include, but are not limited to: at least one processing unit 1410, at least one storage unit 1420, a bus 1430 connecting different system components (including storage unit 1420 and processing unit 1410), and a display unit 1440.

[0153] The storage unit stores program code that can be executed by the processing unit 1410, causing the processing unit 1410 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 1410 can execute the video distribution method and instruction interaction method described above.

[0154] Storage unit 1420 may include readable media in the form of volatile storage units, such as random access memory (RAM) 14201 and / or cache memory 14202, and may further include read-only memory (ROM) 14203.

[0155] Storage unit 1420 may also include a program / utility 14204 having a set (at least one) of program modules 14205, such program modules 14205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0156] Bus 1430 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0157] Electronic device 1400 can also communicate with one or more external devices 1500 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1400, and / or any device that enables electronic device 1400 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1450. Furthermore, electronic device 1400 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1460. As shown, network adapter 1460 communicates with other modules of electronic device 1400 via bus 1430. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0158] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0159] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of this disclosure described in the "Exemplary Methods" section above.

[0160] The program product for implementing the above-described method according to embodiments of the present disclosure may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0161] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0162] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0163] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0164] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0165] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0166] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention described herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not invented by this disclosure. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A video distribution system, characterized in that, include: Server-side, first device-side, and second device-side; The first device is the controlled device, and the second device is the master control device. The first device is an offline device; The first device does not support video playback or viewing online; it can only capture video. The server side includes a first communication component corresponding to the first device side, a second communication component corresponding to the second device side, and a data distribution component; The first device is connected end-to-end with the first communication component through the first data channel, and the second device is connected end-to-end with the second communication component through the second data channel. The first device is configured to send the raw bitstream to the first communication component through the first data channel; The first communication component is configured to send a target video stream corresponding to the original bitstream to the data distribution component; The data distribution component is configured to send the target video stream to the second communication component; The second communication component is configured to distribute the target video stream to the second device via the second data channel.

2. The video distribution system according to claim 1, characterized in that, The first communication component includes a Web RTC communication component and / or a Web Socket communication component; the first data channel includes a Web RTC data channel and / or a WebSocket data channel.

3. The video distribution system according to claim 1, characterized in that, The second communication component includes a Web RTC communication component; the second data channel includes a Web RTC data channel.

4. The video distribution system according to claim 1, characterized in that, The server-side also includes address identification services; The first device is also configured to establish a first data channel between the first device and the first communication component through the address identification service.

5. The video distribution system according to claim 4, characterized in that, The first device is further configured to: establish a first communication connection between the first device and the address identification service, and send a first address identification request to the address identification service based on the first communication connection; The second device is further configured to: establish a second communication connection between the second device and the address identification service, and send a second address identification request to the address identification service based on the second communication connection.

6. The video distribution system according to claim 5, characterized in that, The first communication connection is a first UDP connection, and the second communication connection is a second UDP connection.

7. The video distribution system according to claim 5, characterized in that, The server-side also includes a controller component; The address identification service is configured to: respond to the first address identification request, identify the first virtual communication address included in the first address identification request, obtain the first real communication address, and send the first real communication address to the first device. The first device is further configured to: construct a first session description protocol packet based on a first real communication address; and send the first session description protocol packet to the controller component to request the establishment of the first data channel; The controller component is configured to: enable the first communication component corresponding to the first device; The address identification service is further configured to: identify the second virtual communication address of the first communication component to obtain the second real communication address; The first communication component is configured to: construct a second session description protocol packet based on the second real communication address; and establish a first data channel based on the first session description protocol packet and the second session description protocol packet.

8. The video distribution system according to claim 7, characterized in that, The first real communication address includes a first IP address and a first port address; the second real communication address includes a second IP address and a second port address. The first device is further configured to: construct a first candidate address set based on the first IP address and the first port address, and construct a first session description protocol packet based on the first candidate address set; The first communication component is configured to: construct a second candidate address set based on the second IP address and the second port address, and construct a second session description protocol packet based on the second candidate address set; use the first session description protocol packet as a remote connection object, and perform address matching on the first session description protocol packet and the second session description protocol packet to obtain a first target candidate address pair; The first data channel is established based on the first target candidate address pair.

9. The video distribution system according to claim 7, characterized in that, The first communication component is further configured to: establish a first node connection interface, and obtain a first time node for establishing the first node connection interface and a second time node for receiving the first session description protocol packet; Calculate the time difference between the first time node and the second time node; When the time difference is determined to be greater than a preset time threshold, the establishment of the first data channel is determined to have failed; when the time difference is determined to be less than or equal to the preset time threshold, the establishment of the first data channel is determined to have succeeded. If it is determined that the first data channel has failed to be established, the first communication connection is disconnected.

10. The video distribution system according to claim 1, characterized in that, The first device also includes a data collector and a transcoder; The data acquisition device is configured to acquire raw video streams and / or raw audio streams; The transcoder is configured to: transcode the original video stream and / or the original audio stream to obtain original video data packets and / or original audio data packets, and obtain the original bitstream based on the original video data packets and / or original audio data packets; The first device is also configured to send the raw bitstream to the first communication component via the first data channel.

11. The video distribution system according to claim 1, characterized in that, The server also includes a first encoder-decoder; The first communication component is further configured to: determine whether the original bitstream needs to be transcoded based on the original parameter information of the original bitstream that the first device can produce and the target parameter information that the first communication component can support, and send the original bitstream to the first encoder-decoder when it is determined that transcoding is required; The first encoder-decoder is further configured to transcode the original bitstream according to the target parameter information to obtain a target video stream corresponding to the original bitstream, and to transmit the target video stream to the first communication component.

12. The video distribution system according to claim 11, characterized in that, The first encoder-decoder is further configured to: call the encoding and decoding algorithm included in the first encoder-decoder to resample the original audio RTP packets included in the original bitstream to obtain redundant audio data, and obtain the target video stream based on the transcoded original bitstream and the redundant audio data.

13. The video distribution system according to claim 1, characterized in that, The server also includes a second encoder / decoder and a neural network model; The first communication component is also configured to send the raw bitstream to the second encoder-decoder; The second encoder-decoder is configured to: determine the original frame rate of the original bitstream based on the original parameter information included in the first session description protocol packet, and obtain the key frame to be processed from the original bitstream based on the original frame rate; and transmit the key frame to be processed to the neural network model. The neural network model is configured to: extract the original feature points included in the keyframe to be processed, and identify the target object included in the keyframe to be processed based on the original feature points; construct a label filter based on the target object and the original parameter information, and transmit the label filter to the second encoder-decoder; The second encoder-decoder is further configured to: mark the keyframes to be processed in the original bitstream according to the marking filter to obtain the target video stream, and send the target video stream to the first communication component.

14. The video distribution system according to claim 1, characterized in that, The second device is further configured to: determine the standard jitter-resistant latency required by the server, and send the standard jitter-resistant latency to the second communication component through the second data channel; The second communication component is further configured to: receive multiple target RTP packets sent by the standard jitter-resistant delay data distribution component at intervals, and send the multiple target RTP packets to the second device end through the second data channel.

15. The video distribution system according to claim 14, characterized in that, The second device is further configured to: determine the target anti-jitter latency of the target video stream based on the maximum video transmission frame received within a standard time period corresponding to the target video stream; and determine the standard anti-jitter latency required by the server based on the target anti-jitter latency.

16. The video distribution system according to claim 15, characterized in that, The second device is also configured as follows: The channel transmission rate of the second data channel when transmitting the current target RTP packet in the target video stream is obtained, as well as the real-time video transmission frame of the current target RTP packet within the standard time. The maximum video transmission frame and the average video transmission frame are determined based on the real-time video transmission frames, and the noise variance of the signal transmission noise of the current target RTP packet during transmission is calculated. The target anti-jitter delay is calculated based on the maximum video transmission frame, the average video transmission frame, the channel transmission rate, and the noise variance.

17. The video distribution system according to claim 16, characterized in that, The real-time video transmission frame includes the current video frame and the previous video frame corresponding to the current video frame; The second device is further configured to: calculate the frame delay of the target video stream based on the current frame time of receiving the current video frame and the previous frame time of receiving the previous video frame, and calculate the video frame difference of the target video stream based on the current video frame of the current video frame and the previous video frame of the previous video frame. Based on the frame delay, channel transmission rate, network queuing delay of the current target RTP packet during transmission, and video frame difference, the frame residual is calculated, and the average noise of signal transmission noise is determined based on the frame residual and a preset probability coefficient. The noise variance is determined based on the frame residual and the average noise.

18. The video distribution system according to claim 17, characterized in that, The second device is further configured to: calculate a first proportional coefficient based on the frame index and total number of frames of the current target RTP packet, and calculate the average video transmission frames based on the first proportional coefficient and the current video frame; The frame variance is calculated based on the average video transmission frame and the current video frame, and the maximum video transmission frame is determined based on the frame variance.

19. The video distribution system according to claim 16, characterized in that, The second device is also configured as follows: Calculate the first difference between the maximum video transmission frame and the average video transmission frame, and calculate the first product of the first difference and the reciprocal of the channel transmission rate. Calculate the square root of the noise variance, and calculate the second product between the preset noise figure and the square root. The second difference between the result of the second product operation and the preset noise reduction constant is calculated, and the target anti-jitter delay is obtained based on the second difference and the result of the first product operation.

20. The video distribution system according to claim 15, characterized in that, The second device is also configured to adjust the target anti-jitter latency in real time based on the maximum video transmission frame.

21. The video distribution system according to claim 20, characterized in that, The second device terminal is also configured as follows: The maximum video transmission frame at the current moment is compared with the maximum video transmission frame at the next moment, and when it is determined that the maximum video transmission frame at the current moment is greater than the maximum video transmission frame at the next moment, the target anti-jitter latency is reduced. When it is determined that the maximum video transmission frame at the current moment is less than the maximum video transmission frame at the next moment, the target anti-jitter latency is increased.

22. The video distribution system according to claim 1, characterized in that, The server-side also includes a controller component; The second device is configured to send control commands to the controller component via the second data channel, which are applied to the first device. The controller component is configured to send the control command to the first communication component; The first communication component is configured to send the control command to the first device via the first data channel; The first device is also configured to execute the control command.

23. The video distribution system according to claim 22, characterized in that, The first device is further configured to: parse the control command to obtain the instruction type included in the control command, and execute the target instruction corresponding to the instruction type.

24. The video distribution system according to claim 22, characterized in that, The second device is also configured to: generate the control commands; The controller component is further configured to: determine the target controlled terminal according to the control command, and send the control command to the first communication component corresponding to the target controlled terminal.

25. The video distribution system according to claim 24, characterized in that, The second device is also configured as follows: Obtain the current user identifier of the current controlling user corresponding to the second device terminal, and determine the first device serial number of the first device terminal that the second device terminal can control based on the current user identifier; The control command is generated based on the first device serial number of the first device terminal; or The control command is generated based on the first device serial number and the instruction data input by the current control user according to the current state of the first device.

26. The video distribution system according to claim 25, characterized in that, The first device is also configured to send a first device status event to the controller component via the first data channel; The controller component is also configured to send the first device status event to the second communication component; The second communication component is also configured to send the first device status event to the second device via the second data channel; The second device is also configured to determine and display the current state of the first device based on the first device state event, and to receive instruction data input by the current user based on the displayed current state.

27. The video distribution system according to claim 24, characterized in that, The controller component is further configured to: determine the first device serial number of the first device terminal that the second device terminal can control based on the current user identifier included in the control command, and designate the first device terminal corresponding to the first device serial number as the target controlled terminal; or The target controlled terminal is determined based on the first device serial number included in the control command.

28. The video distribution system according to claim 1, characterized in that, The video distribution system also includes a third device, which is connected end-to-end to a third communication component via a third data channel. The third device is configured to: obtain a token identifier of the interactive page, generate a video acquisition request based on the token identifier, and send the video acquisition request to the controller component; The controller component is further configured to: in response to the video acquisition request, determine the target video stream to be sent based on the token identifier, allocate a channel identifier required for transmitting the target video stream to be sent to the third device, and send the channel identifier to the data distribution component; The data distribution component is configured to push the target video stream to be sent to the third device based on the channel identifier.

29. The video distribution system according to claim 1, characterized in that, The server side also includes a relay server; The first communication component is configured to push the target video stream to a relay server; The relay server is also configured to push the target video stream to a display terminal; wherein the display terminal includes one or more of a second device terminal, a third device terminal, and other display terminals with interactive functions.

30. A command interaction system, characterized in that, include: Server-side, first device-side, and second device-side; The first device is the controlled device, and the second device is the master control device. The first device is an offline device; The first device does not support video playback or viewing online; it can only capture video. The server side includes a controller component, a first communication component corresponding to the first device side, and a second communication component corresponding to the second device side; The first device is connected end-to-end with the first communication component through the first data channel, and the second device is connected end-to-end with the second communication component through the second data channel. The second device is configured to send control commands to the controller component via the second data channel, which act on the first device. The controller component is configured to send the control command to the first communication component; The first communication component is configured to send the control command to the first device via the first data channel; The first device is also configured to execute the control command.

31. The instruction interaction system according to claim 30, characterized in that, The first device is further configured to: parse the control command to obtain the instruction type included in the control command, and execute the target instruction corresponding to the instruction type.

32. The instruction interaction system according to claim 31, characterized in that, The second device is also configured to: generate the control commands; The controller component is further configured to: determine the target controlled terminal according to the control command, and send the control command to the first communication component corresponding to the target controlled terminal.

33. The instruction interaction system according to claim 32, characterized in that, The second device is also configured as follows: Obtain the current user identifier of the current controlling user corresponding to the second device terminal, and determine the first device serial number of the first device terminal that the second device terminal can control based on the current user identifier; The control command is generated based on the first device serial number of the first device terminal; or The control command is generated based on the first device serial number and the instruction data input by the current user based on the current state of the first device.

34. The instruction interaction system according to claim 33, characterized in that, The first device is also configured to send a first device status event to the controller component via the first data channel; The controller component is also configured to send the first device status event to the second communication component; The second communication component is also configured to send the first device status event to the second device via the second data channel; The second device is also configured to determine and display the current state of the first device based on the first device state event, and to receive instruction data input by the current user based on the displayed current state.

35. The instruction interaction system according to claim 32, characterized in that, The controller component is configured to: determine the first device serial number of the first device terminal that the second device terminal can control based on the current user identifier included in the control command, and designate the first device terminal corresponding to the first device serial number as the target controlled terminal; or The target controlled terminal is determined based on the first device serial number included in the control command.

36. A video distribution method, characterized in that, Configured on the server side, the server side includes a first communication component corresponding to the first device side, a second communication component corresponding to the second device side, and a data distribution component. The first device side is connected end-to-end with the first communication component through a first data channel, and the second device side is connected end-to-end with the second communication component through a second data channel. The first device side is the controlled device side, and the second device side is the master control device side. The first device is an offline device; The first device does not support video playback or viewing online; it can only capture video. The video distribution method includes: The first communication component receives the raw bitstream sent by the first device through the first data channel and sends the target video stream corresponding to the raw bitstream to the data distribution component; The data distribution component sends the target video stream to the second communication component; The second communication component pushes the target video stream to the second device via the second data channel.

37. The video distribution method according to claim 36, characterized in that, The server-side also includes a controller component and an address recognition service, and the video distribution method further includes: The address identification service receives a first address identification request sent by a first device, identifies the first virtual communication address included in the first address identification request, obtains a first real communication address, and sends the first real communication address to the first device. The controller component receives a first session description protocol packet generated based on a first real communication address sent by the first device, and starts the first communication component corresponding to the first device. The first communication component constructs a second session description protocol packet and performs address matching on the first session description protocol packet and the second session description protocol packet to obtain a first target candidate address pair; The first communication component establishes the first data channel based on the first target candidate address pair.

38. The video distribution method according to claim 36, characterized in that, The video distribution method further includes: The first communication component determines whether the original bitstream needs to be transcoded based on the original parameter information of the original bitstream that the first device can produce and the target parameter information that the first communication component can support. When it is determined that transcoding is required, the original bitstream is sent to the first encoder-decoder. The first encoder-decoder transcodes the original bitstream according to the target parameter information to obtain the target video stream.

39. The video distribution method according to claim 38, characterized in that, The video distribution method further includes: The first encoder-decoder calls the encoding and decoding algorithm included in the first encoder-decoder to resample the original audio RTP packets included in the original bitstream to obtain redundant audio data, and obtains the target video stream based on the transcoded original bitstream and the redundant audio data.

40. The video distribution method according to claim 36, characterized in that, The video distribution method further includes: The first communication component sends the raw bitstream to the second encoder / decoder, which determines the raw frame rate of the raw bitstream based on the raw parameter information included in the first session description protocol packet. The second encoder-decoder obtains the key frame to be processed from the original bitstream according to the original frame rate, and transmits the key frame to be processed to the neural network model; The neural network model extracts the original feature points included in the key frame to be processed, and identifies the target object included in the key frame to be processed based on the original feature points; The neural network model constructs a label filter based on the target object and the original parameter information, and transmits the label filter to the second encoder-decoder; The second encoder-decoder marks the keyframes to be processed in the original bitstream according to the marking filter to obtain the target video stream, and sends the target video stream to the first communication component.

41. The video distribution method according to claim 36, characterized in that, The video distribution method further includes: In response to a video acquisition request sent by a third device, the controller component determines the target video stream to be sent based on the token identifier included in the video acquisition request. The controller component allocates a channel identifier to the third device for transmitting the target video stream to be sent, and sends the channel identifier to the data distribution component; The data distribution component pushes the target video stream to be sent to the third device according to the channel identifier.

42. The video distribution method according to claim 36, characterized in that, The video distribution method further includes: The controller component pushes the target video stream to the relay server, so that the relay server pushes the target video stream to the display terminal; The display terminal includes one or more of a second device terminal, a third device terminal, and other display terminals with interactive functions.

43. A method for instruction interaction, characterized in that, Configured on the server side, the server side includes a first communication component corresponding to the first device side, a second communication component corresponding to the second device side, and a controller component. The first device side is connected to the first communication component end-to-end through a first data channel, and the second device side is connected to the second communication component end-to-end through a second data channel. The first device side is the controlled device side, and the second device side is the master control device side. The first device is an offline device; The first device does not support video playback or viewing online; it can only capture video. The instruction interaction method includes: The controller component receives control commands sent by the second device terminal to the first device terminal via the second data channel, and sends the control commands to the first communication component; The first communication component sends the control command to the first device through the first data channel, so that the first device executes the control command.

44. The instruction interaction method according to claim 43, characterized in that, Sending the control command to the first communication component includes: The target controlled terminal is determined according to the control command, and the control command is sent to the first communication component corresponding to the target controlled terminal.

45. The instruction interaction method according to claim 43, characterized in that, The control command is generated by the second device in the following manner: Obtain the current user identifier of the current controlling user corresponding to the second device terminal, and determine the first device serial number of the first device terminal that the second device terminal can control based on the current user identifier; The control command is generated based on the first device serial number of the first device terminal; or The control command is generated based on the first device serial number and the instruction data input by the current user based on the current state of the first device.

46. ​​The instruction interaction method according to claim 45, characterized in that, The instruction interaction method further includes: The controller component receives a first device status event sent by the first device terminal and sends the first device status event to the second communication component; The second communication component sends the first device status event to the second device through the second data channel, so that the second device can determine and display the current status of the first device based on the first device status event, and receive the instruction data input by the current user based on the displayed current status.

47. The instruction interaction method according to claim 44, characterized in that, Determining the target controlled terminal according to the control command includes: Based on the current user identifier included in the control command, determine the first device serial number of the first device terminal that the second device terminal can control, and designate the first device terminal corresponding to the first device serial number as the target controlled terminal; or The target controlled terminal is determined based on the first device serial number included in the control command.

48. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the video distribution method according to any one of claims 36-42 and the instruction interaction method according to any one of claims 43-47.

49. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the video distribution method of any one of claims 36-42 and the instruction interaction method of any one of claims 43-47 by executing the executable instructions.

Images