Multimedia data processing method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a multimedia data processing method and device, electronic equipment, a computer readable storage medium and a computer program product. The method is applied to a plug-flow server. The method comprises the steps that first multimedia data corresponding to a first transmission protocol and sent by a robot are received, and the first multimedia data are obtained by the robot through analysis processing of subscription data corresponding to a multimedia collection component; performing protocol conversion processing on the first multimedia data to obtain second multimedia data corresponding to a second transmission protocol; and establishing a communication connection based on the second transmission protocol with at least one terminal, and executing stream pushing processing corresponding to the second multimedia data to the at least one terminal through the communication connection. According to the method and the device, the subscription data receiving process and the stream pushing process can be decoupled, and the data stream pushing convenience and flexibility in a robot multimedia data transmission scene are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a multimedia data processing method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] The robot video display transmission system is a complex system, which is usually used in remote monitoring, remote operation, video conferencing, security monitoring and various automation applications. In related technologies, the Web Real-Time Communication (Webrtc) protocol is usually used to realize video transmission between the robot and the client. Traditional Webrtc video transmission is based on point-to-point transmission. The application on the robot end requires deep coupling business development with the client based on the Webrtc protocol, which is not conducive to business expansion. Summary of the invention

[0003] The embodiments of the present application provide a multimedia data processing method, device, electronic device, computer-readable storage medium and computer program product, which can decouple the subscription data receiving process from the streaming process, and improve the convenience and flexibility of data streaming in robot multimedia data transmission scenarios.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The embodiment of the present application provides a multimedia data processing method, which is applied to a streaming server, including:

[0006] Receiving first multimedia data corresponding to a first transmission protocol sent by the robot, wherein the first multimedia data is obtained by the robot parsing subscription data corresponding to a multimedia acquisition component;

[0007] Performing protocol conversion processing on the first multimedia data to obtain second multimedia data corresponding to a second transmission protocol;

[0008] A communication connection based on the second transmission protocol is established with at least one terminal, and streaming processing corresponding to the second multimedia data is performed to the at least one terminal through the communication connection.

[0009] The embodiment of the present application also provides a multimedia data processing method, which is applied to a robot, comprising:

[0010] Obtain subscription data corresponding to the multimedia acquisition component;

[0011] Parsing the subscription data to obtain first multimedia data;

[0012] The first multimedia data is sent to a streaming server based on the first transmission protocol, so that the streaming server performs protocol conversion processing on the first multimedia data to obtain second multimedia data corresponding to a second transmission protocol, establishes a communication connection based on the second transmission protocol with at least one terminal, and performs streaming processing corresponding to the second multimedia data to the at least one terminal through the communication connection.

[0013] The embodiment of the present application provides a multimedia data processing device, which is applied to a streaming server, including:

[0014] A receiving module, configured to receive first multimedia data corresponding to a first transmission protocol sent by the robot, wherein the first multimedia data is obtained by the robot parsing subscription data corresponding to the multimedia acquisition component;

[0015] a conversion module, configured to perform protocol conversion processing on the first multimedia data to obtain second multimedia data corresponding to a second transmission protocol;

[0016] The streaming module is used to establish a communication connection based on the second transmission protocol with at least one terminal, and perform streaming processing corresponding to the second multimedia data to the at least one terminal through the communication connection.

[0017] In the above scheme, the streaming module is also used to perform the following processing for each of the terminals: sending a service interface corresponding to the second multimedia data to the terminal; in response to receiving a first request signaling generated by the terminal, creating a first instance for sending the second multimedia data based on the second transmission protocol, and setting a first remote description of the first instance based on the first request signaling, wherein the first request signaling is generated by the terminal based on the service interface; generating a first response signaling including the first remote description, and sending the first response signaling to the terminal, so that the terminal creates a second instance for receiving the second multimedia data based on the second transmission protocol based on the first response signaling; when the creation of the first instance and the second instance is completed, determining that the communication connection is established.

[0018] In the above scheme, the streaming module is also used to encode the second multimedia data and add the encoded second multimedia data to the first instance; create a second request signaling corresponding to the second multimedia data based on the service interface, wherein the second request signaling includes address information of the second multimedia data; based on the communication connection of the second transmission protocol, send the second request signaling to the terminal, so that the second instance of the terminal obtains the encoded second multimedia data in the first instance based on the address information, and decodes and plays the encoded second multimedia data.

[0019] The above-mentioned device also includes a recording module, which is used to respond to receiving a recording request from the terminal, start a multimedia data recording mode according to a first configuration file, record the second multimedia data, obtain a recording file, and send a response message carrying the recording file to the terminal.

[0020] The embodiment of the present application further provides a multimedia data processing device, which is applied to a robot and includes:

[0021] An acquisition module, used to acquire subscription data corresponding to the multimedia acquisition component;

[0022] A parsing module, used for parsing the subscription data to obtain first multimedia data;

[0023] A sending module is used to send the first multimedia data to a streaming server based on the first transmission protocol, so that the streaming server performs protocol conversion processing on the first multimedia data to obtain second multimedia data corresponding to the second transmission protocol, establish a communication connection based on the second transmission protocol with at least one terminal, and perform streaming processing corresponding to the second multimedia data to the at least one terminal through the communication connection.

[0024] In the above scheme, the acquisition module is also used for the robot to register in the node manager, wherein the multimedia acquisition component is in a registered state in the node manager, and the multimedia acquisition component has published a first topic to the node manager; when the robot is successfully registered in the node manager, it subscribes to the first topic from the node manager; in response to the multimedia acquisition component publishing the subscription data to the first topic, the subscription data corresponding to the multimedia acquisition component is obtained from the first topic.

[0025] In the above solution, the acquisition module is further used to respond to the registration instruction of the terminal, and the robot is registered in the node manager, or the robot is registered in the node manager according to the second configuration file.

[0026] In the above solution, the acquisition module is further used to subscribe to the first topic from the node manager in response to a topic subscription instruction of the terminal, or to subscribe to the first topic from the node manager according to the second configuration file.

[0027] In the above scheme, the parsing module is also used to extract the code stream of the subscription data to obtain a multimedia code stream; extract data from the multimedia code stream to obtain original audio data and original video data; align and merge the original audio data and the original video data to obtain the first multimedia data.

[0028] In the above scheme, the parsing module is also used to align and merge the original audio data and the original video data to obtain an aligned and merged result; and to adjust the aligned and merged result based on the color value channel to obtain the first multimedia data.

[0029] The above-mentioned device also includes a control module, which is used to send the configuration address information of the multimedia acquisition component to the terminal; in response to receiving a data acquisition instruction for the configuration address information sent by the terminal, control the multimedia acquisition component to be in a working state; in response to receiving a data stop acquisition instruction for the configuration address information sent by the terminal, control the multimedia acquisition component to be in a closed state.

[0030] An embodiment of the present application provides an electronic device, including:

[0031] A memory for storing computer executable instructions or computer programs;

[0032] The processor is used to implement a multimedia data processing method provided in an embodiment of the present application when executing the computer executable instructions or computer programs stored in the memory.

[0033] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which are used to implement a multimedia data processing method provided by an embodiment of the present application when executed by a processor.

[0034] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, a multimedia data processing method provided by an embodiment of the present application is implemented.

[0035] The embodiments of the present application have the following beneficial effects:

[0036] The first multimedia data corresponding to the first transmission protocol sent by the robot is received through the push streaming server, wherein the first multimedia data is obtained by the robot parsing the subscription data corresponding to the multimedia acquisition component. In the related art, multimedia data is directly obtained from the robot through the transmission mode of point-to-point connection, so that the acquisition of multimedia data is limited by the distance of the robot local area network. Therefore, by saving the multimedia data in the server, the acquisition of multimedia data is no longer limited by the distance of the robot local area network, which can effectively solve the distance limitation of multimedia data transmission; the first multimedia data is processed by protocol conversion to obtain the second multimedia data corresponding to the second transmission protocol, and a communication connection based on the second transmission protocol is established with at least one terminal, and the push streaming processing corresponding to the second multimedia data is performed to at least one terminal. In the related art, the robot and the terminal are connected point-to-point through the local area network, so each multimedia data of the robot can only be transmitted in real time with one terminal at the same time. In contrast, the present solution saves the multimedia data in the server, so that the terminal that obtains the multimedia data is no longer limited by the number, and the subscription data receiving process and the push streaming process can be decoupled through the push streaming server, so that the push streaming solution can be more flexible. In summary, the embodiments of the present application can effectively solve the distance limitation of multimedia data transmission, realize real-time transmission of the same multimedia data to more than one terminal, and improve the flexibility of the streaming solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a schematic diagram of the architecture of a multimedia data processing system 100 provided in an embodiment of the present application;

[0038] Figure 2A is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present application;

[0039] Figure 2B is a schematic diagram of the structure of an electronic device 700 provided in an embodiment of the present application;

[0040] Figure 3 is a flowchart of a multimedia data processing method provided in an embodiment of the present application;

[0041] Figure 4 is a flowchart of a multimedia data processing method provided in an embodiment of the present application;

[0042] Figure 5 It is a timing flow diagram of the multimedia data processing method provided in an embodiment of the present application;

[0043] Figure 6 It is a process diagram of the multimedia data processing method provided in the embodiment of the present application in the real-time video transmission scenario. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0045] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0046] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0047] In the following description, the terms "first\second\..." involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\..." can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0049] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0050] 1) Web Real-Time Communication (Webrtc) transmission technology: Webrtc transmission technology is a technology that supports web browsers for real-time audio and video communication and data sharing, allowing audio and video communication to be carried out directly between web browsers without the need for additional plug-ins or software; in a supported network environment, Webrtc can directly establish a connection between two clients, reducing the transit link of data transmission and thus reducing latency; Webrtc is based on a series of standard protocols to ensure the real-time and security of communication. Webrtc can be widely used in various scenarios such as online calls, real-time games, and distance education.

[0051] 2) Topic data: Topic data refers to the real-time image or video stream data published by the camera node under the Publish-Subscribe Pattern.

[0052] 3) Real-Time Messaging Protocol (RTMP): RTMP is a network protocol used for real-time data transmission. It is widely used in the field of streaming media transmission, especially in online live broadcast and real-time audio and video communication. RTMP is usually used with streaming media servers.

[0053] In the related technologies, Webrtc video transmission technology is based on point-to-point transmission. When applied on the robot side, it is necessary to develop business based on deep coupling between the Webrtc protocol and the terminal, which is not conducive to subsequent business expansion. Moreover, the robot side is developed based on the Robot Operating System (ROS) protocol. The camera image drive data is published through the ros topic. If other terminals want to obtain the corresponding video data, they need to subscribe to the topic. After the subscription is completed, the communication connection between the robot and the terminal is established through the Webrtc protocol. Since the business coupling degree of the direct connection with the robot is high, it is difficult to solve the signaling exchange during the communication connection process and the subsequent business expansion needs.

[0054] In addition, the robot-side video display transmission system realizes point-to-point connection between the robot and the terminal through the local area network, and the normal Webrtc real-time video transmission can only be transmitted in two directions in real time. One-way transmission requires additional interactive processing. The business coupling degree directly realized between the robot and the terminal is relatively high, and it is difficult to realize the expansion of subsequent business; and because the Webrtc file transmission system can only transmit in two directions in real time, for special application scenarios, such as live broadcast scenarios, it is necessary to subscribe to or switch multiple cameras at the same time, which is difficult to achieve only through Webrtc.

[0055] In view of this, the embodiments of the present application provide a multimedia data processing method, device, electronic device, computer-readable storage medium and computer program product, which can decouple the subscription data receiving process from the streaming process, and improve the convenience and flexibility of data streaming in robot multimedia data transmission scenarios. The electronic device provided in the embodiments of the present application can be implemented as a server, or implemented by a server and a terminal in collaboration. The following is an example of the multimedia data processing method provided by the embodiment of the present application being implemented in collaboration by a server and a terminal.

[0056] For example, see Figure 1 , Figure 1 1 is a schematic diagram of the architecture of a multimedia data processing system 100 provided in an embodiment of the present application, in order to realize supporting a multimedia data processing application, such as Figure 1 As shown, the multimedia data processing system 100 includes: a server 200, a network 300, a robot end 400 (for example, various types of robots), and a terminal 500. The robot end 400 and the terminal 500 are respectively connected to the server 200 via the network 300, wherein the network 300 may be a local area network or a wide area network, or a combination of the two; the terminal 500 is a terminal associated with a user, and a client 510 runs on the terminal 500. The client 510 may be various types of clients, for example, a client or a browser related to real-time display of video.

[0057] In some embodiments, the robot end 400 obtains the subscription data corresponding to the multimedia acquisition component, and parses and sorts the subscription data to obtain first multimedia data; then, the robot end 400 transmits the first multimedia data to the server 200 through the network 300 based on the first transmission protocol; then, after receiving the first multimedia data corresponding to the first transmission protocol sent by the robot end 400, the server 200 performs protocol conversion processing on the first multimedia data to obtain second multimedia data corresponding to the second transmission protocol; then, the server 200 establishes a communication connection based on the second transmission protocol with at least one terminal 500, and transmits the second multimedia data to the terminal 500 through the network 300; then, the terminal 500 decodes the received second multimedia data to obtain the decoded multimedia data, and displays it on the client 510.

[0058] It should be noted that the technical solution provided in this application can be applied to various scenarios, such as remote video, robot visual interaction and other robot video real-time transmission scenarios.

[0059] In some embodiments, taking the remote video scene as an example, first, the robot obtains the subscription data corresponding to the multimedia acquisition component, and parses and sorts the subscription data to obtain the corresponding first video stream data; then, the robot transmits the first video stream data to the streaming server through the first transmission protocol; then, the streaming server performs protocol conversion processing on the first video stream data to obtain second video stream data corresponding to the second transmission protocol; then, the streaming server establishes a communication connection with at least one terminal according to the second transmission protocol, and transmits the second video stream data to the corresponding terminal according to the established communication connection; thereafter, the terminal decodes the acquired second video stream data to obtain the decoded video stream data, and plays it on the client interface of the terminal to complete the remote video transmission of the current scene.

[0060] In other embodiments, the embodiments of the present application can also be implemented with the aid of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or a local area network to achieve data calculation, storage, processing, and sharing.

[0061] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources.

[0062] For example, Figure 1 The server 200 in the example can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The robot end 400 can be various types of robots (such as remote control robots, safety detection robots, etc.). The terminal 500 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car terminal, etc., but is not limited to this. The robot end 400, the terminal 500 and the server 200 can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiments of the present application.

[0063] The structure of the electronic device provided in the embodiment of the present application is further described below. Taking the electronic device as a server as an example, see Figure 2A , Figure 2A is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present application, Figure 2A The electronic device 600 shown includes: at least one processor 610, a memory 640, and at least one network interface 620. The various components in the electronic device 600 are coupled together via a bus system 630. It is understood that the bus system 630 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 630 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 630 is not described in detail. Figure 2A Various buses are labeled as bus system 630 .

[0064] The processor 610 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0065] The memory 640 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 640 may optionally include one or more storage devices that are physically remote from the processor 610.

[0066] The memory 640 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 640 described in the embodiment of the present application is intended to include any suitable type of memory.

[0067] In some embodiments, memory 640 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0068] Operating system 641, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0069] A network communication module 642, for reaching other computing devices via one or more (wired or wireless) network interfaces 620, exemplary network interfaces 620 include: Bluetooth, Wireless Authentication (WiFi), and Universal Serial Bus (USB);

[0070] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2A The multimedia data processing device 643 stored in the memory 640 is shown, which can be software in the form of programs and plug-ins, including the following software modules: a receiving module 6431, a conversion module 6432, a streaming module 6433 and a recording module 6434. These modules are logical, so they can be arbitrarily combined or further divided according to the functions implemented. It should be pointed out that in Figure 2A For the sake of convenience, all the above modules are shown at once, but it should not be considered that the multimedia data processing device 643 excludes the implementation that can only include the receiving module 6431, the conversion module 6432 and the streaming module 6433. The functions of each module will be explained below.

[0071] The structure of the electronic device for implementing the multimedia data processing method provided in the embodiment of the present application is further described below. Taking the electronic device 700 as an example, see Figure 2B , Figure 2B is a schematic diagram of the structure of an electronic device 700 provided in an embodiment of the present application, Figure 2B The electronic device 700 shown includes a multimedia data processing device 755, which can be software in the form of a program or plug-in, including the following software modules: an acquisition module 7551, a parsing module 7552, a sending module 7553, and a control module 7554. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. It should be noted that in Figure 2B For the sake of convenience, all the above modules are shown at once, but it should not be considered that the multimedia data processing device 755 excludes the implementation that can only include the acquisition module 7551, the parsing module 7552 and the sending module 7553. The functions of each module will be explained below.

[0072] It should be noted that Figure 2B The electronic device 700 shown includes a processor 710, a network interface 720, a bus system 740, an operating system 751, and a network communication module 752. Figure 2A The corresponding modules included in have the same structure and the same function, and the embodiments of the present application will not be repeated here.

[0073] also, Figure 2BThe electronic device 700 shown also includes a user interface 730. The user interface 730 includes one or more output devices 731 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 730 also includes one or more input devices 732, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0074] Figure 2B The memory 750 in the illustrated electronic device 700 also includes a presentation module 753 and an input processing module 754 .

[0075] a presentation module 753 for enabling presentation of information via one or more output devices 731 (e.g., display screen, speaker, etc.) associated with the user interface 730 (e.g., a user interface for operating peripherals and displaying content and information);

[0076] The input processing module 754 is used to detect one or more user inputs or interactions from one of the one or more input devices 732 and translate the detected inputs or interactions.

[0077] The multimedia data processing method provided in the embodiment of the present application will be specifically described below in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0078] See also Figure 3 , Figure 3 is a flowchart of a multimedia data processing method provided in an embodiment of the present application. The multimedia data processing method is applied to a streaming server. Figure 3 The steps shown are explained.

[0079] In step 101, first multimedia data corresponding to a first transmission protocol sent by a robot is received.

[0080] Here, the first multimedia data is obtained by the robot parsing and processing the subscription data corresponding to the multimedia acquisition component.

[0081] It should be noted that the multimedia acquisition component can be a sensor component such as a built-in camera, microphone, etc. of the robot, or a sensor component such as a camera, microphone, etc. external to the robot; the first multimedia data can be the original audio and video data collected by the multimedia acquisition component, or it can be the audio and video data after preliminary processing, and no specific limitation is made here.

[0082] In some embodiments, after the robot obtains the corresponding subscription data through the multimedia acquisition component, it parses the subscription data to obtain first multimedia data; then, the first multimedia data is encapsulated according to the first transmission protocol to obtain a packaged first multimedia data packet; then, the robot sends the packaged first multimedia data packet to the streaming server for receipt by the streaming server; then, after receiving the first multimedia data packet, the streaming server will decode the first multimedia data packet and store the decoded first multimedia data in the streaming server.

[0083] For example, taking a robot for outdoor environment detection as an example, first, the robot collects video data through the built-in camera and encapsulates the video data into a data packet in RTMP format; then, the robot sends the RTMP data packet to the streaming server. After receiving the RTMP data packet, the streaming server decodes the RTMP data packet to obtain the decoded video data, and stores the decoded video data in the streaming server for subsequent terminal calls and viewing.

[0084] In some embodiments, before executing step 101, the following processing can also be performed: implement a handshake operation between the robot and the streaming server based on the first transmission protocol, wherein the handshake operation includes exchanging the first transmission protocol version information, timestamp and random data to ensure that the robot and the streaming server can correctly understand and process the data; after the handshake is successful, the robot sends a connection command message of the first transmission protocol to the streaming server, requesting to establish a connection with the streaming media application configured in the streaming server; after receiving the connection command message, the streaming server will send a confirmation window size message and a bandwidth setting message to the robot to complete the communication connection between the robot and the streaming server. In this way, a safe and reliable communication connection can be established between the robot and the server, thereby improving the overall performance of the system and the user experience.

[0085] For example, the robot sends a handshake request (including version information of the first transmission protocol, timestamp, random data, etc.) to the streaming server. After receiving the handshake request from the robot, the streaming server parses the handshake request and generates a response message (including version information of the first transmission protocol on the streaming server side, timestamp of the streaming server, and random data generated by the streaming server). Then, after receiving the response message from the streaming server, the robot verifies the information in the response message. When the robot verifies successfully, it indicates that the RTMP handshake between the robot and the streaming server is successful. Then, the robot sends a connection command message to the streaming server, requesting to establish a connection with the streaming media application in the streaming server. Subsequently, after receiving the connection command message from the robot, the streaming server sends a confirmation window size and bandwidth setting message to the robot. Finally, after receiving the confirmation window size message and bandwidth setting message sent by the streaming server, the robot adjusts the sending strategy of the robot according to the parameters parsed from the message to confirm that the connection between the robot and the streaming server is successfully established.

[0086] In step 102, protocol conversion is performed on the first multimedia data to obtain second multimedia data corresponding to the second transmission protocol.

[0087] In some embodiments, first, the first multimedia data is parsed to extract the original data streams of the video and audio and related metadata (such as timestamp, frame type, etc.) in the first multimedia data; then, the parsed original data streams are re-encapsulated according to the requirements of the second transmission protocol, including packaging the video and audio data into the data packet format of the second transmission protocol, thereby obtaining the second multimedia data of the second transmission protocol.

[0088] For example, taking the case where the first multimedia data is a video stream in RTMP format and the second multimedia data is a video stream in Webrtc format, after receiving the video stream in RTMP format, the streaming server parses the video stream in RTMP format and obtains H.264-encoded video frames and AAC-encoded audio frames, as well as metadata such as the timestamp of each frame; then, the video frames and audio frames are encapsulated according to the Webrtc specification, where the video frames may be encapsulated into multiple Webrtc video data packets, each of which contains a portion of the video frame data, and the audio frames are also encapsulated into audio data packets, each of which adds Webrtc header information (such as serial number, timestamp, packet type, etc.), thereby obtaining video stream data in Webrtc format. In this way, the video stream in RTMP format is successfully converted to Webrtc format, and is played in real time on the terminal based on the Webrtc protocol, and can be transmitted in a network environment with lower latency and higher quality.

[0089] In step 103, a communication connection based on a second transmission protocol is established with at least one terminal.

[0090] It should be noted that when a communication connection based on the second transmission protocol is established with more than one terminal, a communication connection of the same transmission protocol can be established with each terminal (for example, a communication connection is established with each terminal based on the Webrtc protocol), or a communication connection of a different transmission protocol can be established with each terminal (for example, a communication connection is established with terminal A based on the Webrtc protocol, and a communication connection is established with terminal B based on the RTMP protocol; or a communication connection is established with terminal A and terminal B based on the Webrtc protocol, and a communication connection is established with terminal C based on the RTMP protocol). No specific limitation is made here.

[0091] In some embodiments, the above step 103 can be implemented in the following manner: performing the following processing for each terminal: sending a service interface corresponding to the second multimedia data to the terminal; in response to receiving a first request signaling generated by the terminal, creating a first instance for sending the second multimedia data based on the second transmission protocol, and setting a first remote description of the first instance based on the first request signaling, wherein the first request signaling is generated by the terminal based on the service interface; generating a first response signaling including the first remote description, and sending the first response signaling to the terminal, so that the terminal creates a second instance for receiving the second multimedia data based on the second transmission protocol based on the first response signaling; when the first instance and the second instance are created, determining that the communication connection is established. In this way, the connection process is simplified through clear handshake and connection command messages, making the connection establishment faster and more direct, shortening the user's waiting time; and the system is easier to maintain and expand through standardized handshake and connection processes, which helps to quickly identify and solve problems, thereby facilitating subsequent system upgrades and functional expansions.

[0092] In some embodiments, first, the streaming server sends a service interface to the terminal, wherein the service interface includes all information and rules required to establish a communication connection, and the service interface can be a URL, an API endpoint, or an identifier in other forms; then, after the terminal receives the service interface, it generates a request signaling according to the information provided by the service interface, wherein the request signaling includes the terminal's willingness to establish a communication connection and some initialization parameters (such as the expected transmission protocol version, supported codecs, etc.); then, after receiving the request signaling from the terminal, the streaming server creates a first instance for sending the second multimedia data based on the second transmission protocol (such as Webrtc) according to the information in the request signaling, and sets the remote description of the first instance based on the request signaling, wherein the first instance The first instance is a communication endpoint of a streaming server, and the remote description of the first instance includes relevant information of the terminal (such as the IP address, port number, and key for encrypted communication of the terminal); then, the streaming server generates a response signaling including the remote description of the first instance, and sends the generated response signaling to the terminal, wherein the response signaling is used to indicate that the streaming server is ready to establish a communication connection with the terminal, and provides connection information related to the terminal and the streaming server; thereafter, after receiving the response signaling, the terminal creates a second instance for receiving second multimedia data based on a second transmission protocol according to the response signaling, wherein the second instance is the communication endpoint of the terminal, and when the first instance of the streaming server and the second instance of the terminal are successfully created, it is determined that a communication connection has been successfully established between the streaming server and the terminal.

[0093] For example, taking an online education live broadcast platform as an example, after the streaming server obtains the real-time transmitted teacher's teaching video, first, the streaming server sends a service interface URL for establishing a video call to each student's terminal; then, after receiving the URL, the student terminal generates a request signaling according to the URL and sends it to the streaming server, wherein the request signaling includes the student terminal's willingness to establish a communication connection and the respective terminal device information; then, after receiving the request signaling sent by the student terminal, the streaming server creates an RTCPeerConnection instance as the first instance, and the streaming server updates the remote description of the RTCPeerConnection instance (including IP address, port number, encryption key, etc.) according to the request signaling, wherein the first instance is used to send video data to the student terminal. Here, each student terminal can be divided into In addition to creating a dedicated RTCPeerConnection instance, a public RTCPeerConnection instance can also be created for all student terminals, which is not specifically limited here; then, the streaming server generates a response signaling containing the remote description of the RTCPeerConnection instance, and sends it to the student terminals respectively, wherein the response signaling is used to indicate that the streaming server is ready to establish a communication connection with the student terminal and provide relevant connection information of the streaming server; thereafter, after receiving the response signaling, the student terminal creates an RTCPeerConnection instance as a second instance according to the response signaling, wherein the second instance is used to receive the video data sent by the streaming server, and when the first instance of the streaming server and the second instance of the student terminal are successfully created, it is determined that the communication connection between the streaming server and the student terminal has been successfully established.

[0094] In step 104, streaming processing corresponding to the second multimedia data is performed to at least one terminal via a communication connection.

[0095] In some embodiments, the above step 104 can be implemented in the following ways: encode the second multimedia data and add the encoded second multimedia data to the first instance; create a second request signaling corresponding to the second multimedia data based on the service interface, wherein the second request signaling includes the address information of the second multimedia data; based on the communication connection of the second transmission protocol, send the second request signaling to the terminal, so that the second instance of the terminal obtains the encoded second multimedia data in the first instance based on the address information, and decodes and plays the encoded second multimedia data. By encoding the multimedia data, the amount of encoded data is smaller, the use of storage space can be optimized and the efficiency of data transmission can be improved; at the same time, by creating an instance to realize the sending and transmission between multimedia data, the transmission of multimedia data is more secure and reliable. In this way, the efficiency and reliability of data transmission can be effectively improved.

[0096] In some embodiments, first, the second multimedia data is encoded using a specific encoding algorithm (such as H.264, H.265, etc.) to generate an encoded multimedia data file; then, the encoded multimedia data file is uploaded to the first instance of the streaming server for subsequent access and distribution; then, according to the definition of the service interface, a request signaling containing the address information of the second multimedia data is generated, wherein the address information of the second multimedia data is the storage path or URL of the encoded data on the server; then, the request signaling is sent to the terminal via the selected transmission protocol; thereafter, after receiving the request signaling, the second instance of the terminal parses the address information and initiates a request to the first instance to obtain the encoded multimedia data file; finally, the decoded multimedia data is played using a media player or a corresponding decoding library.

[0097] For example, taking the online education live broadcast platform as an example, the video content of the course is transmitted to the student terminal. First, the streaming server uses the H.264 encoding algorithm to compress and encode the course video to reduce the size of the video file; then, the encoded video file is uploaded to the RTCPeerConnection instance (i.e., the first instance) of the streaming server and stored in a specific path; then, the streaming server generates a request signaling containing the video file address (such as http: / / example.com / course / video1.mp4); then, the streaming server sends the request signaling to the student terminal through the Webrtc protocol; after that, the RTCPeerConnection instance (i.e., the second instance) of the student terminal receives the request signaling, parses the address of the video file, and initiates a request to the streaming server to obtain the encoded video file; finally, the terminal decodes the obtained video file using the H.264 decoding algorithm through the RTCPeerConnection instance, and plays the decoded video content through the media player. In this way, multimedia data can be transmitted from the server to the terminal in real time, efficiently, and accurately, and played on the terminal, providing users with a smooth multimedia playback experience.

[0098] In some embodiments, after executing the above step 104, the following processing can also be performed: in response to receiving a recording request from the terminal, the multimedia data recording mode is started according to the first configuration file, and the second multimedia data is recorded to obtain a recording file, and a response message carrying the recording file is sent to the terminal. In this way, the user can record the multimedia data according to his own needs, which ensures the flexibility and scalability of the system and can effectively improve the user's satisfaction.

[0099] In some embodiments, first, when a specific network port of the streaming server is connected or a recording request is received from a terminal, the streaming server is triggered to read a first configuration file and configure a recording environment according to the parameters and settings in the first configuration file; then, after the recording environment is configured, a multimedia data recording mode is started to prepare for recording; subsequently, in the recording mode, the streaming server or a first instance in the streaming server starts to capture second multimedia data and encodes, compresses, and processes the second multimedia data according to the settings in the first configuration file, wherein the recording process may involve multiple steps such as audio and video synchronization, data buffering, and error handling; thereafter, the streaming server or the first instance in the streaming server writes the recorded multimedia data to a disk or other storage medium to generate a recording file, wherein the format of the recording file may be MP4, AVI, WAV, and the like; finally, the streaming server or the first instance in the streaming server generates a response message and sends it to the terminal via the network, and after receiving the response message, the terminal can parse the information of the recording file and download or access the recording file as desired.

[0100] For example, taking the online live broadcast system as an example, first, the user clicks the "Start Recording" button on the conference interface (i.e., the terminal), and the terminal sends a recording request to the streaming server; then, after receiving the recording request, the streaming server reads the pre-configured recording configuration file (for example, the resolution is 1080p, the frame rate is 30fps, the audio sampling rate is 44.1kHz, etc.), and starts the audio and video recording mode according to these settings; then, in the recording mode, the streaming server starts to capture the audio and video data in the meeting, encodes and compresses it, so as to ensure that the quality and size of the recorded file meet the user's requirements; then, after the recording is completed, the streaming server saves the processed audio and video data as an MP4 file and stores it in the specified location of the streaming server; finally, the streaming server generates a response message, wherein the response message contains information such as the URL and size of the recorded file, and sends it to the user's terminal through the network; finally, after the terminal receives the response message, the user can click the link to download the recorded file, or play it directly in the browser.

[0101] See also Figure 4 , Figure 4 is a flowchart of a multimedia data processing method provided in an embodiment of the present application. The multimedia data processing method is applied to a robot and is combined with Figure 4 The steps shown are explained.

[0102] In step 201, subscription data corresponding to the multimedia acquisition component is obtained.

[0103] In some embodiments, first, the acquired multimedia data type and the corresponding multimedia collection component are determined; then, according to application requirements, subscription rules or conditions (such as data format, quality, timestamp, event trigger, etc.) are defined; thereafter, a subscription relationship is established between the collection component and the robot through an association mechanism (such as API call, event monitoring, message queue, etc.) to obtain the subscription data of the collection component based on the established subscription relationship.

[0104] For example, taking the smart home scenario as an example, the robot's built-in camera is used as a multimedia acquisition component. The user wants to obtain real-time video data captured by the robot's camera with a resolution of 1080p and a frame rate of 30fps. The user can set a subscription request in the mobile application (i.e., the terminal), indicating that he wants to subscribe to the camera's real-time video data, and provides a corresponding callback interface for receiving data. The smart robot associates the camera's video data with the application's subscription request through an API call or event monitoring mechanism. When the camera captures video data that meets the subscription rules (i.e., the resolution is 1080p and the frame rate is 30fps), the robot can obtain the corresponding video data of the camera based on the established association.

[0105] In some embodiments, the above step 201 can be implemented in the following manner: the robot registers with the node manager, wherein the multimedia collection component is in a registered state with the node manager, and the multimedia collection component has published a first topic to the node manager; when the robot successfully registers with the node manager, the robot subscribes to the first topic from the node manager; in response to the multimedia collection component publishing subscription data to the first topic, the multimedia collection component obtains the subscription data corresponding to the first topic. In this way, through the interaction between the robot, the node manager and the multimedia collection component, efficient data sharing and real-time processing are achieved, which can effectively improve the flexibility and reliability of the system.

[0106] In some embodiments, first, a user or a control system sends a registration instruction to the robot through a terminal (such as a computer, a mobile phone, a control panel, etc.), requesting the robot to register with the node manager; then, after receiving the registration instruction, the robot sends a registration request to the node manager, and after the node manager verifies the information, it registers the robot as a node in the system, wherein the registration request includes the robot's identity information, function description, etc.; and, before the robot is registered, the multimedia acquisition component has registered with the node manager and published a first topic for transmitting the multimedia data it has collected; then, when the robot is successfully registered with the node manager, the robot can send a registration request to the node manager according to pre-configuration or dynamic decision. The manager subscribes to the first topic; after that, the multimedia acquisition component continuously acquires multimedia data and publishes the data as subscription data to the first topic, wherein the subscription data includes video frames, audio samples, images, etc.; finally, the node subscribed to the topic can obtain the subscription data from the topic, where the node manager forwards the subscription data on the first topic to all nodes subscribed to the topic or the node subscribed to the topic directly establishes a communication connection with the node that publishes the topic, so that the subscription data on the first topic can be directly obtained from the node that publishes the topic. Here, the node subscribed to the topic includes a robot. After receiving the data, the robot can process it as needed, such as displaying video, analyzing images, storing data, etc.

[0107] For example, take the intelligent inspection robot as an example. A built-in camera is used as a multimedia acquisition component to shoot videos during the inspection process. First, the user sends a registration command to the intelligent inspection robot through the control panel, requiring it to register with the ROS Master node. Then, after receiving the registration command, the intelligent inspection robot sends a registration request to the ROS Master node, including the robot's identity information (such as name, type, function, etc.). After the Master node verifies this information, it registers the intelligent inspection robot as a node in the ROS system. At the same time, before the intelligent inspection robot is registered, its camera has been registered with the ROS Master node and published a URL named / camera / The first topic of image_raw is used to transmit raw image data. Subsequently, after the intelligent inspection robot successfully registers, it subscribes to the topic / camera / image_raw to the ROS Master node according to the pre-configuration, indicating that it wishes to receive raw image data from this topic. After that, the camera continues to shoot video and publishes each frame of image as subscription data to the topic / camera / image_raw. Finally, the ROS Master node forwards the image data on the topic / camera / image_raw to the intelligent inspection robot. After receiving this data, the robot can perform further processing, such as displaying the video stream, analyzing the image content, detecting anomalies, etc.

[0108] In some embodiments, the robot registers with the node manager as described above, which can be implemented in the following manner: the robot registers with the node manager in response to a registration instruction from the terminal, or the robot registers with the node manager according to the second configuration file. In this way, the robot can flexibly register with the node manager according to actual needs, thereby participating in a wider robot system or network.

[0109] In some embodiments, the user or the control system sends an explicit registration instruction to the robot through a terminal (such as a computer, a mobile device, a remote control panel, etc.), where the registration instruction may include the robot's identity information, the address of the node manager to be registered, or other relevant parameters; then, the communication system on the robot receives the registration instruction from the terminal and parses the information therein; finally, the robot uses the parsed information (such as the identity identifier, the node manager address) to send a registration request to the node manager. After verifying the robot's identity and information, the node manager registers it as a valid node in the system and allocates necessary resources and permissions.

[0110] For example, taking the warehouse management system as an example, first, the user selects the "Register New Robot" function through the terminal interface of the warehouse management system, and enters the serial number or other identification information of the robot; then, the terminal system sends a network message containing registration instructions and necessary parameters (such as the node manager address) to the handling robot; after that, after the handling robot receives the message, it parses the registration instructions and parameters, and then sends a registration request to the ROS Master node in the warehouse; finally, after verifying the identity of the handling robot, the ROS Master registers it as a node in the system and allows it to start communicating and collaborating with other nodes.

[0111] In other embodiments, before the robot is deployed or started, a configuration file (i.e., a second configuration file) containing registration information is prepared for it, wherein the configuration file includes the robot's name, type, functional description, address of the node manager, credentials required for registration, etc.; then, when the robot is started or initialized, its software system reads the second configuration file and parses the registration information therein; finally, the robot uses the information read from the configuration file to send a registration request to the node manager, and the node manager verifies the robot's identity and information, completes the registration process, and allocates resources and permissions.

[0112] For example, taking the automatic navigation scenario as an example, during the development stage of the automatic navigation robot, a configuration file containing registration information, such as robot_config.yaml, is generated; then, when the automatic navigation robot is started, its software system automatically reads the robot_config.yaml file and parses out the registration information; finally, the robot uses this information to send a registration request to the ROS Master, completes the registration process, and prepares to receive instructions or data from other nodes.

[0113] In some embodiments, the above-mentioned subscription to the first topic from the node manager can be implemented in the following manner: in response to the topic subscription instruction of the terminal, the node manager subscribes to the first topic, or subscribes to the first topic from the node manager according to the second configuration file. In this way, the node manager can flexibly select different subscription schemes, which can enhance the compatibility of the system with different terminal instructions and configuration files.

[0114] In some embodiments, first, the user or control system sends a clear topic subscription instruction through a terminal (such as a computer, mobile application, remote control interface, etc.), wherein the topic subscription instruction includes the name of the topic to be subscribed (such as / sensor / data), and possible other parameters, such as the robot's identity, message type, etc.; then, the communication system on the robot or node receives the subscription instruction from the terminal, and parses out the topic name and other relevant information therein; thereafter, the robot or node uses the parsed information to send a subscription request to the node manager (such as ROS Master) to indicate the receipt of messages on the specified topic; finally, after receiving the subscription request, the node manager will verify the identity of the robot and the validity of the request, and add the robot to the subscriber list of the corresponding topic. When a message is published to the topic, the node manager will forward the message to all robots that have subscribed to the current topic.

[0115] For example, taking the smart home scenario as an example, the user controls the robot and selects the "View living room camera" function; then, the robot sends a subscription instruction to the node manager of the smart home system (such as ROS Master), requesting to subscribe to the topic published by the living room camera (such as / camera / living_room / image_raw); then, after receiving the subscription instruction, a node in the smart home system (such as the node responsible for video processing) sends a subscription request to ROS Master; then, after verifying the request, ROSMaster adds the node to the subscriber list of the / camera / living_room / image_raw topic; finally, when the living room camera publishes new video frames, these video frames will be forwarded by ROS Master to the robot that subscribes to the current topic.

[0116] In other embodiments, first, before the robot is deployed or started, a configuration file (i.e., a second configuration file) containing registration information is prepared for it, wherein the configuration file includes the name of the topic that the robot / node needs to subscribe to, and possible other subscription parameters; then, during the startup or initialization process, the robot reads the second configuration file and parses the subscription information therein; thereafter, based on the parsed subscription information, the robot automatically sends a subscription request to the node manager, requesting to subscribe to the topics listed in the configuration file; finally, after verifying the validity of the subscription request, the node manager adds the robot to the subscriber list of the corresponding topic and ensures that it can receive messages of the current topic.

[0117] For example, taking the automatic navigation scenario as an example, during the development phase of the automatic navigation robot, a configuration file containing registration information is generated, such as robot_config.yaml, in which the configuration file lists the names of topics that need to be subscribed, such as / sensor / radar / data; then, when the automatic navigation robot is started, the software system reads the subscription_config.yaml file and parses the topics that need to be subscribed; after that, the robot automatically sends a subscription request to the ROS Master, requesting to subscribe to the / sensor / radar / data topic; finally, after verifying the request, the ROS Master adds the automatic navigation robot to the subscriber list of the / sensor / radar / data topic. When the image sensor publishes new data, the newly published data will be forwarded to the automatic navigation robot by the ROS Master.

[0118] In step 202, the subscription data is parsed to obtain first multimedia data.

[0119] In some embodiments, when the data received by the robot is encoded, the data is decoded, wherein the decoding process includes decompression, decryption, format conversion and other operations to restore the original form or processable form of the data; then, the required multimedia information is extracted from the decoded data, such as extracting image frames from a byte stream, extracting sampling points from audio data, separating different media streams from composite data, etc.; then, the extracted multimedia data is further converted (for example, converting image data from one color space to another color space, adjusting the sampling rate or bit depth of audio data, etc.) to obtain first multimedia data.

[0120] For example, taking the robot collecting the current scene image as an example, the robot subscribes to the / camera / video_stream topic through ROS Master. The topic is published by the built-in camera. When the camera captures a new video frame, the video frame is published as a message to the / camera / video_stream topic, and the robot receives the original byte stream containing the video frame; then, the robot's video processing module decodes the byte stream of the received video frame to obtain the decoded data; then, the video processing module extracts the pixel value of each color channel (such as RGB or YUV) from the decoded data, and discards or stores metadata (such as timestamp, frame number, etc.); finally, the video processing module can also convert the extracted pixel values ​​to obtain the final decoded data.

[0121] In some embodiments, the above step 202 can be implemented in the following ways: extracting the code stream of the subscription data to obtain the multimedia code stream; extracting data from the multimedia code stream to obtain the original audio data and the original video data; aligning and merging the original audio data and the original video data to obtain the first multimedia data. In this way, by performing efficient code stream extraction on the subscription data and accurate alignment and merging on the original audio and video data, high-quality multimedia data is generated, which not only improves the quality of multimedia applications, but also improves the efficiency of data processing and transmission, and enhances the flexibility and reliability of the system.

[0122] In some embodiments, first, a code stream containing multimedia data is received from a subscribed topic, and these code streams are parsed using a special decoder or parsing tool to separate the video code stream and the audio code stream; then, the video code stream is decompressed by a video decoder to obtain the original video data, and the audio code stream is decompressed by an audio decoder to obtain the original audio data; finally, the original video data and the original audio data at the corresponding time point are merged to generate a multimedia data packet containing synchronized audio and video information, i.e., the first multimedia data.

[0123] For example, the robot receives a code stream containing multimedia data from a subscribed topic, and parses the code stream through a built-in parsing tool to obtain a video code stream and an audio code stream; then, the video code stream is decoded by a video decoder to restore the corresponding image frames, and the restored image frames are merged to obtain the original video data, and the audio code stream is decoded by an audio decoder to restore the corresponding audio sampling data, and the restored audio sampling data are merged to obtain the original audio data; finally, by comparing the timestamps of the original audio data and the original audio data, the synchronization alignment between the audio and the video is ensured, and the synchronized audio and video data are packaged into a unified multimedia data packet, that is, the first multimedia data.

[0124] In some embodiments, the above-mentioned alignment and merging of the original audio data and the original video data to obtain the first multimedia data can be achieved in the following manner: aligning and merging the original audio data and the original video data to obtain an alignment and merging result; and adjusting the alignment and merging result based on the color value channel to obtain the first multimedia data. In this way, the generated first multimedia data not only ensures the synchronization of audio and video, but also provides visual improvements, so that the viewing experience is more in line with specific needs or preferences.

[0125] In some embodiments, first, the timestamp information in the audio and video data is read, where the timestamp is used to indicate the position of the data frame on the timeline; then, the audio and video data are synchronously adjusted according to the timestamp so that the playback speeds of the two are consistent and the audio and video content at the same time point corresponds; finally, the synchronized audio and video data are merged into a single multimedia stream (i.e., the aligned merge result).

[0126] For example, suppose there is a 5-second video and corresponding audio. The video consists of a series of video frames, each of which has a timestamp (such as 0 seconds, 0.033 seconds, 0.067 seconds, etc.), and the audio also has a similar timestamp; by aligning these timestamps, ensure that the first frame of the video (0 seconds) corresponds to the beginning of the audio (0 seconds), the second frame of the video (0.033 seconds) corresponds to the corresponding part of the audio, and so on, and finally generate a synchronized multimedia file.

[0127] In one embodiment, after obtaining the merged single multimedia stream, the visual multimedia stream is decomposed into different color value channels, such as red (R), green (G), and blue (B) channels, or more advanced color spaces such as YUV; then, each color value channel is independently adjusted, such as increasing brightness, adjusting contrast, changing color saturation, etc.; finally, the adjusted color value channels are resynthesized into audio and video data (i.e., the first multimedia data).

[0128] For example, assuming that you want to increase the brightness of a video, first, decompose the video into three channels: R, G, and B; then, linearly increase the brightness value of each channel (for example, add 10% to each pixel value); finally, re-synthesize the adjusted R, G, and B channels into a video.

[0129] In step 203, first multimedia data is sent to a streaming server based on a first transmission protocol.

[0130] Here, the first multimedia data is sent to the streaming server so that the streaming server performs protocol conversion processing on the first multimedia data to obtain second multimedia data corresponding to the second transmission protocol, establishes a communication connection based on the second transmission protocol with at least one terminal, and performs streaming processing corresponding to the second multimedia data to at least one terminal.

[0131] It should be noted that the execution process of step 203 is similar to the process of step 101 above. For details, please refer to the implementation process of step 101, which will not be repeated here.

[0132] In some embodiments, after executing the above step 203, the following processing may also be performed: sending the configuration address information of the multimedia acquisition component to the terminal; in response to receiving the data acquisition instruction for the configuration address information sent by the terminal, controlling the multimedia acquisition component to be in a working state; in response to receiving the data stop acquisition instruction for the configuration address information sent by the terminal, controlling the multimedia acquisition component to be in a closed state. In this way, by sending the configuration address information of the multimedia acquisition component to the terminal and controlling the switch state of the acquisition component according to the terminal instruction, remote control and real-time response are realized, resource management is optimized, and operation convenience and data acquisition efficiency are improved.

[0133] In some embodiments, first, the robot initializes the multimedia acquisition component and assigns a unique configuration address information to it, wherein the configuration address information may be an IP address, a MAC address, a device ID, or any information that can uniquely identify the component; then, the robot sends the configuration address information to a terminal (which may be a remote control device, a mobile phone APP, a computer program, etc.) through a communication network (such as the Internet, a local area network, etc.), where the communication network completes the transmission through a certain protocol (such as HTTP, TCP / IP, WebSocket, etc.); then, the multimedia acquisition component or a control unit connected thereto will continue to detect instructions from the terminal, and when a data acquisition instruction for specific configuration address information is received from the terminal, the robot parses the data acquisition instruction, and controls the corresponding multimedia acquisition component to enter a working state according to the parsed information, and starts collecting data (such as video, audio, images, etc.); when a stop acquisition instruction for specific configuration address information is received from the terminal, the robot parses the stop acquisition instruction, and controls the corresponding multimedia acquisition component to enter a closed state according to the parsed information, and stops data acquisition.

[0134] For example, take the intelligent control of a robot as an example, which includes a built-in camera as a multimedia acquisition component; first, the robot initializes the camera and assigns it an IP address (such as 192.168.1.100) as the configuration address information, and sends the assigned IP address to the user's mobile terminal through the local area network; then, the user enters the IP address of the camera on the mobile terminal and sends a "start acquisition" command. After receiving the command, the robot parses it out that it is for the camera with an IP address of 192.168.1.100, and controls the camera to start and start collecting video data; when the user sends the "stop acquisition" command again on the mobile terminal for the same IP address (192.168.1.100), the robot receives the command, parses it out that the camera with an IP address of 192.168.1.100 is working, and controls the camera to shut down and stop video data acquisition.

[0135] Next, an exemplary application of the embodiment of the present application in a practical application scenario will be described. The exemplary application describes the specific implementation process of the multimedia data processing method in a robot real-time video transmission scenario.

[0136] In some embodiments, see Figure 5 , Figure 5 1 is a timing flow chart of the multimedia data processing method provided in the embodiment of the present application. Figure 5 Describe the specific implementation process.

[0137] Phase 1: The robot starts the camera driver service and subscribes to the video topic data.

[0138] In some embodiments, first, according to the requirements of the camera driver, corresponding configuration is performed before starting the driver (for example, setting parameters such as resolution and frame rate); then, the camera driver service is started by running the node; then, after starting the camera driver service, the status of the camera node and the published topic data are checked through control commands; finally, the robot obtains the corresponding video topic data by subscribing to the camera node.

[0139] Phase 2: The web terminal edits the camera subscription topic for the robot.

[0140] It should be noted that in order to edit the camera subscription topic on the web terminal, it is necessary to establish an interactive environment between the web terminal and the robot ROS system. This can be achieved through the ROS Web interface tool (such as rosbridge_suite), or you can develop a simple web server by yourself, receive input from the web terminal, and execute corresponding ROS commands. No specific limitations are made here.

[0141] Phase 3: The robot receives video data from the subscribed topic, parses and processes the data, and pushes it to the streaming server.

[0142] In some embodiments, first, the robot receives the video data of the camera corresponding to the subscribed topic, and performs pre-processing operations such as image enhancement and compression as required; then, the processed data is encoded to generate data in a video stream format, such as H.264 or H.265; then, the encoded video stream is pushed to the streaming server through a streaming library or streaming tool, such as FFmpeg, rtmpdump or GStreamer.

[0143] Phase 4: The streaming server converts the RTMP format video stream into WebRTC format.

[0144] In some embodiments, after receiving the RTMP format video stream, the streaming server uses a streaming media server that supports RTMP to Webrtc conversion, such as Jitsi Videobridge, Janus Gateway, MediaSoup or Red5Pro, etc.; then, configures the RTMP input on the streaming server, such as the RTMP application name and stream name; then, the streaming server converts the RTMP format video stream into the Webrtc format to pass it to the web terminal.

[0145] Phase 5: The robot provides streaming configuration address information to the web terminal.

[0146] In some embodiments, the robot provides the address information of the streaming configuration to the web terminal (i.e., determines the address information of the camera that needs to be obtained), the web terminal generates a corresponding API request identifier and sends it to the robot, and the robot controls the robot to stop or start receiving video frames according to the API request identifier provided by the web terminal to control whether the camera continues to stream.

[0147] Stage 6: The web terminal displays the video in real time through the WebRTC streaming address and SDK.

[0148] In some embodiments, the web terminal obtains the streaming video stream data according to the Webrtc streaming address transmitted by the streaming server, and decodes the streaming video stream data through the SDK and displays and plays it in real time.

[0149] In some embodiments, see Figure 6 , Figure 6 This is a schematic diagram of the process of the multimedia data processing method provided by the embodiment of the present application in the video real-time transmission scenario. Figure 6 The specific process of robot real-time video transmission is introduced.

[0150] In step 301, the robot subscribes to topic data of multiple cameras.

[0151] It should be noted that in the robot, a message queue or publish-subscribe mode is used to receive topic data from different cameras. Here, some message middleware (such as RabbitMQ, Apache Kafka, etc.) can be used to set a subscriber for each camera to detect the corresponding topic data.

[0152] In step 302, the robot is deployed through the container, the topic data is obtained and the topic data is decoded to obtain the original audio and video data.

[0153] Here, the relevant modules of the robot are deployed using containers, and the corresponding camera topic data is subscribed and received through the configuration file or terminal API. The audio and video codec is integrated in the robot to parse the subscribed topic data and convert it into original audio and video data.

[0154] In step 303, the robot merges the parsed audio and video data, and parses out normal video frames by setting the color value channel through the OpenCV library.

[0155] Here, the OpenCV library is used to process the audio and video data, merge the audio data and the video data, parse out continuous normal video frames, and adjust and parse the color value channels of the normal video frames to ensure the quality of the parsed images.

[0156] In step 304, the parsed normal video frames are pushed to the streaming server.

[0157] Here, the extracted normal video frames are encapsulated into a video stream in RTMP format, and then the video stream is pushed to the streaming server through the FFmpeg tool (audio and video codec); then, the streaming server converts the corresponding video stream in RTMP format into a video stream in Webrtc format.

[0158] In step 305, a communication connection is established between the streaming server and the Web terminal.

[0159] Here, first, the Web terminal implements the interface service call through the API of the streaming server to obtain the information required for the Webrtc connection (such as SDK, etc.); then, the Web terminal uses the HTTP transmission protocol to exchange signaling information between the web terminal and the streaming server according to the response signaling exchange protocol in the SDK, thereby completing the Webrtc connection between the web terminal and the streaming server.

[0160] In step 306, the streaming server transmits the audio and video data through the WebRTC channel.

[0161] Here, the audio and video data will be transmitted to the web terminal in real time through the Webrtc channel.

[0162] After step 306, the robot can also provide the configured address information to the web terminal. The web terminal will send an API request identifier to the robot based on the provided configured address information. The robot can stop or start receiving video frames based on the API request identifier to control whether the camera continues to push the stream.

[0163] After step 306, the streaming server may start the DVR recording mode according to the settings in the configuration file to record the video stream data.

[0164] The following further describes the exemplary structure of the multimedia data processing device 643 provided in the embodiment of the present application implemented as a software module. In some embodiments, for example, Figure 2A As shown, the software modules stored in the multimedia data processing device 643 of the memory 640 may include: a receiving module 6431, a conversion module 6432 and a streaming module 6433. The receiving module 6431 is used to receive the first multimedia data corresponding to the first transmission protocol sent by the robot, wherein the first multimedia data is obtained by the robot parsing the subscription data corresponding to the multimedia acquisition component; the conversion module 6432 is used to perform protocol conversion processing on the first multimedia data to obtain the second multimedia data of the second transmission protocol; the streaming module 6433 is used to establish a communication connection based on the second transmission protocol for at least one terminal, and perform streaming processing corresponding to the second multimedia data to at least one terminal through the communication connection.

[0165] In some embodiments, the streaming module 6433 is also used to perform the following processing for each terminal: sending a service interface corresponding to the second multimedia data to the terminal; in response to receiving a first request signaling generated by the terminal, creating a first instance for sending the second multimedia data based on a second transmission protocol, and setting a first remote description of the first instance based on the first request signaling, wherein the first request signaling is generated by the terminal based on the service interface; generating a first response signaling including the first remote description, and sending the first response signaling to the terminal, so that the terminal creates a second instance for receiving the second multimedia data based on the second transmission protocol based on the first response signaling; when the creation of the first instance and the second instance is completed, determining that the communication connection establishment is completed.

[0166] In some embodiments, the streaming module 6433 is also used to encode the second multimedia data and add the encoded second multimedia data to the first instance; create a second request signaling corresponding to the second multimedia data based on the service interface, wherein the second request signaling includes address information of the second multimedia data; based on the communication connection of the second transmission protocol, send the second request signaling to the terminal, so that the second instance of the terminal obtains the encoded second multimedia data in the first instance based on the address information, and decodes and plays the encoded second multimedia data.

[0167] In some embodiments, the multimedia data processing device 643 further includes a recording module 6434. The recording module 6434 is configured to, in response to receiving a recording request from a terminal, start a multimedia data recording mode according to a first configuration file, perform recording processing on the second multimedia data, obtain a recording file, and send a response message carrying the recording file to the terminal.

[0168] The following further describes the exemplary structure of the multimedia data processing device 755 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2B As shown, the software modules stored in the multimedia data processing device 755 of the memory 750 may include: an acquisition module 7551, a parsing module 7552, and a sending module 7553. The acquisition module 7551 is used to acquire the subscription data corresponding to the multimedia acquisition component; the parsing module 7552 is used to parse the subscription data to obtain the first multimedia data; the sending module 7553 is used to send the first multimedia data to the streaming server based on the first transmission protocol, so that the streaming server performs protocol conversion processing on the first multimedia data to obtain the second multimedia data corresponding to the second transmission protocol, establish a communication connection based on the second transmission protocol with at least one terminal, and perform streaming processing corresponding to the second multimedia data to at least one terminal.

[0169] In some embodiments, the acquisition module 7551 is also used for the robot to register with the node manager, wherein the multimedia acquisition component is in a registered state in the node manager, and the multimedia acquisition component has published a first topic to the node manager; when the robot successfully registers with the node manager, it subscribes to the first topic from the node manager; in response to the multimedia acquisition component publishing subscription data to the first topic, the subscription data corresponding to the multimedia acquisition component is obtained from the first topic.

[0170] In some embodiments, the acquisition module 7551 is further used to register the robot with the node manager in response to a registration instruction from the terminal, or to register the robot with the node manager according to the second configuration file.

[0171] In some embodiments, the acquisition module 7551 is further used to subscribe to the first topic from the node manager in response to a topic subscription instruction of the terminal, or to subscribe to the first topic from the node manager according to the second configuration file.

[0172] In some embodiments, the parsing module 7552 is also used to extract the code stream of the subscription data to obtain a multimedia code stream; extract data from the multimedia code stream to obtain original audio data and original video data; align and merge the original audio data and the original video data to obtain first multimedia data.

[0173] In some embodiments, the parsing module 7552 is further used to perform alignment and merging processing on the original audio data and the original video data to obtain an alignment and merging result; and perform adjustment processing on the alignment and merging result based on the color value channel to obtain the first multimedia data.

[0174] In some embodiments, the multimedia data processing device 755 further includes a recording module 7554. The control module 7554 is used to send the configuration address information of the multimedia acquisition component to the terminal; in response to receiving a data acquisition instruction for the configuration address information sent by the terminal, control the multimedia acquisition component to be in a working state; in response to receiving a data stop acquisition instruction for the configuration address information sent by the terminal, control the multimedia acquisition component to be in a closed state.

[0175] It should be noted that the description of the device in the embodiment of the present application is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, so it will not be repeated. Figure 3 ,or Figure 4 The present invention should be understood by referring to the description of any one of the accompanying drawings.

[0176] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the multimedia data processing method described in the embodiment of the present application.

[0177] The present application embodiment provides a computer-readable storage medium, in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will execute the multimedia data processing method provided by the present application embodiment, for example, Figure 3 ,or Figure 4 The multimedia data processing method is shown.

[0178] In some embodiments, the computer-readable storage medium may be a ferroelectric random access memory (FRAM), ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM) and other memories; it may also be various devices including one or any combination of the above memories.

[0179] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0180] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0181] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0182] To summarize, through the embodiments of the present application, the first multimedia data corresponding to the first transmission protocol sent by the robot is received through the streaming server, wherein the first multimedia data is obtained by the robot parsing and processing the subscription data corresponding to the multimedia acquisition component. In the related technology, multimedia data is directly obtained from the robot through a point-to-point connection transmission method, so that the acquisition of multimedia data is subject to the distance limitation of the robot's local area network. Therefore, by saving the multimedia data to the server, the acquisition of multimedia data is no longer subject to the distance limitation of the robot's local area network, which can effectively solve the distance limitation of multimedia data transmission; the first multimedia data is subjected to protocol conversion processing to obtain second multimedia data corresponding to the second transmission protocol, a communication connection based on the second transmission protocol is established with at least one terminal, and streaming processing corresponding to the second multimedia data is executed to at least one terminal. In the related art, the robot and the terminal are connected point-to-point through a local area network, so each multimedia data of the robot can only be transmitted in real time with one terminal at a time. In contrast, the present solution saves the multimedia data in the server, so that the terminals that obtain the multimedia data are no longer limited in number. Therefore, the present solution saves the multimedia data in the server and performs streaming processing with at least one terminal through a second transmission protocol. This can effectively solve the distance limitation of multimedia data transmission while realizing real-time transmission of the same multimedia data with more than one terminal.

[0183] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A multimedia data processing method, characterized in that: The method is applied to a streaming server, and the method includes: Receiving first multimedia data corresponding to a first transmission protocol sent by the robot, wherein the first multimedia data is obtained by the robot parsing subscription data corresponding to a multimedia acquisition component; Performing protocol conversion processing on the first multimedia data to obtain second multimedia data corresponding to a second transmission protocol; A communication connection based on the second transmission protocol is established with at least one terminal, and streaming processing corresponding to the second multimedia data is performed to the at least one terminal through the communication connection.

2. The method according to claim 1, characterized in that The establishing a communication connection based on the second transmission protocol with at least one terminal includes: The following processing is performed for each terminal: Sending a service interface corresponding to the second multimedia data to the terminal; In response to receiving a first request signaling generated by the terminal, creating a first instance for sending the second multimedia data based on the second transmission protocol, and setting a first remote description of the first instance based on the first request signaling, wherein the first request signaling is generated by the terminal based on the service interface; generating a first response signaling including the first remote description, and sending the first response signaling to the terminal, so that the terminal creates a second instance for receiving the second multimedia data based on the second transmission protocol based on the first response signaling; When the first instance and the second instance are created, it is determined that the communication connection is established.

3. The method according to claim 2, characterized in that The performing streaming processing corresponding to the second multimedia data to the at least one terminal through the communication connection includes: encoding the second multimedia data, and adding the encoded second multimedia data to the first instance; Creating a second request signaling corresponding to the second multimedia data based on the service interface, wherein the second request signaling includes address information of the second multimedia data; Based on the communication connection of the second transmission protocol, the second request signaling is sent to the terminal, so that the second instance of the terminal obtains the encoded second multimedia data in the first instance based on the address information, and decodes and plays the encoded second multimedia data.

4. The method according to claim 1, characterized in that: The method further comprises: In response to receiving a recording request from the terminal, starting a multimedia data recording mode according to a first configuration file; The second multimedia data is recorded to obtain a recording file, and a response message carrying the recording file is sent to the terminal.

5. A multimedia data processing method, characterized in that: The method is applied to a robot, and the method comprises: Obtain subscription data corresponding to the multimedia acquisition component; Parsing the subscription data to obtain first multimedia data; The first multimedia data is sent to a streaming server based on the first transmission protocol, so that the streaming server performs protocol conversion processing on the first multimedia data to obtain second multimedia data corresponding to a second transmission protocol, establishes a communication connection based on the second transmission protocol with at least one terminal, and performs streaming processing corresponding to the second multimedia data to the at least one terminal through the communication connection.

6. The method according to claim 5, characterized in that The obtaining of subscription data corresponding to the multimedia acquisition component includes: The robot is registered with the node manager, wherein the multimedia collection component is in a registered state with the node manager, and the multimedia collection component has published a first topic to the node manager; When the robot is successfully registered with the node manager, subscribing to the first topic from the node manager; In response to the multimedia collection component publishing the subscription data to the first topic, the subscription data corresponding to the multimedia collection component is acquired from the first topic.

7. The method according to claim 6, characterized in that The robot is registered with the node manager, including: In response to a registration instruction from the terminal, the robot registers with the node manager, or the robot registers with the node manager according to a second configuration file; Subscribing to the first topic from the node manager includes: In response to a topic subscription instruction of the terminal, the first topic is subscribed from the node manager, or the first topic is subscribed from the node manager according to the second configuration file.

8. The method according to claim 5, characterized in that The parsing and processing the subscription data to obtain the first multimedia data includes: Extracting a code stream from the subscription data to obtain a multimedia code stream; Extracting data from the multimedia code stream to obtain original audio data and original video data; The original audio data and the original video data are aligned and merged to obtain the first multimedia data.

9. The method according to claim 8, characterized in that The aligning and merging the original audio data and the original video data to obtain the first multimedia data includes: Performing alignment and merging processing on the original audio data and the original video data to obtain an alignment and merging result; The alignment and merging result is adjusted based on the color value channel to obtain the first multimedia data.

10. The method according to claim 5, characterized in that The method further comprises: Sending the configuration address information of the multimedia acquisition component to the terminal; In response to receiving a data collection instruction for the configuration address information sent by the terminal, controlling the multimedia collection component to be in a working state; In response to receiving a data stop collection instruction for the configuration address information sent by the terminal, the multimedia collection component is controlled to be in a closed state.

11. A multimedia data processing device, characterized in that: The device is applied to a streaming server, and comprises: A receiving module, configured to receive first multimedia data corresponding to a first transmission protocol sent by the robot, wherein the first multimedia data is obtained by the robot parsing subscription data corresponding to the multimedia acquisition component; a conversion module, configured to perform protocol conversion processing on the first multimedia data to obtain second multimedia data corresponding to a second transmission protocol; The streaming module is used to establish a communication connection based on the second transmission protocol with at least one terminal, and perform streaming processing corresponding to the second multimedia data to the at least one terminal through the communication connection.

12. A multimedia data processing device, characterized in that: The device is applied to a robot, and comprises: An acquisition module, used to acquire subscription data corresponding to the multimedia acquisition component; A parsing module, used for parsing the subscription data to obtain first multimedia data; A sending module is used to send the first multimedia data to a streaming server based on the first transmission protocol, so that the streaming server performs protocol conversion processing on the first multimedia data to obtain second multimedia data corresponding to the second transmission protocol, establish a communication connection based on the second transmission protocol with at least one terminal, and perform streaming processing corresponding to the second multimedia data to the at least one terminal through the communication connection.

13. An electronic device, characterized in that: include: A memory for storing computer executable instructions or computer programs; The processor is used to implement the multimedia data processing method according to any one of claims 1 to 10 when executing the computer executable instructions or computer programs stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer programs are executed by a processor, the multimedia data processing method according to any one of claims 1 to 10 is implemented.

15. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer programs are executed by a processor, the multimedia data processing method according to any one of claims 1 to 10 is implemented.

Images