Video streaming media processing device for intelligent patrol robot

Abstract

The invention relates to a video streaming media processing device for an intelligent patrol robot, and belongs to the field of video streaming media processing. The device provided by the invention comprises a video acquisition end and a streaming media server. The video acquisition end can output an RTSP protocol video stream through a visible light camera, can also pull and decode the video stream through FFmpeg, encodes the video stream processed by intelligent algorithms such as target detection, target tracking, panoramic stitching and the like, and then outputs the RTSP protocol video stream. The streaming media server side pulls RTSP protocol video streams through FFmpeg, converts the RTSP protocol video streams into RTMP protocol video streams and then transmits the RTMP protocol video streams to the SRS frame, the SRS frame transmits the RTMP protocol video streams to the monitoring end in a webRTC protocol mode, the video streams are played in a monitoring hall, the surrounding environment state of the intelligent patrol robot is monitored in real time, and then the purpose of security protection of specified targets and areas is achieved.

Description

Technical Field

[0001] The invention belongs to the field of video streaming media processing, and in particular relates to a video streaming media processing device for an intelligent patrol robot. Background Art

[0002] The SRS (Simple Real-Time Media Server) framework is an open source streaming media server that supports multiple streaming media protocols such as RTMP, HTTP-FLV, webRTC, etc., and can provide efficient and stable real-time audio and video services. The SRS framework also supports the integration of tools and services such as FFmpeg and Nginx, and supports optimizing server performance through coroutine technology. The core functions of the SRS framework are developed based on the C++ language. As a high-performance programming language, C++ is very suitable for building server applications that need to handle a large number of concurrent connections and high data throughput. It can support the SRS framework to achieve large-scale real-time audio and video streaming. The SRS framework supports multiple operating system platforms such as Linux, Windows, and MacOS. The Linux platform provides better system resource management and network performance, so the SRS framework has higher operating efficiency on the Linux platform.

[0003] FFmpeg is a powerful and widely used audio and video processing tool, mainly used to record, convert audio and video, and convert them into streams. The tool contains the audio and video codec library libavcodec. FFmpeg can be used to obtain video information, monitor video decomposition into push sequences, re-encode video, etc. The FFmpeg tool needs to be developed on the Linux platform and can be compiled and run in operating system environments such as Windows and MacOS.

[0004] At present, the SRS framework does not support the RTSP protocol, and the existing streaming media service solutions fail to provide the function of integrating FFmpeg to process video streams in real time and transmit them efficiently. In addition, there are bottlenecks in the real-time processing and transmission of multiple video streams, and it is impossible to effectively support the real-time monitoring and data processing requirements of intelligent patrol robots for complex environments. Therefore, the present invention uses the FFmpeg tool to convert the RTSP protocol video stream output by the visible light camera into an RTMP protocol video stream, and then transmits it to the SRS framework for processing. The FFmpeg tool can also realize the decoding before processing and the encoding after processing of the RTSP protocol video stream. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] The technical problem to be solved by the present invention is how to provide a video streaming media processing device for an intelligent patrol robot to solve the shortcomings of existing streaming media services in real-time video transmission of the intelligent patrol robot.

[0007] (II) Technical solution

[0008] In order to solve the above technical problems, the present invention proposes a video streaming media processing device for an intelligent patrol robot, the method comprising the following steps:

[0009] S1. The video acquisition end obtains the RTSP protocol video stream in two ways. The first way is to directly obtain the RTSP protocol video stream output by the visible light camera of the monitoring intelligent patrol robot. The second way is to pull the video stream output by the visible light camera through the FFmpeg framework and decode it, then encode the video stream processed by the intelligent algorithm, and then output the RTSP protocol video stream. The intelligent algorithm includes: target detection, target tracking and panoramic stitching.

[0010] S2, the streaming media server pulls the RTSP protocol video stream from the video acquisition end through FFmpeg, converts it into an RTMP protocol video stream, and transmits it to the SRS framework;

[0011] S3, the SRS framework of the streaming media server is transmitted to the monitoring end in the form of webRTC protocol, the video stream is played in the monitoring hall, and the state of the surrounding environment of the intelligent patrol robot is monitored in real time, so as to achieve the purpose of security protection of the designated targets and areas.

[0012] (III) Beneficial effects

[0013] The present invention proposes a video streaming media processing device for an intelligent patrol robot. Compared with the prior art, the video streaming media processing device for an intelligent patrol robot described in the present invention has the following beneficial effects:

[0014] (1) The streaming media processing device proposed in the present invention uses a combination of the SRS framework and the FFmpeg tool to achieve real-time video streaming transmission with a delay of less than several hundred milliseconds, significantly improving the real-time performance of remote monitoring.

[0015] (2) The streaming media processing device proposed in the present invention uses the webRTC protocol at the streaming media server end, supports simultaneous access by multiple users, and realizes stable video streaming services under large-scale concurrency.

[0016] (3) The streaming media processing device proposed in the present invention can realize intelligent video processing at the video acquisition end, including target detection, tracking and panoramic stitching, thereby enhancing the environmental perception ability of the patrol robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a block diagram of the video streaming media processing device for the intelligent patrol robot according to the present invention;

[0018] Figure 2 The effect diagram of the video streaming media processing device for the intelligent patrol robot described in the present invention; (a) is the robot management interface; (b) is the monitoring interface. DETAILED DESCRIPTION

[0019] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0020] The device proposed by the present invention includes a video acquisition end and a streaming media service end.

[0021] S1. The video acquisition end obtains the RTSP protocol video stream in two ways; the first is to directly obtain the RTSP protocol video stream output by the visible light camera of the monitoring intelligent patrol robot; the second is to pull the video stream output by the visible light camera through the FFmpeg framework and decode it, and then encode the video stream processed by intelligent algorithms such as target detection, target tracking, and panoramic stitching, and then output the RTSP protocol video stream;

[0022] S2, the streaming media server pulls the RTSP protocol video stream from the video acquisition end through FFmpeg, converts it into an RTMP protocol video stream, and transmits it to the SRS framework;

[0023] S3, the SRS framework of the streaming media server is transmitted to the monitoring end in the form of webRTC protocol, the video stream is played in the monitoring hall, and the state of the surrounding environment of the intelligent patrol robot is monitored in real time, so as to achieve the purpose of security protection of the designated targets and areas.

[0024] Embodiment 1:

[0025] A video streaming media processing device designed by the present invention is for an intelligent patrol robot. Aiming at the shortcomings of existing streaming media services in real-time video transmission of intelligent patrol robots, the present invention aims to realize a low-latency, multi-user access, efficient video streaming transmission service through the combination of SRS framework and FFmpeg tool, so as to meet the security needs of designated targets and areas.

[0026] The present invention designs a streaming media processing device for intelligent patrol robots. The device is based on the SRS open source streaming media server, configures the IP address and port of the video stream, transmits the audio and video data collected by the visible light camera to the monitoring hall through the webRTC protocol, and displays it in real time, thereby realizing low-latency and high-efficiency transmission of the video stream.

[0027] Method 1: Pull the original video stream

[0028] S11. Steps for deploying the SRS framework:

[0029] ①Update system packages and install build-essential dependencies;

[0030] ②Download SRS source code, compile and install;

[0031] ③Configure the SRS configuration file srs.conf, set the port, protocol, buffer size, etc. of the video stream.

[0032] Low latency is an important indicator for real-time streaming media transmission. The patent of this invention achieves video streaming transmission with a latency of less than several hundred milliseconds by optimizing the buffer size, using the webRTC protocol and streamlining the data transmission path.

[0033] S12. webRTC is a technical protocol that supports real-time communication between browsers. Steps to deploy the webRTC protocol:

[0034] ① Select the webRTC protocol in the configuration file of the SRS framework.

[0035] ② Use JavaScript to connect to the SRS server through the webRTC API on the web client, and write the front-end code in the Vue.js framework.

[0036] ③Test whether webRTC is successfully connected in the browser and can play the video.

[0037] S13, RTMP protocol, can optimize the transmission efficiency of video streaming services and reduce video latency. Steps for deploying RTMP protocol:

[0038] ① Set the RTMP protocol listening port in the SRS configuration file and define multiple channels for camera transmission.

[0039] ② Use FFmpeg to process the video stream captured by the visible light camera, convert it into RTMP protocol and transmit it to the SRS server.

[0040] Method 2: Pull the video stream after object detection, object tracking, panoramic stitching, etc.

[0041] On the basis of pulling the original video stream, the FFmpeg multimedia framework is added, which can process and convert audio and video files. Specifically, FFmpeg decodes the images and videos collected by the visible light camera, calls yolov5, SSD and other algorithms to achieve target detection, KCF, CFNet and other algorithms to achieve target tracking, or realizes panoramic image stitching after steps such as image acquisition, feature extraction and matching, feature point purification, homography matrix estimation, perspective transformation, image fusion, etc. FFmpeg encodes the processed images and videos, and converts them into RTMP for transmission to the SRS framework. The SRS framework outputs the video stream as a webRTC address accessible in the local area network.

[0042] Embodiment 2:

[0043] The present invention provides a video streaming media processing device for an intelligent patrol robot, which comprises a video acquisition end and a streaming media service end.

[0044] The specific implementation is as follows:

[0045] The intelligent patrol robot is equipped with 4 visible light cameras in total, with the following optional parameters: resolution: 1920×1080@30fps; horizontal field of view: 120°; vertical field of view: 30°; focal length: 1.96mm. The video acquisition end can choose the NVIDIA Orin chip, which has 32G running memory and provides 254TOPS AI computing power. The chip has high-efficiency data processing performance and can effectively support dense computing of artificial intelligence algorithm models such as machine learning and deep learning. The FFmpeg multimedia framework is deployed on the video acquisition end.

[0046] After setting up the hardware environment for the video acquisition end, deploy the SRS framework, webRTC protocol, and RTMP protocol on the streaming media server. Use the SRS framework to push the original video stream or the video stream processed by target detection, target tracking, and panoramic stitching algorithms to the audience. After the streaming media server obtains the webRTC video stream link, the playback program must call the webRTC video playback API interface in conjunction with the JavaScript SDK provided by the SRS framework, and transmit the webRTC protocol address to the HTML video tag. The video stream can be played after the user visits the page.

[0047] The specific steps are as follows:

[0048] S21. Push original video stream

[0049] The streaming media server uses FFmpeg to pull the RTSP protocol video stream of the visible light camera (such as: rtsp: / / 192.168.1.20:554 / StreamId=1), and then FFmpeg pushes it to the RTMP address accessible to the SRS framework. The SRS framework accesses the RTMP address and converts it to the webRTC protocol (such as: webrtc: / / 192.168.1.40 / 4 / 1) to pull the video stream of the visible light camera and finally provide a video stream address of the webRTC protocol.

[0050] S22. Push target detection video stream

[0051] The video capture end uses FFmpeg to pull the RTSP protocol video stream of the visible light camera (such as: rtsp: / / 192.168.1.20:554 / StreamId=2) and decode it, and use it as the input of the target detection algorithm model. After processing, use FFmpeg to encode the video stream and push it to the RTSP address accessible in the local area network. The streaming media server uses FFmpeg to pull the RTSP protocol video stream after the target detection processing is completed on the video capture end, and push it to the RTMP address accessible to the SRS framework. The SRS framework accesses the RTMP address and converts it to the webRTC protocol (such as: webrtc: / / 192.168.1.40 / 4 / 2), and finally provides a webRTC protocol video stream address.

[0052] S23: Push target tracking video stream

[0053] The video capture end uses FFmpeg to pull the RTSP protocol video stream of the visible light camera (such as: rtsp: / / 192.168.1.20:554 / StreamId=3) and decode it, and use it as the input of the target tracking algorithm model. After processing, use FFmpeg to encode the video stream and push it to the RTSP address accessible in the local area network. The streaming media server uses FFmpeg to pull the RTSP protocol video stream after the target detection processing of the video capture end, and push it to the RTMP address accessible to the SRS framework. The SRS framework accesses the RTMP address and converts it to the webRTC protocol (such as: webrtc: / / 192.168.1.40 / 4 / 3), and finally provides a webRTC protocol video stream address.

[0054] S24, push panoramic stitching video stream

[0055] The video acquisition end uses FFmpeg to pull the RTSP protocol video stream of the visible light camera (such as: rtsp: / / 192.168.1.20:554 / StreamId=1, rtsp: / / 192.168.1.20:554 / StreamId=2, rtsp: / / 192.168.1.20:554 / StreamId=3, rtsp: / / 192.168.1.20:554 / StreamId=4) and decode it. After image acquisition, feature extraction and matching, feature point purification, homography matrix estimation, perspective transformation, image fusion and other steps, panoramic image stitching is realized. After processing, FFmpeg is used to encode the video stream and push it to the RTSP address accessible in the local area network. The streaming media server uses FFmpeg to pull the RTSP protocol video stream after the target detection processing of the video acquisition end and push it to the RTMP address accessible to the SRS framework. The SRS framework accesses the RTMP address and converts it into the webRTC protocol (such as webrtc: / / 192.168.1.40 / 4 / 4), and finally provides a video stream address of the webRTC protocol.

[0056] Compared with the prior art, the video streaming media processing device for an intelligent patrol robot described in the present invention has the following beneficial effects:

[0057] (1) The streaming media processing device proposed in the present invention uses a combination of the SRS framework and the FFmpeg tool to achieve real-time video streaming transmission with a delay of less than several hundred milliseconds, significantly improving the real-time performance of remote monitoring.

[0058] (2) The streaming media processing device proposed in the present invention uses the webRTC protocol at the streaming media server end, supports simultaneous access by multiple users, and realizes stable video streaming services under large-scale concurrency.

[0059] (3) The streaming media processing device proposed in the present invention can realize intelligent video processing at the video acquisition end, including target detection, tracking and panoramic stitching, thereby enhancing the environmental perception ability of the patrol robot.

[0060] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A video streaming media processing device for an intelligent patrol robot, characterized in that: The method comprises the following steps: S1. The video acquisition end obtains the RTSP protocol video stream in two ways; the first is to directly obtain the RTSP protocol video stream output by the visible light camera of the monitoring intelligent patrol robot; the second is to pull the video stream output by the visible light camera through the FFmpeg framework and decode it, then encode the video stream processed by the intelligent algorithm, and then output the RTSP protocol video stream; Intelligent algorithms include: target detection, target tracking and panoramic stitching; S2, the streaming media server pulls the RTSP protocol video stream from the video acquisition end through FFmpeg, converts it into an RTMP protocol video stream, and transmits it to the SRS framework; S3, the SRS framework of the streaming media server is transmitted to the monitoring end in the form of webRTC protocol, the video stream is played in the monitoring hall, and the state of the surrounding environment of the intelligent patrol robot is monitored in real time, so as to achieve the purpose of security protection of the designated targets and areas.

2. The video streaming media processing device for the intelligent patrol robot according to claim 1, characterized in that: A total of four visible light cameras are installed on the front, back, left and right sides of the intelligent patrol robot.

3. The video streaming media processing device for the intelligent patrol robot according to claim 1, characterized in that: The video capture end chooses NVIDIA Orin chip.

4. The video streaming media processing device for the intelligent patrol robot according to claim 1, characterized in that: Deploy the SRS framework, webRTC protocol and RTMP protocol on the streaming media server, and push the original video stream or the video stream processed by target detection, target tracking and panoramic stitching algorithm to the audience through the SRS framework. After the streaming media server obtains the webRTC video stream link, the playback program must call the webRTC video playback API interface in conjunction with the JavaScript SDK provided by the SRS framework, and transmit the webRTC protocol address to the HTML video tag. The video stream can be played after the user visits the page.

5. The video streaming media processing device for the intelligent patrol robot according to claim 4, characterized in that: The steps to deploy the SRS framework include: ①Update system packages and install build-essential dependencies; ②Download SRS source code, compile and install; ③Configure the SRS configuration file srs.conf and set the port, protocol and buffer size of the video stream.

6. The video streaming media processing device for the intelligent patrol robot according to claim 4, characterized in that: The steps to deploy the webRTC protocol include: ① Select the webRTC protocol in the configuration file of the SRS framework; ②Use JavaScript to connect to the SRS server through the webRTC API on the web client, and write the front-end code in the Vue.js framework; ③Test whether webRTC is successfully connected in the browser and can play the video.

7. The video streaming media processing device for the intelligent patrol robot according to claim 4, characterized in that: The steps to deploy the RTMP protocol include: ① Set the RTMP protocol listening port in the SRS configuration file and define multiple channels for camera transmission; ② Use FFmpeg to process the video stream captured by the visible light camera, convert it into RTMP protocol and transmit it to the SRS server.

8. The video streaming media processing device for the intelligent patrol robot according to claim 4, characterized in that: Pushing the original video stream includes: the streaming media server uses FFmpeg to pull the RTSP protocol video stream of the visible light camera, and then FFmpeg pushes it to the RTMP address accessible to the SRS framework. The SRS framework accesses the RTMP address and converts it to the webRTC protocol to pull the video stream of the visible light camera, and finally provides a video stream address of the webRTC protocol.

9. The video streaming media processing device for the intelligent patrol robot according to claim 4, characterized in that: Pushing target detection video streams includes: The video acquisition end uses FFmpeg to pull the RTSP protocol video stream of the visible light camera and decode it, and uses it as the input of the target detection algorithm model. After processing, FFmpeg is used to encode the video stream and push it to the RTSP address accessible in the local area network; the streaming media server uses FFmpeg to pull the RTSP protocol video stream after the target detection processing is completed on the video acquisition end, and pushes it to the RTMP address accessible to the SRS framework; the SRS framework accesses the RTMP address and converts it to the webRTC protocol, and finally provides a webRTC protocol video stream address; Push target tracking video stream includes: The video acquisition end uses FFmpeg to pull the RTSP protocol video stream of the visible light camera and decode it, and uses it as the input of the target tracking algorithm model. After processing, FFmpeg is used to encode the video stream and push it to the RTSP address accessible in the local area network; the streaming media server uses FFmpeg to pull the RTSP protocol video stream after the target detection processing of the video acquisition end, and pushes it to the RTMP address accessible to the SRS framework; the SRS framework accesses the RTMP address and converts it to the webRTC protocol, and finally provides a webRTC protocol video stream address; Pushing panoramic stitching video streams includes: The video acquisition end uses FFmpeg to pull the RTSP protocol video stream of the visible light camera and decode it. After image acquisition, feature extraction and matching, feature point purification, homography matrix estimation, perspective transformation, and image fusion steps, panoramic image stitching is achieved. After processing, FFmpeg is used to encode the video stream and push it to an RTSP address accessible in the local area network; the streaming media server uses FFmpeg to pull the RTSP protocol video stream after target detection processing at the video acquisition end, and pushes it to an RTMP address accessible to the SRS framework; the SRS framework accesses the RTMP address and converts it to the webRTC protocol, and finally provides a video stream address of the webRTC protocol.

10. The video streaming media processing device for an intelligent patrol robot according to claim 4, characterized in that: The FFmpeg framework decodes and processes the images and videos captured by the visible light camera, calls the yolov5 and SSD algorithms to achieve target detection, the KCF and CFNet algorithms to achieve target tracking, or realizes panoramic image stitching after image acquisition, feature extraction and matching, feature point purification, homography matrix estimation, perspective transformation, and image fusion steps. FFmpeg encodes the processed images and videos, converts them into RTMP and transmits them to the SRS framework. The SRS framework outputs the video stream as a webRTC address accessible in the local area network.

Images