Distributed video stream data transmission method and system

Abstract

The invention discloses a distributed video stream data transmission method and system, and relates to the technical field of video processing systems. The system comprises an edge device, a cloud server and a local visual terminal, wherein the edge device is used for realizing real-time identification and classification of fruits in an orchard video stream; the cloud server is used for realizing public network accessibility of the video stream pushed by the edge device; and the local visual terminal is used for receiving, displaying and analyzing the fruit detection result from the cloud server in real time. According to the system, efficient detection and real-time visual presentation of the fruit target can be realized.

Description

Technical Field

[0001] This invention relates to the field of video processing system technology, and in particular to a distributed video stream data transmission method and system. Background Technology

[0002] As people's living standards continue to improve, blueberries are increasingly favored by consumers due to their unique flavor and rich nutritional value. However, the small size, soft texture, and dark color of blueberries, along with the insignificant differences in appearance between individuals at different ripening stages, pose significant challenges to manual harvesting and grading. Traditional harvesting methods are inefficient and costly due to the large amount of manpower required and the often unfavorable conditions of high temperature and humidity. Furthermore, manual grading based on visual inspection is susceptible to subjective influences, resulting in low accuracy and difficulty in standardizing and stabilizing fruit quality, thus impacting market competitiveness and product quality.

[0003] Non-destructive testing (NDT) technology for fruits, which obtains information about their appearance or internal quality without damaging their structure, has been widely studied in recent years. This type of technology mainly includes methods such as near-infrared spectroscopy, acoustic vibration, electrical property analysis, image processing, and computer vision. Among these, compared to other NDT methods, computer vision technology, through image acquisition and feature extraction combined with pattern recognition and machine learning algorithms, can efficiently establish a mapping relationship between fruit image features and quality indicators. This enables rapid, accurate, green, and intelligent grading of fruit maturity, offering advantages such as high precision, high speed, and safety / environmental friendliness.

[0004] However, current fruit maturity detection systems still have many shortcomings, such as low detection efficiency, poor visual interaction experience, and complex system deployment, making it difficult to meet the needs of efficient, convenient, and intelligent fruit grading in actual production. Therefore, there is an urgent need to build a high-precision, high-efficiency, and easy-to-deploy fruit maturity detection system to improve the standardization of grading, reduce labor costs, and help the fruit industry improve quality and efficiency. Summary of the Invention

[0005] The technical problem to be solved by the present invention is how to provide a distributed video stream data transmission method and system that can achieve efficient detection and real-time visualization of fruit targets.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a distributed video stream data transmission system, comprising: an edge device, a cloud server, and a local visualization terminal. The edge device is used to realize real-time identification and classification of fruits in the orchard video stream; the cloud server is used to realize the public network accessibility of the video stream pushed by the edge device; and the local visualization terminal is used to receive, display, and analyze the fruit detection results from the cloud server in real time.

[0007] The application further discloses a distributed video stream data transmission method using the transmission system, comprising the following steps:

[0008] S1, real-time identification and classification of fruits in the orchard video stream by the edge device;

[0009] S2, the cloud service realizes the public network accessibility of the video stream pushed by the edge device;

[0010] S3, the local visualization terminal receives, displays and analyzes the fruit detection results from the cloud server in real time.

[0011] The beneficial effects produced by the above technical solutions are that the system and method deploy YOLO-F based on the Jetson AGX Orin platform, realize efficient inference through FP16 quantization acceleration of TensorRT, build a low-delay video transmission pipeline using GStreamer, and realize real-time video pushing using NGINX-RTMP. At the same time, the system builds a graphical interface using PyQt, supports visual display of detection results, MySQL data synchronization and video recording, realizes remote access in a public network environment using FRP, and realizes a closed-loop process from unmanned aerial vehicle video collection to detection result display. The detection speed on the Jetson AGX Orin reaches 134.23 FPS, realizing the unification of high precision, high efficiency and high real-time performance. The overall system shows good stability and adaptability in the field deployment of fruit orchards, verifying its practicality, deployability and popularization value in agricultural scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0012] The application will be described in further detail below with reference to the drawings and specific embodiments.

[0013] Figure 1 is a principle block diagram of the system described in the embodiments of the application;

[0014] Figure 2 is a general processing flowchart of the system described in the embodiments of the application;

[0015] Figure 3 is an interface diagram of the local visualization terminal in the system described in the embodiments of the application;

[0016] Figure 4 is a main flowchart of the method described in the embodiments of the application. DETAILED DESCRIPTION

[0017] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0018] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced according to other embodiments that do not require the specific details described herein. Therefore, the present application is not limited to the embodiments described herein but rather encompasses all embodiments that would normally fall within the scope of the present application.

[0019] As shown in Figure 1 The embodiments of the present application disclose a distributed video stream data transmission system, which mainly includes three parts: an edge device, a cloud server and a local visualization terminal. The edge device is used to realize real-time identification and classification of fruits in orchard video streams. The cloud server is used to realize public network accessibility of the video stream pushed by the edge device. The local visualization terminal is used to receive, display and analyze the fruit detection results from the cloud server in real time.

[0020] As shown in Figure 2 The overall processing steps of the system include the following steps:

[0021] 1) Collect fruit images by a drone, collect high-definition video streams from the drone, and transmit them to a Jetson AGX Orin edge device.

[0022] 2) Use a TensorRT accelerated deployment FP16 quantization model on the edge device to infer the video frames and complete fruit maturity detection.

[0023] 3) Build an efficient encoding pipeline by GStreamer to encode and compress the video frames in real time.

[0024] 4) Deploy NGINX+RTMP servers on the cloud server and the Jetson AGX Orin edge device respectively;

[0025] 5) Combine the flvmux module and the rtmpsink module to push the encoded video stream in real time in the RTMP protocol.

[0026] 6) Use FRP internal network penetration technology to transmit the RTMP stream to Tencent cloud through reverse proxy to realize public network access support.

[0027] 7) Real-time display of fruit detection results in the PyQt visualization system and synchronization of detection data to the MySQL database, supporting video recording and remote backtracking analysis functions.

[0028] The three modules in the system will be described in detail below in combination with specific content:

[0029] Module 1, edge device (including unmanned aerial vehicle), uses Jetson AGX Orin as the main computing platform, deploys a target detection model accelerated by TensorRT, and realizes real-time identification and classification of fruits in the orchard video stream. The target detection model output contains fruit categories, quantities, and corresponding confidence, etc. Key detection indicators, and the processing results are used for subsequent visualization and analysis.

[0030] The edge device connects the camera to collect real-time images. In an embodiment of the present application, a real-time audio and video streaming method based on the GStreamer framework is provided, which calls the command line tool gst-launch-1.0, combines the Linux video input device (Video4Linux, referred to as V4L2) interface, realizes the compression encoding and network transmission of the local collected images, and the specific process is as follows:

[0031] First, the real-time video input stream is obtained from / dev / video0 through the v4l2src element, which is responsible for grabbing uncompressed raw image frames from the system default video capture device. The image frames are buffered through the queue element to improve the stability and asynchronous processing capability of subsequent processing.

[0032] Subsequently, the raw image frames are converted by the videoconvert element to ensure that the output is a unified raw format video / x-raw, with color format I420, and the image resolution is standardized to 640x480 pixels, ensuring the input consistency and compatibility of the subsequent encoding module.

[0033] After that, the image data is input to the x264enc encoder module for H.264 format compression encoding. During the encoding process, the tune parameter is set to zerolatency to adapt to the real-time transmission scenario; the speed-preset is set to ultrafast to minimize the encoding delay; the bitrate is fixed at 500kbps to control the bandwidth occupancy; the key frame interval (key-int-max) is set to 15 to improve the inter-frame compression rate and control the frame refresh frequency; the bframes parameter is set to 0 to disable B frames to reduce decoding delay; and the minimum and maximum quantization parameters (qp-min=20, qp-max=40) are set to balance image quality and encoding efficiency.

[0034] The encoded video stream is imported to the flvmux encapsulator through another queue buffer module for encapsulating the encoded video data into the FLV (Flash Video) format, which is adapted to the transmission requirements of the RTMP (Real-Time Messaging Protocol) protocol.

[0035] Finally, the encapsulated video stream is pushed to the local RTMP server through the rtmpsink module, and the push address is rtmp: / / 192.168.1.102 / live / stream, and the server listens to the default port 1935. Through the above processing flow, the real-time compression and network transmission of the local collected video are realized.

[0036] In the implementation process of the application, the FRP (Fast Reverse Proxy) client module is integrated in the edge device Jetson AGX Orin. The main function of this module is to safely and stably map the RTMP push streaming service (listening port 1935) running locally on the device to the public network access port (such as 6001) of the remote cloud server, so as to realize the public network accessibility of the local private network video stream and break the traditional internal network penetration limit, providing an efficient channel for remote video data acquisition and processing.

[0037] The FRP client establishes a long connection with the corresponding FRP server on the cloud server by configuration, which opens a specified public network port (for example, 6001) on the cloud to forward the access requests from public network users to the RTMP service interface locally on the edge device. This process is completely based on the TCP (Transmission Control Protocol) protocol for communication, which guarantees the order, integrity and high reliability of data transmission, and is suitable for high-definition video stream pushing scenarios with high real-time and data accuracy requirements.

[0038] During data mapping, the FRP system supports automatic reconnection and connection state monitoring functions, which can effectively deal with abnormal situations such as network fluctuations, and improve the stability and maintainability of the overall system. In addition, this mechanism can be used in combination with the dynamic domain name resolution (DDNS) service to further enhance the ease of use and access flexibility of the video service, and adapt to remote data access requirements in various network topologies.

[0039] Through the above-mentioned FRP client and server collaborative mechanism, the video stream collected and processed by the edge device can be stably exposed to external systems or users without the need for a public network IP, providing efficient and secure network support for subsequent remote video display, data acquisition and multi-source information fusion processing.

[0040] Module two, in the embodiments of the application, the cloud server deploys FRP (Fast Reverse Proxy) service end to realize the public network accessibility of the video stream pushed by the edge device. In order to enhance the flexibility of deployment, the scalability of the system and the isolation of the running environment, the FRP service end is encapsulated in a Docker container for running. The containerized deployment mode not only reduces the dependence of the system on the underlying operating system environment, but also effectively isolates the network configuration and the running environment, and improves the stability and maintainability of the service.

[0041] Firstly, the Docker container engine is pre-installed on the cloud server, and the public network listening port 7500 required to be opened is configured to ensure that the FRP service can normally receive the reverse proxy request from the edge device. Subsequently, a FRP service end configuration file is created in the host file system, which contains necessary parameter settings, including bind_port, token, dashboard_port and dashboard_user and dashboard_pwd, etc., for realizing the control and management of network connection.

[0042] Then, the system can pull the FRP Docker image provided by the official or third party. After the construction is completed, the local configuration file is mapped to the specified path inside the container through the mounting mechanism of the Docker container engine, and the port mapping between the host and the container is realized through the -p parameter. By executing the container start command, the FRP service end can run in the background in the container and realize the communication establishment with the edge device.

[0043] In order to ensure the high availability of the service, the system further sets the automatic restart policy of the container (such as --restart=always), so as to ensure that the FRP service can automatically recover and run in the case of server restart or accidental exit of the container. In addition, in order to improve the security and resource scheduling ability of the system, the Docker network isolation mechanism and the container orchestration tool (such as docker-compose) can be combined to uniformly manage and secure the FRP service.

[0044] In a specific embodiment of the application, in order to realize the unified access of the RTMP video stream pushed by the edge device and the stable distribution of the media resource, the system deploys a media stream forwarding server based on NGINX on the cloud server, and integrates the nginx-rtmp-module module in the configuration thereof. The module can support the reception, processing and redistribution of RTMP (Real-Time Messaging Protocol) stream, and meet the core needs of low delay and high concurrency of real-time streaming media business.

[0045] During running, the FRP service deployed in the Docker container first receives the RTMP video stream from the edge device. Port mapping maps the external public network port (such as 6001) to the FRP service listening port (1935) inside the container, and completes the secure forwarding of the edge stream. Subsequently, the NGINX service of the cloud server host machine receives the stream content by listening to the public network port and hands it over to the RTMP module for processing. In the specific configuration, the application block of NGINX defines the RTMP input path (such as / live), and configures the push or pull parameters to facilitate the forwarding and sharing of subsequent content between different terminals.

[0046] To ensure the transmission quality and playback performance of the video stream, the system strictly controls the key parameters of the edge device encoding output. For example, if the video code rate of the edge encoder output is B (unit: kbps) and the video frame rate is F (unit: fps), the data amount transmitted per unit time is:

[0047] (unit: Bytes per frame);

[0048] where D represents the byte size of each image. By reasonably setting bitrate = 500 (kbps), key-int-max = 15, and zero latency mode, the number of delayed frames can be significantly compressed, and the stability and response speed of the video stream during network forwarding can be optimized.

[0049] After receiving the RTMP stream, NGINX can further configure the dash output path to slice the received live stream by time, generate.m3u8 and.ts files for the front-end visualization platform to pull and display in real time, thereby realizing image-level display of the orchard fruit recognition results. This video pull stream process is based on RTMP or HTTP-FLV protocol, ensuring that the front-end terminal can still smoothly obtain the pushed content under different network conditions.

[0050] In addition, the system supports multiple stream concurrent access in terms of scalability, relying on the high-performance event-driven model and multi-thread processing mechanism of NGINX to ensure low memory occupation and stable processing capacity when the number of RTMP connections is large. The system can also cooperate with CDN or distributed proxy servers to distribute the NGINX stream content, thereby realizing the visualization access demand for a large number of user terminals.

[0051] Module three, in a preferred embodiment of the application, the local visualization terminal faces the agricultural terminal user (if the farmer or the management personnel) builds the local visualization system based on the PyQt framework, which is used for receiving, displaying and analyzing the fruit detection results from the cloud server in real time. The front-end interface structure of the system is realized by the multi-window and multi-thread mechanism of PyQt, which has good interface response ability and interactive operation experience. The interface diagram of the local visualization terminal is shown in Figure 3

[0052] The local visualization system connects the real-time unmanned aerial vehicle video stream forwarded by the public IP of the cloud server through the RTMP pull stream mechanism, ensuring stable transmission of the video content to the terminal interface without introducing delay. The system directly integrates the fruit detection model inference results deployed on the edge device, combines the visualization layer for real-time superposition, accurately labels the fruit target categories (such as mature fruit, unripe fruit), quantity and confidence value in each frame of picture, and realizes dynamic monitoring of the fruit growth state.

[0053] To support the judgment and data archiving of the ripening time of the fruit farmers, the system has a built-in local video recording module. The recording function is realized by the VideoRecorder class, which mainly includes output path setting, video encoder initialization, frame-by-frame writing, recording end and file naming, etc. In the specific implementation process, the system uses the cv2.VideoWriter interface to create a.mp4 format video file, and the video parameters support flexible configuration, including resolution (frame_width, frame_height), frame rate (fps) and encoding method (fourcc='mp4v'). Whenever the system user triggers the recording button, a timestamp file name is automatically generated and the current streaming media frame data is captured and stored, effectively forming time-sequenced video materials for long-term management and playback analysis.

[0054] In addition, the system backend synchronously integrates the MySQL database service for structured storage of target detection information output by the model. The target categories, quantity and confidence extracted from each frame of image are written into the database table after processing, realizing flexible query and statistical analysis based on time, region or target category. For example, the time variation trend of the fruit ripening density in a certain region can be analyzed through the SQL aggregation function, or the prediction confidence distribution of the model can be evaluated through the confidence histogram, thereby providing decision support basis for agricultural managers.

[0055] In summary, the visualization terminal takes PyQt as the core, integrates RTMP real-time video pull stream, model inference result superposition display, local video recording and structured database storage, etc., realizes the whole process closed loop from image reception, real-time judgment to data sedimentation, and greatly improves the intelligent level and efficiency of fruit picking time judgment in the agricultural production link.​

[0056] In summary, the present application realizes efficient detection and real-time visualization of fruit targets by constructing a three-layer architecture composed of edge devices, cloud servers and local visualization terminals. The modules of the system are closely coordinated and complementary in function.

[0057] On the edge device side, the target detection model optimized by TensorRT is deployed based on Jetson AGX Orin, and combined with the real-time video stream collected by the camera, the precise identification and classification of fruits are completed. At the same time, the RTMP protocol is used to realize the streaming function, and the FRP client is used to map the local port to the cloud public network, ensuring the stability and real-time performance of data transmission.

[0058] The cloud server side deploys FRP server and NGINX media stream forwarding server through Docker containerization, ensuring the isolation of network environment and the flexibility of service deployment. NIGNX integrates RTMP module to receive and distribute edge streaming video externally, realizing the management ability of high-concurrency video stream request and effectively supporting multi-end access demand.

[0059] The local visualization terminal is built based on PyQt, which accesses the cloud video content through the pull stream mechanism, displays the target detection results in real time, and provides local video recording and database storage functions, realizes the archiving management of key images and target data, and provides an intuitive, stable and intelligent decision-making interface for agricultural users.

[0060] The system realizes the complete deployment link of YOLO-F, and builds a visual real-time detection system integrating model inference, video transmission and result display. The system deploys YOLO-F based on Jetson AGX Orin platform, realizes efficient inference through FP16 quantization acceleration of TensorRT, builds a low-latency video transmission pipeline with GStreamer, and realizes real-time video pushing with NGINX-RTMP. At the same time, the system builds a graphical interface with PyQt, supports visual display of detection results, MySQL data synchronization and video recording; combined with FRP, it realizes remote access in public network environment, effectively breaking the closed-loop process from unmanned aerial vehicle video collection to detection result display.

[0061] The detection speed on Jetson AGX Orin reaches 134.23FPS, realizing the unity of high precision, high efficiency and high real-time performance. The overall system shows good stability and adaptability in the deployment of fruit orchard, verifying its practicality, deployability and popularization value in agricultural scenarios.

[0062] The three modules are connected through a standardized protocol (RTMP, TCP), a unified data interface and a distributed task processing mechanism to form an intelligent fruit monitoring system integrating real-time detection, remote transmission and local visualization, which greatly improves the intelligent level, operation efficiency and data traceability of orchard management and has good popularization value and application prospect.

[0063] Correspondingly, the system described in the present application corresponds to a distributed video stream data transmission method as shown in Figure 4 The embodiment of the present application also discloses a distributed video stream data transmission method, which comprises the following steps:

[0064] S1, real-time identification and classification of fruits in the orchard video stream by the edge device;

[0065] S2, the cloud service realizes the public network accessibility of the video stream pushed by the edge device;

[0066] S3, the local visualization terminal receives, displays and analyzes the fruit detection results from the cloud server in real time. It should be noted that the specific implementation method of the above steps in the method described in the present application can be referred to the system described in the foregoing, and will not be repeated here.

[0067] In summary, the method and system described in the present application can realize efficient detection and real-time visualization of fruit targets.

Claims

1. A distributed video stream data transmission system, characterized in that... include: Edge devices, cloud servers, and local visualization terminals, wherein the edge devices are used to realize real-time identification and classification of fruits in orchard video streams; The cloud server is used to enable public network accessibility of video streams pushed by edge devices; the local visualization terminal is used to receive, display and analyze the fruit detection results from the cloud server in real time.

2. The distributed video stream data transmission system as described in claim 1, characterized in that: The edge device uses Jetson AGX Orin as a computing platform and deploys a target detection model accelerated by TensorRT to achieve real-time identification and classification of fruits in the orchard video stream. The model output includes fruit category, quantity and corresponding confidence level, and the processing results are used for subsequent visualization and analysis.

3. The distributed video stream data transmission system as described in claim 1, characterized in that: The edge device connects to a camera to collect real-time image data, and compresses and encodes the locally acquired images for network transmission using the following method: The v4l2src element obtains the real-time video input stream from / dev / video0 and is responsible for capturing uncompressed raw image frames from the system's default video capture device. The original image frames are format-converted using the videoconvert element, and the image resolution is normalized to a set pixel value. Image data is input to the x264enc encoder module for H.264 format compression encoding; The encoded video stream is imported into the flvmux wrapper via another queue buffer module, which encapsulates the encoded video data into FLV format to meet the transmission requirements of the RTMP protocol. The encapsulated video stream is pushed to the local RTMP server through the rtmpsink module, enabling real-time compression and network transmission of locally acquired video.

4. The distributed video stream data transmission system as described in claim 1, characterized in that: The cloud server deploys an FRP server to enable public network accessibility of video streams pushed by edge devices. The FRP server is encapsulated in a Docker container and runs. Pre-install the Docker container engine on the cloud server and configure the required open public network listening ports to ensure that the FRP server can receive reverse proxy requests from edge devices. Create an FRP server configuration file in the host file system by pulling an official or third-party FRPDocker image; After the build is complete, the local configuration file is mapped to a specified path inside the container through the Docker container engine's mounting mechanism, and the port mapping between the host machine and the container is implemented through the -p parameter; by executing the container startup command, the FRP server runs in the background inside the container, and communication with edge devices is established. Configure the container's automatic restart policy.

5. A distributed video stream data transmission system as described in claim 4, characterized in that: Deploy an NGINX-based media stream forwarding server on a cloud server and integrate the nginx-rtmp-module into its configuration to achieve unified access to RTMP video streams pushed by edge devices and stable distribution of media resources.

6. A distributed video stream data transmission system as described in claim 4, characterized in that: The FRP server deployed in the Docker container engine first receives RTMP video streams from edge devices. Port mapping maps the external public network port to the FRP service listening port inside the container, completing the secure forwarding of the edge stream. The NGINX service on the cloud server host listens on this public network port to receive streaming content and hand it over to the RTMP module for processing; By controlling the key parameters of the edge device's encoding output, assuming the video bitrate output by the edge encoder is B and the video frame rate is F, the amount of data transmitted per unit time is: Where D represents the size of each frame of the image in bytes; After receiving the RTMP stream, NGINX configures the dash output path, slices the received live stream by time, and generates .m3u8 and .ts files for the front-end visualization platform to pull and display in real time, thus realizing image-level display of the orchard fruit recognition results.

7. A distributed video stream data transmission system as described in claim 1, characterized in that: The local visualization terminal is designed for agricultural end-users and is built on the PyQt framework to provide a local visualization system for receiving, displaying and analyzing fruit detection results from the cloud server in real time. The system's front-end interface structure is implemented using PyQt's multi-window and multi-threading mechanisms.

8. A distributed video stream data transmission system as described in claim 7, characterized in that: The local visualization terminal connects to the real-time video stream forwarded by the public IP of the cloud server through the RTMP pull mechanism, directly integrates the inference results of the fruit detection model already deployed on the edge device, and combines them with visualization layers for real-time overlay, annotating the fruit target category, quantity and confidence value detected in each frame, so as to realize dynamic monitoring of the fruit growth status.

9. A distributed video stream data transmission system as described in claim 7, characterized in that: The local visualization terminal has a built-in local video recording module, which is used to retrospectively determine the ripening time of fruits and archive data. The recording function is implemented by the VideoRecorder class, which uses the cv2.VideoWriter interface to create .mp4 format video files. Whenever a system user triggers the record button, it automatically generates a timestamp filename and starts capturing and storing the current streaming media frame data to form time-series video data. The local visualization terminal backend is synchronously integrated with a MySQL database service for structured storage of target detection information output by the model; The target category, quantity, and confidence score extracted from each frame of image are processed and written into a database table to enable querying and statistical analysis based on time, region, or target category.

10. A distributed video stream data transmission method, wherein the method uses the transmission system as described in any one of claims 1-9, characterized in that... Includes the following steps: S1, edge devices perform real-time identification and classification of fruits in the orchard video stream; S2, cloud services enable public network accessibility of video streams pushed by edge devices; S3 is a local visualization terminal that receives, displays, and analyzes fruit detection results from the cloud server in real time.

Images