Engineering machinery camera monitoring storage and multi-process image multiplexing system and method

Abstract

The invention discloses an engineering machinery camera monitoring storage and multi-process image multiplexing system and method, and the system comprises an image collection module which collects an image and outputs the image as a frame image; the image multiplexing module is used for sharing the frame image among a plurality of processes; the video coding module is used for coding the frame image into a main code stream and a sub code stream; the streaming media server is used for forwarding the main code stream and the transcoding sub code stream into a WebRTC stream; the video storage module is used for receiving and storing the main code stream; the remote monitoring module is used for receiving the WebRTC stream and playing the WebRTC stream in real time; the method comprises the steps of camera initialization and image acquisition, memory sharing and semaphore initialization, image acquisition and cyclic writing into a sharing area, multi-process concurrent image reading, video coding and dual-path output, streaming media server forwarding and transcoding, and playing by the remote monitoring module. According to the invention, low-delay wireless transmission of videos, stable video storage and efficient sharing of camera images by a plurality of sensing processes can be realized, and the resource utilization rate and the overall performance of the system are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of engineering machinery, and in particular relates to an engineering machinery camera monitoring storage and multi-process image multiplexing system and method. Background Art

[0002] As construction machinery evolves toward intelligent and unmanned operations, scenarios such as remote control (such as high-precision teleoperation of tower cranes), automated marshaling (such as collaborative mining truck platooning), and autonomous operations (such as intelligent excavator loading) place higher performance demands on visual perception systems. As the core visual perception unit, the performance of cameras directly impacts the environmental perception capabilities and operational safety of construction machinery.

[0003] The current system has the following technical problems:

[0004] 1. Conflict between data exclusivity and multitasking: GMSL, AHD, and other cameras are limited by the driver architecture (such as V4L2), which allows only one process to access the video stream at a time. This prevents other image processing or perception tasks from synchronously acquiring image data.

[0005] 2. The conflict between real-time transmission and high-quality storage: Low-latency wireless transmission (such as WebRTC) requires lower bit rates and compression efficiency, while hard disk recording systems for local storage (such as RTSP recording) require high-quality video. The two have inconsistent requirements for encoding parameters, making it difficult to reuse encoding resources. Deploying independent cameras increases hardware costs and system complexity. Summary of the Invention

[0006] The purpose of the present invention is to provide a system and method for engineering machinery camera monitoring storage and multi-process image multiplexing, which can realize low-latency wireless transmission of video, stable video storage and efficient sharing of camera images by multiple perception processes, significantly improving resource utilization and overall system performance.

[0007] To achieve the above objectives, the present invention provides an engineering machinery camera monitoring storage and multi-process image multiplexing system, comprising:

[0008] Image acquisition module, used to collect images from engineering machinery cameras and output them as frame images;

[0009] An image multiplexing module is connected to the image acquisition module and shares the frame image among multiple processes through the shared memory and semaphore mechanism of the image sharing module;

[0010] The video encoding module is connected to the image multiplexing module and is used to encode frame images. It uses the RTSP protocol to output a main stream for video storage and a sub-stream for video remote monitoring.

[0011] The streaming media server is connected to the video encoding module to forward the main stream and transcode the sub-stream into a WebRTC stream;

[0012] The video storage module is connected to the streaming media server and is used to receive and save the main stream;

[0013] The remote monitoring module is connected to the streaming server to receive the WebRTC stream and play it in real time.

[0014] As a further solution of the present invention: it also includes other image processing processes connected to the image multiplexing module, and the other image processing processes include image recognition process, target detection process and encoding and streaming process, all of which read image data through shared memory.

[0015] As a further solution of the present invention: the image sharing module is based on a single-producer-multiple-consumer mode, including a shared memory buffer and two semaphores, namely sem_empty indicating the writable state of the global shared area and sem_full indicating the readable state of the shared area, wherein the image acquisition module is the producer, and the image multiplexing module, video encoding module, and other image processing processes are all consumers.

[0016] As a further solution of the present invention: the video encoding module adopts a hardware-accelerated video encoder for encoding, and the encoder supports the simultaneous output of two streams, wherein the main stream has a high bit rate and high resolution, and the sub-stream has a low bit rate and low latency.

[0017] As a further solution of the present invention: the video encoding module includes a dynamic transcoding unit, which performs in-situ color space conversion and resolution scaling in the shared memory space when the input image format does not match the encoder requirements.

[0018] As a further solution of the present invention: the video storage module pulls the main code stream through the RTSP protocol and saves it as an MP4 or TS format video file.

[0019] To achieve the above object, the present invention further provides a method based on the above engineering machinery camera monitoring storage and multi-process image multiplexing system, comprising the following steps:

[0020] S1. Camera initialization and image acquisition: Install cameras at key locations on the construction machinery, connect them to the vehicle-mounted main control unit via cables, and perform image acquisition after initialization.

[0021] S2. Shared memory and semaphore initialization: Recreate a shared memory area for image caching in the main control system and establish supporting semaphore resources sem_empty to indicate an empty buffer and sem_full to indicate a full buffer. In the initial state, the image acquisition module writes the first frame of image to the shared memory and updates the sem_full semaphore to 1 and sem_empty to 0.

[0022] S3. Concurrent image reading by multiple processes: Other image processing processes and the video encoding module monitor the sem_full semaphore in a blocking manner. When the image data is ready, the image data in the shared memory is read. After the image is read, other image processing processes maintain the semaphore accordingly.

[0023] S4. Image acquisition loop writes to the shared area: The image acquisition module uses non-blocking acquisition mode to continuously acquire image data. If the shared memory area is still in the "full" state, it waits until all consumers have finished reading and release sem_empty. The acquisition process then writes the next frame of image to the shared memory and updates the semaphore state, achieving decoupling and synchronization between acquisition and distribution.

[0024] S5, Video Encoding and Dual-Output: The video encoding module reads the frame image in the shared memory, performs encoding processing, and is configured in "dual output" mode, using the RTSP protocol for output:

[0025] Output to the streaming media server as sub-stream with low bit rate and low delay parameters;

[0026] Output the main stream to the streaming media server with high bit rate and high quality configuration;

[0027] S6. Streaming server forwarding and transcoding: The streaming server forwards and distributes the received RTSP stream. The main stream is forwarded to the video storage module for archiving and traceability. The sub-stream is transcoded into a WebRTC stream for low-latency access by the remote monitoring module.

[0028] S7. Real-time playback of remote monitoring module: The remote monitoring module establishes a signaling channel with the streaming media server based on the browser or WebRTC player, and obtains the video stream in a point-to-point (P2P) manner, achieving remote real-time preview with millisecond-level low latency, adapting to the visual needs of construction machinery in remote control and operation supervision scenarios.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] Through the shared memory and semaphore mechanism, the time difference between producer writing and consumer reading is ≤33ms (30fps), meeting the frame synchronization accuracy requirements, realizing efficient, zero-copy sharing of image data among multiple processes, solving the camera exclusivity problem, and improving the real-time and parallel performance of the perception system.

[0031] Single-shot encoding supports both low-latency transmission and high-quality storage, taking into account the needs of remote monitoring and operation traceability, avoiding resource waste and improving overall system performance and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a schematic diagram of the structure of the engineering machinery camera monitoring storage and multi-process image multiplexing system of the present invention;

[0033] Figure 2 It is a flow chart of a method based on an engineering machinery camera monitoring storage and multi-process image multiplexing system of the present invention;

[0034] Figure 3 This is a flowchart of the image multiplexing process of the present invention, showing the semaphore synchronization process between the image acquisition module (producer) and multiple processes (consumers). DETAILED DESCRIPTION

[0035] The present invention will be further described below with reference to the accompanying drawings.

[0036] like Figure 1 As shown, the engineering machinery camera monitoring storage and multi-process image multiplexing system includes:

[0037] Image acquisition module, used to collect images from engineering machinery cameras and output them as frame images; the image acquisition module collects image data through the V4L2 interface or a dedicated SDK, and sets the frame rate, resolution, image format, and automatic exposure parameters;

[0038] The image multiplexing module is connected to the image acquisition module and shares the frame image among multiple processes through the shared memory and semaphore mechanism of the image sharing module. The frame image formats in the image sharing module include but are not limited to YUV, RGB, and RAW, and are all transmitted with zero copy through the shared memory.

[0039] The video encoding module is connected to the image multiplexing module and is used to encode frame images. It uses the RTSP protocol to output a main stream for video storage and a sub-stream for video remote monitoring.

[0040] The streaming media server is connected to the video encoding module to forward the main stream and transcode the sub-stream into a WebRTC stream;

[0041] The video storage module is connected to the streaming media server and is used to receive and save the main stream;

[0042] The remote monitoring module is connected to the streaming server and is used to receive WebRTC streams and play them in real time. The remote terminal module includes a web player and an application for achieving low-latency viewing through an RTC connection, with a playback delay of less than 200 milliseconds.

[0043] Furthermore, it also includes other image processing processes connected to the image multiplexing module. The other image processing processes include image recognition process, target detection process and encoding and streaming process, all of which read image data through shared memory.

[0044] Furthermore, the image sharing module is based on a single-producer-multiple-consumer model, including a shared memory buffer and two semaphores, namely sem_empty, which indicates the writable state of the global shared area, and sem_full, which indicates the readable state of the shared area. The image acquisition module is the producer, and the image multiplexing module, video encoding module, and other image processing processes are all consumers.

[0045] Furthermore, the video encoding module uses a hardware-accelerated video encoder for encoding. The encoder supports the simultaneous output of two streams, where the main stream has a high bit rate and high resolution, and the sub-stream has a low bit rate and low latency.

[0046] Furthermore, the video encoding module includes a dynamic transcoding unit that performs in-situ color space conversion and resolution scaling in the shared memory space when the input image format does not match the encoder requirements.

[0047] Preferably, the encoder type of the video encoding module includes:

[0048] Hardware-accelerated encoder: H.264 / H.265 encoder based on GPU / VPU (such as NVIDIA nvenc, Intel Media SDK);

[0049] Software encoder: CPU-based x264 / x265 open source encoder.

[0050] Furthermore, the video storage module pulls the main stream through the RTSP protocol and saves it as an MP4 or TS format video file.

[0051] like Figure 2 As shown, the method based on the above-mentioned engineering machinery camera monitoring storage and multi-process image multiplexing system includes the following steps:

[0052] S1. Camera initialization and image acquisition: Install cameras at key locations on the construction machinery and connect them to the vehicle's main control unit via cables. Use the V4L2 standard interface or the SDK provided by the camera manufacturer to initialize the camera and then perform image acquisition. Initialization includes setting resolution, frame rate, exposure parameters, etc., ensuring exclusive access to the driver layer to guarantee image acquisition stability from the source.

[0053] S2. Shared memory and semaphore initialization: Recreate a shared memory area for image caching in the main control system and establish supporting semaphore resources sem_empty to represent an empty buffer and sem_full to represent a full buffer. These are used to synchronize the image reading and writing processes of the producer and multiple consumers. In the initial state, the image acquisition module writes the first frame of image to the shared memory and updates the sem_full semaphore to 1 and sem_empty to 0.

[0054] S3. Image acquisition loop writes to the shared area: The image acquisition module uses non-blocking acquisition mode to continuously acquire image data. If the shared memory area is still in the "full" state, it waits until all consumers have finished reading and released sem_empty. The acquisition process then writes the next frame of image to the shared memory and updates the semaphore state, achieving decoupling and synchronization between acquisition and distribution.

[0055] S4. Concurrent image reading by multiple processes: Other image processing processes and the video encoding module (i.e., consumers) monitor the sem_full semaphore in a blocking manner. When the image data is ready, the image data in the shared memory is read. After the image is read, no copying is required to ensure high efficiency. Other image processing processes maintain the semaphore accordingly, such as reducing the sem_full count or notifying the producer to release memory space.

[0056] S5, Video Encoding and Dual-Output: The video encoding module reads the frame image in the shared memory, performs encoding processing, and is configured in "dual output" mode, using the RTSP protocol for output:

[0057] Output to the streaming media server as sub-stream with low bit rate and low delay parameters;

[0058] Output the main stream to the streaming media server with high bit rate and high quality configuration;

[0059] S6. Streaming server forwarding and transcoding: The streaming server forwards and distributes the received RTSP stream. The main stream is forwarded to the video storage module for archiving and traceability. The sub-stream is transcoded into a WebRTC stream for low-latency access by the remote monitoring module.

[0060] S7. Remote monitoring module implements playback: The remote monitoring module establishes a signaling channel with the streaming media server based on the browser or WebRTC player, and obtains the video stream in a point-to-point (P2P) manner, achieving remote real-time preview with millisecond-level low latency, adapting to the visual needs of construction machinery in remote control and operation supervision scenarios.

[0061] Example:

[0062] Using a certain type of excavator as the application carrier, the system of the present invention is deployed to implement a video data processing solution with remote monitoring and local storage in parallel. The specific steps are as follows:

[0063] S1: Camera initialization and image acquisition (corresponding to Figure 2 Image acquisition module)

[0064] Three GMSL analog cameras were installed in key operating areas of the excavator, including the cab, boom end, and rear of the vehicle. These cameras were connected to the vehicle's industrial computer via coaxial cables. The industrial computer ran the Linux operating system and identified each camera's device node (e.g., / dev / video0) through the V4L2 interface.

[0065] During initialization, the system sets each camera to a resolution of 1280×720, a frame rate of 30 FPS, and an image format of YUYV. Because the V4L2 driver has exclusive access, the acquisition task is completed independently by an independent acquisition process.

[0066] S2: Shared memory and semaphore initialization (corresponding to Figure 3 Shared area and signal synchronization process in

[0067] During startup, the master program calls shm_open and mmap to create a shared memory area shm_frame (with a capacity of 2 frames, each 1280×720×2 bytes, in YUYV format), and creates named semaphores sem_empty and sem_full, with initial values ​​of 1 and 0, respectively.

[0068] S3: Image acquisition loop writes to the shared area (such as Figure 3 shown)

[0069] Whenever the acquisition process captures a frame of image, it first blocks and waits for sem_empty to be available, then writes the image to the shared memory, and finally uses the broadcast mechanism to notify all consumer processes that the memory is full and release sem_full:

[0070] S4: Multiple processes concurrently read images (such as Figure 3 shown)

[0071] The video encoding module and other image processing processes act as consumers, blocking and waiting for sem_full before reading the image, and releasing sem_empty after reading: multiple consumers share the same frame image, realizing multi-purpose image data reuse and avoiding repeated acquisition or copying.

[0072] S5: Video encoding and dual output

[0073] The video encoding process calls NVIDIA Jetson's hardware encoder nvv4l2h265enc and configures parallel dual-channel output:

[0074] Main stream: high quality, 25fps, H.265, 4Mbps, output as local RTSP stream for NVR to pull and store;

[0075] Substream: low latency, 30fps, H.265, 1Mbps, sent to the streaming server;

[0076] The two channels reuse the same encoding output, saving encoding resources.

[0077] S6: Streaming Media Server Forwarding and Transcoding

[0078] The streaming media server deployed in the vehicle or at the edge receives dual streams:

[0079] The main stream is directly forwarded to the local video recorder (NVR);

[0080] The sub-stream is encapsulated as WebRTC SDP through the built-in transcoding logic of the streaming server, so that remote terminals can access it with low latency through browsers.

[0081] S7: Remote monitoring module real-time playback

[0082] Remote administrators access the WebRTC playback page provided by the streaming server through a browser and establish a signaling connection based on SIP or a custom signaling protocol. The player negotiates through the ICE framework to establish an end-to-end transmission channel (preferring P2P direct connection and enabling TURN relay when penetration fails) and dynamically pulls the low-latency sub-stream encapsulated by WebRTC.

[0083] Real-time remote monitoring of construction machinery operation scenes is achieved, with end-to-end delay ≤ 200ms and video clarity meeting driver assistance needs.

Claims

1. Engineering machinery camera monitoring storage and multi-process image multiplexing system, characterized by: include: Image acquisition module, used to collect images from engineering machinery cameras and output them as frame images; An image multiplexing module is connected to the image acquisition module and shares the frame image among multiple processes through the shared memory and semaphore mechanism of the image sharing module; The video encoding module is connected to the image multiplexing module and is used to encode frame images. It uses the RTSP protocol to output a main stream for video storage and a sub-stream for video remote monitoring. The streaming media server is connected to the video encoding module to forward the main stream and transcode the sub-stream into a WebRTC stream; The video storage module is connected to the streaming media server and is used to receive and save the main stream; The remote monitoring module is connected to the streaming server to receive the WebRTC stream and play it in real time.

2. The engineering machinery camera monitoring storage and multi-process image multiplexing system according to claim 1 is characterized in that: It also includes other image processing processes connected to the image multiplexing module. Other image processing processes include image recognition process, target detection process and encoding and streaming process, all of which read image data through shared memory.

3. The engineering machinery camera monitoring storage and multi-process image multiplexing system according to claim 2 is characterized in that: The image sharing module is based on a single-producer-multiple-consumer model, including a shared memory buffer and two semaphores, namely sem_empty, which indicates the writable state of the global shared area, and sem_full, which indicates the readable state of the shared area. The image acquisition module is the producer, and the video encoding module and other image processing processes are consumers.

4. The engineering machinery camera monitoring storage and multi-process image multiplexing system according to claim 2 is characterized in that: The video encoding module uses a hardware-accelerated video encoder for encoding. The encoder supports the simultaneous output of two streams, where the main stream has a high bit rate and high resolution, and the sub-stream has a low bit rate and low latency.

5. The engineering machinery camera monitoring storage and multi-process image multiplexing system according to claim 4 is characterized in that: The video encoding module includes a dynamic transcoding unit that performs in-situ color space conversion and resolution scaling in the shared memory space when the input image format does not match the encoder requirements.

6. The engineering machinery camera monitoring storage and multi-process image multiplexing system according to claim 2 is characterized in that: The video storage module pulls the main stream through the RTSP protocol and saves it as an MP4 or TS format video file.

7. The method of the engineering machinery camera monitoring storage and multi-process image multiplexing system according to any one of claims 1 to 6, characterized in that: The following steps are involved: S1. Camera initialization and image acquisition: Install cameras at key locations on the construction machinery, connect them to the vehicle-mounted main control unit via cables, and perform image acquisition after initialization. S2. Shared memory and semaphore initialization: Recreate a shared memory area for image caching in the main control system and establish supporting semaphore resources sem_empty to indicate an empty buffer and sem_full to indicate a full buffer. In the initial state, the image acquisition module writes the first frame of image to the shared memory and updates the sem_full semaphore to 1 and sem_empty to 0. S3. Concurrent image reading by multiple processes: Other image processing processes and the video encoding module monitor the sem_full semaphore in a blocking manner. When the image data is ready, the image data in the shared memory is read. After the image is read, other image processing processes maintain the semaphore accordingly. S4. Image acquisition loop writes to the shared area: The image acquisition module uses non-blocking acquisition mode to continuously acquire image data. If the shared memory area is still in the "full" state, it waits until all consumers have finished reading and released sem_empty. The acquisition process then writes the next frame of image to the shared memory and updates the semaphore state, achieving decoupling and synchronization between acquisition and distribution. S5, Video Encoding and Dual-Output: The video encoding module reads the frame image in the shared memory, performs encoding processing, and is configured in "dual output" mode, using the RTSP protocol for output: Output to the streaming media server as sub-stream with low bit rate and low delay parameters; Output the main stream to the streaming media server with high bit rate and high quality configuration; S6. Streaming server forwarding and transcoding: The streaming server forwards and distributes the received RTSP stream. The main stream is forwarded to the video storage module for archiving and traceability. The sub-stream is transcoded into a WebRTC stream for low-latency access by the remote monitoring module. S7. Real-time playback of remote monitoring module: The remote monitoring module establishes a signaling channel with the streaming media server based on the browser or WebRTC player, and obtains the video stream in a point-to-point (P2P) manner, achieving remote real-time preview with millisecond-level low latency, adapting to the visual needs of construction machinery in remote control and operation supervision scenarios.

Images