Multi-channel video processing method and device suitable for engineering machinery

Abstract

The invention discloses a multi-channel video processing method and device suitable for engineering machinery, and belongs to the technical field of image processing, and the method comprises the steps: obtaining multi-channel video frame images of the all-round view of the engineering machinery; image processing tasks are set for the multiple paths of video frame images, and a thread pool based on a UMat resource management method is used for synchronously processing the image processing tasks; and carrying out look-around splicing and display on the multiple paths of video frame images after the processing task is executed. According to the method, the utilization of hardware resources is reduced through a UMat resource management method and a wrong frame detection technology, and the method can be deployed on an embedded platform and can be widely applied to various engineering machines.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a multi-channel video processing method and device suitable for engineering machinery. Background Art

[0002] During the operation of construction machinery (such as excavators and loaders), drivers face significant blind spots, making multi-channel video surveillance and obstacle warning systems crucial for safe operation. However, as the number of video channels increases, traditional video processing methods face issues such as inefficient resource utilization and poor multi-task coordination, driven by the surge in data volume, high real-time requirements, and complex operating conditions of construction machinery.

[0003] Patent publication number CN113535366A discloses a high-performance distributed multi-channel video real-time processing method, which belongs to the field of video processing technology. It includes the following steps: A. Constructing a video processing pipeline, which includes a video decoding module, a video pre-processing module, a video model inference module and a video post-processing module; B. Starting multiple processes / threads: each processing module starts a multi-process mode, and each process starts multiple threads; the number of processes / threads started by each module is determined based on the needs of the business scenario, model performance and software and hardware resource limitations; C. Constructing a shared queue between upstream and downstream modules and setting data access strategies. The video decoding, pre-processing, model inference and post-processing modules of the present invention are all separated, which is simple to use. The specially designed data access mechanism between adjacent modules can significantly improve the efficiency of multi-channel video processing. This invention solves the above-mentioned video processing problems, but it has too high requirements on hardware resources and is not suitable for embedded platforms of mobile vehicles. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a multi-channel video processing method and device suitable for engineering machinery, so as to solve the technical problem that the prior art solutions rely on hardware resources and are not suitable for embedded platforms of mobile vehicles.

[0005] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0006] In a first aspect, the present invention provides a multi-channel video processing method applicable to engineering machinery, comprising:

[0007] Acquire multi-channel video frame images of the surrounding view of the construction machinery;

[0008] Setting image processing tasks for the multiple video frame images, and using a thread pool based on the UMat resource management method to synchronously process the image processing tasks;

[0009] Perform surround stitching and display on the multiple video frame images that have completed the processing task.

[0010] Optionally, acquiring multiple video frame images of the surround view of the engineering machinery includes: allocating an independent thread to each video acquisition task, caching the acquired video data using a circular queue, and extracting video frame images from the cached video data.

[0011] Optionally, the image processing task includes a dedistortion task, and the dedistortion task includes:

[0012] Constructing a distortion model, wherein the distortion model includes a radial distortion model and a tangential distortion model;

[0013] The radial distortion model is:

[0014]

[0015]

[0016] The tangential distortion model is:

[0017]

[0018]

[0019] Where, is the distorted image coordinate, are the undistorted image coordinates, for The distance to the center of the image, is the radial distortion coefficient, is the tangential distortion coefficient;

[0020] Use an iterative method to map the distorted coordinates to the undistorted coordinates until the preset maximum number of iterations is reached:

[0021]

[0022]

[0023] Where, The undistorted image coordinates obtained by iteration .

[0024] Optionally, the image processing task includes a frame error obstacle detection task, and the frame error obstacle detection task includes:

[0025] Perform grayscale processing on the input video frame image to generate a grayscale image;

[0026] Calculate the difference image between the grayscale images of the previous frame and the current frame :

[0027]

[0028] Where, are the grayscale images of the current frame and the previous frame respectively;

[0029] According to the difference image Generate binary image :

[0030]

[0031] Where, For binary images Center coordinates The pixel value of is the difference image Center coordinates Pixel value of is a constant value used to distinguish moving targets from background noise;

[0032] Contour detection using OpenCV:

[0033]

[0034] Where, is the contour detection function, is the list of detected contours;

[0035] Get the moving area based on the contour list :

[0036]

[0037] Where, is the original input video frame image, is the coordinate of the lower left corner of the moving area, is the width and height of the moving area;

[0038] For mobile areas Perform scaling to obtain the target size image :

[0039]

[0040] Where, The input width and height of the pre-trained object detection model;

[0041] Perform object detection using a pre-trained object detection model:

[0042]

[0043] Where, is a pre-trained target detection model, For the test results.

[0044] Optionally, setting image processing tasks for the multiple channels of video frame images, and synchronously processing the image processing tasks using a thread pool based on a UMat resource management method includes:

[0045] For the image processing task of each video frame image, a thread pool is created for parallel operation. Each thread pool obtains Umat from the pool and returns it to the pool after use.

[0046] In a second aspect, the present invention provides a multi-channel video processing device suitable for engineering machinery, comprising:

[0047] An image acquisition module is configured to acquire multi-channel video frame images of the surrounding view of the construction machinery;

[0048] A task processing module is configured to set image processing tasks for multiple channels of video frame images and synchronously process the image processing tasks using a thread pool based on a UMat resource management method;

[0049] The splicing and display module is configured to perform surround splicing and display on the multiple channels of video frame images that have completed the processing task.

[0050] Optionally, setting image processing tasks for the multiple channels of video frame images, and synchronously processing the image processing tasks using a thread pool based on a UMat resource management method includes:

[0051] For the image processing task of each video frame image, a thread pool is created for parallel operation. Each thread pool obtains Umat from the pool and returns it to the pool after use.

[0052] In a third aspect, the present invention provides an electronic device, including a processor and a storage medium;

[0053] The storage medium is used to store instructions;

[0054] The processor is configured to operate according to the instructions to execute the steps of the above method.

[0055] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0056] In a fifth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] The multi-channel video processing method and device provided by this invention, suitable for construction machinery, significantly improves the real-time performance of video processing through thread optimization; manages UMat objects through an object pool, reducing memory allocation and release overhead and improving performance; and reduces the computational workload of YOLO by preliminarily screening moving areas, thereby reducing resource usage. In summary, by reducing hardware resource utilization, the present invention can be deployed on embedded platforms and is widely applicable to various construction machinery. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 1 is a flow chart of a multi-channel video processing method applicable to engineering machinery provided by an embodiment of the present invention;

[0060] Figure 2 This is a flowchart of an implementation of a multi-channel video processing method applicable to engineering machinery provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0062] Example 1:

[0063] like Figure 1 As shown, the present invention provides a multi-channel video processing method applicable to engineering machinery, comprising the following steps:

[0064] Step S1: Acquire multiple video frame images of the surrounding view of the construction machinery.

[0065] Specifically in this embodiment, the multiple video frame images generally include four directions, namely, front, back, left, and right, of the construction machinery. The surround view images can be subsequently spliced ​​through the four video frame images.

[0066] Acquiring multi-channel video frames from construction machinery's surround view involves assigning a separate thread to each video capture task, caching the captured video data using a circular queue, and extracting video frames from the cached video data. This approach not only fully leverages the parallel advantages of multithreading to improve video capture efficiency, but also effectively prevents data loss and ensures the integrity and continuity of video data.

[0067] Step S2: Setting image processing tasks for multiple video frame images, and using a thread pool based on the UMat resource management method to synchronously process the image processing tasks.

[0068] The image processing tasks can be set according to specific usage needs. Specifically, in this embodiment, the image processing tasks include a dedistortion task and a frame error obstacle detection task.

[0069] (1) Dedistortion tasks include:

[0070] (1.1) Constructing a distortion model, which includes a radial distortion model and a tangential distortion model;

[0071] The radial distortion model is:

[0072]

[0073]

[0074] The tangential distortion model is:

[0075]

[0076]

[0077] Where, is the distorted image coordinate, are the undistorted image coordinates, for The distance to the center of the image, is the radial distortion coefficient, is the tangential distortion coefficient;

[0078] (1.2) Use the iterative method to map the distorted coordinates to the undistorted coordinates until the preset maximum number of iterations is reached:

[0079]

[0080]

[0081] Where, The undistorted image coordinates obtained by iteration .

[0082] To reduce the CPU resource usage of iterative methods, the traditional approach is to introduce UMat (a matrix class in OpenCV that unifies CPU and GPU operations, enabling transparent CPU / GPU acceleration and automatically using the GPU for computations) to offload time-consuming computations to the GPU. Since this system is primarily used on embedded platforms and requires simultaneous processing of multiple images, GPU resources are also limited. Therefore, the multi-channel image dedistortion task dynamically allocates resources by creating a thread pool. However, this dynamic allocation can lead to frequent resource release and creation, potentially causing system crashes. Therefore, a UMat object pool resource management method was designed specifically for UMat. In this system's testing experiments, conducted on the RK3568 platform, the frequent release and creation of UMat caused high fluctuations in GPU resource utilization (ranging from 80% to over 100%), leading to system crashes.

[0083] UMat object pool resource management method design description:

[0084] A UMatPtr (UMat pointer) double-ended queue pool (UMat object pool) is created using a C++ deque (a double-ended queue, allowing efficient insertion and deletion operations at both ends). The UMat resource management method initializes and creates 10 objects (preset parameters: a custom script is called from the command line to read the load utilization of the RK3568 GPU, with utilization below 80% as the standard) to build the object pool. This pre-allocation mechanism avoids resource waste caused by frequent release / creation. Two functions are also designed to expose the acquisition and release of UMate objects in the pool. Both creation and release use QMutex (mutex) and QMutexLocker (mutex lock) to achieve mutually exclusive access to the resource pool.

[0085] getUMat (UMate object creation function): Get UMat from the pool for dedistortion calculation of the original images of the four cameras

[0086] releaseUMat (UMat object release function): return it to the pool after use;

[0087] Specific instructions for using the UMat object pool resource management method in the distortion removal process:

[0088] On the RK3568 embedded platform, a thread pool is used to synchronously process dedistortion tasks. This involves creating multiple thread pools for parallel computation. Each thread pool retrieves a UMat from the pool and returns it to the pool after use. Although multiple threads are concurrent, a locking mechanism ensures atomicity during the usage process, avoiding concurrency conflicts. Due to the limited GPU resources on the embedded platform, it is impossible to support more UMats to allocate GPU resources. Repeated creation and release would lead to a surge in GPU usage. Therefore, through the UMat resource management method, UMat resource reuse technology is used to achieve GPU resource reuse. Experimental test results show that GPU utilization is less than 70%, while also reducing CPU resource utilization.

[0089] (2) Error frame obstacle detection tasks include:

[0090] (2.1) grayscale the input video frame image to generate a grayscale image;

[0091] (2.2) Calculate the difference image between the grayscale image of the previous frame and the current frame :

[0092]

[0093] Where, are the grayscale images of the current frame and the previous frame respectively;

[0094] (2.3) According to the difference image Generate binary image :

[0095]

[0096] Where, For binary images Center coordinates The pixel value of is the difference image Center coordinates Pixel value of is a constant value used to distinguish moving targets from background noise;

[0097] (2.4) Contour detection using OpenCV:

[0098]

[0099] Where, is the contour detection function, is the list of detected contours;

[0100] (2.5) Get the moving area based on the contour list :

[0101]

[0102] Where, is the original input video frame image, is the coordinate of the lower left corner of the moving area, is the width and height of the moving area;

[0103] (2.6) For mobile areas Perform scaling to obtain the target size image :

[0104]

[0105] Where, The input width and height of the pre-trained object detection model;

[0106] (2.7) Target detection using pre-trained target detection models:

[0107]

[0108] Where, is a pre-trained target detection model, For the test results.

[0109] Currently, mainstream object detection models, such as the YOLOv5 model, provide detection results that include bounding box coordinates and category probabilities. By comparing the previous and current frames and initially screening for moving areas, the YOLOv5 model's computational workload can be effectively reduced, ultimately lowering resource usage.

[0110] Similar to the dedistortion task described above, on the RK3568 embedded platform, a thread pool is used to synchronously process the error frame detection task. This involves creating multiple thread pools for parallel computation. Each thread obtains a UMat from the pool (UMat object pool) and returns it to the pool after use. Through the UMat resource reuse technology of the UMat object pool resource management method, GPU resources are reused, reducing resource utilization.

[0111] Step S3: performing surround stitching and display on the multi-channel video frame images that have completed the processing task.

[0112] like Figure 2 As shown, in this embodiment, a dedistortion task and an error frame obstacle detection task are given. The dedistortion task can obtain a dedistorted image of multiple video frame images, and the error frame obstacle detection task can obtain the bounding box coordinates and category probabilities of obstacles on multiple video frame images.

[0113] The multi-channel dedistorted images are stitched together based on the camera's intrinsic parameters to produce a surround view image. The bounding box coordinates must first be transformed into the original video frame image, and then transformed into the surround view image based on the camera's intrinsic parameters for annotation and display. The final display contains the surround view image, bounding box coordinates, and category probabilities.

[0114] Further expansion can pre-draw the obstacle coordinate alarm sector area, and display / hide the alarm sector area through coordinates to avoid resource waste caused by repeated redrawing and improve FPS.

[0115] Example 2:

[0116] An embodiment of the present invention provides a multi-channel video processing device suitable for engineering machinery, comprising:

[0117] The image acquisition module is configured to acquire multi-channel video frame images of the surrounding view of the engineering machinery.

[0118] The task processing module is configured to set image processing tasks for multiple video frame images and use a thread pool based on the UMat resource management method to synchronously process image processing tasks; specifically, for the image processing task of each video frame image, a thread pool is created for parallel operation. Each thread pool obtains the Umat in the pool and returns it to the pool after use.

[0119] The splicing display module is configured to perform surround splicing and display of multiple video frame images that have completed processing tasks.

[0120] Example 3:

[0121] An embodiment of the present invention provides an electronic device, including a processor and a storage medium;

[0122] The storage medium is used to store instructions;

[0123] The processor is configured to operate according to the instructions to execute the steps of the above method.

[0124] Example 4:

[0125] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0126] Embodiment 5:

[0127] An embodiment of the present invention provides a computer program product, including a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0128] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0130] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0132] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multi-channel video processing method suitable for engineering machinery, characterized in that: include: Acquire multi-channel video frame images of the surrounding view of the construction machinery; Setting image processing tasks for the multiple video frame images, and using a thread pool based on the UMat resource management method to synchronously process the image processing tasks; Perform surround stitching and display on the multiple video frame images that have completed the processing task.

2. The multi-channel video processing method applicable to engineering machinery according to claim 1, characterized in that: The method of acquiring multi-channel video frame images of the engineering machinery's surround view includes: allocating an independent thread to each video acquisition task, caching the acquired video data using a circular queue, and extracting video frame images from the cached video data.

3. The multi-channel video processing method applicable to engineering machinery according to claim 1, characterized in that: The image processing task includes a dedistortion task, and the dedistortion task includes: Constructing a distortion model, wherein the distortion model includes a radial distortion model and a tangential distortion model; The radial distortion model is: The tangential distortion model is: Where, is the distorted image coordinate, are the undistorted image coordinates, for The distance to the center of the image, is the radial distortion coefficient, is the tangential distortion coefficient; Use an iterative method to map the distorted coordinates to the undistorted coordinates until the preset maximum number of iterations is reached: Where, The undistorted image coordinates obtained by iteration .

4. The multi-channel video processing method applicable to engineering machinery according to claim 1, characterized in that: The image processing task includes a frame error obstacle detection task, and the frame error obstacle detection task includes: Perform grayscale processing on the input video frame image to generate a grayscale image; Calculate the difference image between the grayscale images of the previous frame and the current frame : Where, are the grayscale images of the current frame and the previous frame respectively; According to the difference image Generate binary image : Where, For binary images Center coordinates The pixel value of is the difference image Center coordinates Pixel value of is a constant value used to distinguish moving targets from background noise; Contour detection using OpenCV: Where, is the contour detection function, is the list of detected contours; Get the moving area based on the contour list : Where, is the original input video frame image, is the coordinate of the lower left corner of the moving area, is the width and height of the moving area; For mobile areas Perform scaling to obtain the target size image : Where, The input width and height of the pre-trained object detection model; Perform object detection using a pre-trained object detection model: Where, is a pre-trained target detection model, For the test results.

5. The multi-channel video processing method applicable to engineering machinery according to claim 1, characterized in that: The step of setting an image processing task for the multiple video frame images and synchronously processing the image processing tasks using a thread pool based on a UMat resource management method includes: For the image processing task of each video frame image, a thread pool is created for parallel operation. Each thread pool obtains Umat from the pool and returns it to the pool after use.

6. A multi-channel video processing device suitable for engineering machinery, characterized in that: include: An image acquisition module is configured to acquire multi-channel video frame images of the surrounding view of the construction machinery; A task processing module is configured to set image processing tasks for multiple channels of video frame images and synchronously process the image processing tasks using a thread pool based on a UMat resource management method; The splicing and display module is configured to perform surround splicing and display on the multiple channels of video frame images that have completed the processing task.

7. The multi-channel video processing device suitable for engineering machinery according to claim 6, characterized in that: The step of setting an image processing task for the multiple video frame images and synchronously processing the image processing tasks using a thread pool based on a UMat resource management method includes: For the image processing task of each video frame image, a thread pool is created for parallel operation. Each thread pool obtains Umat from the pool and returns it to the pool after use.

8. An electronic device, characterized in that: including processors and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images