Binocular image high-speed acquisition method based on hardware decoding
Through the combination of hardware decoding and robot operating system ROS, traditional CPU processing cannot meet the problem of unmanned driving real-time and binocular image distortion error, realize high-speed acquisition and high-precision correction of binocular images, and improve unmanned driving visual perception ability.
Patent Information
- Application Number
- CN202410523707.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-28
- Publication Date
- 2024-11-08
AI Technical Summary
The traditional binocular image processing method based on CPU cannot meet the real-time and resource occupation requirements of unmanned driving technology, and binocular image has distortion and assembly error problems, making it difficult to obtain high-precision visual data.
Using a high-speed binocular image acquisition method based on hardware decoding, the binocular image is decoded using a hardware decoder, and preprocessing and correction are combined with the robot operating system ROS and OpenCV, and segmented into left-eye and right-eye images to publish topics to realize real-time data Transmission and correction.
It realizes high-speed acquisition of binocular images of million-level pixels, improves image quality and visual perception capabilities, meets the high-precision visual data needs in the field of unmanned driving, and reduces the use of CPU computing resources.
Smart Images

Figure CN118921448A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision, and in particular to a high-speed binocular image acquisition method based on hardware decoding. Background Art
[0002] At present, the traditional automobile industry is accelerating the research and development of driverless technology based on the development of artificial intelligence. The outstanding role of driverless technology in reducing the driving intensity of drivers and improving driving safety gives this technology a good development prospect.
[0003] Vision-based unmanned driving technology has the advantages of small size, light weight, non-contact, good real-time performance, rich information and high degree of intelligence.
[0004] The monocular vision solution needs to track several adjacent frames of images to obtain the three-dimensional information of the target object, and can only be used in some occasions where the vision requirements are not high. The binocular vision solution can obtain the three-dimensional information of the target object in real time through the parallax principle of two lenses, realizing true stereoscopic vision. Therefore, the binocular vision solution has become the mainstream of vision-based unmanned driving technology.
[0005] The number of pixels in the image of the binocular vision solution is usually in the millions, with high resolution and rich data information. The traditional CPU-based binocular image processing method can no longer meet the requirements of real-time visual navigation and resource occupation. In addition, binocular images often have problems such as distortion and assembly errors. The use of hardware acceleration can significantly improve the performance and efficiency of image processing and play an important role in real-time image processing. Combined with binocular image preprocessing, the imaging defects and assembly errors of the binocular camera can be corrected to ensure measurement accuracy and improve image quality, thereby enhancing resolution and visual perception capabilities. Summary of the invention
[0006] In order to overcome the shortcomings of the prior art, the present invention proposes a high-speed binocular image acquisition method based on hardware decoding, which decodes the binocular image through a special hardware decoder, and then combines the robot operating system ROS and OpenCV to perform image correction preprocessing, and finally publishes the left and right image topics.
[0007] In order to achieve the above-mentioned purpose, the technical solution proposed by the present invention is as follows.
[0008] A binocular image high-speed acquisition method based on hardware decoding comprises the following steps: 1) Start the master node Master, which is responsible for the connection and message communication between nodes, and initializes the sending node Camera_node; 2) Read the binocular image, and then use the hardware decoder nvv4l2decoder to decode the binocular image and store it in the GPU memory. The nvvidconv plug-in stores the binocular image decoded data in the GPU memory into the system memory and converts the binocular image decoded data into an image with a color format of BGRx. The videoconvert plug-in converts the image with a color format of BGRx into the BGR color format supported by OpenCV. 3) Use OpenCV to read the image converted in step 2), then convert the data read by OpenCV into the Message format supported by ROS, and then send the node Camera_node to notify the master node Master to publish the binocular image topic image_raw; 4) Set the sending rate, repeat steps 2) to 3), and continuously send the acquired binocular image messages; 5) Initialize the binocular image segmentation node Camera_splitter, notify the master node Master to subscribe to the binocular image topic image_raw, the topic image_raw has both publishers and subscribers, the master node Master notifies the sending node Camera_node and the binocular image segmentation node Camera_splitter of each other's existence, so as to transmit image messages to each other; 6) If the binocular image segmentation node Camera_splitter receives the image message sent by the sending node Camera_node, if it does not receive the message, it continues to wait for the message to be sent; 7) If a message is received, the callback function imageCallback is triggered to preprocess the image and perform binocular image segmentation; the callback function imageCallback receives the image message parameters and converts the image message read by ROS into the BGR format supported by OpenCV; OpenCV is used to combine the distortion parameters and assembly error internal parameters of the binocular camera to perform correction preprocessing on the image, and then the preprocessed binocular image is segmented into left and right images, and the left and right images are converted into the Message format supported by ROS; based on the binocular image segmentation node Camera_splitter, the master node Master is notified to publish the left image topic left_cam and the right image topic right_cam.
[0009] Step 2) The binocular image is read based on the V4L2 protocol.
[0010] Step 2) The format of the binocular image is MJPEG, H.264, H.265, or JPEG.
[0011] Step 2) The hardware decoder nvv4l2decoder is a GStreamer plug-in for video decoding, which uses a hardware encoding engine to process video data.
[0012] Step 7) The distortion parameters and assembly error internal parameters are obtained based on a single-plane chessboard calibration method.
[0013] Compared with the prior art, the present invention has the following beneficial effects: (1) It does not occupy CPU computing resources and can achieve high-speed acquisition of binocular images with millions of pixels. The acquisition speed is one order of magnitude higher than the traditional CPU-based serial computing method, which can better meet the real-time requirements of unmanned driving technology.
[0014] (2) The robot operating system ROS and OpenCV are used to pre-process and correct the acquired data as well as perform binocular segmentation, correct the imaging defects and assembly errors of the binocular camera, ensure measurement accuracy and improve image quality, enhance resolution and visual perception capabilities, and meet the needs of the unmanned driving field for high-precision visual data.
[0015] (3) By combining the robot operating system ROS to publish left and right eye images, different types of sensor data can be easily integrated, facilitating the rapid development and deployment of data fusion algorithms. In addition, through the topic subscription and publishing mechanism of ROS, real-time transmission and processing of sensor data can be achieved, which is very suitable for application scenarios that require immediate response, such as the field of unmanned driving. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 This is the overall framework diagram of the high-speed binocular image acquisition method based on hardware decoding.
[0017] Figure 2 The figure is a flow chart of the steps of the high-speed acquisition method of binocular images based on hardware decoding. DETAILED DESCRIPTION
[0018] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments, and is implemented based on the technical solution of the present invention, but the protection scope of the present invention is not limited to the following embodiments.
[0019] like Figure 1 As shown, this embodiment provides a high-speed binocular image acquisition method based on hardware decoding, and the acquisition speed is one order of magnitude higher than the traditional CPU-based serial calculation method. The overall framework is divided into a master node Master, a sending node Camera_node, and a binocular image segmentation node Camera_splitter.
[0020] The master node provides naming and registration services for the rest of the nodes in the Robot Operating System (ROS). It keeps track of publishers and subscribers of topics and services. The role of the master node is to enable the various ROS nodes to locate each other. Once these nodes locate each other, they can communicate with each other point-to-point.
[0021] The sending node Camera_node is responsible for acquiring the binocular image in real time and at high speed using the hardware decoding method, and publishing topic messages to transmit binocular image messages. Specifically, it includes decoding the compressed binocular image; copying the acquired binocular image from the GPU memory to the system memory; and converting the binocular image color format from BGRx to BGR.
[0022] The binocular image segmentation node Camera_splitter is responsible for preprocessing the binocular image correction, binocular image segmentation, and publishing the left and right image topics. Specifically, it means correcting the image distortion and assembly error, segmenting the spliced binocular image into the left and right images, converting the left and right images into the Message format supported by ROS, and publishing the left and right image topics.
[0023] This embodiment involves specific steps such as Figure 2 As shown, the steps are as follows: 1) Start the master node Master, which is responsible for the connection and message communication between nodes and initializes the sending node Camera_node.
[0024] 2) Based on the V4L2 protocol, the binocular image compressed and encoded in MJPEG format is read, and then the hardware decoder nvv4l2decoder is used to decode the binocular image in MJPEG format and store it in GPU memory. The nvvidconv plug-in stores the decoded binocular image data in GPU memory into system memory and converts the decoded binocular image data into an image in BGRx color format. The videoconvert plug-in converts the image in BGRx color format into the BGR color format supported by OpenCV.
[0025] 3) Use OpenCV to read the image converted in step (2), then convert the data read by OpenCV into the Message format supported by ROS, and then send the node Camera_node to notify the master node Master to publish the binocular image topic image_raw.
[0026] 4) Set the sending rate and repeat steps (2) to (3) to continuously send the acquired binocular image messages.
[0027] 5) Initialize the binocular image segmentation node Camera_splitter and notify the master node Master to subscribe to the binocular image topic image_raw. The topic image_raw has both publishers and subscribers. The master node Master notifies the sending node Camera_node and the binocular image segmentation node Camera_splitter of each other's existence so that image messages can be transmitted to each other.
[0028] 6) If the binocular image segmentation node Camera_splitter receives the image message sent by the sending node Camera_node, the callback function imageCallback is triggered to preprocess the image and perform binocular image segmentation.
[0029] 7) The callback function imageCallback receives the image message parameters and converts the image message read by ROS into the BGR format supported by OpenCV. Using OpenCV combined with the distortion parameters, assembly errors and other internal parameters of the stereo camera to correct and preprocess the image, it helps to eliminate or reduce measurement errors and improve the efficiency of 3D reconstruction. Then, the preprocessed stereo image is split into left and right images, and the left and right images are converted into the Message format supported by ROS. Based on the stereo image segmentation node Camera_splitter, the master node Master is notified to publish the left image topic left_cam and the right image topic right_cam.
[0030] ROS in step 1) is the abbreviation of Robot Operating System, commonly known as "Robot Operating System". However, it is not a real operating system, but an open source meta-operating system for robots, which provides many functions similar to traditional operating systems: hardware abstraction, underlying device control, common function implementation, inter-process messaging, package management, etc. It also provides related tools and libraries for acquiring, compiling, editing codes, and running programs between multiple computers to complete distributed computing. The master node Master refers to the process that uses the robot operating system ROS for calculations, and is responsible for enabling each ROS node to locate each other.
[0031] Node initialization in step 1) is a basic step in the development of the robot operating system ROS, which refers to preparing the ROS environment for subsequent operations and communications.
[0032] The V4L2 protocol in step 2) is called Video for Linux 2, which is the kernel driver interface for handling video devices in Linux systems. It provides a standardized way for user space programs to communicate and interact with video devices (such as cameras, video acquisition cards, etc.).
[0033] The MJPEG in step 2) is a video compression format that compresses the video by encoding each frame of the image using JPEG instead of using inter-frame coding. This compression method makes the encoder implementation simpler, so it has good applicability on some embedded devices and low-end devices.
[0034] The hardware decoder nvv4l2decoder in step 2) is a GStreamer plug-in for video decoding, which supports multiple video formats H.264, H.265, JPEG and MJPEG. This plug-in integrates with the libv4l2 plug-in interface on the Jetson and DGPU platforms to use the hardware decoding engine to process video data. GStreamer is an open source cross-platform multimedia framework that allows developers to connect the various steps of multimedia processing in a pipeline to achieve the desired effect.
[0035] The nvvidconv plugin in step 2) is a GStreamer plugin for data movement and color format conversion between GPU memory and CPU memory.
[0036] The BGRx in step 2) is a color format in which each color component (red, green, blue) occupies 8 bits, for a total of 32 bits, but it cannot be directly read by OpenCV and requires further color format conversion.
[0037] The videoconvert plugin in step 2) is a component of GStreamer, which is used for video format conversion. The videoconvert plugin can convert audio and video data into different color spaces.
[0038] OpenCV in step 2) is an open source computer vision library that provides interfaces in multiple languages, allowing users to implement various computer vision and machine learning algorithms.
[0039] The topic in step 3) is a common communication mechanism in the robot operating system ROS, which is mainly used for message transmission between different nodes. In ROS, topic communication is based on the publish-subscribe model, that is, one node publishes a message and another node subscribes to the message to receive the message. This communication method is asynchronous and suitable for a variety of scenarios, such as using vision, lidar, and inertial navigation systems to obtain data in robot navigation functions. The robot operating system ROS can send or receive information through topics. A node can publish any number of topics. Through the topic subscription and publishing mechanism of the robot operating system ROS, real-time transmission and processing of sensor data can be achieved. This is especially important for application scenarios that require immediate response, such as autonomous driving and robot navigation.
[0040] The message mechanism in step 3) is one of the foundations of ROS communication. Its essence is a data type. In ROS, messages need to be serialized before transmission, that is, the message object is converted into a byte stream for transmission in the network. After the receiver receives the direct stream, it needs to deserialize, that is, convert the byte stream into the original ROS message object.
[0041] The callback function in step 7) is an important method, which is mainly used to handle asynchronous events, such as the arrival of messages and the response of services. In actual applications, ROS nodes receive messages by subscribing to topics and process these messages by defining callback functions. When a new message arrives, ROS automatically triggers the corresponding callback function and passes the received message as a parameter to the callback function for processing. In summary, the callback function in ROS is a key mechanism for realizing asynchronous communication, event processing, and node operation.
[0042] The correction preprocessing in step 7) is an important step in the binocular vision system. It mainly includes two aspects, camera calibration and stereo matching. In the camera calibration stage, the main process is to determine the internal parameters of the camera, including focal length, principal point coordinates, distortion coefficients, etc. Through this process, the camera's internal parameter matrix and distortion matrix can be obtained. Stereo matching is to obtain the relative position relationship between the left and right cameras by determining the corresponding points in the left and right images, that is, the translation vector t and rotation matrix R of the right camera relative to the left camera. Through camera calibration, the aberration and distortion of the imaging system can be corrected to ensure the accuracy of measurement and improve image quality. Stereo matching can be used to locate and correct components to reduce or eliminate errors in the assembly process.
[0043] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, all of which belong to the protection scope of the present invention. The protection scope of the present invention is given by the attached claims and any equivalent technical solutions thereof.
Claims
1. A high-speed binocular image acquisition method based on hardware decoding, It is characterized in that The following steps are involved: 1) Start the master node Master, which is responsible for the connection and message communication between nodes, and initializes the sending node Camera_node; 2) Read the binocular image, and then use the hardware decoder nvv4l2decoder to decode the binocular image and store it in the GPU memory. The nvvidconv plug-in stores the binocular image decoded data in the GPU memory into the system memory and converts the binocular image decoded data into an image with a color format of BGRx. The videoconvert plug-in converts the image with a color format of BGRx into the BGR color format supported by OpenCV. 3) Use OpenCV to read the image converted in step 2), then convert the data read by OpenCV into the Message format supported by ROS, and then send the node Camera_node to notify the master node Master to publish the binocular image topic image_raw; 4) Set the sending rate, repeat steps 2) to 3), and continuously send the acquired binocular image messages; 5) Initialize the binocular image segmentation node Camera_splitter, notify the master node Master to subscribe to the binocular image topic image_raw, the topic image_raw has both publishers and subscribers, the master node Master notifies the sending node Camera_node and the binocular image segmentation node Camera_splitter of each other's existence, so as to transmit image messages to each other; 6) If the binocular image segmentation node Camera_splitter receives the image message sent by the sending node Camera_node, if it does not receive the message, it continues to wait for the message to be sent; 7) If a message is received, the callback function imageCallback is triggered to pre-process the image and perform binocular image segmentation; The callback function imageCallback receives the image message parameters and converts the image message read by ROS into the BGR format supported by OpenCV. It uses OpenCV combined with the distortion parameters and assembly error internal parameters of the stereo camera to perform correction preprocessing on the image. Then, it uses the preprocessed stereo image to split it into left and right images, and converts the left and right images into the Message format supported by ROS. Based on the stereo image splitter node Camera_splitter, it notifies the master node Master to publish the left image topic left_cam and the right image topic right_cam.
2. According to the binocular image high-speed acquisition method based on hardware decoding described in claim 1, It is characterized in that Step 2) The binocular image is read based on the V4L2 protocol.
3. According to the binocular image high-speed acquisition method based on hardware decoding described in claim 1, It is characterized in that Step 2) The format of the binocular image is MJPEG, H.264, H.265, or JPEG.
4. According to the binocular image high-speed acquisition method based on hardware decoding described in claim 1, It is characterized in that Step 2) The hardware decoder nvv4l2decoder is a GStreamer plug-in for video decoding, which uses a hardware encoding engine to process video data.
5. According to the binocular image high-speed acquisition method based on hardware decoding described in claim 1, It is characterized in that Step 7) The distortion parameters and assembly error internal parameters are obtained based on a single-plane chessboard calibration method.
Citation Information
Patent Citations
Distributed visual SLAM method based on unmanned aerial vehicle platform
CN116228853A
Visual identification positioning and mechanical arm motion track planning method based on binocular camera
CN117301065A
Robotic manipulation methods and systems for executing a domain-specific application in an instrumented environment with electronic minimanipulation libraries
US20160059412A1
