A monitoring method and system based on multi-camera pantograph video image stitching

Abstract

The present invention discloses a monitoring method and system for pantograph video image stitching based on multiple cameras, and the method includes the following specific steps: using a hardware synchronization trigger device to synchronously collect images, and performing image synchronization processing on a processing host; extracting feature points from each collected image to obtain a feature point set to be matched for all images; matching feature points, and converting two images to the same coordinate; obtaining images after matching coordinates, matching areas, and coordinate conversion, and copying and stitching the two images in the projection transformation area; fusing the overlapping boundary cracks of the two images, fusing the colors of the overlapping areas, and cutting the invalid areas after the coordinate conversion; repeating image matching, image copying, and image fusion processing operations, and completely stitching multiple images. The present invention captures high-definition images of the pantograph in real time, which is more conducive to monitoring the working status and integrity abnormalities of the pantograph.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a monitoring method and system based on multi-camera pantograph video image stitching. Background Art

[0002] With the rapid development of urban rail transit in my country, the overhead contact network provides power supply for electrified urban rail transit trains. Electric locomotives obtain electricity through the sliding contact between the pantograph slide and the overhead contact network wire. When the moving pantograph passes through the relatively static overhead contact network, the overhead contact network is disturbed by external forces, thus generating dynamic interaction between the two systems of the pantograph and the overhead contact network. A good pantograph-network relationship is the basic prerequisite for ensuring the safety of urban rail transit operations.

[0003] Traditional pantograph-net relationship monitoring uses video monitoring modules installed on engineering vehicles to simulate the pantograph-net matching relationship. The current mainstream pantograph-net relationship monitoring mainly involves installing high-definition cameras and fill-in lighting equipment on electric passenger cars to monitor the pantograph-net contact screen in real time. Compared with traditional engineering vehicle monitoring systems, mainstream electric passenger car monitoring systems have the characteristics of small installation space and high safety requirements. After traditional mainstream monitoring equipment is transplanted to electric passenger cars, due to installation condition restrictions, there are pain points such as reduced resolution to meet the imaging range, as well as incomplete pantograph-net imaging images and severe distortion, which affects the inspection personnel's judgment on whether the pantograph working status is abnormal. Summary of the invention

[0004] The purpose of the present invention is to solve the deficiencies of the prior art and to provide a monitoring method and system based on multi-camera pantograph video image stitching.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] A monitoring method based on multi-camera pantograph video image stitching includes the following specific steps:

[0007] Use hardware synchronization trigger equipment to synchronously collect images and perform synchronous image processing on the processing host;

[0008] Extract feature points from each collected image to obtain a feature point set to be matched for all images;

[0009] Match the feature points and convert the two images to the same coordinates;

[0010] Obtain the image after matching coordinates, matching area and coordinate transformation, and copy and splice the two images in the projection transformation area;

[0011] The overlapping boundary cracks of the two images are fused, the colors of the overlapping areas are fused, and the invalid areas after coordinate transformation are cropped;

[0012] Repeat the image matching, image copying and image fusion processing operations to completely stitch multiple images.

[0013] The image synchronization process comprises the following specific steps:

[0014] Step 1: First determine whether there is new data in each camera cache queue;

[0015] Step 2: If yes, take out the data at the head of each cache queue;

[0016] Step 3: Determine whether the exposure start time Ts of the data taken out from each camera cache queue is consistent and whether the error is within 1ms. If consistent, perform image stitching processing;

[0017] Step 4: If they are inconsistent, the exposure start time Ts among multiple cameras that is closest to the current moment is selected and retained, and the remaining exposure start times that are earlier than the latest exposure start time are discarded;

[0018] Step 5: Take a new set of data from the first discarded queue and return to step 3 to continue.

[0019] Step 6: If there is no data in the queue, wait 40ms and continue with step 1.

[0020] The feature point extraction adopts the SURF algorithm, and the feature points of each image are extracted through the SURF operator in opencv to obtain the feature point set to be matched for all images.

[0021] The feature points are matched using the FLANN matching method.

[0022] The two images are converted to the same coordinate using the findHomography function in the opencv library.

[0023] A monitoring system based on multi-camera pantograph video image stitching includes a roof acquisition unit and an in-vehicle processing unit; the roof acquisition unit includes multiple groups of high-speed high-definition digital cameras, multiple groups of fill-light equipment and synchronous triggering equipment, and the synchronous triggering equipment is electrically connected to the multiple groups of high-speed high-definition digital cameras and the multiple groups of fill-light equipment respectively; the in-vehicle processing unit includes an electrical control unit and a data processing host, one end of the electrical control unit is connected to the data processing host, and the other end of the electrical control unit is connected to the roof acquisition unit.

[0024] Beneficial effects of the present invention:

[0025] 1. It requires little installation space and can be used for pantograph-net video surveillance in various electric passenger car environments;

[0026] 2. The pantograph imaging coverage is large and there is no blind spot;

[0027] 3. The pantograph imaging of multiple cameras has high pixel and resolution;

[0028] 4. The pantograph image has little distortion and high authenticity. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0030] Figure 1 It is a flowchart of the present invention;

[0031] Figure 2 It is a schematic diagram of the system composition of the present invention;

[0032] Figure 3 It is a schematic diagram of the equipment layout of the present invention;

[0033] In the attached figure: 1-high-definition imaging camera A, 2-high-definition imaging camera B, 3-high-definition imaging camera C, 4-fill light device A, 5-fill light device B, 6-monitoring target. DETAILED DESCRIPTION

[0034] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] A monitoring method based on multi-camera pantograph video image stitching includes multi-camera image synchronous acquisition and multi-image stitching synthesis.

[0037] For synchronous acquisition of images from multiple cameras, in addition to using hardware synchronization triggering equipment to synchronize the exposure of each industrial camera, the time it takes for the images of the cameras that are triggered at the same time to be transmitted to the processing host is also related to the exposure time of each camera, the amount of image data (when designing, try to select cameras with the same resolution size to minimize this difference), and the performance of the transmission channel. In order to ensure that the images of each camera are taken at the same time when stitching images, image synchronization processing is required on the processing host. The main processing methods and processes are as follows:

[0038] 1. Number all cameras: such as Camera1, Camera2, Camera3...;

[0039] 2. Register the exposure start callback function of all cameras. When the camera starts each exposure, it will return a timestamp Ts;

[0040] 3. Register all camera image callback functions. When the camera exposure is completed and the image acquisition is completed, the image data and related image parameters, such as image length, width, image data size Size, exposure time Tb, gain, bit depth, image frame number and other information, will be returned through the image callback function, and the current callback time Te will be recorded at the same time.

[0041] 4. According to the image data size Size returned by the image callback and the transmission bandwidth W (generally 1Gbps) between the industrial camera and the industrial computer, the time Tc required for data transmission can be calculated. That is, Tc=Size / W;

[0042] 5. Based on the camera exposure time Tb and the image transmission time Tc, the total time required for an image from exposure to transmission to the industrial computer can be calculated (of course, the camera's internal data packet time is ignored here, which is processed by the industrial camera's internal hardware and has a very short and stable duration). That is, Ta = Tb + Tc;

[0043] 6. At this time, determine whether the difference Td between the image callback time Te and the current latest exposure start time Ts is equal to or close to the total time Ta of collecting an image. If (Td-Ta) <= 5ms, it is determined that the image data Pic returned by this image callback and the current latest exposure start time are a complete set of image acquisition processes, and the corresponding Ts and Pic are added to the image corresponding to this camera and tied to the end of the queue. If (Td-Ta) > 5ms, it is considered that the image corresponding to the current latest exposure start time Ts is lost (frame loss), and the corresponding Ts and Pic are discarded and not added to the queue.

[0044] At this point, the image cache mechanism for a single industrial camera ends.

[0045] The following is the synchronization processing mechanism for multiple cameras:

[0046] 1. First, determine whether there is new data in each camera cache queue;

[0047] 2. If yes, take out the data at the head of each cache queue;

[0048] 3. Determine whether the exposure start time Ts of the data taken out from each camera cache queue is consistent (within 1ms error). If consistent, perform image stitching processing.

[0049] 4. If they are inconsistent, the exposure start time Ts among multiple cameras that is closest to the current moment is selected and retained, and the remaining exposure start times that are earlier than the latest exposure start time are discarded;

[0050] 5. Take a new set of data from the first discarded queue and return to step 3 to continue execution.

[0051] 6. If there is no data in the queue, wait 40ms and continue to step 1.

[0052] Multi-image stitching is mainly divided into four steps: extracting feature points from each image; matching feature points; image registration and copying; and image fusion processing.

[0053] (1) Image feature point extraction

[0054] The main algorithm used is SURF, which is a robust image recognition and description algorithm. It is a variant of SIFT algorithm. The algorithm steps are roughly the same as SIFT algorithm, but the method used is different and it is more efficient than SIFT algorithm. SURF uses the determinant value of the Hessian matrix as the feature point detection and uses the integral map to accelerate the operation.

[0055] After extracting feature points from each image using the surf operator in OpenCV, we can get the feature point set to be matched for all images.

[0056] (2) Feature point matching

[0057] The feature points of the two images found above are matched using the FLANN matching method to obtain the matching feature points of the two images. FLANN (Fast Approximate Nearest Neighbor Search Library) is the full name of the fast nearest neighbor approximation search function library, which means fast and efficient matching. Feature matching records the feature points (KeyPoint) of the target image and the image to be matched, and constructs a feature quantity (descriptor) based on the feature point set, compares and screens this feature quantity, and finally obtains a mapping set of matching points.

[0058] (3) Image registration and copying

[0059] Image registration is to convert two images to the same coordinates, which is done using the findHomography function in the opencv library. It uses the Homography principle. When two images A and B conform to the same perspective transformation (the process of converting a three-dimensional object in a spatial coordinate system into a two-dimensional image representation), there will be a homography matrix H, and B=A*H. Homography is actually a projection mapping from one plane to another. Through the findHomography function of opencv, we can obtain the homography matrix H. Through the homography matrix, we can obtain the projection transformation area of ​​one image relative to another image. After obtaining the area, the images can be copied and spliced.

[0060] (4) Image fusion

[0061] Since the stitched images are taken by different cameras, there will be slight differences in the imaging lighting color and physical space, so it is necessary to perform image fusion on the overlapping areas of the stitching.

[0062] Image fusion processing mainly involves fusing the boundary cracks of overlapping images, fusing the colors of overlapping areas, and cropping the invalid areas after coordinate transformation. Image fusion processing mainly uses a weighted fusion processing algorithm, that is, the pixels in the overlapping area of ​​the image are added according to a certain weight (a simple average weight method is used for images with little difference, and a linear weighted value method is used according to the value of the X direction of the overlapping area for images with large difference) to synthesize a new image.

[0063] Repeat the above 4 steps to completely stitch multiple images.

[0064] A monitoring system based on multi-camera pantograph video image stitching is mainly divided into two parts: a roof acquisition unit and an in-vehicle processing unit.

[0065] The roof acquisition unit mainly includes multiple sets of high-speed and high-definition digital cameras, fill lighting equipment, and synchronous triggering equipment to monitor the operating status of the pantograph in real time.

[0066] The in-vehicle processing unit is mainly composed of an electrical control unit and a data processing host. The electrical control unit performs electrical control on each device; the data processing host is used to collect and store image data from the roof acquisition unit, and the image data is transmitted to the detection computer via Gigabit Ethernet for analysis, processing, and storage.

[0067] Each high-definition camera on the roof synchronously triggers the fill light device to collect high-definition images of the pantograph. The video image data is transmitted to the data processing host in the car via the gigabit network cable. The data host in the car runs the monitoring software to receive the pantograph images collected by each camera, and uses professional algorithms to process and splice the pantograph images to form a complete pantograph image. Then use the image compression algorithm to save it as a video recording file.

[0068] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements fall within the scope of the present invention to be protected. The scope of the present invention to be protected is defined by the attached claims and their equivalents.

Claims

1. A monitoring method based on multi-camera pantograph video image stitching, It is characterized in that The specific steps include: Use hardware synchronization trigger equipment to synchronously collect images and perform synchronous image processing on the processing host; Extract feature points from each collected image to obtain a feature point set to be matched for all images; Match the feature points and convert the two images to the same coordinates; Obtain the image after matching coordinates, matching area and coordinate transformation, and copy and splice the two images in the projection transformation area; The overlapping boundary cracks of the two images are fused, the colors of the overlapping areas are fused, and the invalid areas after coordinate transformation are cropped; Repeat the image matching, image copying and image fusion processing operations to completely stitch multiple images; The image synchronization process comprises the following specific steps: Step 1: First determine whether there is new data in each camera cache queue; Step 2: If yes, take out the data at the head of each cache queue; Step 3: Determine whether the exposure start time Ts of the data taken out from each camera cache queue is consistent and whether the error is within 1ms. If consistent, perform image stitching processing; Step 4: If they are inconsistent, the exposure start time Ts among multiple cameras that is closest to the current moment is selected and retained, and the remaining exposure start times that are earlier than the latest exposure start time are discarded; Step 5: Take a new set of data from the first discarded queue and return to step 3 to continue execution; Step 6: If there is no data in the queue, wait 40ms and then continue to step 1; The cache mechanism of each camera cache queue includes the following steps: Number all cameras; Register all camera exposure start callback functions, and return a timestamp Ts when each camera exposure starts; Register all camera image callback functions. When the camera exposure is completed and the image acquisition is completed, the image callback function returns the image data and related image parameters, including image length, width, image data size Size, exposure time Tb, gain, bit depth and image frame number information, and records the callback current time Te. According to the image data size Size returned by the image callback and the transmission bandwidth W between the industrial camera and the industrial computer, the time Tc required for data transmission is calculated, which is expressed as: Tc=Size / W; According to the camera exposure time Tb and the image transmission time Tc, the total time required for an image from the start of exposure to the transmission to the industrial computer is calculated, which is expressed as: Ta=Tb+Tc; Determine whether the image callback time Te, the difference Td between the current latest exposure start time Ts, and the total time Ta experienced in acquiring an image are equal to or close to each other; if (Td-Ta)<=5ms, it is determined that the image data Pic returned by this image callback and the current latest exposure start time are a complete set of image acquisition processes, and the corresponding Ts and Pic are added to the image corresponding to this camera and tied to the end of the queue; if (Td-Ta)>5ms, it is considered that the image corresponding to the current latest exposure start time Ts is lost, and the corresponding Ts and Pic are discarded and not added to the queue.

2. A monitoring method based on multi-camera pantograph video image stitching according to claim 1, It is characterized in that The feature point extraction adopts the SURF algorithm, and the feature points of each image are extracted through the SURF operator in opencv to obtain the feature point set to be matched for all images.

3. The monitoring method based on multi-camera pantograph video image stitching according to claim 1, It is characterized in that The feature points are matched using the FLANN matching method.

4. The monitoring method based on multi-camera pantograph video image stitching according to claim 1, It is characterized in that The two images are converted to the same coordinate using the findHomography function in the opencv library.

5. A monitoring system based on multi-camera pantograph video image stitching based on the method according to any one of claims 1 to 4, It is characterized in that It includes a roof acquisition unit and an in-vehicle processing unit; the roof acquisition unit includes multiple groups of high-speed high-definition digital cameras, multiple groups of fill-light equipment and synchronous triggering equipment, and the synchronous triggering equipment is electrically connected to the multiple groups of high-speed high-definition digital cameras and the multiple groups of fill-light equipment respectively; the in-vehicle processing unit includes an electrical control unit and a data processing host, one end of the electrical control unit is connected to the data processing host, and the other end of the electrical control unit is connected to the roof acquisition unit.

Images