A method and device for fast fusion of multi-view surveillance videos

Abstract

The present invention discloses a method and device for rapid fusion of multi-view monitoring videos, which belongs to the technical fields of digital construction, video security supervision, etc. It is realized based on a rapid installation mechanism, and is based on the location planning of high-altitude installation nodes with monitoring on the general plan of the engineering project, so that multi-channel monitoring equipment can fully cover the target area of ​​the project site; the rapid installation mechanism is installed on the installation node of the project site in a horizontal surround; the image feature points in the adjacent monitoring screens are extracted for analysis, and the homography matrix of the positional relationship between the corresponding monitoring devices is rapidly converged and calculated to obtain the corresponding positions and transformation relationships between the corresponding monitoring devices after the overall installation, and then the multi-angle monitoring video screens are preliminarily spliced, and the spliced ​​images are fused. Through the present invention, the efficient, scientific and rapid installation of numerous monitoring equipment on the construction site can be realized, and the numerous and trivial monitoring screens can be simplified into intelligent global monitoring of the construction site through image fusion.

Description

Technical Field

[0001] The present invention belongs to the technical fields of digital construction, video security supervision, etc., and specifically relates to image fusion processing and rapid installation of monitoring equipment. More specifically, it relates to a method and device for rapid fusion of multi-view monitoring videos. Background Art

[0002] Traditional construction site monitoring systems have always had problems such as fragmented video monitoring images, numerous and discontinuous systems, and lack of overall spatial sense; the installation of monitoring equipment is numerous and trivial, and the locations are scattered; the monitoring layout is not scientifically coordinated, and there are problems such as poor consistency of monitoring images. The above factors make it extremely difficult to digitize multiple fragmented monitoring images into an overview of the overall monitoring image in the later stage.

[0003] The panoramic monitoring equipment currently on the market is expensive, has high requirements for location, and has severe side obstruction during installation; it is difficult to meet the above-mentioned supervision needs in modern construction and production.

[0004] Therefore, how to efficiently, scientifically and quickly set up numerous monitoring devices, improve the consistency of monitoring images, and fuse and process images into an overview of the global image monitoring is an urgent problem to be solved. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention proposes a method and device for rapid fusion of multi-view surveillance videos, which can effectively realize the efficient, scientific and rapid installation of numerous surveillance equipment at the construction site, and simplify the numerous and trivial surveillance images into intelligent global monitoring of the construction site through image fusion.

[0006] To achieve the above object, according to one aspect of the present invention, a method for rapid fusion of multi-view surveillance videos is provided, which is implemented based on a rapid installation mechanism, wherein the rapid installation mechanism is provided with a plurality of surveillance devices fixedly installed, and the rapid installation mechanism is provided with a function of rapidly adjusting the embracing size, ensuring that the axis of the rapid installation mechanism and the axis of the installation node are co-horizontal, and the method comprises:

[0007] Based on the location planning of the high-altitude installation nodes with monitoring equipment on the general plan of the project, multiple monitoring equipment can fully cover the target area of ​​the project site;

[0008] Install the quick installation mechanism in a horizontal surround manner on the installation node of the high vertical surface equipment at the project site;

[0009] The image feature points in adjacent monitoring screens are extracted for analysis, and the homography matrix of the positional relationship between the corresponding monitoring devices in the pre-calibration is retrieved for rapid convergence calculation to obtain the corresponding position and transformation relationship between the corresponding monitoring devices after overall installation. The multi-angle monitoring video screens are then preliminarily spliced ​​together, and the spliced ​​images are fused.

[0010] In some optional implementation schemes, after the spliced ​​images are fused, the method further includes:

[0011] The external parameters of the monitoring equipment are integrated in a clockwise manner to calculate H be , H be With H eb Compare and analyze their multiplication rank H be H eb Is it 1? If the difference between the multiplication rank and 1 is within the preset range, the loop detection is initially completed. If the difference between the multiplication rank and 1 is not within the preset range, the loop detection fails, and the external parameters of the monitoring equipment are recalibrated and H is calculated. be With H eb Whether the multiplication rank is 1, where H be The homography matrices H between the N monitoring devices from 1 to N are multiplied in sequence, H be =H 12 H 23 H 34 H 45 … H N-1N , H eb is the homography matrix H of the two monitoring stations N and 1 N1 ;

[0012] Perform pose correction on all frames, and use bundle adjustment to optimize all poses of images in the loop to eliminate accumulated errors.

[0013] In some optional implementation schemes, before retrieving the homography matrix of the positional relationship between the corresponding monitoring devices in the pre-calibration for fast convergence calculation, the method further includes:

[0014] The external parameters of multiple monitoring devices on the fast installation mechanism are pre-calibrated, and the homography matrix H of the relative position relationship between each monitoring device is calculated.

[0015] In some optional implementations, after correcting all poses of the images in the loop using the bundle adjustment method, the method further includes:

[0016] Contour detection and straight line detection are adopted to maintain the structure of curve and straight line structure constraints, constrain similarity transformation, correct the shape of fused and stitched images, and reduce the final image distortion caused by projection distortion.

[0017] In some optional embodiments, the method further comprises:

[0018] The local edge gaps are optimized through the fusion algorithm, and the global color is unified through color extraction, correction and enhancement.

[0019] According to another aspect of the present invention, a multi-view surveillance video rapid fusion device is provided, comprising: a rapid installation mechanism, the rapid installation mechanism is provided with a plurality of surveillance devices fixedly installed, and the rapid installation mechanism is provided with a rapid adjustment of the embracing size, ensuring that the axis of the rapid installation mechanism and the axis of the installation node are co-horizontal;

[0020] Point layout module, which is used to plan the location of high-altitude installation nodes with monitoring equipment based on the general plan of the project, so that multiple monitoring devices can fully cover the target area of ​​the project site;

[0021] A quick installation mechanism, used for horizontal wrap-around installation on the installation nodes of high vertical surface equipment at the project site;

[0022] The fusion module is used to extract image feature points in adjacent monitoring screens for analysis, and to retrieve the homography matrix of the positional relationship between the corresponding monitoring devices in the pre-calibration for rapid convergence calculation, so as to obtain the corresponding position and transformation relationship between the corresponding monitoring devices after overall installation, and then preliminarily splice the multi-angle monitoring video screens and perform fusion processing on the spliced ​​images.

[0023] In some optional embodiments, the device further comprises:

[0024] The loop optimization module is used to fuse the external parameters of the monitoring equipment in a clockwise manner to calculate H be , H be With H eb Compare and analyze their multiplication rank H be H eb Is it 1? If the difference between the multiplication rank and 1 is within the preset range, the loop detection is initially completed. If the difference between the multiplication rank and 1 is not within the preset range, the loop detection fails, and the external parameters of the monitoring equipment are recalibrated and H is calculated. be With H eb Whether the multiplication rank is 1, where H be The homography matrices H between the N monitoring devices from 1 to N are multiplied in sequence, H be =H 12 H 23 H 34H 45 … H N-1N , H eb is the homography matrix H of the two monitoring stations N and 1 N1 ; Perform pose correction on all frames, and use the bundle adjustment method to correct and optimize all poses of the images in the loop to eliminate the accumulated errors.

[0025] In some optional embodiments, the device further comprises:

[0026] The pre-calibration module is used to perform external parameter pre-calibration on multiple monitoring devices on the fast installation mechanism of the fixed installation, and calculate the homography matrix H of the relative position relationship between each monitoring device.

[0027] In some optional embodiments, the device further comprises:

[0028] The affine correction module is used to perform contour detection and straight line detection, maintain the structure of the curve and straight line structure constraints, constrain the similarity transformation, correct the shape of the fused and spliced ​​image, and reduce the final image distortion caused by projection distortion.

[0029] In some optional embodiments, the device further comprises:

[0030] The uniform color optimization module is used to optimize local edge gaps through a fusion algorithm, and to unify the global color through color extraction, correction and enhancement.

[0031] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0032] Using the rapid installation mechanism as the carrier, combined with the pre-calibration, video fusion and image optimization technology of monitoring equipment, the efficient, scientific and rapid installation of numerous monitoring equipment on the construction site can be achieved. The numerous and trivial monitoring images are simplified through image fusion into intelligent overall monitoring of the construction site. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of a method flow provided by an embodiment of the present invention;

[0034] Figure 2 It is a schematic diagram of equipment installation in a specific embodiment of the present invention;

[0035] Figure 3 It is a schematic diagram of monitoring distribution on a quick installation mechanism in a specific embodiment of the present invention;

[0036] Figure 4 It is the original picture of 8 monitoring devices on the quick installation mechanism in the specific embodiment of the present invention;

[0037] Figure 5 Schematic diagram of the loop correction principle in the present invention, wherein (a) is before loop correction, and (b) is after loop correction;

[0038] Figure 6 It is a schematic diagram of the calculation principle of the external parameters of adjacent monitoring devices in the present invention;

[0039] Figure 7 It is the original monitoring picture in the specific embodiment of the present invention;

[0040] Figure 8 It is a monitoring picture after affine transformation and automatic correction processing in a specific embodiment of the present invention;

[0041] Fig. 9 It is the global monitoring picture of the intelligent construction site after the multi-angle monitoring pictures are integrated in the specific embodiment of the present invention;. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0043] The present invention adopts a combination of advanced image processing technology and a rapid installation mechanism to quickly and efficiently install numerous monitoring devices on the construction site. The numerous and trivial monitoring images are simplified into intelligent global monitoring of the construction site through image fusion. The pre-calibration method of the rapid installation mechanism is adopted to effectively reduce the difficulty of later image processing and improve the quality of image fusion. And with tower cranes, lighting lamp poles, etc. as the center, the surrounding monitoring videos are image-fused to achieve the visualization of the construction site plane, which can better overlook the construction site (such as construction surface, road surface, yard, etc.); realize the simultaneous monitoring and dispatching command of multiple construction work surfaces, road surfaces, yards, etc. within the jurisdiction of the project, eliminating the need for users to switch monitoring screens for inspection, thereby improving monitoring capabilities and work efficiency.

[0044] The present invention relates to the technical fields of image fusion processing, photogrammetry, virtual reality enhancement, video security supervision, intelligent equipment, etc., and discloses a method and device for rapid fusion of multi-view surveillance videos. The method specifically comprises the following steps: (1) equipment layout and coverage planning for high-point surveillance; (2) pre-calibration of surveillance equipment based on a rapid installation mechanism; (3) splicing and fusion of multi-angle surveillance video images; (4) image optimization of surveillance loop images; (5) affine change and automatic correction of image images; and (6) uniform color and removal of splicing seams of edges of multiple groups of videos for fusion optimization.

[0045] The present invention realizes the efficient, scientific and rapid installation of numerous monitoring devices at the construction site, and simplifies the numerous and trivial monitoring images into intelligent global monitoring of the construction site through image fusion. The pre-calibration method using a rapid installation mechanism effectively reduces the difficulty of later image processing and improves the quality of image fusion. By combining advanced image processing technology with a rapid installation mechanism, the synchronous monitoring and dispatching command of multiple construction work surfaces, road surfaces, storage yards, etc. within the jurisdiction of the project can be achieved at a glance, eliminating the need for users to switch monitoring screens for inspection, thereby improving monitoring capabilities and work efficiency. The rapid deployment of monitoring equipment, low cost, and efficient image processing make the global monitoring of intelligent construction sites more reliable and practical.

[0046] Figure 1 The present invention provides a method for rapidly fusing multi-view surveillance videos, which includes:

[0047] 1. Equipment layout and coverage planning for high-point monitoring: Based on the location planning of high-altitude installation nodes with monitoring such as tower cranes, construction elevators, and lighting poles on the general plan of the project, multiple monitoring equipment can fully cover the target areas of the project site, such as construction surface, road surface, storage yard, etc.;

[0048] 2. Pre-calibration of monitoring equipment based on quick installation mechanism;

[0049] 3. Multi-angle surveillance video image stitching and fusion;

[0050] 4. Image optimization of monitoring loop screen;

[0051] 5. Affine changes and automatic correction of image;

[0052] 6. Optimize the fusion processing of uniform color and removing seams of multiple video edges.

[0053] In an embodiment of the present invention, the quick installation mechanism is installed in a horizontal surround manner on high vertical equipment such as tower cranes, construction elevators, and lighting poles; the quick installation mechanism is capable of fixedly installing multiple monitoring devices; the quick installation mechanism is capable of quickly adjusting the surrounding size to ensure that the axis of the quick installation mechanism and the axis of the installation node are co-horizontal.

[0054] like Figure 2 As shown, taking the installation on a tower crane as an example, the vertical node has four sides, such as Figure 3 As shown, 8 2.8mm-12mm focal length 1080P infrared gun-type cameras are installed in equal parts at horizontal coaxial angles to form a panoramic coverage centered on the installation node. Figure 4 This is the original picture of 8 monitoring devices in the specific embodiment of the present invention.

[0055] The pre-calibration of monitoring equipment based on the rapid installation mechanism is used to perform external reference pre-calibration (such as Zhang Zhengyou calibration method, etc.) on multiple monitoring equipment before installing vertical equipment (such as tower cranes, construction elevators, lighting street lamp poles, etc.), and use calibration plates, markers, etc. to calculate the homography matrix H of the relative position relationship between each monitoring device. The rapid installation mechanism adopts the pre-calibration method to effectively reduce the difficulty of post-image processing and improve the quality of image fusion. The pre-calibration method is as follows:

[0056] Puv(u,v,1) is the pixel coordinate in the image, Pw(x w ,y w ,z w , 1) is a point in the world coordinate system, such as Figure 6 As shown in the example of two adjacent sets of monitoring images, K is the intrinsic parameter matrix, R is the rotation matrix, and t is the translation vector. They are written together in matrix form as T, called the extrinsic parameter matrix, which represents the transformation from the world coordinate system to the camera coordinate system.

[0057]

[0058] Let's simplify the middle part by recording it as the M matrix:

[0059]

[0060] For the positions of the two monitoring devices, the homography matrix H can be solved:

[0061]

[0062] Using the above method, the homography matrix of the positional relationship between adjacent monitoring devices is calculated in sequence.

[0063] Among them, the multi-angle monitoring video image stitching and fusion is based on the principle of panoramic image imaging, extracts the image feature points in adjacent monitoring images for analysis, and retrieves the homography matrix H of the positional relationship between the corresponding monitoring devices in the pre-calibration for rapid convergence calculation. The corresponding position and transformation relationship between the corresponding monitoring devices after overall installation are calculated, the multi-angle monitoring video images are preliminarily stitched together, and the stitched images are fused.

[0064] Among them, after the multi-channel monitoring images on the quick installation mechanism are spliced ​​and fused, problems such as misalignment and cracks may occur between the tail and the head of the monitoring image. The image optimization of the loopback image can effectively solve potential problems.

[0065] In the embodiment of the present invention, Figure 5 As shown, Figure 5 (a) is before loop correction, (b) is after loop correction. Loop correction uses loop information to eliminate the accumulated error. Loop correction is mainly divided into two steps:

[0066] 1. First, the external parameters of the monitoring equipment are integrated in a clockwise manner to calculate H be (i.e. H begin-end The homography matrices H between the eight monitoring stations from No. 1 to No. 8 are multiplied in sequence, H be =H 12 H 23 H 34 H 45 H 56 H 67 H 78 ), H be With H eb (i.e. H end-begin is the homography matrix H of the two monitoring stations No. 8 and No. 1 81 ) to compare and analyze their multiplication rank (H be H eb ) is 1. The multiplication rank is approximately equal to 1, then Figure 5 (b) The loop detection is initially completed. If the multiplication rank is not close to 1, the loop detection fails, and the external parameters of the monitoring equipment are recalibrated and H is calculated. be With H eb Whether the multiplication rank is 1.

[0067] Among them, the multiplication rank being approximately equal to 1 can be understood as the difference between the multiplication rank and 1 is within the preset range, and the multiplication rank being not close to 1 can be understood as the difference between the multiplication rank and 1 is not within the preset range. The size of the preset range can be determined according to actual needs.

[0068] 2. Finally, all the frame poses are corrected, and the bundle adjustment (BA) method is used to correct and optimize all the poses of the images in the loop to eliminate the accumulated errors.

[0069] In the embodiment of the present invention, the image screen undergoes affine changes and is automatically corrected, and the image may be distorted during shooting, fusion, and change of viewing angle. Contour detection and straight line detection are used, and the structure of the curve and straight line structure is kept constrained to keep drawing the structure, and similarity transformations are constrained to correct the shape of the spliced ​​image and reduce the distortion of the final image caused by projection distortion.

[0070] Figure 7 It is the original monitoring picture in the specific embodiment of the present invention. Figure 8 It is a monitoring picture after affine transformation and automatic correction processing in a specific embodiment of the present invention.

[0071] Fig. 9 It is the global monitoring picture of the intelligent construction site after the multi-angle monitoring pictures are integrated in the specific embodiment of the present invention.

[0072] In the embodiment of the present invention, the edges of multiple groups of videos are uniformly colored and fused to remove seams. Differences such as color aberration and gaps may appear at the image joints. The optimization of the fusion algorithm can effectively solve the problems of local edge gaps. The global color can be made consistent through color extraction, correction, and enhancement.

[0073] The present invention uses a quick installation device as a carrier, combined with the pre-calibration, video fusion, and image optimization technology of monitoring equipment, to achieve efficient, scientific, and rapid installation of numerous monitoring equipment at the construction site. The numerous and trivial monitoring images are simplified through image fusion to form an intelligent global monitoring of the construction site. And the pre-calibration method using a quick installation mechanism effectively reduces the difficulty of later image processing and improves the quality of image fusion. By combining advanced image processing technology with a quick installation mechanism, multiple construction work surfaces, road surfaces, storage yards, etc. within the jurisdiction of the project can be synchronously monitored and dispatched at a glance, eliminating the need for users to switch monitoring screens for inspection, thereby improving monitoring capabilities and work efficiency. Rapid deployment of monitoring equipment, low cost, and efficient image processing are achieved, making the global monitoring of intelligent construction sites more reliable and practical.

[0074] It should be pointed out that, according to the needs of implementation, the various steps / components described in this application can be split into more steps / components, and two or more steps / components or partial operations of steps / components can be combined into new steps / components to achieve the purpose of the present invention.

[0075] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for fast fusion of multi-view surveillance videos, characterized in that: Based on the implementation of a quick installation mechanism, wherein the quick installation mechanism is provided with a plurality of monitoring devices to be fixedly installed, and the quick installation mechanism is provided with a quick adjustment of the embracing size to ensure that the axis of the quick installation mechanism and the axis of the installation node are co-horizontal, the method comprises: Planning and deploying monitoring equipment based on the location of high-altitude installation nodes with monitoring equipment installed on the general plan of the project, so that multiple monitoring equipment can fully cover the target area of ​​the project site; Install the quick installation mechanism in a horizontal surround manner on the installation node of the high vertical surface equipment at the project site; Extract image feature points from adjacent surveillance images for analysis, and retrieve the homography matrix of the positional relationship between the corresponding surveillance devices in the pre-calibration for rapid convergence calculation, and obtain the corresponding position and transformation relationship between the corresponding surveillance devices after overall installation. Then, preliminarily splice the multi-angle surveillance video images, and perform fusion processing on the spliced ​​images. After fusing the spliced ​​images, the method further includes: The external parameters of the monitoring equipment are integrated in a clockwise manner to calculate H be , H be With H eb Compare and analyze their multiplication rank H be H eb Is it 1? If the difference between the multiplication rank and 1 is within the preset range, the loop detection is initially completed. If the difference between the multiplication rank and 1 is not within the preset range, the loop detection fails, and the external parameters of the monitoring equipment are recalibrated and H is calculated. be With H eb Whether the multiplication rank is 1, where H be The homography matrices H between the N monitoring devices from 1 to N are multiplied in sequence, H be =H 12 H 23 H 34 H 45 … H N-1N , H eb is the homography matrix H of the two monitoring stations N and 1 N1 ; Perform pose correction on all frames, and use bundle adjustment to optimize all poses of images in the loop to eliminate accumulated errors.

2. The method according to claim 1, characterized in that Before retrieving the homography matrix of the positional relationship between the corresponding monitoring devices in the pre-calibration for fast convergence calculation, the method further includes: The external parameters of multiple monitoring devices on the fast installation mechanism are pre-calibrated, and the homography matrix H of the relative position relationship between each monitoring device is calculated.

3. The method according to claim 1, characterized in that After correcting all the positions and postures of the images in the loop using the bundle adjustment method, the method further includes: Contour detection and straight line detection are adopted to maintain the structure of curve and straight line structure constraints, constrain similarity transformation, correct the shape of fused and stitched images, and reduce the final image distortion caused by projection distortion.

4. The method according to claim 3, characterized in that The method further comprises: The local edge gaps are optimized through the fusion algorithm, and the global color is unified through color extraction, correction and enhancement.

5. A multi-view surveillance video fast fusion device, characterized in that: include: The quick installation mechanism can be used to fix multiple monitoring devices, and the quick installation mechanism can quickly adjust the embracing size to ensure that the axis of the quick installation mechanism and the axis of the installation node are co-horizontal; Point layout module, which is used to plan and deploy monitoring equipment based on the location of high-altitude installation nodes with monitoring equipment installed on the general plan of the project, so that multiple monitoring equipment can fully cover the target area of ​​the project site; A quick installation mechanism, used for horizontal wrap-around installation on the installation nodes of high vertical surface equipment at the project site; The fusion module is used to extract image feature points in adjacent monitoring images for analysis, and to retrieve the homography matrix of the positional relationship between the corresponding monitoring devices in the pre-calibration for rapid convergence calculation, to obtain the corresponding position and transformation relationship between the corresponding monitoring devices after overall installation, and then to preliminarily splice the multi-angle monitoring video images and perform fusion processing on the spliced ​​images; The device also includes: The loop optimization module is used to fuse the external parameters of the monitoring equipment in a clockwise manner to calculate H be , H be With H eb Compare and analyze their multiplication rank H be H eb Is it 1? If the difference between the multiplication rank and 1 is within the preset range, the loop detection is initially completed. If the difference between the multiplication rank and 1 is not within the preset range, the loop detection fails, and the external parameters of the monitoring equipment are recalibrated and H is calculated. be With H eb Whether the multiplication rank is 1, where H be The homography matrices H between the N monitoring devices from 1 to N are multiplied in sequence, H be =H 12 H 23 H 34 H 45 … H N-1N , H eb is the homography matrix H of the two monitoring stations N and 1 N1 ; Perform pose correction on all frames, and use the bundle adjustment method to correct and optimize all poses of the images in the loop to eliminate the accumulated errors.

6. The device according to claim 5, characterized in that The device also includes: The pre-calibration module is used to perform external parameter pre-calibration on multiple monitoring devices on the fast installation mechanism of the fixed installation, and calculate the homography matrix H of the relative position relationship between each monitoring device.

7. The device according to claim 5, characterized in that The device also includes: The affine correction module is used to perform contour detection and straight line detection, maintain the structure of the curve and straight line structure constraints, constrain the similarity transformation, correct the shape of the fused and spliced ​​image, and reduce the final image distortion caused by projection distortion.

8. The device according to claim 7, characterized in that The device also includes: The uniform color optimization module is used to optimize local edge gaps through a fusion algorithm, and to unify the global color through color extraction, correction and enhancement.

Images