Method and device for fusion of panoramic images of newly added equipment in multi-channel surveillance videos

Abstract

A method and device for panoramic image fusion of newly added equipment images of multi-channel monitoring videos, the method comprising the following steps: 1) after multiple channels of original monitoring images are connected, the single-point multi-channel images are spliced ​​and fused into an overall image as the main image of the reference system; 2) after pre-processing the newly added equipment monitoring image, grid block division is performed; 3) the newly added equipment monitoring image processed by grid block is compared with the block blocks of the monitoring image base map one by one by likelihood, and a block pair indexed by the maximum likelihood is formed; 4) after taking out the block pair with the largest likelihood, feature point matching is performed, and the homography matrix of the block matching pair is calculated to obtain the block position relationship; 5) the homography matrix is ​​verified and calculated; 6) by calculating the optimized homography matrix, the newly added equipment monitoring image and the main image are spliced ​​and fused. The panoramic image fusion technology of the present invention does not require manual intervention, reduces the difficulty of use, and is conducive to implementation and application promotion.

Description

Technical Field

[0001] The present invention relates to video image processing technology, and in particular to a method and device for fusing panoramic images of newly added equipment of multi-channel monitoring videos. Background Art

[0002] The scenes at construction sites are complex and changeable, which will lead to frequent changes in the location of monitoring, such as the need to add or shift. The above factors lead to the fact that during the use of global monitoring, due to the addition and shift of the monitoring location, the monitoring screen used is quite different from the original screen, and the size of the new monitoring screen is quite different from the original screen, which requires frequent human intervention for adjustment and management. There are a large number of identical areas and highly redundant information in the monitoring screen, and the bandwidth pressure of on-site monitoring image transmission is high; the construction site is complex, the adjustment time of monitoring equipment is not fixed, and professional image technicians are used for irregular maintenance, resulting in high overall use costs, which is not conducive to the application and promotion in the fields of digital construction and video security remote implementation.

[0003] Therefore, how to efficiently and scientifically manage the newly added multi-channel surveillance videos and adaptively process the adjusted images is an urgent problem to be solved. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method and device for panoramic picture fusion of newly added equipment images of multi-channel monitoring videos in view of the defects in the prior art.

[0005] The technical solution adopted by the present invention to solve the technical problem is: a method for fusion of panoramic images of newly added equipment of multi-channel monitoring videos, comprising the following steps:

[0006] 1) After multiple channels of original surveillance images are connected, the multiple channels of images at a single point are spliced ​​and fused into an overall image as the main image of the reference system; the main image is processed into grid blocks;

[0007] 2) taking one of the newly added equipment monitoring screens to be spliced, pre-processing the newly added equipment monitoring screen, and then dividing it into grid blocks;

[0008] 3) Taking the main screen as the monitoring screen base map, the newly added equipment monitoring screen processed by gridding and block processing is compared with the block blocks of the monitoring screen base map one by one to form a block pair indexed by likelihood;

[0009] 4) Take out the tile pair with the largest likelihood in the likelihood index to perform feature point matching, calculate the homography matrix of the tile matching pair, and obtain the positional relationship between the tile pairs;

[0010] 5) Verify and calculate the homography matrix of the tile matching pair;

[0011] 6) By verifying the calculated homography matrix, the newly added device monitoring screen is stitched and fused with the main screen. The newly added monitoring screen adaptively fused screen is used as the updated main screen for the next new device adaptive management and panoramic screen fusion.

[0012] According to the above scheme, in step 2), the monitoring screen of the newly added equipment is preprocessed, including image distortion correction and image viewing angle adjustment.

[0013] According to the above scheme, in step 5), the homography matrix of the block matching pair is verified and calculated, and the specific steps are as follows:

[0014] If there are two or more high-likelihood tile pairs and they are adjacent to the tile pair with the highest likelihood, first combine the adjacent tiles into a single tile, and then calculate the homography matrix of the combined tile matching pair; if the calculation result is consistent with that in step 4), the verification is passed;

[0015] The high-likelihood block pair is a block pair whose difference with the likelihood index with the maximum likelihood is within a set range, and the set range is generally 1%;

[0016] Otherwise, the tile pair with the largest likelihood is re-gridded into two or more tiles, and then the homography matrix of each tile matching pair is calculated; if the calculation results of each tile matching pair are consistent with those in step 4), the verification is passed;

[0017] If the verification fails, the splicing is abandoned and the newly added device monitoring screen is put into the processing queue.

[0018] A panoramic picture fusion device for adding equipment images of multiple monitoring videos, comprising:

[0019] The main picture processing module is used to stitch and fuse the multiple original surveillance images at a single point into an overall picture as the main picture of the reference system after multiple channels of original surveillance images are connected; the main picture is processed into grids and blocks;

[0020] A new equipment monitoring screen processing module is used to take out one of the new equipment monitoring screens to be spliced, pre-process the new equipment monitoring screen, and then divide it into grid blocks;

[0021] The splicing analysis module is used to determine the splicing position relationship between the main screen and the newly added device monitoring screen, as follows:

[0022] 1) Taking the main screen as the monitoring screen base map, the newly added equipment monitoring screen processed by gridding and block processing is compared with the block blocks of the monitoring screen base map one by one to form a block pair indexed by likelihood;

[0023] 2) Take out the tile pair with the largest likelihood in the likelihood index to perform feature point matching, calculate the homography matrix of the tile matching pair, and obtain the positional relationship between the tile pairs;

[0024] 3) Verify and calculate the homography matrix of the tile matching pair;

[0025] The stitching module is used to stitch and fuse the newly added device monitoring screen with the main screen by verifying the calculated homography matrix. The newly added monitoring screen adaptively fused screen is used as the updated main screen for the next new device adaptive management and panoramic screen fusion.

[0026] According to the above scheme, the newly added device monitoring image processing module pre-processes the newly added device monitoring image, including image distortion correction and image viewing angle adjustment.

[0027] According to the above scheme, in the splicing analysis module, the homography matrix of the tile matching pair is verified and calculated, and the specific steps are as follows:

[0028] If there are two or more high-likelihood tile pairs and they are adjacent to the tile pair with the highest likelihood, first combine the adjacent tiles into a single tile, and then calculate the homography matrix of the combined tile matching pair; if the calculation result is consistent with the homography matrix in step 4), the verification is passed;

[0029] The high-likelihood block pair is a block pair whose likelihood difference with the block pair with the highest likelihood in the likelihood index is within a set range;

[0030] Otherwise, the tile pair with the largest likelihood is re-gridded into two or more tiles, and the homography matrix of each tile matching pair is calculated; if the calculation results of each tile matching pair are consistent with the homography matrix in step 4), the verification is passed;

[0031] If the verification fails, the splicing is abandoned and the newly added device monitoring screen is put into the queue for splicing processing.

[0032] According to the above scheme, in the splicing analysis module, in the verification calculation, whether the calculation result is consistent with the homography matrix in step 4) is determined by the rotation relationship judgment threshold and the displacement relationship judgment threshold of the corresponding position relationship.

[0033] The beneficial effects produced by the present invention are:

[0034] The present invention eliminates a large number of identical areas and highly redundant information in multi-channel monitoring images, thereby reducing the pressure on the transmission bandwidth of on-site monitoring images; it meets the needs of complex construction sites and irregular adjustment time of monitoring equipment, and newly added equipment adaptive management and panoramic image fusion technology do not require manual intervention, thereby reducing manual maintenance and difficulty of use, which is conducive to implementation and promotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0036] Figure 1 is a method flow chart of an embodiment of the present invention;

[0037] Figure 2 Schematic diagram of image gridding, block processing and matching according to an embodiment of the present invention;

[0038] Figure 3 It is a schematic diagram of adaptive fusion of newly added surveillance images in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0040] like Figure 1 As shown, a method for fusion of panoramic images of newly added equipment of multi-channel surveillance videos includes the following steps:

[0041] 1) After multiple channels of original surveillance images are connected, the multiple channels of images at a single point are spliced ​​and fused into an overall image as the main image of the reference system; the main image is processed into grid blocks;

[0042] In this embodiment, a single-point multi-channel image stitching is set to increase the efficiency of stitching;

[0043] like Figure 2 The single-point multi-channel image splicing fusion picture (main picture) shown in the figure uses numbers (1-28) to represent the content of each block of the grid-based block processing;

[0044] 2) After pre-processing the monitoring screen of the newly added equipment, it is divided into grid blocks;

[0045] like Figure 2 As shown in , the newly added device monitoring screen (1) uses letters and numbers (A, B, 4`, 5`) to represent the content of each block of the grid-based block processing, and the newly added device monitoring screen (2) uses letters and numbers (27`, 26`, E, F) to represent the content of each block of the grid-based block processing.

[0046] 3) Taking the main screen as the monitoring screen base map, the newly added equipment monitoring screen processed by gridding and block processing is compared with the block blocks of the monitoring screen base map one by one to form a block pair indexed by the maximum likelihood;

[0047] If the maximum likelihood value in the maximum likelihood index is less than 50%, the splicing is abandoned and the newly added device monitoring screen is put into the queue to be processed;

[0048] 4) Take out the block pair with the highest likelihood in the maximum likelihood index to perform feature point matching, calculate the homography matrix of the block matching pair, and obtain the block position relationship;

[0049] like Figure 2 As shown, the single-point multi-channel images are spliced ​​and fused into the monitoring screen base map, and grid processing is performed to divide them into 1-28, a total of 28 blocks. The newly added equipment monitoring screen (1) is grid-processed and divided into A, B, 4', 5', a total of 4 blocks.

[0050] The 4 blocks of the newly added equipment monitoring screen (1) and the 28 blocks of the single-point multi-channel screen splicing and fusion screen are compared with each other using the SIFT (Scale Invariant Feature Transform) algorithm to extract the feature point group of each block and find the block pair with the maximum likelihood index.

[0051] 4' and 5' of the newly added equipment monitoring screen (1) are the block pairs with 4 and 5 of the single-point multi-channel screen splicing and fusion screen respectively, which are the maximum likelihood index blocks.

[0052] In the block pair with the largest likelihood, feature point matching is performed between 4' and 4, and between 5' and 5, and the homography matrices of the block matching pairs are calculated as H 4`4 , H 5`5。 Knowing the internal parameter matrices K1 and K2 of the two monitoring hardware, the position relationship between the image block pairs is obtained as [R 4`4 , T 4`4 ] and [R 5`5 , T 5`5 ]. 4` and 4, 5` and 5 are adjacent blocks, and the monitoring screen is far away from the device. [R 4`4 , T 4`4 ]≈[R 5`5 , T 5`5 ].

[0053] 5) Verify and calculate the homography matrix of the tile matching pair;

[0054] In step 5), the homography matrix of the tile matching pair is verified and calculated. The specific steps are as follows:

[0055] If there are two or more high-likelihood tile pairs and they are adjacent to the tile pair with the highest likelihood, first combine the adjacent tiles into a single tile, and then calculate the homography matrix of the combined tile matching pair; if the calculation result is consistent with that in step 4), the verification is passed;

[0056] The high-likelihood block pair is a block pair whose difference with the likelihood index with the maximum likelihood is within a set range, and the set range is generally 1%;

[0057] Otherwise, a smaller grid will be used to re-grid the tile pair with the largest likelihood, dividing it into two or more tiles, and then the homography matrix of each tile matching pair will be calculated; if the calculation results of each tile matching pair are consistent with those in step 4), the verification is passed;

[0058] If the verification fails, the splicing is abandoned and the newly added device monitoring screen is put into the queue for processing;

[0059] like Figure 2 As shown, the single-point multi-channel images are spliced ​​and fused into the monitoring screen base map, and grid processing is performed to divide them into 1-28, a total of 28 blocks. The newly added equipment monitoring screen (1) is grid-processed and divided into A, B, 4', 5', a total of 4 blocks.

[0060] The 4 blocks of the newly added equipment monitoring screen (1) and the 28 blocks of the single-point multi-channel screen splicing and fusion screen are compared with each other using the SIFT (Scale Invariant Feature Transform) algorithm to extract the feature point group of each block and find the block pair with the maximum likelihood index.

[0061] 4' and 5' of the newly added equipment monitoring screen (1) are the block pairs with 4 and 5 of the single-point multi-channel screen splicing and fusion screen respectively, which are the maximum likelihood index blocks.

[0062] In the block pair with the largest likelihood, feature point matching is performed between 4' and 4, and between 5' and 5, and the homography matrices of the block matching pairs are calculated as H 4`4 , H 5`5。 Knowing the internal parameter matrices K1 and K2 of the two monitoring hardware, the position relationship between the image block pairs is obtained as [R 4`4 , T 4`4 ] and [R 5`5 , T 5`5 ]. 4` and 4, 5` and 5 are adjacent blocks, and the monitoring screen is far away from the device. [R 4`4 , T 4`4 ]≈[R 5`5 , T 5`5 ].

[0063] Combine the two adjacent blocks 4` and 5` into a single block 4`5` and combine the two adjacent blocks 4 and 5 into a single block 45 for feature point matching. Calculate the homography matrix of the block matching pair as H 4`5`45。Given the internal parameter matrices K1 and K2 of the two monitoring hardware, the positional relationship between the newly added device monitoring screen (1) and the single-point multi-channel screen splicing and fusion screen is obtained as [R 4`5`45 , T 4`5`45 ].

[0064] Since the block composed of the two adjacent blocks 4' and 5' is large and the calculation takes a long time, we use [R 4`4 , T 4`4 ] or [R 5`5 , T 5`5 ] is brought into the calculation as the preset value to ensure rapid convergence, improve efficiency and achieve the verification purpose.

[0065] By verifying the calculated homography matrix, the newly added device monitoring screen (1) is fused with the single-point multi-channel screen splicing and fusion screen, and the newly added monitoring screen adaptive fusion screen is used as the updated main screen for the next new device adaptive management and panoramic screen fusion.

[0066] The agreement of the calculation results can be judged in the following ways:

[0067] 4' and 4, 5' and 5 are adjacent blocks. Since the monitoring screen is generally far away from the device, [R 4`4 , T 4`4 ]≈[R 5`5 , T 5`5 ].

[0068] Rotation relationship judgment threshold t R =|(R 4`4 -R 5`5 ) / R 5`5 |<0.01.

[0069] Displacement relationship judgment threshold t T =|(T 4`4 -T 5`5 ) / T 5`5 |<0.01.

[0070] Position relationship 4`5`45 , T 4`5`45 ] and [R 4`4 , T 4`4 ] or [R 5`5 , T 5`5 ] also meets the above-mentioned rotation relationship judgment threshold and displacement relationship judgment threshold;

[0071] Similarly, by quickly and adaptively merging the newly added device monitoring screen (2), a schematic diagram of the adaptive fusion of the newly added monitoring image can be obtained.

[0072] 6) By calculating the optimized homography matrix (block position relationship), the newly added device monitoring screen and the main screen are spliced ​​and fused. When splicing images, they are spliced ​​according to the image position relationship; splicing and fusion of the newly added device monitoring screen (1), the newly added device monitoring screen (2) and the single-point multi-channel image splicing fusion screen (main screen) will reduce the repeated redundant information screen, such as Figure 3 As shown, a new monitoring screen adaptive fusion screen is added;

[0073] The newly added monitoring screen adaptive fusion screen is used as the updated main screen for the next new device adaptive management and panoramic screen fusion.

[0074] The present invention can effectively solve the problems that have always existed in traditional construction site monitoring systems, such as scattered video monitoring images, numerous and discontinuous systems, and lack of overall spatial sense. At the same time, it can eliminate a large number of identical areas and highly redundant information in multi-channel monitoring images, reducing the pressure on the bandwidth of on-site monitoring image transmission. It can meet the needs of complex construction sites and irregular adjustment time of monitoring equipment. New equipment adaptive management and panoramic image fusion technology do not require manual intervention, reduce manual maintenance, and reduce the difficulty of use, which is conducive to the implementation and promotion of applications.

[0075] According to the above method, the present invention also provides a panoramic picture fusion device for newly added equipment images of multi-channel monitoring videos, comprising:

[0076] The main picture processing module is used to stitch and fuse the multiple original surveillance images at a single point into an overall picture as the main picture of the reference system after multiple channels of original surveillance images are connected; the main picture is processed into grids and blocks;

[0077] A new device monitoring screen processing module is used to take out one of the new device monitoring screens to be spliced, pre-process the new device monitoring screen, and then divide it into grid blocks; the pre-processing includes image distortion correction and image viewing angle adjustment;

[0078] The splicing analysis module is used to determine the splicing position relationship between the main screen and the newly added device monitoring screen, as follows:

[0079] 1) Taking the main screen as the monitoring screen base map, the newly added equipment monitoring screen processed by gridding and block processing is compared with the block blocks of the monitoring screen base map one by one to form a block pair indexed by likelihood;

[0080] If the maximum likelihood value in the maximum likelihood index is less than 50%, the splicing is abandoned and the newly added device monitoring screen is put into the queue to be processed;

[0081] 2) Take out the tile pair with the largest likelihood in the likelihood index to perform feature point matching, calculate the homography matrix of the tile matching pair, and obtain the positional relationship between the tile pairs;

[0082] 3) Verify and calculate the homography matrix of the tile matching pair;

[0083] Verify the homography matrix of the tile matching pair. The specific steps are as follows:

[0084] If there are two or more high-likelihood tile pairs and they are adjacent to the tile pair with the highest likelihood, first combine the adjacent tiles into a single tile, and then calculate the homography matrix of the combined tile matching pair; if the calculation result is consistent with the homography matrix in step 4), the verification is passed;

[0085] The high-likelihood block pair is a block pair whose likelihood difference with the block pair with the highest likelihood in the likelihood index is within a set range;

[0086] Otherwise, the tile pair with the largest likelihood is re-gridded into two or more tiles, and the homography matrix of each tile matching pair is calculated; if the calculation results of each tile matching pair are consistent with the homography matrix in step 4), the verification is passed;

[0087] If the verification fails, the splicing is abandoned and the newly added device monitoring screen is put into the queue for splicing processing.

[0088] The stitching module is used to stitch and fuse the newly added device monitoring screen with the main screen by verifying the calculated homography matrix. When stitching images, the stitching is performed according to the image position relationship; the newly added monitoring screen adaptively fused screen is used as the updated main screen for the next new device adaptive management and panoramic screen fusion.

[0089] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A method for fusion of panoramic images of newly added equipment in multi-channel surveillance videos, characterized in that: The following steps are involved: 1) After multiple channels of original surveillance images are connected, the multiple channels of images at a single point are spliced ​​and fused into an overall picture as the main picture of the reference system; the main picture is processed into image grids and blocks; 2) taking one of the newly added equipment monitoring screens to be spliced, pre-processing the newly added equipment monitoring screen, and then dividing it into grid blocks; 3) Taking the main screen as the monitoring screen base map, the newly added equipment monitoring screen processed by gridding and block processing is compared with the block blocks of the monitoring screen base map one by one to form a block pair indexed by likelihood; 4) Take out the tile pair with the largest likelihood in the likelihood index to perform feature point matching, calculate the homography matrix of the tile matching pair, and obtain the positional relationship between the tile pairs; 5) Verify and calculate the homography matrix of the tile matching pair; the specific steps are as follows: If there are two or more high-likelihood tile pairs and they are adjacent to the tile pair with the highest likelihood, first combine the adjacent tiles into a single tile, and then calculate the homography matrix of the combined tile matching pair; if the calculation result is consistent with the homography matrix in step 4), the verification is passed; The high-likelihood block pair is a block pair whose likelihood difference with the block pair with the highest likelihood in the likelihood index is within a set range; Otherwise, the tile pair with the largest likelihood is re-gridded into two or more tiles, and the homography matrix of each tile matching pair is calculated; if the calculation results of each tile matching pair are consistent with the homography matrix in step 4), the verification is passed; If the verification fails, the splicing is abandoned and the newly added device monitoring screen is put into the queue for splicing processing; 6) By verifying the calculated homography matrix, the newly added device monitoring screen is stitched and fused with the main screen. The newly added monitoring screen adaptively fused screen is used as the updated main screen for the next new device adaptive management and panoramic screen fusion.

2. The method for fusion of panoramic images of newly added equipment of multi-channel surveillance videos according to claim 1 is characterized in that: In the step 2), the newly added device monitoring screen is preprocessed, including image distortion correction and image viewing angle adjustment.

3. The method for fusion of panoramic images of newly added equipment of multi-channel surveillance videos according to claim 1 is characterized in that: In the step 5), in the verification calculation, whether the calculation result is consistent with the homography matrix in step 4) is judged by the rotation relationship judgment threshold and the displacement relationship judgment threshold of the corresponding position relationship, wherein the rotation relationship is the difference between the rotation matrices of the block pair, and the displacement relationship is the difference between the translation vectors of the block pair. When the rotation relationship judgment threshold and the displacement relationship judgment threshold are both less than the set value, it is judged that the calculation result is consistent with the homography matrix in step 4).

4. A device for fusion of panoramic images of newly added equipment of multi-channel monitoring videos, characterized in that: include: The main picture processing module is used to stitch and fuse multiple channels of original surveillance images into a whole picture as the main picture of the reference system after multiple channels of original surveillance images are connected; Perform image gridding and block processing on the main screen; A new equipment monitoring screen processing module is used to take out one of the new equipment monitoring screens to be spliced, pre-process the new equipment monitoring screen, and then divide it into grid blocks; The splicing analysis module is used to determine the splicing position relationship between the main screen and the newly added device monitoring screen, as follows: 1) Taking the main screen as the monitoring screen base map, the newly added equipment monitoring screen processed by gridding and block processing is compared with the block blocks of the monitoring screen base map one by one to form a block pair indexed by likelihood; 2) Take out the tile pair with the largest likelihood in the likelihood index to perform feature point matching, calculate the homography matrix of the tile matching pair, and obtain the positional relationship between the tile pairs; 3) Verify and calculate the homography matrix of the tile matching pair; the specific steps are as follows: If there are two or more high-likelihood tile pairs and they are adjacent to the tile pair with the highest likelihood, first combine the adjacent tiles into a single tile, and then calculate the homography matrix of the combined tile matching pair; if the calculation result is consistent with the homography matrix in step 2), the verification is passed; The high-likelihood block pair is a block pair whose likelihood difference with the block pair with the highest likelihood in the likelihood index is within a set range; Otherwise, the tile pair with the largest likelihood is re-gridded into two or more tiles, and the homography matrix of each tile matching pair is calculated; if the calculation results of each tile matching pair are consistent with the homography matrix in step 2), the verification is passed; If the verification fails, the splicing is abandoned and the newly added device monitoring screen is put into the queue for splicing processing; The stitching module is used to stitch and fuse the newly added device monitoring screen with the main screen by verifying the calculated homography matrix. The newly added monitoring screen adaptively fused screen is used as the updated main screen for the next new device adaptive management and panoramic screen fusion.

5. The device for merging panoramic images of newly added equipment of multi-channel surveillance videos according to claim 4 is characterized in that: The newly added device monitoring picture processing module pre-processes the newly added device monitoring picture, including image distortion correction and image viewing angle adjustment.

6. The device for merging new equipment images of multi-channel surveillance videos according to claim 4 is characterized in that: In the stitching analysis module, in the verification calculation, whether the calculation result is consistent with the homography matrix in step 2) is judged by the rotation relationship judgment threshold and the displacement relationship judgment threshold of the corresponding position relationship, wherein the rotation relationship is the difference between the rotation matrices of the tile pair, and the displacement relationship is the difference between the translation vectors of the tile pair. When the rotation relationship judgment threshold and the displacement relationship judgment threshold are both less than the set value, it is judged that the calculation result is consistent with the homography matrix in step 2).

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of a method for panoramic picture fusion of newly added device images of multiple surveillance videos as described in any one of claims 1 to 3 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, a method for panoramic image fusion of newly added equipment in multi-channel surveillance videos as described in any one of claims 1 to 3 is implemented.

Images