Multi-channel video real-time fusion method for industrial intellectualization

Abstract

The invention belongs to the technical field of video fusion, and particularly relates to a multi-channel video real-time fusion method for industrial intellectualization. The method comprises the steps of equipment installation, mask manufacturing, first-frame image processing and real-time video fusion, and the first-frame image processing comprises the steps of obtaining video first-frame images corresponding to cameras and processing the video first-frame images to obtain images to be spliced; calculating the initial position of the image to be spliced in the final fusion image and the size of the final fusion image; according to the initial position and the size of the final fusion image, obtaining a left and right black edge adding matrix of each image; uploading the mask template and the black edge matrix from a CPU (Central Processing Unit) end to a GPU (Graphics Processing Unit) end; in the real-time video fusion step, boundaries are sequentially added to each sequence image at the same moment based on CUDA acceleration and fusion is carried out, and a wide-field seamless fusion panoramic image is generated.

Description

Technical Field

[0001] The invention belongs to the technical field of video fusion, and in particular relates to a multi-channel video real-time fusion method oriented to industrial intelligence. Background Art

[0002] Industrial plants have a harsh production environment, are usually under high temperature and high pressure, and require multi-station collaborative operations. An effective video surveillance system can detect and coordinate each production link in real time to ensure the stable operation of the production area. At present, the monitoring system in industrial plants mainly relies on multiple independent cameras for video acquisition. The output of each camera is independently transmitted to the monitoring center. The overlapping area between videos is large. Monitoring personnel need to observe and analyze multiple video streams at the same time, and cannot effectively perceive the global scene, resulting in low efficiency in personnel information processing and difficulty in making decisions quickly and accurately.

[0003] At present, video fusion technology has made certain progress in the fields of smart transportation and virtual-real integration, but there are still many unresolved industry pain points in the industrial field. Video fusion technology based on traffic and security scenes usually has cameras installed at a high position, with a large depth of field and no stereoscopic parallax, and image fusion can be performed based on a relatively simple matching algorithm. In complex industrial scenes, due to the long-distance installation and wide viewing angle of the camera, the same object in the overlapping area presents different forms under different viewing angles, resulting in significant differences in the relevant descriptors of the same points, making it difficult to accurately match. Secondly, some existing technologies have bottlenecks in their ability to process video streams, making it difficult to meet the real-time processing requirements of high-bandwidth, multi-channel videos. Industrial plants occupy a large area, and usually dozens of cameras are required to obtain surveillance videos without blind spots. However, the existing technology has insufficient performance during processing and cannot achieve real-time and accurate fusion. In addition, some existing solutions need to be improved in terms of their ability to adapt to special scenes such as industrial plants and their scalability.

[0004] Therefore, in order to solve the problems existing in traditional monitoring systems in industrial plants, it is urgent to propose a multi-channel video real-time fusion method for industrial intelligence to meet the monitoring needs of industrial production and improve the overall perception of the monitoring system and the efficiency of personnel supervision. Summary of the invention

[0005] In view of the above technical problems, the present invention proposes a multi-channel video real-time fusion method for industrial intelligence; through real-time video fusion technology, the dispersion, information island and information redundancy problems of traditional video surveillance systems and the problem of large video fusion delay are solved.

[0006] The technical solution adopted by the present invention is:

[0007] A multi-channel video real-time fusion method for industrial intelligence, the method comprising:

[0008] Equipment installation: Fixedly install the surveillance cameras, and control the installation position of each surveillance camera so that the images collected by adjacent cameras have overlapping areas;

[0009] Mask making: Obtain the images of each camera with overlapping areas at any same time, perform image preprocessing, and make a mask template;

[0010] First frame image processing: obtain the first frame image of the video corresponding to each camera, process the first frame image of the video, and obtain the image to be spliced; calculate the starting position of the image to be spliced ​​in the final fused image and the size of the final fused image; obtain the black edge matrix added to the left and right of each image according to the starting position and the size of the final fused image; upload the mask template and the black edge matrix from the CPU to the GPU;

[0011] Real-time video fusion: obtain the video stream of each surveillance camera, scale and mask each video stream frame by frame, and upload it to the GPU to generate an image frame sequence; fuse and sort all the image frame sequences to obtain a set of image sequences to be fused, and the order of each image frame sequence in the set of image sequences to be fused is consistent with the actual installation order of the camera; based on CUDA acceleration, add boundaries to each sequence image at the same time and fuse them in turn, and optimize the stitching seams based on the weighted fade-in and fade-out fusion algorithm to eliminate the stitching seams that exist when stitching based on suture lines, and generate a wide-field seamless fusion panoramic image.

[0012] Furthermore, in the equipment installation step, all monitoring cameras are fixed on the pillars in the same posture, and ensure that the equipment in the target production line is in the center position in each monitoring screen, and the focal length of all monitoring cameras is set to the minimum; and there is at least 1 / 3 overlapping area between adjacent cameras.

[0013] Furthermore, the steps of making the mask are specifically as follows:

[0014] Calculate image scaling parameters: obtain the images of each camera at the same time, select a certain camera as the reference, take the size of the device in the image corresponding to the reference camera as the reference, calculate the scaling ratio of adjacent images based on the overlapping area, and obtain the scaling parameters of all images in sequence;

[0015] Seam line selection: According to the position of the device in the overlapping area, the best seam line is manually selected, and the best seam line is selected in the background without foreground devices; then the retention area and deletion area of ​​each image are determined according to the position of the seam line, the pixel value of the retention area is set to 1, and the pixel value of the deletion area is set to 0, and the initial mask template is obtained;

[0016] The initial mask is modified, and the modified mask template is expressed as:

[0017]

[0018] Among them, I(x,y) represents the value of the mask at (x,y); x, y represent the horizontal and vertical coordinates of the image respectively, I0 represents the deleted area with a pixel value of 0, and x seam represents the abscissa of the point on the optimal suture line, d represents the length of the area to be fused, I1 represents the area after the transition area is removed from the original retained area, that is, I1=I-I0-d, and I represents the entire image.

[0019] Furthermore, the steps of processing the first frame image are specifically as follows:

[0020] (1) based on the real-time streaming protocol (RTSP protocol), sequentially obtaining the first frame image of the video corresponding to each camera, scaling the corresponding image according to the scaling parameter obtained in the step of calculating the image scaling parameter, and directly multiplying the scaled image data with the modified mask to obtain the image to be stitched;

[0021] (2) According to the fusion length L between adjacent images fusion Calculate the starting position of each image in the final fused image and the size of the final fused image in turn, and store them for subsequent splicing calls; where the fusion length L between adjacent images fusion The calculation formula is:

[0022]

[0023] in, and They represent the maximum and minimum values ​​of the horizontal coordinates of the optimal suture line, respectively, and d is the length of the area to be fused;

[0024] (3) obtaining the zero matrix to be added to the left and right of each image, i.e., the black edge matrix, according to the starting position and the size of the final fused image;

[0025] Upload the modified mask template and black edge matrix from the CPU to the GPU.

[0026] Furthermore, the real-time video fusion step is specifically as follows:

[0027] Based on the real-time streaming protocol (RTSP protocol), the video streams of each camera are obtained in parallel, and each video stream is scaled and masked frame by frame based on the modified mask template, and the obtained masked video frame images are uploaded to the GPU to generate multiple image frame sequences corresponding to the camera;

[0028] The i-th image in each image frame sequence has the same corresponding time, i ranges from 1 to n, and n is the total number of images in the image frame sequence;

[0029] All image frame sequences are fused and sorted to obtain a set of image sequences to be fused, and the order of each image frame sequence in the set of image sequences to be fused is consistent with the actual installation order of the cameras;

[0030] Based on CUDA acceleration, boundaries are added to each sequence image at the same time and then fused.

[0031] Furthermore, image fusion uses a weighted fade-in and fade-out fusion algorithm to optimize the stitching seams; the fusion weight calculation formula is as follows:

[0032]

[0033] Among them, k is the fusion weight parameter, ranging from 0 to 1, and w is the fusion weight.

[0034] Furthermore, the i-th image in each image frame sequence has the same corresponding time, i ranges from 1 to n, and n is the total number of images in the image frame sequence;

[0035] An initialization time threshold is set in each video stream capture process to ensure that the first frame image in the image frame sequence of each camera is at the same time, and the images obtained before the threshold are discarded and not included in the image frame sequence.

[0036] Beneficial technical effects of the present invention:

[0037] The existing video fusion technology needs to calculate the feature points of each frame image and perform image conversion and sequential fusion, which is time-consuming. In the present invention, a mask is used to replace the above process. When fusion is performed, the mask matrix and the image matrix are multiplied based on multi-threading to directly obtain the image to be fused; by calculating the starting row and column of each image in the final fused image and the size of the final fused image, a 0 matrix is ​​used to fill each image to the final image size, and fusion is performed based on multi-threaded pixel addition. Compared with sequential fusion, the time consumption is greatly reduced and the real-time performance of the algorithm is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a schematic flow chart of the steps of the video fusion method provided by an embodiment of the present invention;

[0039] Figure 2 It is a practical application effect diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following is a specific implementation of the technical solution of the present invention:

[0041] The following technical problems exist in the existing technology: the existing technology is mainly based on the research and application of low depth of field and no stereoscopic parallax, which is not suitable for industrial scenes where the camera is installed far away and there is stereoscopic parallax; secondly, some existing technologies have bottlenecks in the processing ability of video streams, and it is difficult to meet the real-time processing requirements of high-bandwidth and multi-channel videos; industrial plants cover a large area, and usually dozens of cameras are required to obtain surveillance videos without blind spots, while the existing technology has insufficient performance in processing and cannot achieve real-time and accurate fusion;

[0042] The present invention provides an embodiment of a multi-channel video real-time fusion method for industrial intelligence, the method comprising:

[0043] Equipment installation: Fixedly install the surveillance cameras, and control the installation position of each surveillance camera so that the images collected by adjacent cameras have overlapping areas;

[0044] Mask making: Obtain the images of each camera with overlapping areas at any same time, perform image preprocessing, and make a mask template;

[0045] First frame image processing: obtain the first frame image of the video corresponding to each camera, process the first frame image of the video, and obtain the image to be spliced; calculate the starting position of the image to be spliced ​​in the final fused image and the size of the final fused image; obtain the black edge matrix added to the left and right of each image according to the starting position and the size of the final fused image; upload the mask template and the black edge matrix from the CPU to the GPU;

[0046] Real-time video fusion: obtain the video stream of each surveillance camera, scale and mask each video stream frame by frame, and upload it to the GPU to generate an image frame sequence; fuse and sort all the image frame sequences to obtain a set of image sequences to be fused, and the order of each image frame sequence in the set of image sequences to be fused is consistent with the actual installation order of the camera; based on CUDA acceleration, add boundaries to each sequence image at the same time and fuse them in turn, and optimize the stitching seams based on the weighted fade-in and fade-out fusion algorithm to eliminate the stitching seams that exist when stitching based on suture lines, and generate a wide-field seamless fusion panoramic image.

[0047] In this embodiment, in the equipment installation step, all monitoring cameras are fixed on the pillars in the same posture. Preferably, all monitoring cameras are fixed on the pillars in a posture perpendicular to the ground and tilted downward at 45 degrees. It is ensured that the equipment in the target production line is in the center position in each monitoring screen, and the focal length of all monitoring cameras is set to the minimum. There is at least 1 / 3 overlapping area between adjacent cameras to ensure that the video can fully cover the monitoring area.

[0048] The target production line is placed in the center of each screen and the focal length is set to the minimum, which not only increases the distance between the camera and the target, but also minimizes the impact of the camera's fisheye effect on the screen.

[0049] In this embodiment, the steps of making the mask are specifically as follows:

[0050] Calculate image scaling parameters: obtain images of each camera with overlapping areas at the same time. Specifically, select images with neat pictures and no obstruction by people or other equipment for subsequent processing; select a certain camera as the reference camera, take the size of the equipment in the image corresponding to the reference camera as the reference, calculate the scaling ratio of adjacent images based on the overlapping area, and obtain the scaling parameters of all images in turn;

[0051] Seam line selection: According to the position of the device in the overlapping area, the best seam line is manually selected, and the best seam line is selected in the background without foreground devices; then the retention area and deletion area of ​​each image are determined according to the position of the seam line, the pixel value of the retention area is set to 1, and the pixel value of the deletion area is set to 0, and the initial mask template is obtained;

[0052] The initial mask is modified, and the modified mask template is expressed as:

[0053]

[0054] Among them, I(x,y) represents the value of the mask at (x,y); x, y represent the horizontal and vertical coordinates of the image respectively, I0 represents the deleted area with a pixel value of 0, and x seam represents the abscissa of the point on the optimal suture line, d represents the length of the area to be fused, I1 represents the area after the transition area is removed from the original retained area, that is, I1=I-I0-d, and I represents the entire image.

[0055] In this embodiment, the steps of processing the first frame image are specifically as follows:

[0056] (1) based on the real-time streaming protocol (RTSP protocol), sequentially obtaining the first frame image of the video corresponding to each camera, scaling the corresponding image according to the scaling parameter obtained in the step of calculating the image scaling parameter, and directly multiplying the scaled image data with the modified mask to obtain the image to be stitched;

[0057] (2) According to the fusion length L between adjacent images fusion Calculate the starting position of each image in the final fused image and the size of the final fused image in turn, and store them for subsequent stitching calls;

[0058] Among them, the fusion length L between adjacent images fusion The calculation formula is:

[0059]

[0060] in, and They represent the maximum and minimum values ​​of the horizontal coordinates of the optimal suture line, respectively, and d is the length of the area to be fused;

[0061] (3) To speed up the image fusion process, the number of zero matrix rows and columns to be filled on the left and right sides of each image is further calculated according to the starting position and the size of the final fused image.

[0062] That is, get the black border matrix that should be added to the left and right of each image;

[0063] The modified mask template and black edge matrix are uploaded from the CPU to the GPU for subsequent calculations.

[0064] In this embodiment, the real-time video fusion step is specifically as follows:

[0065] Based on the real-time streaming protocol (RTSP protocol), the video streams of each camera are obtained in parallel, and each video stream is scaled and masked frame by frame based on the modified mask template, and the obtained masked video frame images are uploaded to the GPU to generate multiple image frame sequences corresponding to the camera;

[0066] The i-th image in each image frame sequence has the same corresponding time, i ranges from 1 to n, and n is the total number of images in the image frame sequence;

[0067] All image frame sequences are fused and sorted to obtain a set of image sequences to be fused. The order of each image frame sequence in the set of image sequences to be fused is consistent with the actual installation order of the camera. Since all image data have been uploaded to the GPU, CUDA acceleration is used to add boundaries to each sequence image at the same time and fuse them in turn, which can speed up the calculation process, reduce the calculation time, and generate wide-field seamless fused panoramic images more quickly.

[0068] In this embodiment, in order to obtain a fused image with smoother transition and better visual effect, the stitching seams are optimized based on a weighted fade-in and fade-out fusion algorithm to eliminate the stitching seams existing in stitching based on suture lines, and finally a wide-field seamless fused panoramic image is generated.

[0069] The fusion weight calculation formula is as follows:

[0070]

[0071] Among them, k is the fusion weight parameter, ranging from 0 to 1, and w is the fusion weight.

[0072] In this embodiment, the i-th image in each image frame sequence has the same corresponding time, i ranges from 1 to n, and n is the total number of images in the image frame sequence;

[0073] However, due to network and server performance reasons, when the number of video streams reaches six or more, it is impossible to obtain each video image at the same time. Therefore, it is necessary to set an initialization time threshold in each video stream capture process. In this embodiment, an initialization time threshold is set in each video stream capture process according to device performance and network conditions to ensure that the first frame image in the image frame sequence of each camera is at the same time, and the images obtained before the threshold are discarded and not included in the image frame sequence.

[0074] In the present invention, mask fusion and image addition are used to reduce the amount of calculation, and multi-threading is used based on GPU to perform parallel fusion of images, so as to speed up the calculation process and reduce the time consumption of single frame processing: the existing video fusion technology needs to calculate the feature points of each frame image and perform image conversion and sequential fusion, which is time-consuming. In the present invention, a mask is used to replace the above process, and the mask matrix and the image matrix are multiplied based on multi-threading when fusion is performed to directly obtain the image to be fused; in addition, by calculating the starting rows and columns of each image in the final fused image and the size of the final fused image, a 0 matrix is ​​used to fill each image to the size of the final image, and fusion is performed based on multi-threaded pixel addition. Compared with sequential fusion, the time consumption is greatly reduced and the real-time performance of the algorithm is improved.

[0075] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the embodiments of the technology in a steel plant. According to the on-site conditions, the monitoring cameras are installed on pillars 7.5m above the ground, with a camera spacing of 12m or 15m and an arrangement of about 45°. This ensures that the center line of the production line is centered in each image and that adjacent camera images have at least 1 / 3 overlapping areas. Ultimately, a total of 26 cameras are used to monitor the production line and perform video fusion. Images from each camera at the same time are obtained for preprocessing, and a mask template is made. Two servers on site fuse the left and right side monitoring, respectively, and at the same time correct some tilted images caused by the camera position. Multi-threading and GPU parallel processing are used to fuse frame-by-frame images to obtain fused real-time video and play it with the server desktop, such as Figure 2 The following figure shows the actual application effect. The server and the large screen of the operation center are connected by a high-definition video cable, and the fused image is mapped to the large screen of the operation center with almost no delay, so as to realize the supervision and operation guidance of the on-site staff.

[0076] The method provided by the present invention performs video fusion for scenes with stereoscopic parallax, improves image processing speed based on mask templates and GPU parallel acceleration, and realizes the fusion of up to 13 channels of video in a single server.

[0077] The multi-channel video real-time fusion method provided by the present invention adopts mask fusion and image addition to reduce the amount of calculation, and uses multi-threading based on GPU to perform parallel fusion of images, thereby speeding up the calculation process and reducing the time consumption of single frame processing.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-channel video real-time fusion method for industrial intelligence, characterized in that: The method comprises: Equipment installation: Fixedly install the surveillance cameras, and control the installation position of each surveillance camera so that the images collected by adjacent cameras have overlapping areas; Mask making: Obtain the images of each camera with overlapping areas at any same time, perform image preprocessing, and make a mask template; First frame image processing: obtain the first frame image of the video corresponding to each camera, process the first frame image of the video, and obtain the image to be spliced; calculate the starting position of the image to be spliced ​​in the final fused image and the size of the final fused image; obtain the black edge matrix added to the left and right of each image according to the starting position and the size of the final fused image; upload the mask template and the black edge matrix from the CPU to the GPU; Real-time video fusion: obtain the video stream of each surveillance camera, scale and mask each video stream frame by frame, and upload it to the GPU to generate an image frame sequence; fuse and sort all the image frame sequences to obtain a set of image sequences to be fused, and the order of each image frame sequence in the set of image sequences to be fused is consistent with the actual installation order of the camera; based on CUDA acceleration, add boundaries to each sequence image at the same time and fuse them in turn, and optimize the stitching seams based on the weighted fade-in and fade-out fusion algorithm to eliminate the stitching seams that exist when stitching based on suture lines, and generate a wide-field seamless fusion panoramic image.

2. According to the method of claim 1, the method is characterized in that: In the equipment installation step, all monitoring cameras are fixed on the pillars in the same posture, and ensure that the equipment in the target production line is in the center of each monitoring screen, and the focal length of all monitoring cameras is set to the minimum; and there is at least 1 / 3 overlapping area between adjacent cameras.

3. According to the method of claim 1, the method is characterized in that: The steps of making the mask are specifically as follows: Calculate image scaling parameters: obtain the images of each camera at the same time, select a certain camera as the reference, take the size of the device in the image corresponding to the reference camera as the reference, calculate the scaling ratio of adjacent images based on the overlapping area, and obtain the scaling parameters of all images in sequence; Seam line selection: According to the position of the device in the overlapping area, the best seam line is manually selected, and the best seam line is selected in the background without foreground devices; then the retention area and deletion area of ​​each image are determined according to the position of the seam line, the pixel value of the retention area is set to 1, and the pixel value of the deletion area is set to 0, and the initial mask template is obtained; The initial mask is modified, and the modified mask template is expressed as: Among them, I(x,y) represents the value of the mask at (x,y); x, y represent the horizontal and vertical coordinates of the image respectively, I0 represents the deleted area with a pixel value of 0, and x seam Indicates the The horizontal coordinate of the point on the best suture line, d represents the length of the area to be fused, and I1 represents the length of the original The retained area is the area after removing the transition area, that is, I1 = I-I0-d, I is the entire image picture.

4. According to the method of claim 3, the method is characterized in that: The steps of first frame image processing are specifically as follows: (1) obtaining the first frame image of the video corresponding to each camera in sequence based on the real-time streaming protocol, scaling the corresponding image according to the scaling parameter obtained in the step of calculating the image scaling parameter, and directly multiplying the scaled image data by the modified mask to obtain the image to be stitched; (2) According to the fusion length L between adjacent images fusion Calculate the starting position of each image in the final fused image and the size of the final fused image in turn, and store them for subsequent splicing calls; where the fusion length L between adjacent images fusion The calculation formula is: in, and They represent the maximum and minimum values ​​of the horizontal coordinates of the optimal suture line, respectively, and d is the length of the area to be fused; (3) obtaining the zero matrix to be added to the left and right of each image, i.e., the black edge matrix, according to the starting position and the size of the final fused image; Upload the modified mask template and black edge matrix from the CPU to the GPU.

5. According to the method of claim 4, the method is characterized in that: The real-time video fusion step is specifically as follows: The video streams of each camera are obtained in parallel based on the real-time streaming protocol, and each video stream is scaled and masked frame by frame based on the modified mask template, and the obtained masked video frame images are uploaded to the GPU to generate multiple image frame sequences corresponding to the camera; The i-th image in each image frame sequence has the same corresponding time, i ranges from 1 to n, and n is the total number of images in the image frame sequence; All image frame sequences are fused and sorted to obtain a set of image sequences to be fused, and the order of each image frame sequence in the set of image sequences to be fused is consistent with the actual installation order of the cameras; Based on CUDA acceleration, boundaries are added to each sequence image at the same time and then fused.

6. The method for real-time fusion of multi-channel videos for industrial intelligence according to claim 5 is characterized in that: Image fusion uses a weighted fade-in and fade-out fusion algorithm to optimize the stitching seams; The fusion weight calculation formula is as follows: Among them, k is the fusion weight parameter, ranging from 0 to 1, and w is the fusion weight.

7. The method for real-time fusion of multi-channel videos for industrial intelligence according to claim 5 is characterized in that: The i-th image in each image frame sequence has the same corresponding time, i ranges from 1 to n, and n is the total number of images in the image frame sequence; An initialization time threshold is set in each video stream capture process to ensure that the first frame image in the image frame sequence of each camera is at the same time, and the images obtained before the threshold are discarded and not included in the image frame sequence.

Images