A multi-path video real-time fusion method for industrial intelligentization

Abstract

The application belongs to the technical field of video fusion, and particularly relates to a multi-path video real-time fusion method for industrial intelligentization. The method comprises the steps of device installation, mask making, first frame image processing and real-time video fusion. The first frame image processing is to obtain the first frame image of each camera corresponding to the video, process the first frame image of the video, and obtain a to-be-stitched image; the starting position of the to-be-stitched image in the final fusion image and the size of the final fusion image are calculated; the left and right black matrixes to be added for each image are obtained according to the starting position and the size of the final fusion image; the mask template and the black matrixes are uploaded from the CPU end to the GPU end; in the real-time video fusion step, the boundary of each sequence image at the same time is added and fused in turn based on CUDA acceleration, so as to generate a wide-view seamless fusion panoramic image.

Description

Technical Field

[0001] The present invention belongs to the technical field of video fusion, and in particular relates to a multi-channel video real-time fusion method oriented to industrial intelligence. Background Art

[0002] Industrial plants face harsh production environments, often exposed to high temperatures and high pressures, and require multi-station collaboration. An effective video surveillance system can monitor and coordinate each production process in real time to ensure stable operation. Currently, industrial plant surveillance systems rely primarily on multiple independent cameras for video capture. Each camera's output is independently transmitted to a monitoring center, resulting in significant overlap between videos. Monitoring personnel must simultaneously observe and analyze multiple video streams, hindering their ability to effectively perceive the overall situation. This results in inefficient information processing and makes it difficult for personnel to make quick and accurate decisions.

[0003] Video fusion technology has made significant progress in areas such as smart transportation and virtual-reality integration, but many unresolved pain points remain in the industrial sector. Video fusion technologies for transportation and security scenarios typically use high-mounted cameras with a wide depth of field and no stereoscopic parallax, enabling image fusion based on relatively simple matching algorithms. However, in complex industrial scenarios, the long-range mounting and wide viewing angles of cameras mean that the same object in overlapping areas appears different from different perspectives. This results in significant discrepancies in the descriptors associated with identical points, making accurate matching difficult. Furthermore, some existing technologies face bottlenecks in video stream processing capabilities, making them unable to meet the demands of high-bandwidth, multi-channel video processing in real time. Industrial plants occupy large areas, typically requiring dozens of cameras to capture comprehensive surveillance video. However, existing technologies suffer from insufficient processing performance, making real-time and accurate fusion impossible. Furthermore, some existing solutions need to be improved in terms of adaptability and scalability to these specialized scenarios.

[0004] Therefore, in order to solve the problems existing in traditional monitoring systems in industrial plants, it is urgent to propose a multi-channel video real-time fusion method for industrial intelligence to meet the monitoring needs of industrial production and improve the overall perception of the monitoring system and the efficiency of personnel supervision. Summary of the Invention

[0005] In response to the above technical problems, the present invention proposes a real-time fusion method for multi-channel videos for industrial intelligence; through real-time video fusion technology, the dispersion, information island and information redundancy problems of traditional video surveillance systems and the problem of large video fusion delay are solved.

[0006] The technical solution adopted in the present invention is:

[0007] A multi-channel video real-time fusion method for industrial intelligence, the method comprising:

[0008] Equipment installation: Fixedly install the surveillance cameras and control the installation position of each surveillance camera so that the images collected by adjacent cameras have overlapping areas;

[0009] Mask production: Obtain overlapping images from all cameras at any given moment, perform image preprocessing, and produce a mask template.

[0010] First-frame image processing: Obtain the first-frame image of the video corresponding to each camera, process the first-frame image of the video to obtain the image to be stitched; calculate the starting position of the image to be stitched in the final fused image and the size of the final fused image; obtain the black edge matrix for each image based on the starting position and the size of the final fused image; upload the mask template and the black edge matrix from the CPU to the GPU;

[0011] Real-time video fusion: The video streams of each surveillance camera are obtained, and each video stream is scaled and masked frame by frame, and uploaded to the GPU to generate an image frame sequence. All image frame sequences are fused and sorted to obtain a set of image sequences to be fused. The order of each image frame sequence in the set of image sequences to be fused is consistent with the actual installation order of the cameras. Based on CUDA acceleration, boundaries are added to each sequence image at the same time and fused. The stitching seams are optimized based on the weighted fade-in and fade-out fusion algorithm to eliminate the stitching seams that exist when stitching based on suture lines, and a wide-field seamless fusion panoramic image is generated.

[0012] Furthermore, in the equipment installation step, all surveillance cameras are fixed on the pillars in the same posture, and ensure that the equipment in the target production line is in the center position in each monitoring screen, and the focal length of all surveillance cameras is set to the minimum; and there is at least 1 / 3 overlapping area between adjacent cameras.

[0013] Furthermore, the steps of making the mask are specifically as follows:

[0014] Calculate image scaling parameters: Obtain images from all cameras at the same time, select a camera as a reference, use the size of the device in the image corresponding to the reference camera as a reference, calculate the scaling ratio of adjacent images based on the overlapping area, and obtain the scaling parameters of all images in sequence;

[0015] Seam line selection: Based on the location of the devices in the overlapping area, the optimal seam line is manually selected, and the optimal seam line is selected in the background without foreground devices. Then, based on the location of the seam line, the retained and deleted areas of each image are determined. The pixel values ​​of the retained area are set to 1, and the pixel values ​​of the deleted area are set to 0 to obtain the initial mask template.

[0016] The initial mask is modified, and the modified mask template is expressed as:

[0017]

[0018] Among them, I(x,y) represents the value of the mask at (x,y); x, y represent the horizontal and vertical coordinates of the image respectively, I0 represents the deleted area with a pixel value of 0, and x seam represents the abscissa of the point on the optimal suture line, d represents the length of the area to be fused, I1 represents the area after removing the transition area from the original retained area, that is, I1=I-I0-d, and I is the entire image.

[0019] Furthermore, the steps of processing the first frame image are specifically as follows:

[0020] (1) obtaining the first frame of the video corresponding to each camera in sequence based on the real-time streaming protocol (RTSP protocol), scaling the corresponding image according to the scaling parameter obtained in the step of calculating the image scaling parameter, and directly multiplying the scaled image data with the modified mask to obtain the image to be spliced;

[0021] (2) According to the fusion length L between adjacent images fusion Calculate the starting position of each image in the final fused image and the size of the final fused image in turn, and store them for subsequent splicing calls; among them, the fusion length L between adjacent images fusion The calculation formula is:

[0022]

[0023] in, and They represent the maximum and minimum values ​​of the horizontal coordinate of the optimal suture line, respectively, and d is the length of the area to be fused;

[0024] (3) Obtaining the zero matrix to be added to the left and right sides of each image, i.e., the black edge matrix, according to the starting position and the size of the final fused image;

[0025] Upload the modified mask template and black edge matrix from the CPU to the GPU.

[0026] Furthermore, the real-time video fusion step is specifically as follows:

[0027] The system acquires the video streams of each camera in parallel based on the Real-Time Streaming Protocol (RTSP), scales and masks each video stream frame by frame based on the modified mask template, and uploads the obtained masked video frame images to the GPU to generate multiple image frame sequences corresponding to the camera.

[0028] The i-th image in each image frame sequence has the same corresponding time, i ranges from 1 to n, and n is the total number of images in the image frame sequence;

[0029] All image frame sequences are fused and sorted to obtain a set of image sequences to be fused. The order of each image frame sequence in the set of image sequences to be fused is consistent with the actual installation order of the cameras.

[0030] Based on CUDA acceleration, boundaries are added to each sequence image at the same moment and then fused.

[0031] Furthermore, image fusion uses a weighted fade-in and fade-out fusion algorithm to optimize the stitching seams; the fusion weight calculation formula is as follows:

[0032]

[0033] Among them, k is the fusion weight parameter, ranging from 0 to 1, and w is the fusion weight.

[0034] Furthermore, the i-th image in each image frame sequence corresponds to the same time, i ranges from 1 to n, and n is the total number of images in the image frame sequence;

[0035] An initialization time threshold is set in each video stream capture process to ensure that the first frame image in the image frame sequence of each camera is at the same time, and the images obtained before the threshold are discarded and not included in the image frame sequence.

[0036] Beneficial technical effects of the present invention:

[0037] Existing video fusion technology requires calculating feature points for each frame of image and performing image conversion and sequential fusion, which is time-consuming. However, the present invention uses a mask to replace the above process. When performing fusion, the mask matrix and the image matrix are multiplied based on multi-threading to directly obtain the image to be fused; by calculating the starting rows and columns of each image in the final fused image and the size of the final fused image, using a 0 matrix to fill each image to the final image size, and fusion based on multi-threaded pixel addition, compared with sequential fusion, the time consumption is greatly reduced and the real-time performance of the algorithm is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 1 is a schematic diagram of the steps of the video fusion method provided by an embodiment of the present invention;

[0039] Figure 2 This is a practical application effect diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following are specific implementations of the technical solution of the present invention:

[0041] The following technical problems exist in existing technologies: existing technologies are mainly based on research and application under conditions of low depth of field and no stereo parallax, which is not suitable for industrial scenarios where cameras are installed far away and stereo parallax exists; secondly, some existing technologies have bottlenecks in their video stream processing capabilities, making it difficult to meet the real-time processing requirements of high-bandwidth, multi-channel videos; industrial plants occupy large areas, and dozens of cameras are usually required to obtain comprehensive surveillance videos. However, existing technologies have insufficient processing performance and cannot achieve real-time and accurate fusion;

[0042] The present invention provides an embodiment of a multi-channel video real-time fusion method for industrial intelligence, the method comprising:

[0043] Equipment installation: Fixedly install the surveillance cameras and control the installation position of each surveillance camera so that the images collected by adjacent cameras have overlapping areas;

[0044] Mask production: Obtain overlapping images from all cameras at any given moment, perform image preprocessing, and produce a mask template.

[0045] First-frame image processing: Obtain the first-frame image of the video corresponding to each camera, process the first-frame image of the video to obtain the image to be stitched; calculate the starting position of the image to be stitched in the final fused image and the size of the final fused image; obtain the black edge matrix for each image based on the starting position and the size of the final fused image; upload the mask template and the black edge matrix from the CPU to the GPU;

[0046] Real-time video fusion: The video streams of each surveillance camera are obtained, and each video stream is scaled and masked frame by frame, and uploaded to the GPU to generate an image frame sequence. All image frame sequences are fused and sorted to obtain a set of image sequences to be fused. The order of each image frame sequence in the set of image sequences to be fused is consistent with the actual installation order of the cameras. Based on CUDA acceleration, boundaries are added to each sequence image at the same time and fused. The stitching seams are optimized based on the weighted fade-in and fade-out fusion algorithm to eliminate the stitching seams that exist when stitching based on suture lines, and a wide-field seamless fusion panoramic image is generated.

[0047] In this embodiment, in the equipment installation step, all surveillance cameras are fixed on the pillars in the same posture. Preferably, all surveillance cameras are fixed on the pillars in a posture perpendicular to the ground and tilted downward at 45 degrees. It is ensured that the equipment in the target production line is in the center position in each monitoring screen, and the focal length of all surveillance cameras is set to the minimum. There is at least 1 / 3 overlapping area between adjacent cameras to ensure that the video can fully cover the monitoring area.

[0048] Wherein, the target production line is in the center position in each picture, and the focal length is set to the minimum, which not only increases the distance between the camera and the target, but also minimizes the fisheye effect of the camera on the picture.

[0049] In the embodiment, the step of manufacturing the mask is specifically:

[0050] Calculating image scaling parameters: obtaining images of each camera at the same time with overlapping areas, specifically, selecting images with clean pictures, no personnel, and no other equipment obstruction for subsequent processing; selecting a certain camera as a reference camera, taking the size of the device in the image corresponding to the reference camera as a reference, and calculating the scaling ratio of adjacent images based on the overlapping area, and sequentially obtaining the scaling parameters of all images;

[0051] Suture selection: according to the position of the device in the overlapping area, the best suture is manually selected, and the best suture is selected at the background without foreground device; then the retention area and the deletion area of each image are determined according to the position of the suture, the pixel value of the retention area is set to 1, and the pixel value of the deletion area is set to 0, to obtain an initial mask template;

[0052] The initial mask is modified, and the modified mask template is represented as:

[0053]

[0054] Wherein, I(x,y) represents the value of the mask at (x,y); x and y represent the horizontal and vertical coordinates of the image respectively, I0 represents the deletion area with a pixel value of 0, x seam represents the horizontal coordinate of the point on the best suture, d represents the length of the to-be-fused area, I1 represents the area after removing the transition area from the original retention area, that is, I1=I-I0-d, and I is the entire image.

[0055] In the embodiment, the step of processing the first frame image is specifically:

[0056] (1) Based on the real-time streaming protocol (rtsp protocol), sequentially obtain the first frame image of each camera corresponding video, scale the corresponding image according to the scaling parameters obtained in the step of calculating the image scaling parameters, and multiply the scaled image data directly with the modified mask to obtain the to-be-stitched image;

[0057] (2) According to the fusion length L fusion between adjacent images, sequentially calculate the starting position of each image in the final fusion image and the size of the final fusion image, and store it for subsequent stitching call;

[0058] Wherein, the fusion length L fusion between adjacent images is calculated as follows:

[0059]

[0060] in, and They represent the maximum and minimum values ​​of the horizontal coordinate of the optimal suture line, respectively, and d is the length of the area to be fused;

[0061] (3) To speed up the image fusion process, the number of zero matrix rows and columns to be filled on the left and right sides of each image is further calculated based on the starting position and the size of the final fused image.

[0062] That is, get the black border matrix that should be added to the left and right of each image;

[0063] The modified mask template and black edge matrix are uploaded from the CPU to the GPU for subsequent calculations.

[0064] In this embodiment, the real-time video fusion steps are specifically as follows:

[0065] The system acquires the video streams of each camera in parallel based on the Real-Time Streaming Protocol (RTSP), scales and masks each video stream frame by frame based on the modified mask template, and uploads the obtained masked video frame images to the GPU to generate multiple image frame sequences corresponding to the camera.

[0066] The i-th image in each image frame sequence has the same corresponding time, i ranges from 1 to n, and n is the total number of images in the image frame sequence;

[0067] All image frame sequences are fused and sorted to obtain a set of image sequences to be fused. The order of each image frame sequence in the set is consistent with the actual installation order of the cameras. Since all image data has been uploaded to the GPU, CUDA acceleration is used to add boundaries to each sequence image at the same time and fuse them in sequence, which can speed up the calculation process, reduce computing time, and generate wide-field seamless fused panoramic images more quickly.

[0068] In this embodiment, in order to obtain a fused image with smoother transition and better visual effect, the stitching seams are optimized based on a weighted fade-in and fade-out fusion algorithm to eliminate the stitching seams that exist when stitching based on suture lines, and finally generate a wide-field seamless fused panoramic image.

[0069] The fusion weight calculation formula is as follows:

[0070]

[0071] Among them, k is the fusion weight parameter, ranging from 0 to 1, and w is the fusion weight.

[0072] In this embodiment, the i-th image in each image frame sequence has the same corresponding time, i ranges from 1 to n, and n is the total number of images in the image frame sequence;

[0073] However, due to network and server performance reasons, when the number of video streams reaches six or more, it is impossible to obtain the video images of each channel at the same time. Therefore, it is necessary to set an initialization time threshold in each video stream capture process. In this embodiment, the initialization time threshold is set in each video stream capture process according to the device performance and network conditions to ensure that the first frame image in the image frame sequence of each camera is at the same time, and the images obtained before the threshold are discarded and not included in the image frame sequence.

[0074] In the present invention, mask fusion and image addition are adopted to reduce the amount of calculation, and multi-threading is used based on GPU to perform parallel fusion of images, thereby speeding up the calculation process and reducing the time consumption of single-frame processing: the existing video fusion technology requires calculating feature points for each frame image and performing image conversion and sequential fusion, which is time-consuming. In the present invention, masks are used to replace the above process, and the mask matrix and the image matrix are multiplied based on multi-threading when fusion is performed to directly obtain the image to be fused; in addition, by calculating the starting rows and columns of each image in the final fused image and the size of the final fused image, a 0 matrix is ​​used to fill each image to the size of the final image, and fusion is performed based on multi-threaded pixel addition. Compared with sequential fusion, the time consumption is greatly reduced and the real-time performance of the algorithm is improved.

[0075] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the embodiments of the technology in a certain steel plant. According to the on-site conditions, the monitoring cameras are installed on pillars 7.5m from the ground, with a camera spacing of 12m or 15m and an arrangement of about 45°. This ensures that the center line of the production line is centered in each image and that adjacent camera images have at least 1 / 3 overlapping areas. Ultimately, a total of 26 cameras are used to monitor the production line and perform video fusion. Images from each camera at the same moment are obtained for preprocessing, and a mask template is produced. Two servers on site fuse the left and right side monitoring, respectively, and at the same time correct some tilted images caused by the camera position. Multi-threading and GPU parallel processing are used to fuse the frames, obtaining the fused real-time video and playing it with the server desktop. Figure 2 The following diagram shows the actual application effect. Using a high-definition video cable to connect the server and the operation center's large screen, the fused image is mapped to the operation center's large screen with virtually no delay, enabling on-site staff to monitor and provide operational guidance.

[0076] The method provided by the present invention performs video fusion for scenes with stereoscopic parallax, improves image processing speed based on mask templates and GPU parallel acceleration, and realizes the fusion of up to 13 channels of video in a single server.

[0077] The multi-channel video real-time fusion method provided by the present invention adopts mask fusion and image addition to reduce the amount of calculation, and uses multi-threading based on GPU to perform parallel fusion of images, thereby speeding up the calculation process and reducing the time consumption of single-frame processing.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-channel video real-time fusion method for industrial intelligence, characterized by: The method comprises: Equipment installation: Fixedly install the surveillance cameras and control the installation position of each surveillance camera so that the images collected by adjacent cameras have overlapping areas; Mask production: Obtain overlapping images from all cameras at any given moment, perform image preprocessing, and produce a mask template. First-frame image processing: Obtain the first-frame image of the video corresponding to each camera, process the first-frame image of the video to obtain the image to be stitched; calculate the starting position of the image to be stitched in the final fused image and the size of the final fused image; obtain the black border matrix to be added to the left and right of each image based on the starting position and the size of the final fused image; upload the mask template and the black border matrix from the CPU to the GPU; Real-time video fusion: The system obtains video streams from each surveillance camera, performs frame-by-frame scaling and masking on each stream, and uploads them to the GPU to generate an image frame sequence. All image frame sequences are fused and sorted to obtain a set of image sequences to be fused. The order of the image frames in the set is consistent with the order in which the cameras are actually installed. Using CUDA acceleration, boundaries are added to each image sequence at the same moment and fused. A weighted fade-in / fade-out fusion algorithm is used to optimize the stitching seams, eliminating the seams that exist when stitching based on suture lines, generating a wide-field seamlessly fused panoramic image. The steps of mask making are specifically as follows: Calculate image scaling parameters: Obtain images from all cameras at the same time, select a camera as a reference, use the size of the device in the image corresponding to the reference camera as a reference, calculate the scaling ratio of adjacent images based on the overlapping area, and obtain the scaling parameters of all images in sequence; Seam line selection: Based on the location of the devices in the overlapping area, the optimal seam line is manually selected, and the optimal seam line is selected in the background without foreground devices. Then, based on the location of the seam line, the retained and deleted areas of each image are determined. The pixel values ​​of the retained area are set to 1, and the pixel values ​​of the deleted area are set to 0 to obtain the initial mask template. The initial mask is modified, and the modified mask template is expressed as: Among them, I(x,y) represents the value of the mask at (x,y); x, y represent the horizontal and vertical coordinates of the image respectively, I0 represents the deleted area with a pixel value of 0, and x seam Indicates the The horizontal coordinate of the point on the best suture line, d represents the length of the area to be fused, and I1 represents the length of the original The retained area is the area after removing the transition area, that is, I1 = I-I0-d, where I is the entire image. picture; The steps of processing the first frame image are specifically as follows: (1) Based on the real-time streaming protocol, the video headers corresponding to each camera are obtained in turn. Frame image, scaling the corresponding image according to the scaling parameter obtained in the step of calculating the image scaling parameter, and directly multiplying the scaled image data with the modified mask to obtain the image to be stitched; (2) According to the fusion length L between adjacent images fusion Calculate the starting position of each image in the final fused image and the size of the final fused image in turn, and store them for subsequent splicing calls; among them, the fusion length L between adjacent images fusion The calculation formula is: in, and They represent the maximum and minimum values ​​of the horizontal coordinate of the optimal suture line, respectively, and d is the length of the area to be fused; (3) Obtaining the zero matrix to be added to the left and right sides of each image, i.e., the black edge matrix, according to the starting position and the size of the final fused image; Upload the modified mask template and black edge matrix from the CPU to the GPU.

2. The method for real-time fusion of multiple video channels for industrial intelligence according to claim 1, characterized in that: During the equipment installation steps, all surveillance cameras are fixed on the pillars in the same posture, ensuring that the equipment in the target production line is centered in each surveillance screen, and the focal length of all surveillance cameras is set to the minimum; and there is at least 1 / 3 overlapping area between adjacent cameras.

3. The method for real-time fusion of multiple video channels for industrial intelligence according to claim 1, characterized in that: The real-time video fusion steps are specifically as follows: The video streams of each camera are acquired in parallel based on the real-time streaming protocol. Each video stream is scaled and masked frame by frame based on the modified mask template. The masked video frame images are uploaded to the GPU to generate multiple image frame sequences corresponding to the camera. The i-th image in each image frame sequence has the same corresponding time, i ranges from 1 to n, and n is the total number of images in the image frame sequence; All image frame sequences are fused and sorted to obtain a set of image sequences to be fused. The order of each image frame sequence in the set of image sequences to be fused is consistent with the actual installation order of the cameras. Based on CUDA acceleration, boundaries are added to each sequence image at the same moment and then fused.

4. The method for real-time fusion of multiple video channels for industrial intelligence according to claim 3 is characterized in that: Image fusion uses a weighted fade-in and fade-out fusion algorithm to optimize the stitching seams; The fusion weight calculation formula is as follows: Among them, k is the fusion weight parameter, ranging from 0 to 1, and w is the fusion weight.

5. The method for real-time fusion of multiple video channels for industrial intelligence according to claim 3 is characterized in that: The i-th image in each image frame sequence has the same corresponding time, i ranges from 1 to n, and n is the total number of images in the image frame sequence; An initialization time threshold is set in each video stream capture process to ensure that the first frame image in the image frame sequence of each camera is at the same time, and the images obtained before the threshold are discarded and not included in the image frame sequence.

Images