Wharf storage yard global visual video fusion intelligent inspection method and platform thereof

Abstract

The invention relates to the technical field of inspection management, in particular to a wharf storage yard global visual video fusion intelligent inspection method and platform, and the method specifically comprises the steps: S1, video fusion processing; s2, video preprocessing; s3, extracting and matching feature points; s4, estimating a homography matrix; s5, coordinate transformation; s6, performing image fusion processing; s7, carrying out post-treatment; according to the system and the method, the refinement level of storage yard management is improved, multiple functions such as business collaboration, commanding and dispatching, monitoring and early warning, risk control and equipment maintenance are integrated, powerful technical support is provided for safety production and management of a storage yard, and the comprehensive application technology innovation point of a video fusion technology based on a three-dimensional scene is fully embodied.

Description

Technical Field

[0001] The present invention relates to the field of inspection management technology, and in particular to a terminal yard full-area visualized video fusion intelligent inspection method and platform. Background Art

[0002] As a buffer zone for cargo transfer at dry bulk terminals, the yard plays an important role in terminal transportation. Personnel and vehicle safety management and daily belt inspections at the yard are one of the key factors that determine the terminal's service capacity. At present, management is mainly carried out through traditional video surveillance combined with manual inspections. Existing video surveillance systems often lack the integration and display of macroscopic location information. The video data generated by each camera is usually scattered, isolated, fragmented, and incoherent, making it difficult to form a macroscopic grasp of the entire monitoring area. This scattered data form requires monitoring personnel to spend a lot of time and energy to analyze the video images of each camera, making it difficult to quickly and accurately understand the real-time situation of the entire monitoring area. The information disclosed in this background technology section is intended only to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form of implication that the information constitutes prior art already known to those skilled in the art. Summary of the invention

[0003] The purpose of the present invention is to provide a visual video fusion intelligent inspection method and platform for the whole area of ​​the terminal yard, and to realize the comprehensive visual monitoring and intelligent management of the yard terminal by integrating video fusion, video twin, AI analysis and IoT perception, geographic information, big data and other related technologies. The platform not only improves the level of refinement of yard management, but also integrates multiple functions such as business collaboration, command and dispatch, monitoring and early warning, risk control, equipment maintenance, etc., providing strong technical support for the safe production and management of the yard, and fully reflecting the comprehensive application technology innovation of video fusion technology based on three-dimensional scenes.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] The present invention provides a terminal yard global visualized video fusion intelligent inspection method, which comprises:

[0006] S1. Video fusion processing

[0007] Adopt 3D geographic information scene video stitching and fusion technology, use the technical principle of image stitching to stitch videos, obtain the position relationship between fixed cameras, perform geometric transformation on images, and fuse 3D spatial information to obtain real-time 3D panoramic video;

[0008] S2. Video preprocessing

[0009] Perform brightness correction, distortion correction and cylindrical transformation preprocessing on each frame of the collected overlapping area to improve the matching accuracy and stitching effect;

[0010] S3. Feature point extraction and matching

[0011] Relying on the feature points with rich texture information in the image, the coordinate transformation relationship between adjacent images is established, and a unified coordinate transformation is performed to obtain the panoramic stitching result;

[0012] S4. Homography matrix estimation

[0013] The RANSAC algorithm is used to eliminate mismatched points in the matching process;

[0014] S5. Coordinate transformation

[0015] After calculating the SIFT matching of the image pair after the cylindrical transformation and obtaining the homography relationship, the coordinate transformation is performed by the following formula to transform the image into a unified coordinate system;

[0016] S6. Image fusion processing

[0017] Adopt grayscale pixel weighted average algorithm to perform stitching fusion;

[0018] S7, post-processing

[0019] Includes: Bundle Adjustment and Exposure Compensation.

[0020] Preferably, in S1, it includes:

[0021] Image preprocessing (geometric correction, enhancement, denoising) of multi-channel video images;

[0022] Register (stitch, crop) the processed images;

[0023] Establish transformation models (color fusion, perspective transformation);

[0024] Perform unified coordinate transformation;

[0025] Perform image spatiotemporal fusion and multi-source sensor device fusion;

[0026] The fused images are used for data analysis and business applications in three-dimensional space scenes.

[0027] Preferably, in S2,

[0028] The brightness correction is a histogram equalization algorithm, which calculates the global overall contrast distribution of the image and adaptively expands the image brightness range through a probability equalization approach to improve the image contrast;

[0029] Among them, the probability of each gray level appearing in the image is:

[0030] p(x)=x s / N;

[0031] Where N is the total number of pixels in the image, x s is the number of pixels with gray value x, thus obtaining the cumulative distribution function:

[0032]

[0033] Thus, t s Map to the [0,256] interval to obtain a normalized balanced histogram, and then obtain a new pixel value for each pixel in the traversed image;

[0034] The distortion correction is performed by solving the distortion coefficient through the chessboard calibration algorithm to correct the image distortion. The formula is as follows:

[0035]

[0036] Among them, r 2 =x 2 +y 2 , k1, k2 are distortion coefficients;

[0037] The cylindrical transformation directly projects the plane image onto the cylindrical surface, so that multiple images are spliced ​​to form a circular cylinder;

[0038] The cylindrical projection transformation is as follows:

[0039]

[0040] Where s=f, and f is the focal length of the camera.

[0041] Preferably, S3 includes:

[0042] S31, scale space extreme value detection: use the two-dimensional Gaussian kernel function to convolve with the image to obtain the pyramid image;

[0043] The two-dimensional Gaussian kernel function is:

[0044]

[0045] For a two-dimensional image, the scale space at different scales can be obtained by convolving the image with a Gaussian kernel:

[0046] L(x,y,σ)=G(x,y,σ)*I(x,y)

[0047] When detecting scale space extremes, a single pixel needs to be compared with 8 neighboring pixels of the same scale and 9*2 neighboring pixels of the corresponding position of the adjacent scale;

[0048] S32, accurately determining the position and scale of key points by fitting a three-dimensional quadratic function, while removing low-contrast key points and unstable edge response points;

[0049] S33, using the gradient direction distribution characteristics of the pixels in the neighborhood of the key point to specify a direction parameter for each key point, so that the operator has rotation invariance;

[0050]

[0051] θ(x,y)=αtg2((L(x,y+1)-L(x,y-1)) / (L(x+1,y)-L(x-1,y)))

[0052] The above formula is the modulus and direction formula at the pixel (x, y);

[0053] S34, generate SIFT descriptor, then take 8*8 window with key point center, the arrow of each small grid represents the gradient direction of the pixel, the length of the arrow represents the gradient modulus, calculate the gradient direction histogram in 8 directions on each 4*4 small block, draw the accumulated value of each gradient direction, form seed point, each seed point has 8-direction direction vector information as the value of descriptor;

[0054] The matching of feature points is mainly obtained based on the Euclidean distance of the SIFT descriptor. The points with a distance less than a certain threshold are regarded as matching points, and the corresponding coordinates of the matching points are recorded;

[0055]

[0056] Preferably, in S4, it includes:

[0057] S41, randomly select a subset S1 of N data points from the data set;

[0058] S42, solving model M1 for the selected subset S1;

[0059] S43, estimating the distance between M1 and the remaining data points, and recording the number of points whose distance is within the threshold;

[0060] S44, if the number of points is greater than a certain ratio, the iteration is completed, otherwise return to step A;

[0061] The RANSAC algorithm is used to eliminate mismatched point pairs and obtain the perspective relationship model H based on the maximum a posteriori probability.

[0062] Preferably, in S6, it includes:

[0063] Assume that the image seam line is taken as the center line, and the distance between the pixel and the seam line is d;

[0064] R′=d×R1+(1-d)×R2

[0065] G′=d×G1+(1-d)×G2

[0066] B′=d×B1+(1-d)×B2;

[0067] R, G, B are the red, green, and blue channel pixel values ​​respectively. The new pixel value of the spliced ​​area is obtained by weighting the distance between the pixel values ​​of the two adjacent images.

[0068] The present invention also provides a global visualized video fusion intelligent inspection platform, which adopts the global visualized video fusion intelligent inspection method for the terminal yard.

[0069] By adopting the above technical solution, the present invention has the following beneficial effects:

[0070] The present invention not only improves the level of refinement of storage yard management, but also integrates multiple functions such as business collaboration, command and dispatch, monitoring and early warning, risk control, and equipment maintenance, providing strong technical support for safe production and management of the storage yard, and fully reflecting the comprehensive application technology innovation of video fusion technology based on three-dimensional scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0072] Figure 1 It is a schematic diagram of the overall architecture of the global visualization video fusion intelligent inspection platform of the present invention;

[0073] Figure 2 It is a schematic diagram of the technical route of the global visualization video fusion intelligent inspection platform of the present invention;

[0074] Figure 3 It is a schematic diagram of the video fusion processing framework of the global visual video fusion intelligent inspection platform of the present invention;

[0075] Figure 4 A schematic diagram of the histogram equalization provided by the present invention;

[0076] Figure 5 A schematic diagram of lens distortion provided by the present invention;

[0077] Figure 6 A schematic diagram of barrel distortion correction provided by the present invention;

[0078] Figure 7 A schematic diagram of the cylindrical transformation provided by the present invention;

[0079] Figure 8 A schematic diagram of the cylindrical projection provided by the present invention;

[0080] Fig. 9 A schematic diagram of the structure of the scale space local extremum detection provided by the present invention;

[0081] Fig.10 A schematic diagram of generating a feature vector from key points provided by the present invention;

[0082] Fig.11 A schematic diagram of Sift feature matching provided by the present invention;

[0083] Fig.12 A top view of the cylindrical splicing provided by the present invention;

[0084] Fig.13 A schematic diagram of grayscale weighted average provided by the present invention;

[0085] Fig.14 A schematic diagram of bundling adjustment provided by the present invention;

[0086] Fig.15 A schematic diagram of the bundling adjustment result provided by the present invention. DETAILED DESCRIPTION

[0087] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0088] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0089] Embodiment 1

[0090] The first embodiment provides a method for intelligent inspection of a terminal yard using global visualized video fusion, which includes:

[0091] S1. Video fusion processing

[0092] The three-dimensional geographic information scene video stitching and fusion technology uses the technical principles of image stitching to stitch videos. By obtaining the positional relationship between fixed cameras, the image is geometrically transformed and the three-dimensional spatial information is fused to obtain real-time three-dimensional panoramic video.

[0093] Combination Figure 3 As shown in the figure, the video fusion process mainly includes three parts: video preprocessing, image transformation, and spatial fusion. First, the geometric correction of multiple video images is preprocessed, and the processed images are spliced ​​and cropped, and the model is transformed and the spatial coordinates are unified. Finally, the spatiotemporal images are fused, and the fused images are used for data analysis and business applications in three-dimensional space scenes.

[0094] S2. Video preprocessing

[0095] The preprocessing mainly performs brightness correction, distortion correction, cylindrical transformation and other preprocessing on each frame of the collected overlapping areas to improve the matching accuracy and stitching effect.

[0096] Brightness correction mainly solves the problem that the input image is too bright or too dark, or the brightness difference between adjacent images is large, and improves the number and quality of feature point detection. It can calculate the matching relationship more accurately. The commonly used brightness correction method is the histogram equalization algorithm. Figure 4 As shown in FIG, this method calculates the global overall contrast distribution of the image, adaptively expands the image brightness range through the idea of ​​probability equalization, and improves the image contrast.

[0097] The probability of each gray level appearing in the image is:

[0098] p(x)=x s / N

[0099] Where N is the total number of pixels in the image, x s is the number of pixels with gray value x. Thus, the cumulative distribution function can be obtained.

[0100]

[0101] Thus, t s Map it to the interval [0,256] to get the normalized balanced histogram. Then get the new pixel value for each pixel in the traversal image.

[0102] Combination Figures 5 to 8 As shown in the figure, image distortion is the error in the geometric position of the image point on the image plane. This is caused by the bending of the central projection ray because the imaging system cannot make the image and the actual dance strictly meet the pinhole imaging model (or central projection relationship) in the full field of view. Distortion is divided into radial distortion and tangential distortion.

[0103] The distortion coefficient is solved by chessboard calibration algorithm to correct image distortion.

[0104]

[0105] where r 2=x 2 +y 2 , k1, k2 are distortion coefficients.

[0106] Cylinder transformation directly projects the plane image onto the cylinder, so that multiple images are spliced ​​together to form a circular cylinder.

[0107] The cylindrical projection transformation is as follows:

[0108]

[0109] Where s=f, f is the focal length of the camera.

[0110] S3. Feature point extraction and matching

[0111] Relying on the feature points with rich texture information in the image, the coordinate transformation relationship between adjacent images is established, and the panoramic stitching result is obtained by unified coordinate transformation. The main popular feature matching algorithms include SIFT, SURF, FREAK, ORB, Harris, etc. Among them, SIFT is a scale-invariant image feature with high robustness to image scaling, rotation and local noise. The extraction steps are as follows: Figures 9 to 11 As shown:

[0112] A. Scale space extremum detection. A two-dimensional Gaussian kernel function is used to convolve with the image to obtain a pyramid image.

[0113] The two-dimensional Gaussian kernel function is:

[0114]

[0115] For a two-dimensional image, the scale space at different scales can be obtained by convolving the image with a Gaussian kernel:

[0116] L(x,y,σ)=G(x,y,σ)*I(x,y)

[0117] When detecting scale space extreme values, a single pixel needs to be compared with 8 neighboring pixels around the same scale and 9*2 neighboring pixels around the corresponding position of the adjacent scale.

[0118] B. Accurately determine the location and scale of key points by fitting a three-dimensional quadratic function, and remove low-contrast key points and unstable edge response points.

[0119] C. Use the gradient direction distribution characteristics of the pixels in the neighborhood of the key point to specify the direction parameter for each key point, so that the operator is rotation-invariant.

[0120]

[0121] θ(x,y)=αtg2((L(x,y+1)-L(x,y-1)) / (L(x+1,y)-L(x-1,y)))

[0122] The above formula is the modulus and direction formula at the pixel (x, y).

[0123] D. Generate SIFT descriptor. Next, take an 8*8 window with the key point as the center. The arrow of each small grid represents the gradient direction of the pixel. The length of the arrow represents the gradient modulus. Calculate the gradient direction histogram in 8 directions on each 4*4 small block, draw the accumulated value of each gradient direction, form a seed point, and each seed point has 8-direction direction vector information as the value of the descriptor.

[0124] The matching of feature points is mainly obtained based on the Euclidean distance of the SIFT descriptor. Points with distances less than a certain threshold are regarded as matching points, and the corresponding coordinates of the matching points are recorded.

[0125]

[0126] S4. Homography matrix estimation

[0127] Generally speaking, local feature detection algorithms often have matching errors, which will affect the accuracy of homography matrix estimation. Therefore, it is particularly important to eliminate mismatched points in the matching and obtain an accurate homography matrix.

[0128] The commonly used RANSAC (Random Sample Consensus) algorithm is used. The algorithm steps are as follows:

[0129] A. Randomly select a subset S1 of N data points from the data set

[0130] B. Solve model M1 for the selected subset S1.

[0131] C. Estimate the distance between M1 and the remaining data points, and record the number of points whose distance is within the threshold.

[0132] D. If the number of points is greater than a certain ratio, the iteration is completed, otherwise return to step 1.

[0133] From the above, we can see that the RANSAC algorithm can eliminate mismatched point pairs and obtain the perspective relationship model H based on the maximum a posteriori probability.

[0134] S5. Coordinate transformation

[0135] Combination Fig.12 As shown, after calculating SIFT matching for the image pair after the cylindrical transformation and obtaining the homography relationship, the coordinate transformation can be performed by the following formula to transform the image into a unified coordinate system.

[0136]

[0137] S6. Image fusion processing

[0138] The result after coordinate transformation often shows blur, color deviation and other phenomena at the seam position due to camera sensor or homography matrix errors, so image fusion technology is needed to process the seam area. The so-called image fusion is to integrate different images and use the complementary information between multiple images to improve image recognizability and clarity, thereby obtaining a more reliable, accurate and comprehensive image overview of the scene target, which is suitable for human eye observation or further necessary technical processing using a computer. The grayscale pixel weighted average algorithm is used for seam fusion. Combined with Fig.13 As shown, the algorithm principle is as follows:

[0139] Assume that the image seam line is taken as the center line and the distance between the pixel and the seam line is d.

[0140] R′=d×R1+(1-d)×R2

[0141] G′=d×G1+(1-d)×G2

[0142] B′=d×B1+(1-d)×B2

[0143] R, G, B are the red, green, and blue channel pixel values ​​respectively. The new pixel value of the stitching area is obtained by weighting the distance between the pixel values ​​of the two adjacent images.

[0144] S7, post-processing, including: bundle adjustment and exposure compensation, combined Fig.14 and Fig.15 shown.

[0145] Bundling adjustment, since the homography matrix cannot be obtained very accurately, it is related to the accuracy of the image corner position, the image corner matching error and the error in solving the homography matrix. In addition, the multiplication of the homography matrix will bring cumulative errors, causing position offsets when multiple images are stitched together, and they are not strictly on a straight line. Here we use bundling adjustment to solve this problem.

[0146] Concatenate the first image with the rightmost image and calculate the difference between (x1, y1) and (x n ,y n ) is the equation of the straight line, y = ax + b. Then the complete stitched image is rotated to obtain the stitching result on a horizontal line, and the middle area is cropped to obtain the final stitching result.

[0147] Exposure compensation: Due to differences in sensor types and camera fields of view, the brightness of the stitching positions of the two adjacent images we obtain is generally different. After stitching is completed, we can see an obvious "transition zone". Through exposure compensation, the brightness of the adjacent image areas is adjusted to make the transition smoother. The algorithm principle is as follows:

[0148] A. Calculate the average brightness of the left and right areas of the seam G1, G2.

[0149] B. Brightness ratio calculation.

[0150]

[0151] C. Brightness gain compensation.

[0152] R′=Rη

[0153] G′=Gη

[0154] B′=Bη

[0155] In summary, this embodiment is based on the video fusion of three-dimensional space scenes, and adopts multi-channel distortion video correction based on three-dimensional geographic information to spatially informatize the spliced ​​and fused video. This technology first considers the real-time nature of video images, realizes multi-video synchronization through time calibration, and at the same time adopts advanced three-dimensional space image matrix transformation and distortion correction processing, and performs video three-dimensional space matching through image affine transformation.

[0156] At the same time, it corrects various video distortions, including irregular image distortions caused by camera visual axis tilt and field of view cone, fisheye image distortion, etc., matches the three-dimensional spatial information pixels of the corrected video, and fuses the video to the spatial position it should be in the three-dimensional scene through texture fusion technology. It realizes multi-angle transformation of viewing angles to observe real-time video, and performs scaling, movement, and aspect ratio transformation on video objects, providing reliable video image display technology support to adapt to screen display peripherals of various scales.

[0157] Embodiment 2

[0158] This embodiment also provides a global visualization video fusion intelligent inspection platform, which adopts the global visualization video fusion intelligent inspection method for the terminal yard.

[0159] 1. Overall architecture:

[0160] like Figure 1 As shown in the figure, based on the current situation and management needs of the yard terminal, a full-domain visual video fusion intelligent inspection platform is designed, which mainly realizes the fusion of three-dimensional virtual simulation and monitoring video, business data and multi-source perception data, and carries out comprehensive command and control business applications on this basis.

[0161] (1) Data support

[0162] It mainly provides the necessary data resources for the system, including basic data and business data. Basic data, including remote sensing images, digital elevation models (DEMs), electronic maps, SHP data and 3D models, are used to build a 3D geographic information system. Business data includes security monitoring video data, multi-source perception and other business data, which are used to realize 3D geographic information visualization business applications.

[0163] (2) Technical support

[0164] It mainly provides technical support for the realization of the virtual-real fusion application of geographic information scene video. The basic core is the three-dimensional geographic information system (3DGIS) platform, which carries remote sensing images, digital elevation models DEM, electronic maps, SHP data and three-dimensional models, security monitoring video data, multi-source perception and other business data, and supports the virtual-real fusion of three-dimensional geographic information scene video, video path plan planning automatic cruise scheduling, two- and three-dimensional integrated linkage and positioning browsing, historical video three-geographic information scene synchronous backtracking, business-specific spatial analysis tool support, multi-source business data spatial scene visual fusion and other visualization business applications.

[0165] (3) Business support

[0166] Through the visualization functions such as virtual-real fusion of three-dimensional geographic information scene videos, automatic cruise scheduling of video path plan planning, two- and three-dimensional integrated linkage and positioning browsing, synchronous backtracking of three geographic information scenes in historical videos, support of business-specific spatial analysis tools, and visual fusion of multi-source business data spatial scenes, a three-dimensional geographic information video spatiotemporal perception fusion prevention and control system application is built.

[0167] 2. Technical route:

[0168] like Figure 2 As shown, 3D video fusion technology is an innovative application system that integrates 3D spatial information, video splicing and fusion, multi-source data perception and other technologies and applications.

[0169] According to the real terrain and building conditions, a 3D scene simulation model is constructed, matching the standard geographic coordinate system (longitude, latitude, altitude), and the surveillance video is accurately spliced ​​and integrated according to the scene to realize the unified infrastructure of 3D geographic information system visualization spatiotemporal data fusion. Based on this architecture, actual business applications are carried out with the support of 3D geographic information visualization technology.

[0170] 3. Technical Implementation:

[0171] By using high-tech technologies that combine virtuality and reality, such as 3D panoramic video fusion and integrated layout of 3D space scenes, innovative applications of the fusion of video and spatial position are realized on the basis of traditional video surveillance. This includes unified processing of the video collected by the camera. In order to ensure the panoramic video display effect, the video images of each fused surveillance camera are corrected, and the video images are uniformly processed in terms of image ratio, brightness, contrast, etc. to improve the consistency of the camera video images; this includes establishing a geometric mapping relationship between real-time video and 3D space scene positions, realizing the fusion and matching of split-lens surveillance video data collected by the front-end camera with the 3D space scene mapping, etc.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent inspection of the whole area of ​​a terminal yard by visualizing video fusion, characterized in that: include: S1. Video fusion processing Adopt 3D geographic information scene video stitching and fusion technology, use the technical principle of image stitching to stitch videos, obtain the position relationship between fixed cameras, perform geometric transformation on images, and fuse 3D spatial information to obtain real-time 3D panoramic video; S2. Video preprocessing Perform brightness correction, distortion correction and cylindrical transformation preprocessing on each frame of the collected overlapping area to improve the matching accuracy and stitching effect; S3. Feature point extraction and matching Relying on the feature points with rich texture information in the image, the coordinate transformation relationship between adjacent images is established, and a unified coordinate transformation is performed to obtain the panoramic stitching result; S4. Homography matrix estimation The RANSAC algorithm is used to eliminate mismatched points in the matching process; S5. Coordinate transformation After calculating the SIFT matching of the image pair after the cylindrical transformation and obtaining the homography relationship, the coordinate transformation is performed by the following formula to transform the image into a unified coordinate system; S6. Image fusion processing Adopt grayscale pixel weighted average algorithm to perform stitching fusion; S7, post-processing Includes: Bundle Adjustment and Exposure Compensation.

2. The method for intelligent inspection of the whole area of ​​the terminal yard by visual video fusion according to claim 1 is characterized in that: In S1, it includes: Image preprocessing of multi-channel video images; Register the processed images; Establish transformation model; Perform unified coordinate transformation; Perform image spatiotemporal fusion and multi-source sensor device fusion; The fused images are used for data analysis and business applications in three-dimensional space scenes.

3. The method for intelligent inspection of the whole area of ​​the terminal yard by visual video fusion according to claim 1 is characterized in that: In S2, The brightness correction is a histogram equalization algorithm, which calculates the global overall contrast distribution of the image and adaptively expands the image brightness range through a probability equalization approach to improve the image contrast; Among them, the probability of each gray level appearing in the image is: p(x)=x s / N; Where N is the total number of pixels in the image, x s is the number of pixels with gray value x, thus obtaining the cumulative distribution function: Thus, t s Map to the [0,256] interval to obtain a normalized balanced histogram, and then obtain a new pixel value for each pixel in the traversed image; The distortion correction is performed by solving the distortion coefficient through the chessboard calibration algorithm to correct the image distortion. The formula is as follows: Among them, r 2 =x 2 +y 2 , k1, k2 are distortion coefficients; The cylindrical transformation directly projects the plane image onto the cylindrical surface, so that multiple images are spliced ​​to form a circular cylinder; The cylindrical projection transformation is as follows: Wherein, s=f, f is the focal length of the camera.

4. The method for intelligent inspection of the whole area of ​​the terminal yard by visual video fusion according to claim 1 is characterized in that: In S3, this includes: S31, scale space extreme value detection: use the two-dimensional Gaussian kernel function to convolve with the image to obtain the pyramid image; The two-dimensional Gaussian kernel function is: For a two-dimensional image, the scale space at different scales can be obtained by convolving the image with a Gaussian kernel: L(x,y,σ)=G(x,y,σ)*I(x,y) When detecting scale space extremes, a single pixel needs to be compared with 8 neighboring pixels of the same scale and 9*2 neighboring pixels of the corresponding position of the adjacent scale; S32, accurately determining the position and scale of key points by fitting a three-dimensional quadratic function, while removing low-contrast key points and unstable edge response points; S33, using the gradient direction distribution characteristics of the pixels in the neighborhood of the key point to specify a direction parameter for each key point, so that the operator has rotation invariance; θ(x,y)=atg2((L(x,y+1)-L(x,y-1)) / (L(x+1,y)-L(x-1,y))) The above formula is the modulus and direction formula at the pixel (x, y); S34, generate SIFT descriptor, then take 8*8 window with key point center, the arrow of each small grid represents the gradient direction of the pixel, the length of the arrow represents the gradient modulus, calculate the gradient direction histogram in 8 directions on each 4*4 small block, draw the accumulated value of each gradient direction, form seed point, each seed point has 8-direction direction vector information as the value of descriptor; The matching of feature points is mainly obtained based on the Euclidean distance of the SIFT descriptor. The points with a distance less than a certain threshold are regarded as matching points, and the corresponding coordinates of the matching points are recorded; 5. The method for intelligent inspection of the whole area of ​​the terminal yard by visual video fusion according to claim 1 is characterized in that: In S4, it includes: S41, randomly select a subset S1 of N data points from the data set; S42, solving model M1 for the selected subset S1; S43, estimating the distance between M1 and the remaining data points, and recording the number of points whose distance is within the threshold; S44, if the number of points is greater than a certain ratio, the iteration is completed, otherwise return to step A; The RANSAC algorithm is used to eliminate mismatched point pairs and obtain the perspective relationship model H based on the maximum a posteriori probability.

6. The method for intelligent inspection of the whole area of ​​the terminal yard by visual video fusion according to claim 1 is characterized in that: In S6, it includes: Assume that the image seam line is taken as the center line, and the distance between the pixel and the seam line is d; R′=d×R1+(1-d)×R2 G′=d×G1+(1-d)×G2 B′=d×B1+(1-d)×B2; R, G, B are the red, green, and blue channel pixel values ​​respectively. The new pixel value of the stitching area is obtained by weighting the distance between the pixel values ​​of the two adjacent images.

7. A global visual video fusion intelligent inspection platform, characterized in that: A global visualization video fusion intelligent inspection platform method as described in any one of claims 1 to 6 is adopted.

Images