A wide-area ocean panoramic fusion method and system based on multi-source data

Abstract

The present application provides a method and system for wide-area ocean panoramic fusion based on multi-source data, the method comprising: acquiring multiple images and extracting calibration points of the images, calculating the parameters of the camera, selecting a reference image from the image using the minimum spanning tree method, and sorting the remaining images; using a direct linear transformation algorithm to obtain the transformation matrix between two images with overlapping areas in all images, and obtaining the projection transformation relationship between all remaining images and the reference image; using a bundle adjustment method to optimize the parameters and obtain the calibrated camera parameters; and using an improved graph cut algorithm to perform splicing and fusion processing on multiple images to generate a panoramic fusion image of the sea surface. The calibration point residuals are collaboratively optimized using the light speed adjustment method to obtain the calibrated camera parameters, thereby constraining the overall geometric structure of the camera. The improved graph cut algorithm can be applied to the sea surface splicing application scenario to achieve the fusion of sea surface panoramic images.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a wide-area ocean panoramic fusion method and system based on multi-source data. Background Art

[0002] Panorama technology is a form of virtual reality. In general, panorama refers to a wide-angle image representation of the real world, which can be a painting, a photograph, a video or a three-dimensional model of a scene. The panorama mentioned in this invention refers to a panoramic image representation based on photography or video. Panorama stitching technology is a virtual reality technology that generates realistic graphics based on image drawing technology. Panorama stitching technology mainly realizes a panoramic view of the scene by stitching images.

[0003] In recent years, panoramic image technology has been applied to the monitoring of the sea surface environment. Among them, camera parameter calibration is the core step of panoramic image stitching, and its accuracy has a crucial impact on the fusion effect of wide-area sea surface images. In addition, the target detection module needs to integrate visible light and thermal imaging effects. Therefore, the visible light camera and thermal imaging camera loaded on the device must be jointly calibrated.

[0004] The camera device rotates coaxially at a constant speed and scans one circle. One device scanning cycle is the imaging cycle of a panoramic image. The cycle time can be set, usually less than 100s. The image acquisition radius is 3km and the scanning area is 25km 2 During the scanning process, due to the calibration accuracy problem, the four directions of each image and the overlapping areas of other images are not very accurate. If simply superimposed, it will cause obvious ghosting and blurring.

[0005] In view of this, it is of great significance to design a wide-area ocean panoramic fusion method and system that can correct the transformation matrix to constrain the overall geometric structure of the camera and solve the ghost problem. Summary of the invention

[0006] The embodiment of the present application proposes a wide-area ocean panoramic fusion method and system based on multi-source data to solve the technical problems mentioned in the above background technology section.

[0007] In a first aspect, an embodiment of the present application provides a wide-area ocean panorama fusion method based on multi-source data, the method comprising the following steps:

[0008] S110, acquiring multiple images and extracting calibration points of the images, calculating the intrinsic parameters and extrinsic parameters of the camera, selecting a reference image from the images using a minimum spanning tree method, and sorting the remaining images;

[0009] S120, using a direct linear transformation algorithm to obtain a transformation matrix between two images with overlapping areas in all images, and to obtain a projection transformation relationship between all remaining images and the reference image;

[0010] S130, optimizing the intrinsic parameters and the extrinsic parameters using the bundle adjustment method to obtain calibrated camera parameters; and

[0011] S140, using an improved graph cutting algorithm to perform stitching and fusion processing on the multiple images to generate a panoramic fusion image of the sea surface.

[0012] This application uses the light speed adjustment method to collaboratively optimize the calibration point residuals obtained by multi-source sensors, obtain the calibrated camera parameters, and constrain the overall geometric structure of the camera. The improved graph cut algorithm can be applied to the sea surface stitching application scenario and can realize the fusion of sea surface panoramic images.

[0013] In some embodiments, in step S140, an improved graph cut algorithm is used to perform splicing and fusion processing on multiple images, and the data item energy loss function E in the improved graph cut algorithm is d As shown below:

[0014]

[0015] Among them, l p represents the label value at pixel p, According to p l The value of determines the different endpoints, α is the weight coefficient, and σ is the standard deviation of the Gaussian distance.

[0016] The improved graph cut algorithm uses the Gaussian distance of image coordinates and pixel gradient to jointly constrain the classification results of graph cut, which helps to find the segmentation line to separate the overlapping areas more evenly. The segmentation line tends to be perpendicular to the direction of camera movement, which is more suitable for the stitching task of sea surface videos.

[0017] In some embodiments, in step S120, a direct linear transformation algorithm is used to obtain images I having overlapping areas in all images. i and image I j The transformation matrix H between ij , and obtain the projection transformation relationship between all remaining images and the reference image Where K represents a total of K images, and the projection transformation relationship includes intrinsic parameters and extrinsic parameters.

[0018] This operation can be used to obtain the initial values ​​of the intrinsic and extrinsic parameters of the camera device, so that the accurate values ​​of the intrinsic and extrinsic parameters can be obtained later through the adjustment optimization method.

[0019] In some embodiments, in step S120, the kth calibration point is obtained from the image Ii Transform to image I i The resulting error is expressed by the following formula:

[0020]

[0021] The error value of the feature points between all images is calculated by the following formula:

[0022]

[0023] Among them, I(i) represents all images with image I i A set of images with overlapping areas, K(i,j) represents image I i and image I j All feature points with the same name.

[0024] In some embodiments, in step S130, the nonlinear least squares algorithm Levenberg-Marquardt is used to calculate the error value to obtain the projection transformation relationship. The parameter matrix with the minimum value The smallest parameter matrix is ​​used as the calibrated camera parameters.

[0025] This operation can obtain the transformation matrix corrected by the global bundle adjustment method, thereby constraining the overall geometric structure of the camera.

[0026] In some embodiments, in step S140, the plurality of images are stitched and fused using an improved graph cut algorithm, further comprising:

[0027] The label value of each pixel p of any pair of images I0 and I1 with overlapping areas is marked as l p , where the label value l∈{0,1}, determines whether the pixel p satisfies the following formula to minimize the energy function E:

[0028]

[0029] If the above formula is satisfied and the pixel with the final label value of 1 appears in the panoramic fusion image of the sea surface;

[0030] Among them, Ed indicates that the label value is l p Es represents the smoothing term between all neighboring pixels associated with pixel p, λ is an important factor between the data term and the smoothing term Es, and p and q represent different pixels.

[0031] The graph cut algorithm is used to obtain the stitching seams of the projection transformation, and the best stitching seams are selected for image stitching and fusion.

[0032] In some embodiments, in step S140, the formula of the smoothing term Es is:

[0033] E s (p,l p ,q,l q )=|l p -l q |(D(p)+D(q))

[0034] If l p = l q , then the smoothing term Es is equal to 0, if l p Not equal to l q , then the size of the smoothing term is determined by the size of the formula D, where D is:

[0035]

[0036]

[0037] Among them, l p is the label value of pixel p, l q is the label value of pixel q, D represents the sum of the differences of the image in the overlapping area being labeled with different label values, α is an important factor, α=2.

[0038] In some embodiments, acquiring multiple images and extracting calibration points of the images, and calculating the intrinsic parameters and extrinsic parameters of the camera include the following steps:

[0039] S111, obtaining checkerboard images for camera calibration taken from multiple viewpoints;

[0040] S112, extracting corner points from the chessboard calibration object image captured by the camera;

[0041] S113. Calculate the intrinsic parameters and extrinsic parameters of the camera according to the acquired corner point information.

[0042] This operation can quickly obtain the intrinsic and extrinsic parameters of the camera for subsequent operations.

[0043] In a second aspect, the present application provides a wide-area ocean panoramic fusion system based on multi-source data, the system comprising the following modules:

[0044] an acquisition module configured to acquire a plurality of images and extract calibration points of the images, calculate intrinsic parameters and extrinsic parameters of the camera, select a reference image from the images using a minimum spanning tree method, and sort the remaining images; and

[0045] a projection module configured to obtain a transformation matrix between two images having overlapping areas among all the images using a direct linear transformation algorithm, and to obtain a projection transformation relationship between all the remaining images and the reference image; and

[0046] an optimization module configured to optimize the intrinsic parameters and extrinsic parameters using a bundle adjustment method to obtain calibrated camera parameters; and

[0047] The stitching module is configured to use an improved graph cutting algorithm to stitch and fuse the multiple images to generate a panoramic fused image.

[0048] In a third aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, any of the above methods is implemented.

[0049] The embodiment of the present application provides a wide-area ocean panoramic fusion method and system based on multi-source data, which corrects the transformation matrix between images with overlapping areas through the global bundle adjustment method to constrain the overall geometric structure of the camera; and adopts an improved graph cut algorithm to jointly constrain the classification results of the graph cut through the Gaussian distance and pixel gradient of the image coordinates to help find the best stitching seam, so that the stitching seam tends to be perpendicular to the direction of the camera device, thereby meeting the stitching of the sea surface image. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0051] Figure 1 is a flow chart of a wide-area ocean panorama fusion method based on multi-source data according to the present application;

[0052] Figure 2 is a schematic diagram of a wide-area ocean panoramic fusion system based on multi-source data according to the present application;

[0053] Figure 3 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0054] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0055] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0056] Figure 1 The flowchart of the wide-area ocean panorama fusion method based on multi-source data of the present application is shown as follows: Figure 1 As shown, the method 100 comprises the following steps:

[0057] S110, acquiring multiple images and extracting calibration points of the images, calculating the intrinsic parameters and extrinsic parameters of the camera, selecting a reference image from the images using a minimum spanning tree method, and sorting the remaining images.

[0058] In this embodiment, the reference image I can be selected using the minimum spanning tree method according to the geometric topological relationship between the images. R , and sort the remaining input images. The geometric topological relationship between images can be roughly estimated using the phase correlation method to obtain translation and rotation parameters. The phase correlation method is a frequency-domain-based image registration algorithm with extremely high computational efficiency.

[0059] In this embodiment, acquiring multiple images and extracting calibration points of the images includes: obtaining an image set I through a set of sensors loaded by a given camera device (such as a visible light camera, a thermal imaging camera) k {k=1,...,K}, extract the calibration points of these K images (such as chessboard corner point information), and then calculate the intrinsic parameters and extrinsic parameters of the camera. Specifically, it also includes the following sub-steps:

[0060] S111, obtaining checkerboard images for camera calibration taken from multiple viewpoints, wherein the calibration object is a checkered series film calibration plate, the camera is fixed at a suitable angle, and the checkerboard calibration object image is taken, and the direction of the calibration object is adjusted to take multiple photos in different directions with the camera.

[0061] S112, extracting corner points of the chessboard calibration object image taken by the camera, wherein the extraction can be automatic or the user can be allowed to perform interactive chessboard corner point extraction.

[0062] S113, calculating the intrinsic parameters and extrinsic parameters of the camera according to the acquired corner point information, wherein the calculation can be performed quickly through the MATLAB calibration toolbox.

[0063] Continue to refer Figure 1 , the method 100 further includes:

[0064] S120, using a direct linear transformation algorithm to obtain a transformation matrix between two images with overlapping areas in all images, and to obtain a projection transformation relationship between all remaining images and the reference image;

[0065] S130, optimizing the intrinsic parameters and the extrinsic parameters using the bundle adjustment method to obtain calibrated camera parameters;

[0066] In this embodiment, in step S120, a direct linear transformation algorithm (DLT) is used to calculate and obtain the image I having the overlapping area in all images. i and image I j The transformation matrix H between ij , and obtain the projection transformation relationship between all remaining images and the reference image Wherein, K represents a total of K images acquired, and the projection transformation relationship includes intrinsic parameters and extrinsic parameters.

[0067] In step S120, further, the kth calibration point is obtained from image I i Transform to image I i The resulting error is expressed by the following formula:

[0068]

[0069] The error value of the feature points between all images is calculated by the following formula:

[0070]

[0071] Among them, I(i) represents all images with image I i A set of images with overlapping areas, K(i,j) represents image I i and image I j All feature points with the same name.

[0072] Further, in step S130, the nonlinear least squares algorithm Levenberg-Marquardt can be used to calculate the error value, and finally the projection transformation relationship is obtained. The parameter matrix with the minimum value The minimum parameter matrix is ​​used as the calibrated camera parameters.

[0073] The use of direct linear transformation algorithm for calculation has the following significant characteristics: first, the image space coordinates are directly transformed into the object space coordinates, and the initial values ​​of the internal and external orientation elements are not required; second, the original observed image point coordinates are directly used to effectively compensate for system errors. The main differences from conventional camera calibration are: considering the impact of electrical errors on calibration; the algorithm requires fast real-time and stable and reliable.

[0074] The calibration of camera parameters is the core step of panoramic image stitching, and its accuracy has a crucial impact on the fusion effect of wide-area sea surface images. Moreover, the target detection module needs to fuse the effects of visible light and thermal imaging. Therefore, the visible light camera and thermal imaging camera loaded on the device must be jointly calibrated. Since the bundle adjustment method can collaboratively optimize the residuals of the marker points obtained by multi-source sensors during the calibration process. Therefore, the present application adopts bundle adjustment to realize the calibration of the camera. Through the operation of steps S110-S130, the transformation matrix corrected by the global bundle adjustment method can be obtained, thereby constraining the overall geometric structure of the camera.

[0075] Continue to refer Figure 1 , the method 100 further includes:

[0076] S140, using an improved graph cutting algorithm to perform stitching and fusion processing on the multiple images to generate a panoramic fusion image of the sea surface.

[0077] In this embodiment, an improved graph cutting algorithm is used to perform splicing and fusion processing on multiple images, which specifically includes:

[0078] The label value of each pixel p of any pair of images I0 and I1 with overlapping areas is marked as l p , where the label value l∈{0,1}, determines whether the pixel p satisfies the following formula to minimize the energy function E:

[0079]

[0080] If the above formula is satisfied and the pixel with the final label value of 1 appears in the panoramic fusion image of the sea surface;

[0081] Among them, E d Indicates that the label value is l p The energy loss function of pixel p on the data item, Es represents the smoothing term between all neighboring pixels associated with pixel p. Usually, four-neighborhood connection is used. The smoothing term Es is the loss function at the seam of the overlapping area of ​​two images. λ is an important factor between the data item and the smoothing term Es. p and q represent different pixels.

[0082] Among them, the smoothing term Es represents the penalty for the discontinuity between pixel p and other nodes q in the domain. The formula of the smoothing term Es is as follows:

[0083] E s (p,l p ,q,l q )=|l p -l q |(D(p)+D(q))

[0084] If l p = lq , then the smoothing term Es is equal to 0, if l p Not equal to l q , then the size of the smoothing term is determined by the size of the formula D, where D is:

[0085]

[0086]

[0087] Among them, l p is the label value of pixel p, l q is the label value of pixel q, D represents the sum of the differences of the images in the overlapping area being labeled with different label values, α is an important factor that determines the influence of the two types of loss functions on the ability, α = 2.

[0088] In the embodiment of the present application, the sea surface image has only horizontal or vertical translation transformation, so the position of the endpoint is dynamically selected according to the translation direction between image frames. The midpoints of the two boundaries perpendicular to the displacement direction in the image are selected as the two endpoints of the graph cut model. Therefore, the data item energy loss function in the traditional graph cut algorithm is improved. Specifically, the data item energy loss function E in the improved graph cut algorithm is d As shown below:

[0089]

[0090] Among them, l p represents the label value at pixel p, According to p l The values ​​determine different endpoints, α is the weight coefficient, and σ is the standard deviation of the Gaussian distance. p =0, then the gradient of pixel p in image I0 is used. Otherwise, the gradient of image I1 is selected, that is, the gradient value of pixel p in the two images is compared. The larger the gradient, the smaller the loss function, and the greater the probability that the edge is cut.

[0091] in, According to p l The value of determines different endpoints, p0 is the S endpoint, p1 is the T endpoint, α is a weight coefficient, and σ is the standard deviation of the Gaussian distance. Image Segmentation (Graph Cuts) The graph has two more vertices on the basis of the ordinary graph. These two vertices are represented by the symbols "S" and "T" respectively, and are called terminal vertices. The improved graph cut algorithm uses the Gaussian distance of the image coordinates and the pixel gradient to jointly constrain the classification results of the graph cut. It helps to find the segmentation line to separate the overlapping areas more evenly, and the segmentation line tends to be perpendicular to the direction of camera movement, which is more suitable for the stitching task of sea surface videos.

[0092] This application uses the light speed adjustment method to collaboratively optimize the calibration point residuals obtained by multi-source sensors, obtain the calibrated camera parameters, and constrain the overall geometric structure of the camera. The improved graph cut algorithm is used to find the best segmentation line, which is suitable for sea surface splicing application scenarios and realizes the fusion of sea surface panoramic images.

[0093] Further references Figure 2 As an implementation of the above method, the present application provides an embodiment of a wide-area ocean panoramic fusion system based on multi-source data. Figure 1 Corresponding to the method embodiment shown, the system can be specifically applied to various electronic devices. The system 200 includes:

[0094] An acquisition module 210 configured to acquire multiple images and extract calibration points of the images, calculate intrinsic parameters and extrinsic parameters of the camera, select a reference image from the images using a minimum spanning tree method, and sort the remaining images; and

[0095] A projection module 220, configured to obtain a transformation matrix between two images having overlapping areas in all the images using a direct linear transformation algorithm, and to obtain a projection transformation relationship between all the remaining images and the reference image; and

[0096] An optimization module 230, configured to optimize the intrinsic parameters and extrinsic parameters using a bundle adjustment method to obtain calibrated camera parameters; and

[0097] The stitching module 240 is configured to use an improved graph cut algorithm to stitch and fuse the multiple images to generate a panoramic fused image.

[0098] In the embodiment of the present application, the system takes the sea surface environment around uninhabited islands and reefs and the monitoring needs of intrusion targets as the starting point. The system can simultaneously collect images of different depths of field through multiple image acquisition devices in the vertical direction, and seamlessly stitch and fuse them into a panoramic image. The system has the characteristics of intuitive imaging, real-time warning, and stable performance. The image acquisition device rotates coaxially at a certain speed and scans one circle. One device scanning cycle is the imaging cycle of a panoramic image. The cycle time can be set, generally less than 100s. The image acquisition radius is 3km and the scanning area is 25km. 2 The system specifically includes a multi-source sensor joint calibration system and a panoramic image fusion system based on fast graph cutting. It can be widely used in unmanned wide-area sea surfaces for border defense, coastal defense, and islands and reefs, and can achieve real-time wide-area panoramic and intrusion target detection, promoting the innovation of intelligent monitoring technology for unmanned islands and reefs.

[0099] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any of the above methods is implemented.

[0100] like Figure 3 As shown, the computer system 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage part 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the system 300 are also stored. The CPU 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0101] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed, so that a computer program read therefrom is installed into the storage section 308 as needed.

[0102] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 309, and / or installed from the removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable medium described in the present application can be a computer-readable signal medium or a computer-readable medium or any combination of the above two. The computer-readable medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus or a device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0103] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0104] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] The modules involved in the embodiments described in the present application may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes an acquisition module, an analysis module, and an output module. The names of these modules do not constitute limitations on the modules themselves in some cases.

[0106] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A wide-area ocean panorama fusion method based on multi-source data, characterized in that: The following steps are involved: S110, acquiring multiple images and extracting calibration points of the images, calculating intrinsic parameters and extrinsic parameters of the camera, selecting a reference image from the images using a minimum spanning tree method, and sorting the remaining images; S120, using a direct linear transformation algorithm to obtain a transformation matrix between two images having overlapping areas in all the images, and to obtain a projection transformation relationship between all the remaining images and the reference image; S130, optimizing the intrinsic parameters and extrinsic parameters by using a bundle adjustment method to obtain calibrated camera parameters; as well as S140, using an improved graph cutting algorithm to perform splicing and fusion processing on the multiple images to generate a panoramic fused image of the sea surface; The improved graph cutting algorithm is used to perform splicing and fusion processing on the multiple images. The data item energy loss function E in the improved graph cutting algorithm is d As shown below: Among them, l p Represents the label value at pixel p, p lp According to p l The value of determines different endpoints, α is the weight coefficient, and σ is the standard deviation of the Gaussian distance; The improved graph cutting algorithm is used to perform splicing and fusion processing on the multiple images, and also includes: The label value of each pixel p of any pair of images I0 and I1 with overlapping areas is marked as l p , where the label value l∈{0,1}, determines whether the pixel p satisfies the following formula for minimizing the energy function E: If the above formula is satisfied and the pixel point with the final label value of 1 appears in the sea surface panoramic fusion image; Among them, E d Indicates that the label value is l p The energy loss function of pixel p on the data item, E s represents the smoothness term between all neighboring pixels associated with pixel p, and λ is the data term and the smoothness term E s The important factors between them, p and q represent different pixels.

2. The wide-area ocean panorama fusion method based on multi-source data according to claim 1, characterized in that: In step S120, a direct linear transformation algorithm is used to obtain images I having overlapping areas in all the images. i and image I j The transformation matrix H between ij , and obtaining the projection transformation relationship between all the remaining images and the reference image Wherein K represents a total of K images, and the projection transformation relationship includes the intrinsic parameters and the extrinsic parameters.

3. The wide-area ocean panorama fusion method based on multi-source data according to claim 2 is characterized in that: In step S120, the kth calibration point is obtained from image I i Transform to image I i The resulting error is expressed by the following formula: The error value of the feature points between all images is calculated by the following formula: Among them, I(i) represents all images with image I i A set of images with overlapping areas, K(i,j) represents image I i and image I j All feature points with the same name.

4. The wide-area ocean panorama fusion method based on multi-source data according to claim 3 is characterized in that: In step S130, the nonlinear least squares algorithm Levenberg-Marquardt is used to calculate the error value to obtain the projection transformation relationship The parameter matrix with the minimum value The minimum parameter matrix is ​​used as the calibrated camera parameters.

5. The wide-area ocean panorama fusion method based on multi-source data according to claim 1, characterized in that: In step S140, the formula of the smoothing term Es is: E s (p,l p ,q,l q )=|l p -l q |·(D(p)+D(q)) If l p = l q , then the smoothing term E s is equal to 0, if l p Not equal to l q , then the size of the smoothing term is determined by the size of the formula D, where D is: Among them, l p is the label value of pixel p, l q is the label value of pixel q, D represents the sum of the differences of the image in the overlapping area being labeled with different label values, α is an important factor, α=2.

6. The wide-area ocean panorama fusion method based on multi-source data according to claim 1, characterized in that: In step S110, a plurality of images are acquired and calibration points of the images are extracted to calculate the intrinsic parameters and extrinsic parameters of the camera, including the following steps: S111, obtaining checkerboard images for camera calibration taken from multiple viewpoints; S112, extracting corner points from the chessboard calibration object image captured by the camera; S113. Calculate the intrinsic parameters and extrinsic parameters of the camera according to the acquired corner point information.

7. A wide-area ocean panoramic fusion system based on multi-source data, characterized in that: Includes the following modules: An acquisition module configured to acquire multiple images and extract calibration points of the images, calculate intrinsic parameters and extrinsic parameters of the camera, select a reference image from the images using a minimum spanning tree method, and sort the remaining images; as well as A projection module, configured to obtain a transformation matrix between two images having overlapping areas in all the images using a direct linear transformation algorithm, and to obtain a projection transformation relationship between all the remaining images and the reference image; as well as An optimization module configured to optimize the intrinsic parameters and extrinsic parameters using a bundle adjustment method to obtain calibrated camera parameters; as well as A stitching module is configured to stitch and fuse the multiple images using an improved graph cut algorithm to generate a panoramic fused image of the sea surface; The improved graph cutting algorithm is used to perform splicing and fusion processing on the multiple images. The data item energy loss function E in the improved graph cutting algorithm is d As shown below: Among them, l p Represents the label value at pixel p, p lp According to p l The value of determines different endpoints, α is the weight coefficient, and σ is the standard deviation of the Gaussian distance; The improved graph cutting algorithm is used to perform splicing and fusion processing on the multiple images, and also includes: The label value of each pixel p of any pair of images I0 and I1 with overlapping areas is marked as l p , where the label value l∈{0,1}, determines whether the pixel p satisfies the following formula for minimizing the energy function E: If the above formula is satisfied and the pixel point with the final label value of 1 appears in the sea surface panoramic fusion image; Among them, E d Indicates that the label value is l p The energy loss function of pixel p on the data item, E s represents the smoothness term between all neighboring pixels associated with pixel p, and λ is the data term and the smoothness term E s The important factors between them, p and q represent different pixels.

8. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 6 when executed by a processor.

Images