A method and system for consistent calibration of internal and external parameters of underwater cameras and array image stitching

Abstract

The invention discloses a method and system for consistent calibration of internal and external parameters of an underwater camera and array image stitching, and relates to the technical field of underwater structure detection, comprising: based on a feature detection mechanism of a SURF algorithm, using a multi-scale identifiable speckle calibration pattern to consistently calibrate the internal and external parameters of a camera; using the calibrated internal and external parameters of the camera to correct the optical center and optical axis of an image, obtaining a pixel translation value of the array image, and realizing accurate correction of the optical center and optical axis of the image; determining a maximum inscribed rectangle according to the obtained pixel translation value, and calculating the camera index value and pixel index value of all points in the rectangle; inputting the collected underwater structure surface image into an image stitching algorithm, and outputting a stitching result of the underwater structure surface array image. Compared with a traditional artificial underwater exploration method, the invention can realize rapid and accurate detection of underwater structure surfaces, not only improving detection accuracy, but also improving detection efficiency and saving human resources.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater structure detection, and in particular to a method and system for calibrating consistent internal and external parameters of an underwater camera and stitching array images. Background Art

[0002] China has more than 1 million bridges, and their safe operation and maintenance are basic requirements. Currently, structural inspections focus more on structures exposed to the air and ignore hidden underwater structures. Existing underwater structure inspections mainly rely on divers carrying more than 100 kilograms of equipment to explore underwater structures. Then the inspectors take pictures and observe the surface of the underwater structures to determine the type and severity of defects. This method is limited by underwater lighting conditions and the capabilities of human divers, relies on the subjectivity of inspectors, and has a low accuracy rate. In addition, manual inspections require divers to explore underwater structural defects in turbulent waters, which has a low degree of intelligence, a long inspection time, high costs, and high risks. Summary of the invention

[0003] In order to solve the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide a method and system for consistent calibration of internal and external parameters of an underwater camera and array image stitching, which can realize rapid image acquisition and accurate image stitching of structural underwater defects.

[0004] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a method for calibrating consistent internal and external parameters of an underwater camera and stitching array images, the method comprising the following steps:

[0005] Receiving a multi-scale recognizable speckle calibration pattern, and using the multi-scale recognizable speckle calibration pattern to consistently calibrate the internal and external parameters of a camera to obtain a calibrated camera, wherein the multi-scale recognizable speckle calibration pattern is generated based on a feature detection mechanism of a SURF algorithm;

[0006] Receiving an underwater array image, and correcting the optical center and optical axis of the underwater array image using the calibrated camera internal and external parameters to obtain a pixel translation value of the underwater array image;

[0007] According to the pixel translation value of the underwater array image, the maximum inscribed rectangle is determined, and the camera index value and the pixel index value of all the points in the maximum inscribed rectangle are calculated; according to the camera index value and the pixel index value of all the points in the maximum inscribed rectangle, the nearest point interpolation is performed on the underwater array image to obtain the mapping relationship between the calculated pixel point and the camera pixel point;

[0008] Based on the mapping relationship between calculated pixels and camera pixels, the corrected underwater structure surface array image is input into a pre-established image stitching algorithm model, and the stitching result of the underwater structure surface array image is output.

[0009] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the multi-scale identifiable speckle calibration pattern is generated based on the reverse engineering of the scaling process in SURF detection, and based on the reversed SURF scaling process, multi-scale noise images of the same size are used to synthesize the calibration pattern, allowing the camera to acquire and detect features at different distances, and extracting features of the image detected by the camera and the calibration pattern at the same time.

[0010] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the use of a multi-scale identifiable speckle calibration pattern to consistently calibrate the intrinsic and extrinsic parameters of the camera includes: speckle pattern generation, calibration plate design, and consistent calibration of the intrinsic and extrinsic parameters of the camera.

[0011] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: a process for correcting the optical center and optical axis of the underwater array image: calculating the image homography matrix and optical center coordinates respectively by using the intrinsic and extrinsic parameters of the camera calibration, performing optical axis correction on each image using the homography matrix, making all corrected optical axes parallel, calculating the distance from the corrected optical center to the surface of the structure by using the z coordinates in all calculated optical center coordinates, correcting the optical centers of all cameras to be on the same plane, and finally calculating the pixel translation value between the camera principal points based on the physical size distance between the optical centers for subsequent array camera image stitching.

[0012] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: the underwater array image acquisition device is an underwater planar array camera, and a camera optical method is used to obtain an array two-dimensional image of the underwater structure and its surface as the underwater array image.

[0013] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of determining a maximum inscribed rectangle according to a pixel translation value of the underwater array image:

[0014] After obtaining the pixel translation value of each underwater array image, the array camera image is translated according to the pixel translation value of the underwater array image, and the maximum inscribed rectangle of the area occupied by the array camera image is calculated.

[0015] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of calculating the camera index values ​​and pixel index values ​​of all points in the maximum inscribed rectangle:

[0016] For all pixel coordinates in the maximum inscribed rectangle ( x , y ) to the camera's main point C i Distance for:

[0017]

[0018] in, C ix and C iy Camera C i Ideal camera principal point x and y Coordinates, then ( x , y ) camera index value c ( x , y , C i )for:

[0019]

[0020] Among them, argmin means finding the distance Minimum camera index i ;

[0021] For determining the camera C i The pixel point ( x , y ), using the nearest neighbor interpolation method to calculate its position in the camera C i The pixel position in the image ( x’ , y’ ):

[0022]

[0023]

[0024] in, round The function is a rounding function.

[0025] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a calculation process of inputting the corrected underwater structure surface array image into a pre-established image stitching algorithm model and outputting a stitching result of the underwater structure surface array image:

[0026] For all pixels of the image points in the maximum inscribed rectangle Pixel ( x , y ) have:

[0027]

[0028] in i is the camera index value of the image point, (x’ , y’ ) is the pixel index value of the image point.

[0029] In a second aspect, in order to achieve the above-mentioned purpose, the present invention discloses an underwater camera internal and external parameter consistent calibration and array image stitching system, comprising:

[0030] A camera calibration module, used for receiving a multi-scale recognizable speckle calibration pattern, and using the multi-scale recognizable speckle calibration pattern to consistently calibrate the internal and external parameters of the camera to obtain a calibrated camera, wherein the multi-scale recognizable speckle calibration pattern is generated based on a feature detection mechanism of a SURF algorithm;

[0031] The image correction module is used to receive underwater array images, correct the optical center and optical axis of the underwater array images with the calibrated camera internal and external parameters, and use the homography matrix to correct the optical axis of each image. After correction, all optical axes are parallel. The distance from the corrected optical center to the surface of the structure is calculated by calculating the z coordinate of all optical center coordinates, and the optical centers of all cameras are corrected to be on the same plane. Finally, the pixel translation value between the camera principal points is calculated according to the physical size distance between the optical centers, so as to facilitate the subsequent array camera image stitching.

[0032] The image mapping module is used to determine the maximum inscribed rectangle according to the pixel translation value of the underwater array image, calculate the camera index value and the pixel index value of all points in the maximum inscribed rectangle; perform the nearest point interpolation on the underwater array image according to the camera index value and the pixel index value of all points in the maximum inscribed rectangle, and obtain the mapping relationship between the calculated pixel point and the camera pixel point;

[0033] The image stitching module is used to input the corrected underwater structure surface array image into a pre-established image stitching algorithm model based on the mapping relationship between the calculated pixels and the camera pixels, and output the stitching result of the underwater structure surface array image.

[0034] In conjunction with the second aspect, in certain implementations of the second aspect, the system further includes: the camera calibration module includes a multi-scale recognizable speckle calibration pattern generated based on the reverse engineering of the scaling process in SURF detection, and based on the reversed SURF scaling process, a multi-scale noise image of the same size is used to synthesize the calibration pattern, allowing the camera to acquire and detect features at different distances, and extracting features of the image detected by the camera and the calibration pattern at the same time;

[0035] The camera calibration module uses a multi-scale identifiable speckle calibration pattern to calibrate the camera's internal and external parameters consistently, including: speckle pattern generation, calibration plate design, and consistent calibration of the camera's internal and external parameters;

[0036] The image correction module corrects the optical center and optical axis of the underwater array image: by using the intrinsic and extrinsic parameters of the camera calibration to calculate the image homography matrix and the optical center coordinates, the pixel translation value of the image is further calculated;

[0037] The underwater array image acquisition device of the image correction module is an underwater planar array camera, which uses camera optics to obtain an array two-dimensional image of the underwater structure and its surface as the underwater array image;

[0038] The image mapping module determines the process of the maximum inscribed rectangle based on the pixel translation value of the underwater array image:

[0039] After obtaining the pixel translation value of each underwater array image, the array camera image is translated according to the pixel translation value of the underwater array image, and the maximum inscribed rectangle of the area occupied by the array camera image is calculated;

[0040] The image mapping module calculates the camera index value and pixel index value of all points in the maximum inscribed rectangle:

[0041] For all pixel coordinates in the maximum inscribed rectangle ( x , y ) to the camera's main point C i Distance for:

[0042]

[0043] in, C ix and C iy Camera C i Ideal camera principal point x and y Coordinates, then ( x , y ) camera index value c ( x , y , C i )for:

[0044]

[0045] Among them, argmin means finding the distance Minimum camera index i ;

[0046] For determining the camera C i The pixel point ( x , y), using the nearest neighbor interpolation method to calculate its position in the camera C i The pixel position in the image ( x’ , y’ ):

[0047]

[0048]

[0049] in, round The function is a rounding function.

[0050] The image stitching module inputs the corrected underwater structure surface array image into the pre-established image stitching algorithm model and outputs the calculation process of the stitching result of the underwater structure surface array image:

[0051] For all pixels of the image points in the maximum inscribed rectangle Pixel ( x , y ) have:

[0052]

[0053] in i is the camera index value of the image point, ( x’ , y’ ) is the pixel index value of the image point.

[0054] Beneficial effects of the present invention:

[0055] The present invention proposes a model for calibrating the internal and external parameters of a camera based on a speckle calibration pattern, which can accurately and consistently calibrate the internal and external parameters of an array camera and export them at one time, thereby improving the convenience of camera calibration; the present invention proposes a model for rapid stitching and detection of underwater array images, which improves the efficiency of optical detection of underwater structures and the accuracy of image stitching; the present invention uses an array camera to detect underwater structures, which can improve detection efficiency compared to traditional manual underwater exploration. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0057] Figure 1 It is a schematic flow chart of the method of the present invention;

[0058] Figure 2 It is a flow chart of calibrating camera internal and external parameters based on speckle calibration pattern in the present invention;

[0059] Figure 3 This is a flow chart of underwater array image optical center and optical axis correction in the present invention;

[0060] Figure 4 This is a flowchart of the rapid splicing of underwater array images of the present invention;

[0061] Figure 5 This is a result diagram of the rapid stitching of underwater array images according to the present invention;

[0062] Figure 6 It is a schematic diagram of the system structure of the present invention;

[0063] Figure 7 It is an implementation diagram of a practical verification case of the present invention;

[0064] Figure 8 It is a schematic diagram of the speckle calibration pattern and calibration image actually used in the present invention. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] Embodiment 1:

[0067] The following is an introduction to the relevant terms involved in the embodiments of the present application:

[0068] SURF algorithm introduction:

[0069] SURF uses the determinant value of the Hessian matrix for feature point response detection and uses integral images to accelerate the calculation; SURF's descriptor is based on the 2D discrete wavelet transform response Haar wavelet and effectively utilizes the integral image.

[0070] like Figure 1 As shown, a method for calibrating consistent internal and external parameters of an underwater camera and stitching array images comprises the following steps:

[0071] S101: receiving a multi-scale recognizable speckle calibration pattern, and using the multi-scale recognizable speckle calibration pattern to consistently calibrate the internal and external parameters of a camera to obtain a calibrated camera, wherein the multi-scale recognizable speckle calibration pattern is generated based on a feature detection mechanism of a SURF algorithm;

[0072] By generating noise images of different sizes and resizing them to have the same size, images with noise of different scales are added together to form detectable features with a large number of different scales;

[0073] The specific underwater array image acquisition device is an underwater planar array camera, which uses camera optics to obtain array two-dimensional images of underwater structures and their surfaces;

[0074] For the technical solution of this embodiment, Figure 2 As shown in the figure, the flow chart of calibrating camera internal and external parameters based on speckle calibration pattern is mainly used to calibrate the internal and external parameters of the six cameras of the array once; the main process includes speckle pattern generation, calibration plate design and consistent calibration of camera internal and external parameters, speckle pattern generation relies on the inverse process of SURF detection, synthesizes multi-scale noise images to generate multi-scale recognizable speckle images as calibration patterns, adjusts the size of the calibration pattern so that it is combined with the rigid body as a calibration plate for camera calibration, and then uses the array camera to collect calibration images of the calibration plate. For the internal parameters of a single camera, the calibration image is feature matched after SURF feature detection, and then the distortion parameters are calculated, and finally the camera internal parameters are derived. For the external parameters of multiple cameras, the calculated camera internal parameters are used to extract the position of the initial camera, optimize the camera external parameters, and finally the camera external parameters are derived after verification and calibration, which is used to accurately derive the internal and external parameters of six cameras at one time;

[0075] Based on the reverse engineering of the scaling process in SURF detection, the generated speckle pattern contains multi-scale identifiable features, and the camera internal and external parameters are consistently calibrated: based on the inverted SURF scaling process, multi-scale noise images of the same size are used to synthesize the calibration pattern, allowing the camera to acquire and detect features at different distances, and at the same time extract the features of the image detected by the camera and the calibration pattern. The distance ratio inspection method is used to match the features according to the similarity of the feature descriptors. The basic matrix of the single-camera intrinsic parameter calibration is first performed. The multi-camera extrinsic parameter calibration rotation matrix and translation matrix provided by the single-camera calibration can improve the simplicity and accuracy of the camera internal and external parameter calibration.

[0076] S102: receiving an underwater array image, and correcting the optical center and optical axis of the underwater array image using the calibrated camera internal and external parameters to obtain a pixel translation value of the underwater array image;

[0077] For the technical solution of this embodiment, Figure 3 As shown in the figure, the optical center and optical axis correction flow chart of underwater structure array image: First, import the internal reference Ai and external reference Ri, Ti calibrated by the array camera. In the process of optical axis correction, use the projection relationship between world coordinates and image coordinates, assume that the optical center position remains unchanged before and after image correction, and determine the image homography matrix ,in The ideal intrinsic parameter matrix of the camera, It is a unit matrix, which means that the optical axis of the camera after correction is perpendicular to the plane of the calibration plate. The optical axis correction is performed on each image to solve the optical axis correction problem. In the optical center correction process, it is assumed that the optical center position changes before and after the image correction. After the optical axis correction is completed, the optical axes of the cameras are parallel. The optical center coordinates are determined. , to solve the optical center correction problem; after the optical axis correction and the optical center position correction, the physical distance between the optical centers is calculated, and the pixel translation between the camera principal points is calculated and output. The pixel translation value is relative to the first camera, which is convenient for the subsequent fast image stitching.

[0078] S103: determining a maximum inscribed rectangle according to the pixel translation value of the underwater array image, and calculating the camera index value and the pixel index value of all points in the maximum inscribed rectangle; performing nearest point interpolation on the underwater array image according to the camera index value and the pixel index value of all points in the maximum inscribed rectangle, and obtaining a mapping relationship between the calculated pixel point and the camera pixel point;

[0079] After obtaining the image pixel translation value of each image, the array image is translated according to the translation value, the maximum inscribed rectangle of the area occupied by the array camera image is calculated, the camera index value of all pixels in the maximum inscribed rectangle is calculated, and the pixel index value of the camera image point corresponding to all pixels in the maximum inscribed rectangle is calculated. Combined with the underwater structure surface image collected by the planar array camera during underwater operation, the final stitching image of the array camera is obtained to restore the underwater structure surface.

[0080] The camera is specifically a 3×2 planar array camera equipped with an LED light source and a working distance of about 160mm;

[0081] The calculation process of calculating the camera index value and pixel index value of all points in the maximum inscribed rectangle is as follows:

[0082] For all pixel coordinates in the maximum inscribed rectangle ( x , y ) to the camera's main point C i Distance for:

[0083]

[0084] in, C ix and C iy Camera C i Ideal camera principal point x and y Coordinates, then ( x , y ) camera index value c (x , y , C i )for:

[0085]

[0086] Among them, argmin means finding the distance Minimum camera index i ;

[0087] For determining the camera C i The pixel point ( x , y ), using the nearest neighbor interpolation method to calculate its position in the camera C i The pixel position in the image ( x’ , y’ ):

[0088]

[0089]

[0090] in, round The function is a rounding function, which is used to round the coordinates to the nearest integer.

[0091] S104: Based on the mapping relationship between the calculated pixels and the camera pixels, the corrected underwater structure surface array image is input into a pre-established image stitching algorithm model, and the stitching result of the underwater structure surface array image is output.

[0092] The corrected underwater structure surface array image is input into the pre-established image stitching algorithm model, and the calculation process of the stitching result of the underwater structure surface array image is output:

[0093] For all pixels of the image points in the maximum inscribed rectangle Pixel ( x , y ) have:

[0094]

[0095] in i is the camera index value of the image point, ( x’ , y’ ) is the pixel index value of the image point.

[0096] For the technical solution of this embodiment, Figure 4As shown, the flowchart of rapid stitching of structured underwater array images, multiple real-time images taken by the array camera need to be stitched to obtain a stitched image of the camera array. First, the largest image rectangle is determined according to the resolution of the image, and then the image pixels are translated using the derived image pixel translation value. Then, the maximum inscribed rectangle is calculated by searching the boundaries around the center point. The next step is to calculate the camera index value, traverse all the pixels of the maximum inscribed rectangle, and judge it. If the pixel point only exists in one camera, the camera index value is the camera number. If the pixel point exists in multiple cameras, it is necessary to calculate the distance between the point and the ideal principal points of multiple cameras. The one with a closer distance is the camera index value of the point, and the pixel index value of all pixels in the maximum inscribed rectangle is derived. The last step is to calculate the pixel index value and perform image stitching. Input the original stitched image, use the camera index value of all pixels in the maximum inscribed rectangle to search for the corresponding stitched image, calculate the pixel index value of the corresponding pixel of the original image by interpolation of adjacent points, and establish a one-to-one correspondence between the pixel position in each original camera image and the pixel position in the stitched image. Finally, fast image stitching is performed, which can optimize processing time at the cost of additional storage space.

[0097] The array image is translated according to the image pixel translation value, and the center point searches for the boundaries around to calculate the maximum inscribed rectangle; the camera index value of each pixel point of the maximum inscribed rectangle is calculated. If the pixel point only exists in one camera image, the camera index value is the camera number. If the pixel point exists in multiple cameras, the number of the point that is closer to the ideal principal point of multiple cameras is the camera index value; the specific pixel position of a camera image corresponding to the pixel point is calculated, and the mapping of the pixel point to the pixel point of a specific camera is calculated using the neighboring point interpolation method, which improves the stitching efficiency of the underwater array image without the need for feature extraction;

[0098] Specifically, the present invention is further described below by way of embodiments: Figure 5 This is an exemplary embodiment of the results of rapid stitching of underwater structure surface array images using the method proposed in the present invention. The proposed method can better restore the surface features of the underwater structure when stitching multiple underwater array images without the need for feature extraction. It has strong accuracy and robustness and has good application prospects in actual underwater structure detection and image stitching tasks.

[0099] Embodiment 2: In the second aspect, as Figure 6 As shown, in order to achieve the above-mentioned purpose, the present invention discloses an underwater camera internal and external parameter consistent calibration and array image stitching system, comprising:

[0100] A camera calibration module 11 is used to receive a multi-scale recognizable speckle calibration pattern, and use the multi-scale recognizable speckle calibration pattern to consistently calibrate the internal and external parameters of the camera to obtain a calibrated camera, wherein the multi-scale recognizable speckle calibration pattern is generated based on a feature detection mechanism of a SURF algorithm;

[0101] The image correction module 12 is used to receive the underwater array image, and correct the optical center and optical axis of the underwater array image using the calibrated camera internal and external parameters to obtain the pixel translation value of the underwater array image;

[0102] The image mapping module 13 is used to determine the maximum inscribed rectangle according to the pixel translation value of the underwater array image, calculate the camera index value and the pixel index value of all the points in the maximum inscribed rectangle; perform the nearest point interpolation on the underwater array image according to the camera index value and the pixel index value of all the points in the maximum inscribed rectangle, and obtain the mapping relationship between the calculated pixel point and the camera pixel point;

[0103] The image stitching module 14 is used to input the corrected underwater structure surface array image into a pre-established image stitching algorithm model based on the mapping relationship between the calculated pixels and the camera pixels, and output the stitching result of the underwater structure surface array image.

[0104] In combination with the second aspect, in some implementations of the second aspect, the system further includes: the camera calibration module 11 includes a multi-scale recognizable speckle calibration pattern generated based on the reverse engineering of the scaling process in SURF detection, and based on the reversed SURF scaling process, a multi-scale noise image of the same size is used to synthesize the calibration pattern, allowing the camera to acquire and detect features at different distances, and extracting features of the image detected by the camera and the calibration pattern at the same time

[0105] The camera calibration module 11 uses a multi-scale identifiable speckle calibration pattern to calibrate the camera's internal and external parameters in a consistent manner, including: speckle pattern generation, calibration plate design, and consistent calibration of the camera's internal and external parameters;

[0106] The image correction module 12 corrects the optical center and optical axis of the underwater array image by calculating the image homography matrix and the optical center coordinates respectively by using the intrinsic parameters and extrinsic parameters of the camera calibration, thereby further calculating the pixel translation value of the image;

[0107] The underwater array image acquisition device of the image correction module 12 is an underwater planar array camera, which uses a camera optical method to obtain an array two-dimensional image of the underwater structure and its surface as the underwater array image;

[0108] The image mapping module 13 determines the process of the maximum inscribed rectangle according to the pixel translation value of the underwater array image:

[0109] After obtaining the pixel translation value of each underwater array image, the array camera image is translated according to the pixel translation value of the underwater array image, and the maximum inscribed rectangle of the area occupied by the array camera image is calculated;

[0110] The image mapping module 13 calculates the camera index values ​​and pixel index values ​​of all points in the maximum inscribed rectangle:

[0111] For all pixel coordinates in the maximum inscribed rectangle ( x , y ) to the camera's main point C i Distance for:

[0112]

[0113] in, C ix and C iy Camera C i Ideal camera principal point x and y Coordinates, then ( x , y ) camera index value c ( x , y , C i )for:

[0114]

[0115] Among them, argmin means finding the distance Minimum camera index i ;

[0116] For determining the camera C i The pixel point ( x , y ), using the nearest neighbor interpolation method to calculate its position in the camera C i The pixel position in the image ( x’ , y’ ):

[0117]

[0118]

[0119] in, round The function is a rounding function, which is used to round the coordinates to the nearest integer.

[0120] The image stitching module 14 inputs the corrected underwater structure surface array image into a pre-established image stitching algorithm model, and outputs the calculation process of the stitching result of the underwater structure surface array image:

[0121] For all pixels of the image points in the maximum inscribed rectangle Pixel ( x , y ) have:

[0122]

[0123] in i is the camera index value of the image point, ( x’ , y’ ) is the pixel index value of the image point.

[0124] Embodiment 3: The third aspect, as Figure 7 As shown, in order to achieve the above-mentioned purpose, the present invention adopts actual verification cases to verify the feasibility of the invention, including:

[0125] The camera system is a 3×2 planar array camera device, with LED light sources distributed in a diamond shape around the camera module, a working distance of 160 mm, and a 100 cm×100 cm concrete slab placed in a water tank for underwater inspection;

[0126] Figure 8 Based on the feature detection mechanism of SURF algorithm, the generated speckle calibration pattern contains multi-scale identifiable speckle calibration patterns and calibration images collected by array cameras. Six cameras simultaneously collect the speckle calibration images, calibrate the camera internal and external parameters consistently, and derive the camera internal parameter matrix and the rotation matrix and translation matrix of the camera extrinsic parameters.

[0127] Figure 5 In order to obtain the camera internal and external parameters based on the consistent calibration method of camera internal and external parameters, the camera internal and external parameters are used to correct the image to obtain the corrected optical center, optical axis and pixel translation value. The pixel translation value is then used to calculate the maximum inscribed rectangle. The camera index value and pixel index value of the image point are then calculated to obtain the image mapping relationship. Finally, the six underwater structure surface images are quickly stitched together through the image stitching algorithm to obtain the final stitching image.

[0128] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0129] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by the processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0130] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0131] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A method for calibrating consistent internal and external parameters of an underwater camera and stitching array images, characterized in that: The method comprises the following steps: Receiving a multi-scale recognizable speckle calibration pattern, and using the multi-scale recognizable speckle calibration pattern to consistently calibrate the internal and external parameters of a camera to obtain a calibrated camera, wherein the multi-scale recognizable speckle calibration pattern is generated based on a feature detection mechanism of a SURF algorithm; The multi-scale identifiable speckle calibration pattern is generated based on the reverse engineering of the scaling process in SURF detection. Based on the reversed SURF scaling process, multi-scale noise images of the same size are used to synthesize the calibration pattern, allowing the camera to acquire and detect features at different distances, and extract features of the image detected by the camera and the calibration pattern at the same time; Receiving an underwater array image, and correcting the optical center and optical axis of the underwater array image using the calibrated camera internal and external parameters to obtain a pixel translation value of the underwater array image; According to the pixel translation value of the underwater array image, the maximum inscribed rectangle is determined, and the camera index value and the pixel index value of all the points in the maximum inscribed rectangle are calculated; according to the camera index value and the pixel index value of all the points in the maximum inscribed rectangle, the nearest point interpolation is performed on the underwater array image to obtain the mapping relationship between the calculated pixel point and the camera pixel point; Based on the mapping relationship between calculated pixels and camera pixels, the corrected underwater structure surface array image is input into a pre-established image stitching algorithm model, and the stitching result of the underwater structure surface array image is output.

2. The method for calibrating consistent internal and external parameters of an underwater camera and stitching array images according to claim 1, characterized in that: The method of calibrating the internal and external parameters of a camera by using a multi-scale identifiable speckle calibration pattern includes: generating a speckle pattern, designing a calibration plate, and calibrating the internal and external parameters of the camera.

3. The method for calibrating consistent internal and external parameters of an underwater camera and stitching array images according to claim 1, characterized in that: The process of correcting the optical center and optical axis of the underwater array image is as follows: the image homography matrix and optical center coordinates are calculated respectively by using the intrinsic parameters and extrinsic parameters of the camera calibration, the optical axis of each image is corrected by using the homography matrix, all the corrected optical axes are parallel, the distance from the corrected optical center to the surface of the structure is calculated by using the z coordinates in all the calculated optical center coordinates, the optical centers of all cameras are corrected to be on the same plane, and finally the pixel translation value between the camera principal points is calculated according to the physical size distance between the optical centers, which is used for subsequent array camera image stitching.

4. The method for calibrating consistent internal and external parameters of an underwater camera and stitching array images according to claim 3, characterized in that: The underwater array image acquisition device is an underwater planar array camera, which uses a camera optical method to obtain an array two-dimensional image of the underwater structure and its surface as the underwater array image.

5. The method for calibrating consistent internal and external parameters of an underwater camera and stitching array images according to claim 1, characterized in that: The process of determining the maximum inscribed rectangle according to the pixel translation value of the underwater array image: After obtaining the pixel translation value of each underwater array image, the array camera image is translated according to the pixel translation value of the underwater array image, and the maximum inscribed rectangle of the area occupied by the array camera image is calculated.

6. The method for calibrating consistent internal and external parameters of an underwater camera and stitching array images according to claim 1, characterized in that: The calculation process of the camera index value and pixel index value of all points in the maximum inscribed rectangle is as follows: For all pixel coordinates (x, y) in the maximum inscribed rectangle to the camera principal point C i The distance d(x,y,C i )for: Among them, C ix and C iy Camera C i The x and y coordinates of the ideal camera principal point are then the camera index value c(x,y,C i )for: c(x,y,C i )=argmin i {d(x,y,C i )} Among them, argmin means finding the distance d(x,y,C i )The smallest camera index i; For the camera C i The pixel point (x, y) is calculated using the nearest neighbor interpolation method at camera C. i Pixel position (x',y') in the image: x′=round(x-C i ) y′=round(y-C i ) Among them, the round function is a rounding function.

7. The method for calibrating consistent internal and external parameters of an underwater camera and stitching array images according to claim 1, characterized in that: The calculation process of inputting the corrected underwater structure surface array image into a pre-established image stitching algorithm model and outputting the stitching result of the underwater structure surface array image: For all image points in the maximum inscribed rectangle, the pixels Pixel(x,y) have: Pixel(x,y)=Pixel i (x′,y′) Where i is the camera index value of the image point, and (x',y') is the pixel index value of the image point.

8. An underwater camera internal and external parameter consistent calibration and array image stitching system, characterized in that: include: A camera calibration module, used for receiving a multi-scale recognizable speckle calibration pattern, and using the multi-scale recognizable speckle calibration pattern to consistently calibrate the internal and external parameters of the camera to obtain a calibrated camera, wherein the multi-scale recognizable speckle calibration pattern is generated based on a feature detection mechanism of a SURF algorithm; The camera calibration module contains multi-scale recognizable speckle calibration patterns generated based on the reverse engineering of the scaling process in SURF detection. Based on the reversed SURF scaling process, multi-scale noise images of the same size are used to synthesize the calibration pattern, allowing the camera to acquire and detect features at different distances, and extract features of the image detected by the camera and the calibration pattern at the same time; An image correction module is used to receive an underwater array image, and use the calibrated camera internal and external parameters to correct the optical center and optical axis of the underwater array image to obtain a pixel translation value of the underwater array image; The image mapping module is used to determine the maximum inscribed rectangle according to the pixel translation value of the underwater array image, calculate the camera index value and the pixel index value of all points in the maximum inscribed rectangle; perform the nearest point interpolation on the underwater array image according to the camera index value and the pixel index value of all points in the maximum inscribed rectangle, and obtain the mapping relationship between the calculated pixel point and the camera pixel point; The image stitching module is used to input the corrected underwater structure surface array image into a pre-established image stitching algorithm model based on the mapping relationship between the calculated pixels and the camera pixels, and output the stitching result of the underwater structure surface array image.

9. The underwater camera internal and external parameter consistent calibration and array image stitching system according to claim 8, characterized in that: The camera calibration module uses a multi-scale identifiable speckle calibration pattern to calibrate the camera's internal and external parameters consistently, including: speckle pattern generation, calibration plate design, and consistent calibration of the camera's internal and external parameters; The process of correcting the optical center and optical axis of the underwater array image in the image correction module: by using the intrinsic and extrinsic parameters of the camera calibration to calculate the image homography matrix and the optical center coordinates respectively, the homography matrix is ​​used to correct the optical axis of each image, and all the corrected optical axes are parallel. The distance from the corrected optical center to the surface of the structure is calculated by the z coordinate of all the calculated optical center coordinates, and the optical centers of all cameras are corrected to be on the same plane. Finally, the pixel translation value between the principal points of the cameras is calculated according to the physical size distance between the optical centers, which is used for subsequent array camera image stitching; The underwater array image acquisition device of the image correction module is an underwater planar array camera, which uses camera optics to obtain an array two-dimensional image of the underwater structure and its surface as the underwater array image; The image mapping module determines the process of the maximum inscribed rectangle based on the pixel translation value of the underwater array image: After obtaining the pixel translation value of each underwater array image, the array camera image is translated according to the pixel translation value of the underwater array image, and the maximum inscribed rectangle of the area occupied by the array camera image is calculated; The image mapping module calculates the camera index value and pixel index value of all points in the maximum inscribed rectangle: For all pixel coordinates (x, y) in the maximum inscribed rectangle to the camera principal point C i The distance d(x,y,C i )for: Among them, C ix and C iy Camera C i The x and y coordinates of the ideal camera principal point are then the camera index value c(x,y,C i )for: c(x,y,C i )=argmin i {d(x,y,C i )} Among them, argmin means finding the distance d(x,y,C i )The smallest camera index i; For the camera C i The pixel point (x, y) is calculated using the nearest neighbor interpolation method at camera C. i Pixel position (x',y') in the image: x′=round(x-C i ) y′=round(y-C i ) Among them, the round function is a rounding function; The image stitching module inputs the corrected underwater structure surface array image into the pre-established image stitching algorithm model and outputs the calculation process of the stitching result of the underwater structure surface array image: For all image points in the maximum inscribed rectangle, the pixels Pixel(x,y) have: Pixel(x,y)=Pixel i (x′,y′) Where i is the camera index value of the image point, and (x',y') is the pixel index value of the image point.

Images