Multi-focus image alignment method and system, computer readable storage medium and computer program product

Abstract

The invention relates to the technical field of image processing, and discloses a multi-focus image alignment method and system, a computer readable storage medium and a computer program product. The method comprises the following steps: establishing an alignment calculation model based on a preset first template image according to a shot calibration plate multi-focus image and a focusing position thereof; collecting a multi-focus image set of the target shooting object, and recording a focusing position corresponding to each multi-focus image in the multi-focus image set; and according to the established alignment calculation model, calculating a conversion image of the multi-focus image of the target shooting object based on the selected second template image. According to the method, an alignment calculation model can be established through calibration, and the defect of geometric distortion caused by focal length change of a traditional image alignment method is overcome; besides, the method ensures the alignment precision, also significantly improves the calculation efficiency, avoids the time-consuming problem existing in a traditional feature matching method, and is suitable for processing high-resolution and large-scale image data.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, system, computer-readable storage medium and computer program product for multi-focus image alignment. Background Art

[0002] With the rapid development of digital image processing technology, multi-focus image fusion has become an important research direction in the field of image processing. Multi-focus image fusion technology is usually used to combine a series of images with different focal points into an image with a wider depth of field. The core of this technology is to capture multi-focus images of different focal areas by changing the focal length, and merge the images so that the final fused image can contain clear details in all images, thereby achieving a higher depth of field and a wider clear range than a single image.

[0003] Common methods of adjusting focus include using a liquid lens to change the diopter to quickly adjust the focus, or using a motion mechanism and a fixed-focus lens to change the focus distance. However, these methods may cause problems such as changes in image magnification and field of view deviation, resulting in geometric misalignment between images. When performing image fusion, the geometric misalignment between images will cause the final fused image to have edge blur and misalignment. Therefore, how to achieve precise alignment before image fusion is a key issue in this technical field.

[0004] At present, image alignment technology is widely used in image stitching, image registration and other fields, but most methods do not take into account the geometric deformation caused by focal length changes in multi-focus images. Traditional image alignment methods generally rely on feature point matching. This method can work effectively in some simple cases, such as alignment of images with small depth of field changes and clear textures, but when faced with large depth of field changes or unclear image textures, inaccurate matching, misalignment or distortion often occur.

[0005] At the same time, image alignment methods based on feature matching usually take a long time in the feature extraction and matching process, especially when the image size is large or there are many feature points. For example, when algorithms such as SIFT and SURF extract feature points in high-resolution images and match them, the amount of calculation is very large, and the matching process may involve a large amount of computing and storage resources, resulting in inefficient alignment process. Summary of the invention

[0006] The object of the present invention is to provide a method, system, computer-readable storage medium and computer program product for multi-focus image alignment, so as to solve or at least partially solve the technical problems mentioned in the above background technology.

[0007] To achieve this object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a method for aligning a multi-focus image, comprising:

[0009] Take at least three multi-focus images of the calibration plate with different focal lengths, and record the focus position corresponding to each multi-focus image of the calibration plate;

[0010] According to the captured calibration plate multi-focus image and its focus position, an alignment calculation model based on a predetermined first template image is established; wherein the first template image is one of the at least three calibration plate multi-focus images with different focal lengths;

[0011] Collect a multi-focus image set of the target object, and record the focus position corresponding to each multi-focus image in the multi-focus image set;

[0012] Calculate, according to the established alignment calculation model, a transformed image of the multi-focus image of the target object based on the selected second template image; wherein the second template image is an image in the multi-focus image set;

[0013] The smallest valid common area ROI in the transformed image is cropped to obtain the aligned multi-focus image.

[0014] Optionally, the step of establishing an alignment calculation model based on a predetermined first template image according to the captured multi-focus image of the calibration plate and its focus position specifically includes:

[0015] Assume that the multi-focus image of the calibration plate is represented as S i , the first template image is represented as S r , S i The corresponding focus position is G i ; Where i, r are the serial numbers of the multi-focus images of the calibration plate;

[0016] Calculate S using the ECC algorithm i With S r The affine transformation matrix M i ;M i And the alignment calculation model are expressed as:

[0017] S′ i (x,y)=f[S i (x,y),G i , G0];

[0018] Among them, S i ' indicates S i The image obtained after the conversion of the calculation model by alignment;

[0019] Use the least squares method to find G i With M i The relationship between them can be expressed as:

[0020] a 11 =f1(G i ), a 12 =f2(G i ), a 13 =f3(G i ), a 21 =f4(G i ), a 22 =f5(G i ), a 23 =f6(G i );

[0021] Among them, f1, f2, f3, f4, f5 and f6 are based on G i The obtained function;

[0022] The alignment calculation model is transformed into:

[0023]

[0024] Where [x, y] is the pixel coordinate in the multifocus image to be aligned, and [x′, y′] is the pixel coordinate in the transformed image.

[0025] Optionally, the step of establishing an alignment calculation model based on a predetermined first template image according to the captured multi-focus image of the calibration plate and its focus position specifically includes:

[0026] Assume that the multi-focus image of the calibration plate is represented as S i , the first template image is represented as S r , S i The corresponding focus position is G i ; Where i, r are the serial numbers of the multi-focus images of the calibration plate;

[0027] Use the transformation of the reference point coordinates on the calibration plate to calculate the corresponding affine transformation matrix M i ; Wherein, the reference point coordinates are the center or corner coordinates;

[0028] The calculation method of the affine transformation matrix specifically includes:

[0029] If the reference point coordinates are selected as the circle center coordinates, the multi-focus image S is obtained by the Hough circle finding algorithm. i All the center coordinates of the circle; the center coordinates are marked as (x ij ,y ij ), i is the serial number of the multi-focus image of the calibration plate, and j is the serial number of the center of the circle in the field of view;

[0030] Set the affine transformation matrix M i for:

[0031]

[0032] S r The jth circle center coordinate (x rj ,y rj ) is mapped to other calibration plate multi-focus images S i The jth circle center coordinate (x ij ,y ij ), establish the following coordinate mapping equation:

[0033]

[0034] For each multi-focus image of the calibration plate, j coordinate mapping equations can be obtained through the coordinate mapping equation, and a coordinate mapping equation group is constructed;

[0035] By solving the coordinate mapping equations, we can obtain M i The parameter a in 11 、a 12 、a 13 、a 21 、a 22 and a 23 ;

[0036] Use the least squares method to find G i With M i The relationship between the parameters in is expressed as:

[0037] a 11 =f1(G i ), a 12 =f2(G i ), a 13 =f3(G i ), a 21 =f4(G i ), a 22 =f5(G i ), a 23 =f6(G i );

[0038] Among them, f1, f2, f3, f4, f5 and f6 are based on G i The obtained function;

[0039] Then the alignment calculation model is transformed into:

[0040]

[0041] Where [x, y] is the pixel coordinate in the multifocus image to be aligned, and [x′, y′] is the pixel coordinate in the transformed image.

[0042] Optionally, the capturing of at least three calibration plate multi-focus images with different focal lengths and recording the focus position corresponding to each calibration plate multi-focus image specifically includes:

[0043] Fixing a camera at a predetermined first shooting position and aligning it with a calibration plate; wherein the camera is equipped with a liquid lens capable of controllably adjusting the diopter;

[0044] The liquid lens is controlled to adjust different diopters, at least three multi-focus images of the calibration plate with different focal lengths are obtained by photographing with a camera, and the diopters corresponding to each multi-focus image of the calibration plate are recorded.

[0045] Optionally, S i Corresponding focus position G i Expressed as:

[0046] G i =△dpt i =dpt i -dpt r ; Among them, dpt i The recorded S i Corresponding diopter, dpt r is the diopter corresponding to the first template image;

[0047] △dpt i and the affine transformation matrix M i The elements in the equation are linear relations, which can be expressed as:

[0048] a ij =k×△dpt i +b;

[0049] For the affine transformation matrix M i Each element a ij , after minimizing the following error sum of squares, k and b are obtained by least squares fitting; the error sum of squares and the calculation formulas of k and b are expressed as follows:

[0050] The sum of squared errors is expressed as:

[0051]

[0052] The calculation formula for k is:

[0053]

[0054] The calculation formula of b is:

[0055]

[0056] Optionally, before establishing the alignment calculation model based on the predetermined first template image according to the captured multi-focus image of the calibration plate and its focus position, the method further includes:

[0057] The clearest multi-focus image of the calibration plate is selected as the first template image.

[0058] Optionally, the acquiring of a multi-focus image set of the target object and recording a focus position corresponding to each multi-focus image in the multi-focus image set specifically includes:

[0059] Fixing the camera at a predetermined second shooting position and aiming the camera at the target object;

[0060] Control the liquid lens to adjust different diopters, obtain a set of multi-focus images of the target object through camera shooting, and record the diopters corresponding to each multi-focus image;

[0061] Selecting an image from a group of acquired multi-focus images of the target object as the second template image;

[0062] Assume that the multi-focus image of the target object is P k , the second template image is P s , P k Corresponding focus position G k Expressed as:

[0063] G k =△dpt k =dpt k -dpt s ; Among them, dpt k The recorded P k Corresponding diopter, dpt s is the diopter corresponding to the second template image; k, s are the serial numbers of the multi-focus images of the target object;

[0064] The method of calculating the multi-focus image of the target object based on the transformed image of the selected second template image according to the established alignment calculation model is specifically as follows:

[0065] Through the established alignment calculation model, the multi-focus image P of the target object is k Transform to obtain the multi-focus image P k The transformed image P′ based on the second template image k .

[0066] Optionally, the cropping of the smallest effective common area ROI in the transformed image to obtain the aligned multi-focus image specifically includes:

[0067] Calculate the ROI area in all transformed images;

[0068] Find the intersection of all ROI areas and obtain the smallest valid common area ROI common ;

[0069] Crop the ROI in each transformed image separately common The images of the parts are aligned to obtain a multifocus image.

[0070] Optionally, the calculation method of the ROI area in the converted image specifically includes:

[0071] Transform the image P′ k Convert to the corresponding grayscale image G′ k , and based on the grayscale image G′ k The size of sets the initial boundary of the ROI area and limits the scope of the initial search area;

[0072] By binarizing the grayscale image G′ k Convert to a mask and calculate the grayscale image G′ based on the mask k The number of zero-valued pixels in the eight-neighborhood of each pixel in the image is used to determine the boundary of the ROI area and obtain the transformed image P′ k ROI area;

[0073] The step of cropping the image of the ROI portion in the transformed image to obtain the aligned multi-focus image specifically includes:

[0074] For each image P′ k , crop ROI common The aligned multi-focus image P″ is generated k ,Right now:

[0075] P″ k =P k (ROI common ).

[0076] Optionally, the calibration plate is a circular calibration plate or a checkerboard calibration plate.

[0077] Optionally, the target object is a milling cutter.

[0078] In a second aspect, the present invention provides a system for aligning multi-focus images, comprising:

[0079] A shooting component, used for shooting at least three multi-focus images of a calibration plate with different focal lengths;

[0080] A storage unit, electrically connected to the shooting component, and used to store the captured multi-focus images of the calibration plate and the focus position corresponding to each recorded multi-focus image of the calibration plate;

[0081] A model building unit, electrically connected to the storage unit, and used to establish an alignment calculation model based on a predetermined first template image according to the captured multi-focus image of the calibration plate and its focus position; wherein the first template image is one of the at least three multi-focus images of the calibration plate with different focal lengths;

[0082] The shooting component is further used to shoot a multi-focus image set of the target object; the storage unit is further used to store the shot multi-focus image set and the focus position corresponding to each multi-focus image in the recorded multi-focus image set;

[0083] An image alignment calculation unit, electrically connected to the storage unit, for calculating a transformed image of the multi-focus image of the target object based on the selected second template image according to the established alignment calculation model; wherein the second template image is an image in the multi-focus image set;

[0084] The image cropping unit is electrically connected to the image alignment calculation unit and is used to crop the smallest effective common area ROI in the transformed image to obtain an aligned multi-focus image.

[0085] Optionally, the method for constructing the alignment calculation model specifically includes:

[0086] Assume that the multi-focus image of the calibration plate is represented as S i , the first template image is represented by S r , S i The corresponding focus position is G i ; Where i, r are the serial numbers of the multi-focus images of the calibration plate;

[0087] Calculate S using the ECC algorithm i With S r The affine transformation matrix M i ;M i And the alignment calculation model are expressed as:

[0088]

[0089] Among them, S i ' indicates S i The image obtained after the conversion of the calculation model by alignment;

[0090] Use the least squares method to find G i With M i The relationship between them can be expressed as:

[0091] a 11 =f1(G i ), a 12 =f2(G i ), a 13 =f3(Gi ), a 21 =f4(G i ), a 22 =f5(G i ), a 23 =f6(G i );

[0092] Among them, f1, f2, f3, f4, f5 and f6 are based on G i The obtained function;

[0093] The alignment calculation model is transformed into:

[0094]

[0095] Where [x, y] is the pixel coordinate in the multifocus image to be aligned, and [x′, y′] is the pixel coordinate in the transformed image.

[0096] Optionally, the method for constructing the alignment calculation model specifically includes:

[0097] Assume that the multi-focus image of the calibration plate is represented as S i , the first template image is represented as S r , S i The corresponding focus position is G i ; Where i, r are the serial numbers of the multi-focus images of the calibration plate;

[0098] Use the transformation of the reference point coordinates on the calibration plate to calculate the corresponding affine transformation matrix M i ; Wherein, the reference point coordinates are the center or corner coordinates;

[0099] The calculation method of the affine transformation matrix specifically includes:

[0100] If the reference point coordinates are selected as the circle center coordinates, the multi-focus image S is obtained by the Hough circle finding algorithm. i All the center coordinates of the circle; the center coordinates are marked as (x ij ,y ij ), i is the serial number of the multi-focus image of the calibration plate, and j is the serial number of the center of the circle in the field of view;

[0101] Set the affine transformation matrix M i for:

[0102]

[0103] S r The jth circle center coordinate (x rj ,y rj ) is mapped to other calibration plate multi-focus images S i The jth circle center coordinate (xij ,y ij ), establish the following coordinate mapping equation:

[0104]

[0105] For each multi-focus image of the calibration plate, j coordinate mapping equations can be obtained through the coordinate mapping equation, and a coordinate mapping equation group is constructed;

[0106] By solving the coordinate mapping equations, we can obtain M i The parameter a in 11 、a 12 、a 13 、a 21 、a 22 and a 23 ;

[0107] Use the least squares method to find G i With M i The relationship between the parameters in is expressed as:

[0108] a 11 =f1(G i ), a 12 =f2(G i ), a 13 =f3(G i ), a 21 =f4(G i ), a 22 =f5(G i ), a 23 =f6(G i );

[0109] Among them, f1, f2, f3, f4, f5 and f6 are based on G i The obtained function;

[0110] Then the alignment calculation model is transformed into:

[0111]

[0112] Where [x, y] is the pixel coordinate in the multifocus image to be aligned, and [x′, y′] is the pixel coordinate in the transformed image.

[0113] Optionally, the method for shooting the multi-focus image of the calibration plate is specifically as follows:

[0114] Fixing a camera at a predetermined first shooting position and aligning it with a calibration plate; wherein the camera is equipped with a liquid lens capable of controllably adjusting the diopter;

[0115] The liquid lens is controlled to adjust different diopters, at least three multi-focus images of the calibration plate with different focal lengths are obtained by photographing with a camera, and the diopters corresponding to each multi-focus image of the calibration plate are recorded.

[0116] Optionally, S i Corresponding focus position G i Expressed as:

[0117] G i =△dpt i =dpt i -dpt r ; Among them, dpt i The recorded S i Corresponding diopter, dpt r is the diopter corresponding to the first template image;

[0118] △dpt i and the affine transformation matrix M i The elements in the equation are linear relations, which can be expressed as:

[0119] a ij =k×△dpt i +b;

[0120] For the affine transformation matrix M i Each element a ij , after minimizing the following error sum of squares, k and b are obtained by least squares fitting; the error sum of squares and the calculation formulas of k and b are expressed as follows:

[0121] The sum of squared errors is expressed as:

[0122]

[0123] The calculation formula for k is:

[0124]

[0125] The calculation formula of b is:

[0126]

[0127] Optionally, the first template image is selected as the clearest multi-focus image of the calibration plate.

[0128] Optionally, the method for acquiring the multi-focus image set of the target object is specifically:

[0129] Fixing the camera at a predetermined second shooting position and aiming the camera at the target object;

[0130] Control the liquid lens to adjust different diopters, obtain a set of multi-focus images of the target object through camera shooting, and record the diopters corresponding to each multi-focus image;

[0131] The conversion method of the conversion image is specifically as follows:

[0132] Selecting an image from a group of acquired multi-focus images of the target object as the second template image;

[0133] Assume that the multi-focus image of the target object is P k , the second template image is P s , P k Corresponding focus position G k Expressed as:

[0134] G k =△dpt k =dpt k -dpt s ; Among them, dpt k The recorded P k Corresponding diopter, dpt s is the diopter corresponding to the second template image; k, s are the serial numbers of the multi-focus images of the target object;

[0135] The image alignment calculation unit uses the established alignment calculation model to align the multi-focus image P of the target object. k Transform to obtain the multi-focus image P k The transformed image P based on the second template image k '.

[0136] Optionally, the image cropping unit is specifically used to:

[0137] Calculate the ROI area in all transformed images;

[0138] Find the intersection of all ROI areas and obtain the smallest valid common area ROI common ;

[0139] Crop the ROI in the transformed image common The method further comprises: calculating the smallest effective common area ROI in all transformed images to obtain an aligned multi-focus image.

[0140] Transform the image P′ k Convert to the corresponding grayscale image G′ k , and based on the grayscale image G k The size of sets the initial boundary of the ROI area and limits the scope of the initial search area;

[0141] By binarizing the grayscale image G′ k Convert to a mask and calculate the grayscale image G′ based on the mask k The number of zero-valued pixels in the eight-neighborhood of each pixel in the image is used to determine the boundary of the ROI area and obtain the transformed image P′. k ROI area;

[0142] The step of cropping the image of the ROI portion in the converted image to obtain the aligned multi-focus image specifically includes:

[0143] For each image P′ k , crop ROI common The aligned multi-focus image P″ is generated k ,Right now:

[0144] P″ k =P k (ROI common ).

[0145] Optionally, the calibration plate is a circular calibration plate or a checkerboard calibration plate.

[0146] Optionally, the target object is a milling cutter.

[0147] In a third aspect, the present invention further provides a computer-readable storage medium, in which at least one instruction is stored, and the instruction is loaded and executed by a processor to implement the multi-focus image alignment method as described above.

[0148] In a fourth aspect, the present invention further provides a computer program product, comprising a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the above-mentioned method of multi-focus image alignment is implemented.

[0149] Compared with the prior art, the present invention has the following beneficial effects:

[0150] A multi-focus image alignment method provided by the present invention can establish an alignment calculation model through calibration, thereby overcoming the shortcomings of traditional image alignment methods when facing geometric distortion caused by focal length changes; while ensuring the alignment accuracy, the method also significantly improves the calculation efficiency, avoids the time-consuming problem existing in traditional feature matching methods, and is suitable for processing high-resolution and large-scale image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0151] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0152] Figure 1 A flowchart of a multi-focus image alignment method provided by an embodiment of the present invention.

[0153] Figure 2 A schematic diagram of the architecture of a multi-focus image alignment system provided by an embodiment of the present invention.

[0154] Figure 3 A picture of a shooting component in operation provided by an embodiment of the present invention.

[0155] Figure 4 A multi-focus image of a portion of the calibration plate captured by an embodiment of the present invention.

[0156] Figure 5 A method based on dpt provided in an embodiment of the present invention r The picture with =0 is the first template image, and the experimental data diagram is recorded in the process of calculating the affine transformation matrix corresponding to the multi-focus images of other calibration plates using the ECC algorithm.

[0157] Figure 6 A multi-focus image set of a target object captured by an embodiment of the present invention.

[0158] Figure 7 For Figure 6 The transformed image obtained by aligning one of the images in the multifocus image set.

[0159] Figure 8 For Figure 7 The aligned multifocus image is obtained by cropping the transformed image in .

[0160] Figure 9a This is the fusion effect of the multi-focus image set without alignment operation.

[0161] Figure 9b This is the fusion effect diagram of the multi-focus image set after alignment operation through the ECC algorithm.

[0162] Fig.9c This is a diagram showing the fusion effect of a multi-focus image set after an alignment operation is performed using the alignment calculation model provided by an embodiment of the present invention.

[0163] Fig.10a An image mask provided by an embodiment of the present inventionk Schematic diagram of .

[0164] Fig.10b A schematic diagram of a boundary of a ROI area that needs to be updated is provided in an embodiment of the present invention.

[0165] Fig.10c is an image P′ provided by an embodiment of the present invention k ROI k Schematic diagram of .

[0166] In the figure:

[0167] 10. Shooting component; 20. Storage unit; 30. Model building unit; 40. Image alignment calculation unit; 50. Image cropping unit; 60. Image fusion unit. DETAILED DESCRIPTION

[0168] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0169] Embodiment 1:

[0170] Please refer to Figure 1 , Figure 1 A flowchart of a multi-focus image alignment method provided by an embodiment of the present invention, the method specifically includes:

[0171] Step 110: Take at least three multi-focus images of the calibration plate with different focal lengths, and record the focus position corresponding to each multi-focus image of the calibration plate.

[0172] In this embodiment, step 110 is specifically implemented as follows:

[0173] Step 111, fixing the camera at a predetermined first shooting position and aligning it with the calibration plate; wherein the camera is equipped with a liquid lens capable of controllably adjusting the diopter;

[0174] Step 112: Control the liquid lens to adjust different diopters, obtain at least three multi-focus images of the calibration plate with different focal lengths through the camera, and record the diopters corresponding to each multi-focus image of the calibration plate.

[0175] For ease of understanding, for example, the multi-focus image of the calibration plate is represented as S i , S i The corresponding focus position is G i; Where i is the serial number of the multi-focus image of the calibration plate;

[0176] S i Corresponding focus position G i Expressed as:

[0177] G i =△dpt i =dpt i -dpt r ; Among them, dpt i The recorded S i Corresponding diopter, dpt r is the diopter corresponding to the first template image.

[0178] Please refer to Figure 3 , Figure 3 A picture of a camera component in operation provided by an embodiment of the present invention, Figure 4 A multi-focus image of a portion of the calibration plate captured by an embodiment of the present invention.

[0179] Step 120: Establish an alignment calculation model based on a predetermined first template image according to the captured multi-focus image of the calibration plate and its focus position.

[0180] In step 120, the first template image is one of at least three multi-focus images of a calibration plate with different focal lengths;

[0181] For ease of understanding, illustratively, the first template image is represented as S r , r is the serial number of the multi-focus image of the calibration plate;

[0182] As an optional implementation, step 120 is specifically implemented as follows:

[0183] Step 121: Calculate S using the ECC algorithm i With S r The affine transformation matrix M i .

[0184] ECC stands for Enhanced Correlation Coefficient. The ECC algorithm is an optimization algorithm commonly used in image registration. It solves image transformation by minimizing the difference in correlation coefficients between images. Specifically, the ECC algorithm estimates the geometric transformation relationship between images by optimizing an objective function based on image grayscale information, and finds the optimal transformation matrix through iterative calculation to minimize the relative error between images. Since multi-focus images are usually obtained by shooting the same scene with a camera at different focal lengths. Focal length changes mainly affect depth of field, and the impact of focal length changes on image geometry is usually limited to translation, rotation, or slight scale changes, so it is sufficient to use affine transformation to describe this relationship.

[0185] Specifically, Mi and the alignment calculation model are expressed as:

[0186] S′ i (x,y)=f[S i (x,y),G i , G0];

[0187] Among them, S i ' indicates S i The image obtained after the conversion of the calculation model by alignment;

[0188] Step 122: Use the least squares method to find G i With M i The relationship between.

[0189] Changes in focus may cause a slight shift in the center of the image. For example, when the focus is close, objects in the image may shift slightly relative to the camera's viewpoint, and this shift can be accounted for by the translation component a of the transformation matrix. 13 and a 23 When the camera focal length changes, it may introduce slight rotation or scaling changes, which can be expressed by the rotation of the transformation matrix (a 11 , a 12 ) and scale changes (a 21 , a 22 ) to describe.

[0190] The least squares method is a mathematical optimization method that is usually used to fit a set of experimental data to minimize the sum of squared errors between the actual data points and the fitted curve. Assuming that there is a certain relationship (which may be linear or nonlinear) between the focus position and the elements of the affine transformation matrix, the goal is to find this relationship using the least squares method, which can be expressed as:

[0191] a 11 =f1(G i ), a12 =f2(G i ), a 13 =f3(G i ), a 21 =f4(G i ), a 22 =f5(G i ), a 23 =f6(G i );

[0192] Among them, f1, f2, f3, f4, f5 and f6 are based on G i Specifically, these functions can be obtained by fitting the experimental data using the least squares method, and represent the influence of the focus position change on the transformation matrix elements.

[0193] Step 123: transform the alignment calculation model into:

[0194]

[0195] Where [x, y] is the pixel coordinate in the multifocus image to be aligned, and [x′, y′] is the pixel coordinate in the transformed image.

[0196] It should be noted that △dpt i and the affine transformation matrix M i The elements in the equation are linear relations, which can be expressed as:

[0197] ai j =k×△dpt i +b;

[0198] For the affine transformation matrix M i Each element a ij , after minimizing the following error sum of squares, k and b are obtained by least squares fitting; the error sum of squares and the calculation formulas of k and b are expressed as follows:

[0199] The sum of squared errors is expressed as:

[0200]

[0201] The calculation formula for k is:

[0202]

[0203] The calculation formula of b is:

[0204]

[0205] Please refer to Figure 5 , Figure 5 A method based on dpt provided in an embodiment of the present inventionr = 0 is the first template image, and the experimental data diagram recorded in the process of calculating the affine transformation matrix corresponding to the multi-focus images of other calibration plates using the ECC algorithm. It should be noted that in this embodiment, when shooting the multi-focus image of the calibration plate, the position of the camera lens is adjusted so that dpt r = 0 is the clearest multi-focus image of the calibration plate, which is then used as a template;

[0206] As another optional implementation, step 120 may also be implemented as follows:

[0207] Use the transformation of the reference point coordinates on the calibration plate to calculate the corresponding affine transformation matrix M i ;

[0208] The calibration plate is a circular calibration plate or a checkerboard calibration plate, and the reference point coordinates are the center of the circle or the corner point coordinates;

[0209] Taking the calibration plate as a circular calibration plate and the reference point coordinates as the center coordinates as an example, the calculation method of the affine transformation matrix specifically includes:

[0210] Step 121', obtaining a multi-focus image S i The coordinates of all the circle centers in .

[0211] The coordinates of the center of the circle are marked as (x ij ,y ij ), i is the serial number of the multi-focus image of the calibration plate, and j is the serial number of the center of the circle in the field of view;

[0212] Step 122', set the affine transformation matrix M i for:

[0213]

[0214] S r The jth circle center coordinate (x rj ,y rj ) is mapped to other calibration plate multi-focus images S i The jth circle center coordinate (x ij ,y ij ), establish the following coordinate mapping equation:

[0215]

[0216] Step 123 ′: for each multi-focus image of the calibration plate, j coordinate mapping equations can be obtained through the coordinate mapping equation, and a coordinate mapping equation group is constructed.

[0217] Step 124', by solving the coordinate mapping equations, obtain M i The parameter a in 11 、a12 、a 13 、a 21 、a 22 and a 23 .

[0218] Step 125', use the least squares method to find G i With M i The relationship between the parameters in .

[0219] In step 125', a 11 =f1(G i ), a 12 =f2(G i ), a 13 =f3(G i ), a 21 =f4(G i ), a 22 =f5(G i ), a 23 =f6(G i );

[0220] Among them, f1, f2, f3, f4, f5 and f6 are based on G i The obtained function;

[0221] Then transform the alignment calculation model into:

[0222]

[0223] Where [x, y] is the pixel coordinate in the multifocus image to be aligned, and [x′, y′] is the pixel coordinate in the transformed image.

[0224] Step 130: Acquire a multi-focus image set of the target object, and record the focus position corresponding to each multi-focus image in the multi-focus image set.

[0225] For example, taking the target object to be photographed as a milling cutter as an example, step 130 is specifically implemented as follows:

[0226] Step 131: fix the camera at a predetermined second shooting position and aim it at the target object.

[0227] Step 132: Control the liquid lens to adjust different diopters, obtain a group of multi-focus images of the target object through camera photography, and record the diopters corresponding to each multi-focus image.

[0228] Please refer to Figure 6 , Figure 6 A multi-focus image set of a target object captured by an embodiment of the present invention.

[0229] Step 140: Calculate, according to the established alignment calculation model, a transformed image of the multi-focus image of the target object based on the selected second template image.

[0230] The second template image is an image in the multi-focus image set; therefore, before step 140, it is necessary to select an image from a group of acquired multi-focus images of the target object as the second template image.

[0231] For ease of understanding, let the multi-focus image of the target object be P k , the second template image is P s , P k Corresponding focus position G k Expressed as:

[0232] G k =△dpt k =dpt k -dpt s ; Among them, dpt k The recorded P k Corresponding diopter, dpt s is the diopter corresponding to the second template image; k, s are the serial numbers of the multi-focus images of the target object;

[0233] Step 140 is specifically implemented as follows:

[0234] Through the established alignment calculation model, the multi-focus image P of the target object is k Transform to obtain the multi-focus image Pk based on the second template image transformation image P′ k .

[0235] Please refer to Figure 7 , Figure 7 For Figure 6 The transformed image obtained by aligning one of the images in the multifocus image set.

[0236] Step 150: Crop the smallest effective common area ROI in the transformed image to obtain an aligned multi-focus image.

[0237] Step 150 specifically includes:

[0238] Step 151: Calculate the ROI areas in all transformed images.

[0239] The calculation method of the ROI area in the conversion image specifically includes:

[0240] Transform the image P′ k Convert to the corresponding grayscale image G′ k , and based on the grayscale image G′ kThe size of sets the initial boundary of the ROI area and limits the scope of the initial search area;

[0241] By binarizing the grayscale image G′ k Convert to a mask and calculate the grayscale image G′ based on the mask k The number of zero-valued pixels in the eight-neighborhood of each pixel in the image is used to determine the boundary of the ROI area and obtain the transformed image P′ k ROI area.

[0242] For example, G′ k and P′ k The relationship can be expressed as:

[0243] G′ k (y,x)=0.299·P′ kB (y,x)+0.587·P′ kG (y,x)+0.114·P′ kR (y,x);

[0244] P′ kB (y,x),P′ kG (y,x),P′ kR (y,x) are the image P′ k The values ​​of the blue, green, and red channels at the (y, x) position.

[0245] Convert the image to a mask by binarization k ,in:

[0246]

[0247] The threshold is usually a set constant value used to distinguish the foreground from the background; here, all non-black pixels (values ​​greater than 0) are set to 255, so the threshold>0. Fig.10a As shown, it is an image mask provided by an embodiment of the present invention. k Schematic diagram of .

[0248] Assuming that the initial boundaries of the ROI region are set based on the size of the image, usually a ratio of the image, four initial boundaries are set here:

[0249] leftBoundary=0.1·n k , rightBoundary=0.9·n k , topBoundary=0.1·m k , bottomBoundary=0.9·m k ;

[0250] where n k is the image G′ k The number of rows, m k is the image G′ k The number of columns.

[0251] The ROI boundary is updated by calculating the number of zero-valued pixels in the eight-neighborhood of each pixel in the image; specifically, Fig.10b As shown, Fig.10b A schematic diagram of an ROI region boundary that needs to be updated is provided for an embodiment of the present invention; if there are 5 zero-valued pixels in the eight neighborhood pixels of a certain pixel point, the point is considered to be a boundary point and the ROI region boundary needs to be updated.

[0252] For image P′ k For each pixel (y, x) in , count the number of zero-valued pixels in its neighborhood. If zeroCount(y, x) = 5, update the region of interest boundary:

[0253] LU kx =min(LU kx ,x)if x <leftBoundary,

[0254] RL kx =max(RL kx ,x)if x>rightBoundary,

[0255] LU ky =min(LU ky ,y)if y <topBoundary,

[0256] RL ky =max(RL ky ,y))if y>bbottomBoundary;

[0257] Among them, LU kx and LU ky ROI k The x and y coordinates of the upper left corner, RL kx and RL ky ROI k The x-coordinate and y-coordinate of the lower right corner. The initial value is: LU kx =0;LU ky =0; RL kx =m k RL ky =n k ;

[0258] At this time, for the image P′ k , ROI k=(LU kx ,LU ky ,RL kx ,RL ky );ROI k As attached Fig.10c As shown, this is equivalent to taking the maximum inscribed rectangle of the white area.

[0259] It should be noted that, in particular, if no valid boundary is found during the ROI calculation process, or there are no valid foreground pixels in a certain area, then the entire image is returned as the ROI, that is:

[0260] ROI k =(0,0,m k ,n k ).

[0261] Step 152: Obtain the intersection of all ROI regions to obtain the smallest effective common region ROI common .

[0262] ROI common It can be expressed as:

[0263] ROI common =ROI0∩ROI1∩…∩ROI k =(LU cx , L.U. cy , RL cx , RL cy );

[0264] The parameters in the above formula are expressed as follows:

[0265] LU cx = max{LU 0x , L.U. 1x ,…,LU kx};

[0266] LU cy = max{LU 0y ,LU iy ,…,LU ky};

[0267] RL cx =min{RL 0x ,RL 1x ,…,R Lk};

[0268] RL cy =min{RL 0y ,RL 1y ,…,RL ky}.

[0269] Step 153: Crop the ROIs in each transformed image common The images of the parts are aligned to obtain a multifocus image.

[0270] For each image P′ k , crop ROI common The aligned multi-focus image P″ is generated k ,Right now:

[0271] P″ k =P′ k (ROI common ).

[0272] Please refer to Figure 8 , Figure 8 For Figure 7 The aligned multifocus image is obtained by cropping the transformed image in .

[0273] After step 150, the method further includes:

[0274] Align the image P″ k Perform fusion processing.

[0275] It should be noted that there are many ways to fuse images. This embodiment uses the Laplace full-view depth fusion algorithm to combine the clear areas of images of different depths to generate a full-view depth image containing all focal plane information. It is understandable that image fusion is not the focus of the present invention, so its implementation process will not be repeated here.

[0276] Please refer to Figure 9a , Figure 9b and Fig.9c , Figure 9a This is the fusion effect diagram of the multi-focus image set without alignment operation. Figure 9b This is the fusion effect diagram of the multi-focus image set after the alignment operation through the ECC algorithm. Fig.9c This is a fusion effect diagram of a multi-focus image set after an alignment operation is performed using the alignment calculation model provided by an embodiment of the present invention;

[0277] According to experimental comparison ( Figure 9b and Fig.9c ), the ECC alignment algorithm (iteration number is 5) and the alignment calculation model of the present invention respectively take 37935.2ms and 152.37ms for the alignment calculation of 35 multi-focus images with a pixel size of 3072*2048. It can be seen that the alignment model of the present invention can significantly reduce the calculation time while ensuring the accuracy.

[0278] In summary, the multi-focus image alignment method provided in this embodiment has at least the following beneficial effects:

[0279] 1) Improve alignment accuracy: This method establishes an alignment calculation model through calibration, so that the images can still be accurately aligned under the geometric deformation background of multi-focus images. Unlike traditional feature point matching methods, this invention can effectively deal with problems such as image magnification changes and field of view deviation caused by focal length changes, avoid edge blur and image dislocation during fusion, and ensure the clarity and integrity of the fused image.

[0280] 2) Enhanced computational efficiency: Traditional feature point matching methods have high computational costs when processing high-resolution images, while the present invention achieves efficient alignment by calibrating and establishing the mapping relationship between the focus position and the transformation matrix. This process does not require feature extraction and matching frame by frame, significantly reduces computational time, improves processing efficiency, and is particularly suitable for alignment tasks of large-scale image data.

[0281] Embodiment 2:

[0282] Please refer to Figure 2 , Figure 2 A schematic diagram of the architecture of a multi-focus image alignment system provided by an embodiment of the present invention.

[0283] The system specifically includes:

[0284] The shooting component 10 is used to shoot at least three multi-focus images of the calibration plate with different focal lengths;

[0285] The storage unit 20 is electrically connected to the shooting assembly 10 and is used to store the captured multi-focus images of the calibration plate and the focus position corresponding to each recorded multi-focus image of the calibration plate;

[0286] The model building unit 30 is electrically connected to the storage unit 20 and is used to establish an alignment calculation model based on a predetermined first template image according to the captured multi-focus image of the calibration plate and its focus position; wherein the first template image is one of at least three multi-focus images of the calibration plate with different focal lengths;

[0287] The photographing component 10 is also used to photograph a multi-focus image set of a target object; the storage unit 20 is also used to store the photographed multi-focus image set and the focus position corresponding to each multi-focus image in the recorded multi-focus image set;

[0288] The image alignment calculation unit 40 is electrically connected to the storage unit 20 and is used to calculate the transformed image of the multi-focus image of the target object based on the selected second template image according to the established alignment calculation model; wherein the second template image is an image in the multi-focus image set;

[0289] The image cropping unit 50 is electrically connected to the image alignment calculation unit 40 and is used to crop the smallest effective common area ROI in the transformed image to obtain an aligned multi-focus image;

[0290] The image fusion unit 60 is electrically connected to the image cropping unit 50 and the storage unit 20, respectively, and is used to fuse the aligned multi-focus images to generate a full-view depth image containing all focal plane information.

[0291] Since the method for aligning the multi-focus images has been described in detail in the first embodiment, it will not be described in detail in this embodiment.

[0292] The multi-focus image alignment system provided in this embodiment can establish an alignment calculation model through calibration, overcoming the shortcomings of traditional image alignment methods in the face of geometric distortion caused by focal length changes. While ensuring alignment accuracy, it significantly improves calculation efficiency and avoids the time-consuming problem existing in traditional feature matching methods, and is suitable for processing high-resolution and large-scale image data.

[0293] Embodiment three:

[0294] This embodiment further provides a computer-readable storage medium, in which at least one instruction is stored. The instruction is loaded and executed by a processor to implement a multi-focus image alignment method as described in the first embodiment.

[0295] Since the method for aligning the multi-focus images has been described in detail in the first embodiment, it will not be described in detail in this embodiment.

[0296] Embodiment 4:

[0297] The present invention also provides a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, a multi-focus image alignment method as described in Example 1 is implemented.

[0298] Since the method for aligning the multi-focus images has been described in detail in the first embodiment, it will not be described in detail in this embodiment.

[0299] Those skilled in the art can understand that all or part of the steps of the above embodiments can be completed by hardware, or by a program to instruct the relevant hardware to complete, and the program can be stored in a computer-readable storage medium, and the storage medium mentioned above can be a read-only memory, a disk or an optical disk, etc. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0300] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for aligning multi-focus images, characterized in that: include: Take at least three multi-focus images of the calibration plate with different focal lengths, and record the focus position corresponding to each multi-focus image of the calibration plate; According to the captured calibration plate multi-focus image and its focus position, an alignment calculation model based on a predetermined first template image is established; wherein the first template image is one of the at least three calibration plate multi-focus images with different focal lengths; Collect a multi-focus image set of the target object, and record the focus position corresponding to each multi-focus image in the multi-focus image set; Calculate, according to the established alignment calculation model, a transformed image of the multi-focus image of the target object based on the selected second template image; wherein the second template image is an image in the multi-focus image set; The smallest valid common area ROI in the transformed image is cropped to obtain the aligned multi-focus image.

2. A method for aligning multi-focus images according to claim 1, characterized in that: The step of establishing an alignment calculation model based on a predetermined first template image according to the captured multi-focus image of the calibration plate and its focus position specifically includes: Assume that the multi-focus image of the calibration plate is represented as S i , the first template image is represented as S r , S i The corresponding focus position is G i ; Where i, r are the serial numbers of the multi-focus images of the calibration plate; Calculate S using the ECC algorithm i With S r The affine transformation matrix M i ;M i And the alignment calculation model are expressed as: S′ i (x,y)=f[S i (x,y),G i ,G0]; Among them, S i ' indicates S i The image obtained after the conversion of the calculation model by alignment; Use the least squares method to find G i With M i The relationship between them can be expressed as: a 11 =f1(G i ),a 12 =f2(G i ),a 13 =f3(G i ),a 21 =f4(G i ),a 22 =f5(G i ),a 23 =f6(G i ); Among them, f1, f2, f3, f4, f5 and f6 are based on G i The obtained function; The alignment calculation model is transformed into: Where [x, y] is the pixel coordinate in the multifocus image to be aligned, and [x′, y′] is the pixel coordinate in the transformed image.

3. The method for aligning multi-focus images according to claim 1, characterized in that: The step of establishing an alignment calculation model based on a predetermined first template image according to the captured multi-focus image of the calibration plate and its focus position specifically includes: Assume that the multi-focus image of the calibration plate is represented as S i , the first template image is represented as S r , S i The corresponding focus position is G i ; Where i, r are the serial numbers of the multi-focus images of the calibration plate; Use the transformation of the reference point coordinates on the calibration plate to calculate the corresponding affine transformation matrix M i ; Wherein, the reference point coordinates are the center or corner coordinates; The calculation method of the affine transformation matrix specifically includes: If the reference point coordinates are selected as the circle center coordinates, the multi-focus image S is obtained by the Hough circle finding algorithm. i All the center coordinates of the circle; the center coordinates are marked as (x ij ,y ij ), i is the serial number of the multi-focus image of the calibration plate, and j is the serial number of the center of the circle in the field of view; Set the affine transformation matrix M i for: S r The jth circle center coordinate (x rj ,y rj ) is mapped to other calibration plate multi-focus images S i The jth circle center coordinate (x ij ,y ij ), establish the following coordinate mapping equation: For each multi-focus image of the calibration plate, j coordinate mapping equations can be obtained through the coordinate mapping equation, and a coordinate mapping equation group is constructed; By solving the coordinate mapping equations, we can obtain M i The parameter a in 11 、a 12 、a 13 、a 21 、a 22 and a 23 ; Use the least squares method to find G i With M i The relationship between the parameters in is expressed as: a 11 =f1(G i ),a 12 =f2(G i ),a 13 =f3(G i ),a 21 =f4(G i ),a 22 =f5(G i ),a 23 =f6(G i ); Among them, f1, f2, f3, f4, f5 and f6 are based on G i The obtained function; Then transform the alignment calculation model into: Where [x, y] is the pixel coordinate in the multifocus image to be aligned, and [x′, y′] is the pixel coordinate in the transformed image.

4. A method for aligning multi-focus images according to claim 2 or 3, characterized in that: The shooting of at least three calibration plate multi-focus images with different focal lengths and recording the focus position corresponding to each calibration plate multi-focus image specifically includes: Fixing a camera at a predetermined first shooting position and aligning it with a calibration plate; wherein the camera is equipped with a liquid lens capable of controllably adjusting the diopter; The liquid lens is controlled to adjust different diopters, at least three multi-focus images of the calibration plate with different focal lengths are obtained by photographing with a camera, and the diopters corresponding to each multi-focus image of the calibration plate are recorded.

5. The method for aligning multi-focus images according to claim 4, characterized in that: S i Corresponding focus position G i Expressed as: G i =△dpt i =dpt i -dpt r ; Among them, dpt i The recorded S i Corresponding diopter, dpt r is the diopter corresponding to the first template image; △dpt i and the affine transformation matrix M i The elements in the equation are linear relations, which can be expressed as: a ij =k×△dpt i +b; For the affine transformation matrix M i Each element a ij , after minimizing the following error sum of squares, k and b are obtained by least squares fitting; the error sum of squares and the calculation formulas of k and b are expressed as follows: The sum of squared errors is expressed as: The calculation formula for k is: The calculation formula of b is:

6. The method for aligning multi-focus images according to claim 5, characterized in that: Before establishing the alignment calculation model based on the predetermined first template image according to the captured multi-focus image of the calibration plate and its focus position, the method further includes: The clearest multi-focus image of the calibration plate is selected as the first template image.

7. The method for aligning multi-focus images according to claim 5, characterized in that: The step of collecting a multi-focus image set of the target object and recording a focus position corresponding to each multi-focus image in the multi-focus image set specifically includes: Fixing the camera at a predetermined second shooting position and aiming the camera at the target object; Control the liquid lens to adjust different diopters, obtain a set of multi-focus images of the target object through camera shooting, and record the diopters corresponding to each multi-focus image; Selecting an image from a group of acquired multi-focus images of the target object as the second template image; Assume that the multi-focus image of the target object is P k , the second template image is P s , P k Corresponding focus position G k Expressed as: G k =△dpt k =dpt k -dpt s ; Among them, dpt k The recorded P k Corresponding diopter, dpt s is the diopter corresponding to the second template image; k, s are the serial numbers of the multi-focus images of the target object; The method of calculating the multi-focus image of the target object based on the transformed image of the selected second template image according to the established alignment calculation model is specifically as follows: Through the established alignment calculation model, the multi-focus image P of the target object is k Transform to obtain the multi-focus image P k The transformed image P′ based on the second template image k .

8. A method for aligning multi-focus images according to claim 7, characterized in that: The step of cropping the smallest effective common area ROI in the transformed image to obtain the aligned multi-focus image specifically includes: Calculate the ROI area in all transformed images; Find the intersection of all ROI areas and obtain the smallest valid common area ROI common ; Crop the ROI in each transformed image separately common The images of the parts are aligned to obtain a multifocus image.

9. The method for aligning multi-focus images according to claim 8, characterized in that: The calculation method of the ROI area in the conversion image specifically includes: Transform the image P′ k Convert to the corresponding grayscale image G′ k , and based on the grayscale image G′ k Size setting ROI The initial boundary of the region, which defines the scope of the initial search area; By binarizing the grayscale image G′ k Convert to a mask and calculate the grayscale image G′ based on the mask k The number of zero-valued pixels in the eight-neighborhood of each pixel in the image is used to determine the boundary of the ROI area and obtain the transformed image P′ k ROI area; The step of cropping the image of the ROI portion in the transformed image to obtain the aligned multi-focus image specifically includes: For each image P′ k , crop ROI common The aligned multi-focus image P″ is generated k ,Right now: P″ k =P′ k (ROI common )。 10. The method for aligning multi-focus images according to claim 4, characterized in that: The calibration plate is a circular calibration plate or a checkerboard calibration plate.

11. The method for aligning multi-focus images according to claim 1, characterized in that: The target object is a milling cutter.

12. A system for aligning multi-focus images, characterized in that: include: A shooting component, used for shooting at least three multi-focus images of a calibration plate with different focal lengths; A storage unit, electrically connected to the shooting component, for storing the captured multi-focus images of the calibration plate and the focus position corresponding to each recorded multi-focus image of the calibration plate; A model building unit, electrically connected to the storage unit, and used to establish an alignment calculation model based on a predetermined first template image according to the captured multi-focus image of the calibration plate and its focus position; wherein the first template image is one of the at least three multi-focus images of the calibration plate with different focal lengths; The shooting component is further used to shoot a multi-focus image set of the target object; the storage unit is further used to store the shot multi-focus image set and the focus position corresponding to each multi-focus image in the recorded multi-focus image set; An image alignment calculation unit, electrically connected to the storage unit, for calculating a transformed image of the multi-focus image of the target object based on the selected second template image according to the established alignment calculation model; wherein the second template image is an image in the multi-focus image set; The image cropping unit is electrically connected to the image alignment calculation unit and is used to crop the smallest effective common area ROI in the transformed image to obtain an aligned multi-focus image.

13. A multi-focus image alignment system according to claim 12, characterized in that: The method for constructing the alignment calculation model specifically includes: Assume that the multi-focus image of the calibration plate is represented as S i , the first template image is represented by S r , S i The corresponding focus position is G i ; Where i, r are the serial numbers of the multi-focus images of the calibration plate; Calculate S using the ECC algorithm i With S r The affine transformation matrix M i ;M i And the alignment calculation model are expressed as: S′ i (x,y)=f[S i (x,y),G i ,G0]; Among them, S i ' indicates S i The image obtained after the conversion of the calculation model by alignment; Use the least squares method to find G i With M i The relationship between them can be expressed as: a 11 =f1(G i ),a 12 =f2(G i ),a 13 =f3(G i ),a 21 =f4(G i ),a 22 =f5(G i ),a 23 =f6(G i ); Among them, f1, f2, f3, f4, f5 and f6 are based on G i The obtained function; The alignment calculation model is transformed into: Where [x, y] is the pixel coordinate in the multifocus image to be aligned, and [x′, y′] is the pixel coordinate in the transformed image.

14. The multi-focus image alignment system according to claim 12, characterized in that: The method for constructing the alignment calculation model specifically includes: Assume that the multi-focus image of the calibration plate is represented as S i , the first template image is represented by S r , S i The corresponding focus position is G i ; Where i, r are the serial numbers of the multi-focus images of the calibration plate; Use the transformation of the reference point coordinates on the calibration plate to calculate the corresponding affine transformation matrix M i ; Wherein, the reference point coordinates are the center or corner coordinates; The calculation method of the affine transformation matrix specifically includes: If the reference point coordinates are selected as the circle center coordinates, the multi-focus image S is obtained by the Hough circle finding algorithm. i All the center coordinates of the circle; the center coordinates are marked as (x ij ,y ij ), i is the serial number of the multi-focus image of the calibration plate, and j is the serial number of the center of the circle in the field of view; Set the affine transformation matrix M i for: S r The jth circle center coordinate (x rj ,y rj ) is mapped to other calibration plate multi-focus images S i The jth circle center coordinate (x ij ,y ij ), establish the following coordinate mapping equation: For each multi-focus image of the calibration plate, j coordinate mapping equations can be obtained through the coordinate mapping equation, and a coordinate mapping equation group is constructed; By solving the coordinate mapping equations, we can obtain M i The parameter a in 11 、a 12 、a 13 、a 21 、a 22 and a 23 ; Use the least squares method to find G i With M i The relationship between the parameters in is expressed as: a 11 =f1(G i ),a 12 =f2(G i ),a 13 =f3(G i ),a 21 =f4(G i ),a 22 =f5(G i ),a 23 =f6(G i ); Among them, f1, f2, f3, f4, f5 and f6 are based on G i The obtained function; Then transform the alignment calculation model into: Where [x, y] is the pixel coordinate in the multifocus image to be aligned, and [x′, y′] is the pixel coordinate in the transformed image.

15. A multi-focus image alignment system according to claim 13 or 14, characterized in that: The method for shooting the multi-focus image of the calibration plate is specifically as follows: Fixing a camera at a predetermined first shooting position and aligning it with a calibration plate; wherein the camera is equipped with a liquid lens capable of controllably adjusting the diopter; The liquid lens is controlled to adjust different diopters, at least three multi-focus images of the calibration plate with different focal lengths are obtained by photographing with a camera, and the diopters corresponding to each multi-focus image of the calibration plate are recorded.

16. A multi-focus image alignment system according to claim 15, characterized in that: S i Corresponding focus position G i Expressed as: G i =△dpt i =dpt i -dpt r ; Among them, dpt i The recorded S i Corresponding diopter, dpt r is the diopter corresponding to the first template image; △dpt i and the affine transformation matrix M i The elements in the equation are linear relations, which can be expressed as: a ij =k×△dpt i +b; For the affine transformation matrix M i Each element a ij , after minimizing the following error sum of squares, k and b are obtained by least squares fitting; the error sum of squares and the calculation formulas of k and b are expressed as follows: The sum of squared errors is expressed as: The calculation formula for k is: The calculation formula of b is:

17. A multi-focus image alignment system according to claim 16, characterized in that: The first template image is selected as dpt r =0 calibration plate multifocus image.

18. The multi-focus image alignment system according to claim 16, characterized in that: The method for collecting the multi-focus image set of the target object is specifically as follows: Fixing the camera at a predetermined second shooting position and aiming the camera at the target object; Control the liquid lens to adjust different diopters, obtain a set of multi-focus images of the target object through camera shooting, and record the diopters corresponding to each multi-focus image; The conversion method of the conversion image is specifically as follows: Selecting an image from a group of acquired multi-focus images of the target object as the second template image; Assume that the multi-focus image of the target object is P k , the second template image is P s , P k Corresponding focus position G k Expressed as: G k =△dpt k =dpt k -dpt s ; Among them, dpt k The recorded P k Corresponding diopter, dpt s is the diopter corresponding to the second template image; k, s are the serial numbers of the multi-focus images of the target object; The image alignment calculation unit uses the established alignment calculation model to align multiple Focus image P k Transform to obtain the multi-focus image P k The converted image 'P' based on the second template image k .

19. A multi-focus image alignment system according to claim 18, characterized in that: The image cropping unit is specifically used for: Calculate the ROI area in all transformed images; Find the intersection of all ROI areas and obtain the smallest valid common area ROI common ; Crop the ROI in the transformed image common Part of the image is taken to obtain the aligned multi-focus image.

20. The multi-focus image alignment system according to claim 19, characterized in that: The calculation of the smallest effective common area ROI in all transformed images specifically includes: Transform the image P′ k Convert to the corresponding grayscale image G′ k , and based on the grayscale image G′ k Size setting ROI The initial boundary of the region, which defines the scope of the initial search area; By binarizing the grayscale image G′ k Convert to a mask and calculate the grayscale image G′ based on the mask k The number of zero-valued pixels in the eight-neighborhood of each pixel in the image is used to determine the boundary of the ROI area and obtain the transformed image P′ k ROI area; The step of cropping the image of the ROI portion in the transformed image to obtain the aligned multi-focus image specifically includes: For each image P′ k , crop ROI common The aligned multi-focus image P″ is generated k ,Right now: P" k =P′ k (ROI common )。 21. The multi-focus image alignment system according to claim 15, characterized in that: The calibration plate is a circular calibration plate or a checkerboard calibration plate.

22. The multi-focus image alignment system according to claim 12, characterized in that: The target object is a milling cutter.

23. A computer-readable storage medium, wherein at least one instruction is stored in the storage medium, characterized in that: The instructions are loaded and executed by the processor to implement a multi-focus image alignment method as described in any one of claims 1-11.

24. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, a multi-focus image alignment method as described in any one of claims 1 to 11 is implemented.

Images