2D image mixing method based on spatio-temporal feature information fusion and storage medium

Abstract

The invention relates to the technical field of image processing, in particular to a spatial-temporal feature information fused 2D image mixing method and a storage medium. Comprising the following steps: S1, acquiring a 2D original image; s2, preprocessing the original image; s3, feature point mixed information registration: extracting feature points of the image, and mapping the feature points needing registration and splicing into a unified 2D plane after processing and transformation by establishing a mixed space-time transformation matrix of edge feature information; s4, feature point information fusion: establishing a space-time fusion feature matrix, performing fusion of space-time mixed feature information, and completing edge splicing; s5, outputting a fused image; s6, performing post-processing on the fused image to enable the gray threshold of the mixed connection area to be included in the gray threshold of the original image; and S7, outputting a fused mixed splicing image. According to the method, the problems of poor precision and large error of complex image splicing can be well solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a 2D image mixing method and a storage medium for fusing spatiotemporal feature information. Background Art

[0002] With the continuous development of science and technology, machine vision technology has developed rapidly. It can be applied to automobile production, electronic circuits, food production and other fields. Machine vision technology is used to collect data and realize functions such as defect detection, visual measurement, and text recognition. In the production process of some large-sized workpieces, machine vision technology is used to identify and detect them. However, due to the large size of the workpiece, the camera cannot collect the complete workpiece at one time. It is necessary to collect multiple images multiple times and stitch them together to form a complete image.

[0003] Image stitching requires a set of images to be spatially matched and aligned. The key technologies involved in image stitching are how to find the overlapping parts of the images and how to stitch the images with overlapping parts, namely image registration and image fusion. Image registration methods are divided into region-based image registration and feature-based image registration. Among region-based image registration, the simpler one is the grayscale value-based image registration. By obtaining the grayscale values ​​of the model and the image itself, the similarity metrics such as the cross-correlation coefficient or the sum of squares between them are calculated. The normalization of the cross-correlation coefficient is robust to linear brightness changes, but is very sensitive to noise, occlusion or nonlinear illumination changes in the image, and often fails to obtain accurate results or even fails to search for the target. Summary of the invention

[0004] In order to solve the technical problems of poor precision and large errors in the prior art of complex 2D optical image stitching, the present invention provides a 2D image blending method and a storage medium for fusing spatiotemporal feature information.

[0005] To achieve the above object, the technical solution of the present invention is as follows:

[0006] On the one hand, the present invention provides a 2D image blending method for fusing spatiotemporal feature information, comprising:

[0007] S1, collect 2D original images;

[0008] S2, preprocessing the original image;

[0009] S3, feature point mixed information registration: extract the feature points of the image, establish a mixed spatiotemporal transformation matrix of edge feature information, and map the feature points that need to be registered and spliced ​​into a unified 2D plane after processing and transformation;

[0010] S4, feature point information fusion: establish a spatiotemporal fusion feature matrix, fuse the spatiotemporal mixed feature information, and complete edge splicing;

[0011] S5. Output the fused image.

[0012] Furthermore, the preprocessing of the original image includes: denoising, grayscale processing and feature morphological operation.

[0013] Furthermore, the denoising includes Gaussian filtering and median filtering.

[0014] Furthermore, Gaussian filtering denoising is performed using a Gaussian filtering function:

[0015]

[0016] In the above formula, x and y are the pixel coordinates of the original image, and σ represents the standard deviation of the pixel coordinates.

[0017] Furthermore, the median filter uses a 3×3 denoising transformation matrix for denoising.

[0018] Furthermore, the feature morphological operation specifically includes:

[0019] Assume that the original image is represented by the G function, and the linear state equation and observation equation of the G function are:

[0020]

[0021] In the above formula, X(x) is the mathematical vector representation of the state of the original image, Y(x-1) is the feature observation vector at time x-1, W(x-1) is the image noise at time x-1, A represents the linear state amplitude, H represents the noise amplitude, and V(x-1) represents the observation noise. The noise variance matrix of the image can be expressed as:

[0022] Q(x)=E{W(x)W T (x)}

[0023] In the above formula, E is the base of natural logarithm;

[0024] The variance matrix of observation noise can be expressed as:

[0025] R(x)=E{V(x)V T (x)}

[0026] In the above formula, E is the base of natural logarithm;

[0027] According to the gray value of each pixel in the original image G = (x, y), the gray value observation sequence is sorted as: {Y(0), Y(1), Y(2)..., Y(x)}. At this time, the minimum variance matrix of the error estimation can be expressed as:

[0028] P(x / x)=E{e(x / x)eT(x / x)}

[0029] The a posteriori error estimate is:

[0030] e(x / x)=X(x)-X(x / x)

[0031] The a priori error estimate can be expressed as:

[0032] e(x / x-1)=X(x)-X(x / x-1)

[0033] The prior error equation matrix can be obtained as:

[0034] P(x / x-1)=E{e(x / x-1)e T (x / x-1)}

[0035] The posterior estimate based on Kalman filtering is:

[0036] X(x / x)=X(x / x-1)+K(x)[Y(x)-HX(x / x-1)]

[0037] Where K(x) is the singular matrix of X(x);

[0038] After deformation:

[0039] e(x / x)=X(x)-X(x / x)

[0040] =[IK(x)H]e(x / x-1)-K(x)V(x)

[0041] Then we can get:

[0042] P(x / x)=E[e(x / x)e T (x / x)]

[0043] =E{[IK(x)H]e(x / x-1)e T (x / x-1)[IK(x)H] T}-

[0044] E{K(x)V(x)e T (x / x-1)V T (x)K T (x)}-

[0045] E{[IK(x)H]e(x / x-1)V T (x)K T (x)}+

[0046] E{K(x)V(x)V T (x)K T (x)}

[0047] Simplified to:

[0048] P(x / x)=[IK(x)H]P(x / x-1)[IK(x)H] T +K(x)R(x)K T (x)

[0049] The calculated error variance matrix is ​​expressed as:

[0050] P(x / x)=[IK(x)H]P -1 (x / x-1).

[0051] Furthermore, before performing the feature morphological operation, a geometric transformation is performed on the denoised image, and the geometric transformation includes: translation and rotation.

[0052] Furthermore, the translation matrix used for translation processing is:

[0053]

[0054] Furthermore, after the feature morphological operation, similarity transformation and hybrid transformation are performed on the image.

[0055] Furthermore, the matrix for similarity transformation is:

[0056]

[0057] Furthermore, the matrix for the hybrid transformation is:

[0058]

[0059] Furthermore, step S3 specifically includes: after preprocessing the original image, performing mixed feature information transformation, the corresponding mixed feature information transformation equation is:

[0060] G2(x, y) = g1(x-x0, y-y0)

[0061] The feature point information that needs to be fused and registered is mixed and registered, and the corresponding mapping relationship is expressed as:

[0062]

[0063] The corresponding mapping transformation matrix is ​​expressed as:

[0064]

[0065] The registration model of the mapping transformation relationship is:

[0066]

[0067] Further, according to the hybrid spatiotemporal transformation matrix created in step S3, the corresponding matrix transformation is expressed as:

[0068] Furthermore, the spatiotemporal fusion feature matrix expression is:

[0069]

[0070] The spatiotemporal mixed feature information is fused sequentially to complete edge splicing.

[0071] Furthermore, the fused image is expressed as:

[0072] G(x,y)=(1-α)G r (x,y)+αG t (x, y).

[0073] Furthermore, the 2D image mixing method further includes:

[0074] S6, post-processing the fused image so that the grayscale threshold of the blended area is included in the grayscale threshold of the original image;

[0075] S7, output the fused, mixed and stitched image.

[0076] Furthermore, the grayscale threshold transformation mapping relationship of the mixed area in step S6 is expressed as:

[0077]

[0078] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the 2D image blending method for fusing the above-mentioned spatiotemporal feature information is implemented.

[0079] Compared with the prior art, the present invention has the following beneficial effects:

[0080] The 2D image blending method for fusing spatiotemporal feature information provided by the present invention first preprocesses the collected image, and then performs feature point mixed information alignment on the preprocessed image based on its edge feature information, establishes a mixed spatiotemporal transformation matrix of the edge feature information, and then establishes a spatiotemporal fusion feature matrix to fuse the spatiotemporal mixed feature information to complete edge splicing. The present invention can solve the technical problems of poor precision and large errors in complex image splicing, and can meet the requirements of large-area 2D optical image feature recognition and splicing. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 The figure is a flow chart of the method of the present invention.

[0082] Figure 2 This is an example diagram of the 3×3 median filter transformation before and after the present invention.

[0083] Figure 3 This is a mixed splicing graph of the spatiotemporal fusion feature points in the present invention. DETAILED DESCRIPTION

[0084] The technical solution of the present invention will be clearly described below in conjunction with the accompanying drawings. Obviously, the described embodiments are not all embodiments of the present invention, and all other embodiments obtained by ordinary technicians in the field without making creative work are within the protection scope of the present invention.

[0085] It should be noted that, unless otherwise specifically stated, the relative arrangements of components and steps, and numerical expressions set forth in these embodiments should not be construed as limiting the scope of the present invention.

[0086] The following description of the exemplary embodiments is merely illustrative and is not intended to limit the present invention and its application or use in any sense. Techniques, methods and devices known to ordinary technicians in the relevant field may not be discussed in detail here, but where applicable, these techniques, methods and devices should be considered as part of this specification.

[0087] Embodiment 1

[0088] This embodiment provides a 2D image blending method for fusing spatiotemporal feature information. Figure 1 As shown, including:

[0089] S1, collect 2D original images;

[0090] S2. Preprocessing the original image. Preprocessing the original image mainly includes: denoising, grayscale processing and feature morphological operations.

[0091] Among them, denoising includes Gaussian filtering and median filtering, Gaussian filtering is performed first, and then median filtering is performed.

[0092] Gaussian filter denoising uses Gaussian filter function for processing:

[0093]

[0094] In the above formula, x and y are the pixel coordinates of the original image, and σ represents the standard deviation of the pixel coordinates.

[0095] The median filter uses a 3×3 denoising transformation matrix for denoising; the 3×3 denoising transformation matrix is ​​as follows:

[0096]

[0097] like Figure 2 The figure shows the schematic diagram before and after denoising by median filtering.

[0098] Before performing feature morphological operations, the denoised image is subjected to geometric transformation, which includes translation and rotation. The translation matrix used for translation processing is:

[0099]

[0100] The feature morphological operations specifically include:

[0101] Assume that the original image is represented by the G function, and the linear state equation and observation equation of the G function are:

[0102]

[0103] In the above formula, X(x) is the mathematical vector representation of the state of the original image, Y(x-1) is the feature observation vector at time x-1, W(x-1) is the image noise at time x-1, A represents the linear state amplitude, H represents the noise amplitude, and V(x-1) represents the observation noise. The noise variance matrix of the image can be expressed as:

[0104] Q(x)=E{W(x)W T (x)}

[0105] In the above formula, E is the base of natural logarithm;

[0106] The variance matrix of observation noise can be expressed as:

[0107] R(x)=E{V(x)V T (x)}

[0108] In the above formula, E is the base of natural logarithm;

[0109] According to the gray value of each pixel in the original image G = (x, y), the gray value observation sequence is sorted as: {Y(0), Y(1), Y(2)..., Y(x)}. At this time, the minimum variance matrix of the error estimation can be expressed as:

[0110] P(x / x)=E{e(x / x)eT(x / x)}

[0111] The a posteriori error estimate is:

[0112] e(x / x)=X(x)-X(x / x)

[0113] The a priori error estimate can be expressed as:

[0114] e(x / x-1)=X(x)-X(x / x-1)

[0115] The prior error equation matrix can be obtained as:

[0116] P(x / x-1)=E{e(x / x-1)e T (x / x-1)}

[0117] The posterior estimate based on Kalman filtering is:

[0118] X(x / x)=X(x / x-1)+K(x)[Y(x)-HX(x / x-1)]

[0119] Where K(x) is the singular matrix of X(x);

[0120] After deformation:

[0121] e(x / x)=X(x)-X(x / x)

[0122] =[IK(x)H]e(x / x-1)-K(x)V(x)

[0123] Then we can get:

[0124] P(x / x)=E[e(x / x)e T (x / x)]

[0125] =E{[IK(x)H]e(x / x-1)e T (x / x-1)[IK(x)H] T}-

[0126] E{K(x)V(x)e T (x / x-1)V T (x)K T (x)}-

[0127] E{[IK(x)H]e(x / x-1)V T (x)K T (x)}+

[0128] E{K(x)V(x)V T (x)K T (x)}

[0129] Simplified to:

[0130] P(x / x)=[IK(x)H]P(x / x-1)[IK(x)H] T +K(x)R(x)K T (x)

[0131] The calculated error variance matrix is ​​expressed as:

[0132] P(x / x)=[IK(x)H]P -1 (x / x-1).

[0133] After the feature morphological operation, the image is subjected to similarity transformation and hybrid transformation.

[0134] The matrix for similarity transformation is:

[0135]

[0136] The matrix for the mixed transformation is:

[0137]

[0138] S3, feature point mixed information registration: extract the feature points of the image, establish a mixed spatiotemporal transformation matrix of edge feature information, and map the feature points that need to be registered and spliced ​​into a unified 2D plane after processing and transformation;

[0139] After preprocessing the original image, the mixed feature information transformation is performed, and the corresponding mixed feature information transformation equation is:

[0140] G2(x, y) = g1(x-x0, y-y0)

[0141] The feature point information that needs to be fused and registered is mixed and registered, and the corresponding mapping relationship is expressed as:

[0142]

[0143] The corresponding mapping transformation matrix is ​​expressed as:

[0144]

[0145] The registration model of the mapping transformation relationship is:

[0146]

[0147] S4, feature point information fusion: establish a spatiotemporal fusion feature matrix, fuse the spatiotemporal mixed feature information, and complete edge splicing;

[0148] According to the hybrid spatiotemporal transformation matrix created in step S3, the hybrid spatiotemporal transformation matrix is ​​specifically a median filter transformation matrix, a variance matrix of observation noise, a hybrid transformation matrix, a mapping transformation matrix and a registration model, which can be uniformly expressed as:

[0149]

[0150] The spatiotemporal fusion feature matrix is ​​established based on the feature point information after grayscale and feature morphological operation. The expression of the spatiotemporal fusion feature matrix is:

[0151]

[0152] In this way, the spatiotemporal mixed feature information is fused to complete edge splicing.

[0153] S5, output fused image; the fused image is expressed as:

[0154] G(x,y)=(1-α)G r (x,y)+αG t (x,y).

[0155] S6, post-processing the fused image, graying the fused feature area and processing the mixed area features of the fused image, so that the gray threshold of the mixed area is included in the gray threshold of the original image; the mapping relationship of the gray threshold transformation of the mixed area is expressed as:

[0156]

[0157] S7, output fused mixed spliced ​​image, such as Figure 3 The figure shows an illustration of a spatiotemporal fusion feature point hybrid splicing diagram.

[0158] Embodiment 2

[0159] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the 2D image blending method for fusing spatiotemporal feature information provided in the first embodiment is implemented.

[0160] The above specific implementation methods are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to examples, a person skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A 2D image blending method for fusing spatiotemporal feature information, characterized in that: include: S1, collect 2D original images; S2, preprocessing the original image; S3, feature point mixed information registration: extract the feature points of the image, establish a mixed spatiotemporal transformation matrix of edge feature information, and map the feature points that need to be registered and spliced ​​into a unified 2D plane after processing and transformation; S4, feature point information fusion: establish a spatiotemporal fusion feature matrix, fuse the spatiotemporal mixed feature information, and complete edge splicing; S5. Output the fused image.

2. The 2D image blending method according to claim 1, characterized in that: Preprocessing of the original image includes: denoising, grayscale processing and feature morphological operations; The feature morphological operations specifically include: Assume that the original image is represented by the G function, and the linear state equation and observation equation of the G function are: In the above formula, X(x) is the mathematical vector representation of the state of the original image at time x, X(x) is the mathematical vector representation of the state of the original image at time x-1, Y(x-1) is the feature observation vector at time x-1, W(x-1) is the image noise at time x-1, A represents the linear state amplitude, H represents the noise amplitude, and V(x-1) represents the observation noise. The noise variance matrix of the image can be expressed as: Q(x)=E{W(x)W T (x)} In the above formula, E is the base of natural logarithm; The variance matrix of observation noise can be expressed as: R(x)=E{V(x)V T (x)} In the above formula, E is the base of natural logarithm; According to the gray value of each pixel in the original image G = (x, y), the gray value observation sequence is sorted as: {Y(0), Y(1), Y(2)..., Y(x)}. At this time, the minimum variance matrix of the error estimation can be expressed as: P(x / x)=E{e(x / x)eT(x / x)} The a posteriori error estimate is: e(x / x)=X(x)-X(x / x) The a priori error estimate can be expressed as: e(x / x-1)=X(x)-X(x / x-1) The prior error equation matrix is ​​obtained as: P(x / x-1)=E{e(x / x-1)e T (x / x-1)} The posterior estimate based on Kalman filtering is: X(x / x)=X(x / x-1)+K(x)[Y(x)-HX(x / x-1)] Where K(x) is the singular matrix of X(x); After deformation: e(x / x)=X(x)-X(x / x)=[IK(x)H]e(x / x-1)-K(x)V(x) Then we can get: P(x / x)=E[e(x / x)e T (x / x)]=E{[I-K(x)H]e(x / x-1)e T (x / x-1)[I-K(x)H] T }-E{K(x)V(x)e T (x / x-1)V T (x)K T (x)}-E{[I-K(x)H]e(x / x-1)V T (x)K T (x)}+E{K(x)V(x)V T (x)K T (x)} Simplified to: P(x / x)=[I-K(x)H]P(x / x-1)[I-K(x)H] T +K(x)R(x)K T (x) The calculated error variance matrix is ​​expressed as: P(x / x)=[I-K(x)H]P -1 (x / x-1)。 3. The 2D image blending method according to claim 2, characterized in that: After the feature morphological operation, the image is subjected to similarity transformation and hybrid transformation; The matrix for similarity transformation is: The matrix for the mixed transformation is:

4. The 2D image blending method according to claim 3, characterized in that: Step S3 specifically includes: after preprocessing the original image, performing mixed feature information transformation, the corresponding mixed feature information transformation equation is: G2(x, y) = g1(x-x0, y-y0) The feature point information that needs to be fused and registered is mixed and registered, and the corresponding mapping relationship is expressed as: The corresponding mapping transformation matrix is ​​expressed as: The registration model of the mapping transformation relationship is:

5. The 2D image blending method according to claim 4, characterized in that: According to the hybrid spatiotemporal transformation matrix created in step S3, the corresponding matrix transformation is expressed as:

6. The 2D image blending method according to claim 5, characterized in that: The expression of spatiotemporal fusion feature matrix is: The spatiotemporal mixed feature information is fused sequentially to complete edge splicing.

7. The 2D image blending method according to claim 1, characterized in that: The fused image is represented as: G(x,y)=(1-α)G r (x, y)+αG t (x, y)。 8. The 2D image blending method according to claim 1, characterized in that: The 2D image mixing method further includes: S6, post-processing the fused image so that the grayscale threshold of the blended area is included in the grayscale threshold of the original image; S7, output the fused, mixed and stitched image.

9. The 2D image blending method according to claim 8, characterized in that: The grayscale threshold transformation mapping relationship of the mixed area in step S6 is expressed as:

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the 2D image blending method for fusing spatiotemporal feature information as described in any one of claims 1 to 9.

Images