Door defect image correction method based on improved spatial transformation network STN

Abstract

The application discloses a kind of based on the door defect image correction method of improved space transformation network STN, and the door defect image is collected by camera, image is input to the improved space transformation network STN that has been trained in advance, and the image after data enhancement module processing is sent into parameter prediction module by limiting contrast adaptive histogram equalization through data enhancement module, the K reference points needed for coordinate transformation are obtained;Based on the K reference points obtained by parameter prediction module, the reference point is input into coordinate transformation module, and TPS transformation is carried out by thin sample board interpolation, and the feature sampling window required by pixel acquisition module is generated;Feature sampling window and the image after data enhancement module processing are sent into pixel acquisition module, and the input image is corrected to obtain the final corrected image.The present application can correct the stripe light background in the image to be detected to horizontal or vertical direction, and improve the accuracy of subsequent detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a vehicle door defect image correction method based on an improved spatial transformation network STN. BACKGROUND

[0002] In traditional vehicle workshops, the detection of defects in vehicle doors or even entire vehicles often relies on experienced technicians who use their naked eyes to detect defects. This method is time-consuming and labor-intensive, and its accuracy cannot be effectively guaranteed, making it difficult to meet the requirements of modern high-speed and high-quality production processes. Therefore, defect detection technology based on image processing technology has emerged. Traditional defect detection technology based on image processing often uses cameras to take pictures of the objects being tested, and then uses various techniques to detect defects in the obtained images. For flat objects such as steel and glass, this method has good results. However, for high-brightness curved surfaces such as vehicle doors, conventional light sources cannot make defects such as depressions or protrusions visible, so it is necessary to introduce a striped light background. When the stripes are distorted at the defect location, they can be used as a feature for detection. However, due to the presence of curved vehicle doors, the striped light background will be distorted, which will affect the accuracy of subsequent detection work. Therefore, it is necessary to correct the striped light background to reduce the false detection rate of defect detection, so as to make the entire production process faster and of higher quality.

[0003] The present application relates to the technical field of image processing, in particular to a vehicle door defect image correction method based on an improved spatial transformation network STN. Before improvement, the spatial transformation network STN uses affine transformation to correct images. However, the degree of freedom of affine transformation is limited and cannot fully adapt to the curved features of vehicle doors, so further improvement is needed. The improved spatial transformation network STN converts affine transformation to thin-plate spline interpolation TPS, thereby achieving more flexible and accurate image correction. This method corrects the camera-captured vehicle door defect image by correcting the striped light background in the image to be horizontal or vertical, so as to facilitate subsequent detection work and improve the accuracy of detection results. Through this method, we can more accurately and accurately correct the vehicle door defect image. This can effectively reduce the interference of image correction on subsequent detection work, thereby improving the accuracy and reliability of the detection results. At the same time, the improved spatial transformation network STN can also adapt to more complex curved features, and has better adaptability and flexibility.

[0004] Based on the above discussion, it is of high practical application value to invent a vehicle door defect image correction method based on an improved spatial transformation network STN that enhances image quality. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide a vehicle door defect image correction method based on an improved spatial transformation network STN, which can correct the stripe light background in the image to be detected to be horizontal or vertical, and improve the accuracy of subsequent detection.

[0006] To achieve the above purpose, the technical scheme provided by the present application is: a vehicle door defect image correction method based on an improved spatial transformation network STN, which is composed of a data enhancement module, a parameter prediction module, a coordinate transformation module and a pixel acquisition module, and the improvement is that the data enhancement module and the coordinate transformation module are optimized, for the data enhancement module, the limited contrast adaptive histogram equalization is added to preprocess the image to improve the quality of the image, and for the coordinate transformation module, the thin-plate spline interpolation TPS transformation is used instead of affine transformation as the coordinate transformation method to better adapt to the correction requirements of the vehicle door defect image.

[0007] The specific implementation of the vehicle door defect image correction method includes the following steps:

[0008] 1) Collect the vehicle door defect image through the camera, input the image into the improved spatial transformation network STN which has been trained in advance, and perform limited contrast adaptive histogram equalization through the data enhancement module to improve the clarity and contrast of the vehicle door defect image, improve the uneven illumination problem of the image caused by excessive exposure of the camera and poor focusing, and enhance the image quality;

[0009] 2) Send the image processed by the data enhancement module into the CNN of the parameter prediction module to obtain K reference points required for coordinate transformation; wherein the parameter prediction module is composed of a parameter prediction network localization network, and the convolutional neural network CNN is used to construct the parameter prediction network;

[0010] 3) Based on the K reference points obtained by the parameter prediction module, input the reference points into the coordinate transformation module for thin-plate spline interpolation TPS transformation, and the TPS transformation can perform pixel-level operation on the image to generate the feature sampling window required by the pixel acquisition module;

[0011] 4) Send the feature sampling window and the image processed by the data enhancement module in step 1) into the pixel acquisition module to correct the input image and fill the blank interval of the image caused by the thin-plate spline interpolation TPS transformation, and obtain the final corrected image.

[0012] Further, the data enhancement module applies a contrast limited adaptive histogram equalization method, which is first to divide the image into multiple sub-image blocks, then find the middle boundary gray value in each sub-image block; according to this boundary gray value, the image is divided into high brightness area and low brightness area, and the total number of pixel points in each area is counted; the gray adjustment parameters of the high and low brightness areas are calculated through the pixel point numbers of the high and low brightness areas, and the high and low brightness areas are histogram equalized respectively, effectively solving the problem of uneven image illumination caused by excessive exposure or poor focusing of the camera.

[0013] Further, the parameter prediction module performs the following operations:

[0014] The pixels in the input image are normalized, and a reference point c is defined by (x, y), where x represents the horizontal coordinate of the reference point and y represents the vertical coordinate of the reference point. The parameter prediction network locates K basic reference points through regression, so 2K outputs are needed. The overall reference point coordinate set is where is a set of real numbers, is a matrix of size 2xK, and c1, c2,..., c K represents the first to the Kth reference point, using a normalized coordinate system, with the origin of the coordinate system being the center of the input image, and the reference point (x, y) value in the interval range of [-1, 1];

[0015] The parameter prediction module is composed of a parameter prediction network localization network, which is constructed by using a convolutional neural network CNN. The pytorch library is used to build this network. The convolutional neural network CNN includes Conv traditional convolutional layer, Maxpool maximum pooling layer and FC fully connected layer. The entire network structure is composed of four convolutional layers, one maximum pooling layer and two fully connected layers. This structure can efficiently predict parameters, and the design features of the convolutional neural network CNN are considered in the construction process;

[0016] For the four convolutional layers, the first convolutional layer has 64 filters, a 3x3 filter size, 1 padding, and a 1 stride. The second, third, and fourth convolutional layers have 2 filters each, meaning the next three convolutional layers have twice the number of filters as the previous one, while their filter size, padding, and stride are the same as the first convolutional layer. The max-pooling layer has 512 filters, a 2x2 filter size, 1 padding, and a 1 stride. The fully connected layer uses tanh as the activation function, ensuring the output vector is within the range of [-1, 1]. This design enables the network to predict parameters more efficiently and improves prediction accuracy.

[0017] Furthermore, the coordinate transformation module performs the following operations:

[0018] Step 1: Define the basic reference points. Before performing the thin template interpolation TPS transformation, first define another set of reference points, called the basic reference points. Let C′ represent the set of basic reference point coordinates, c1′, c2′, ..., c K C′ represents the first to the Kth basic reference points, which are evenly distributed along the top and bottom edges of the corrected image. Since K is a constant and the coordinate system has been normalized, C′ is also always a constant.

[0019] Step 2: Calculate the transformation matrix. After obtaining the global reference point coordinate set C and the basic reference point coordinate set C′ from the parameter prediction network localization network, the transformation matrix... The value can be determined:

[0020]

[0021] In the above formula, T represents the transformation matrix, and T represents the matrix transpose. -1 Represents finding the inversion of a matrix. Represents a 2×(K+3) matrix, Δ C′ The intermediate matrix is ​​calculated from C′:

[0022]

[0023] In the above formula, 1 K×1 It is a vector matrix of size K×1, where all values ​​are 1. 1×K It is a 1×K vector matrix containing only 1 values. It is a product of r i,jThe resulting K×K vector matrix, where:

[0024]

[0025] d i,j =Euclidean(c i ′,c j ′)

[0026] In the above formula, r i,j d represents the element in the i-th row and j-th column of matrix R. i,j Represents the basic reference point c i ′ and c j Euclidean distance between two points;

[0027] Step 3: Generate the feature sampling pane. The feature sampling pane on the corrected image is then used... It means that p i ′=[x′ i ,y′ i ] T x′ represents the i-th pixel on the correction network. i y′ represents the x-coordinate and y-coordinate respectively, N is the number of pixels, and each point p in the feature sampling pane on the corrected image... i ′, can all be found by the following thin template interpolation TPS transform to find the pixel point p corresponding to the feature sampling pane of the input image. i p i =[x i ,y i ] T x represents the i-th pixel in the feature sampling pane of the input image. i ,y i These represent the x-axis and y-axis respectively:

[0028]

[0029]

[0030]

[0031] In the above formula, r′ i,k It is p i and the kth basic reference point c′ k The square Euclidean distance between them, r′ i，K It is p i and the Kth basic reference point c′ K The square Euclidean distance between them, d i,k It is p i and the kth basic reference point c′ k The Euclidean distance between them It is the fitting matrix for the i-th reference point, obtained by traversing the feature sampling panes of the input image. All points in the input image are used to generate a feature sampling pane. The feature sampling pane is differentiable, so the gradient can be backpropagated.

[0032] Furthermore, the pixel sampling module performs the following operations:

[0033] p′ i The pixel values ​​from the input image I p i Bilinear interpolation is performed on the pixels near the target pixel. It can perform interpolation calculations on the four neighboring pixels around the target pixel to obtain the value of the target pixel. By setting all pixel values, the corrected image I′ is obtained.

[0034]

[0035] In the formula, V represents the bilinear sampler, which is also a differentiable module. Thin template interpolation TPS transform allows it to flexibly change the image according to the geometry and features of the input image, thereby achieving more flexible and accurate image correction. Through thin template interpolation TPS transform, defective images in different situations can be corrected, thereby obtaining more accurate and reliable image results.

[0036] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0037] 1. By employing contrast-limited adaptive histogram equalization, compared to unlimited contrast-limited adaptive histogram equalization, noise in relatively uniform areas of the image can be avoided by over-amplifying it, thereby improving image lighting problems caused by camera overexposure and poor focus.

[0038] 2. This method uses an improved Spatial Transform Network (STN) to correct the original car door defect image. Unlike traditional affine transformation, the improved Thin Template Interpolation (TPS) transform can flexibly convert irregular images into high-quality images with horizontal or vertical striped backgrounds, and it has good correction effects for different striped background intervals, different striped background angles, and different camera shooting angles.

[0039] 3. This method uses a Convolutional Neural Network (CNN) to construct a parameter prediction network (localization network). As the most classic convolutional neural network, CNN has the characteristics of high speed, high accuracy, and strong scalability. This network can be optimized to achieve better results.

[0040] 4、The coordinate transformation module used in the method is different from the affine transformation used in the previous space transformation network STN, and the thin plate spline interpolation TPS transformation allows us to flexibly convert irregular images into high-quality images with horizontal or vertical stripes in the light background, so as to realize more accurate and accurate image correction.

[0041] 5、The pixel acquisition module of the method is differentiable, and after the differentiable localization network and the differentiable grid generator, the improved space transformation network STN can perform back propagation error differentiation for training, and the fiducial points and the basic fiducial points can be transformed in both directions, so as to realize more efficient and accurate image correction. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The method logical flowchart of the application.

[0043] Figure 2 The car door defect image with stripe light background collected by the application.

[0044] Figure 3 The car door defect image processed by the limited contrast adaptive histogram equalization.

[0045] Figure 4 The structure diagram of the improved space transformation network STN; in the figure, Input Image I represents the input image, C represents the fiducial points, Rectified Image I' represents the corrected image, Localization Network represents the parameter prediction module, Grid Generator represents the coordinate transformation module, and Sampler represents the pixel acquisition module, representing the feature sampling window.

[0046] Figure 5 The structure diagram of the convolutional neural network CNN.

[0047] Figure 6 The conversion diagram of the fiducial points and the basic fiducial points; in the figure, Input Image I represents the input image, Rectified Image I' represents the corrected image, C represents the fiducial points, C' represents the basic fiducial points, p i representing the input image feature sampling window pixel point, i ' represents the corrected image feature sampling window pixel point, representing the conversion matrix.

[0048] Figure 7 The actual car door defect image correction effect diagram. DETAILED DESCRIPTION

[0049] The application will be further described below with reference to specific embodiments.

[0050] As Figures 1 to 7 shown, the embodiment provides a vehicle door defect image correction method based on an improved spatial transformation network STN, which is composed of a data enhancement module, a parameter prediction module, a coordinate transformation module and a pixel acquisition module. The improvement is that the data enhancement module and the coordinate transformation module are optimized. For the data enhancement module, a limited contrast adaptive histogram equalization is added to preprocess the image to improve the quality of the image. For the coordinate transformation module, a thin-plate spline interpolation TPS transformation is used instead of affine transformation as the coordinate transformation method to better adapt to the correction requirements of the vehicle door defect image.

[0051] The specific implementation of the vehicle door defect image correction method includes the following steps:

[0052] 1) A mechanical arm carries a camera to take pictures of the vehicle door to obtain a defect image with a striped light background, as shown in Figure 2 The image is input into the improved spatial transformation network STN which has been trained in advance. The data enhancement module performs limited contrast adaptive histogram equalization, as shown in Figure 3 to improve the clarity and contrast of the vehicle door defect image, improve the uneven illumination problem caused by excessive camera exposure and poor focusing, and enhance the image quality.

[0053] The data enhancement module applies a limited contrast adaptive histogram equalization method, which is as follows: convert the input image into a luminance grayscale image, divide it into multiple sub-image blocks, and count the number of pixel points at each grayscale level in each sub-image block; find the dividing grayscale level that divides the total number of pixel points in each sub-image block into the closest half, and count the total number of pixel points in the high and low luminance regions of the image; respectively, accumulate the histogram of the high and low luminance regions of each sub-image block, calculate the adjustment parameters of the high and low luminance regions, which are the total number of pixel points in the high and low luminance regions of the sub-image block divided by the total number of pixel points in the sub-image block, multiplied by 255; for the low luminance region pixel point, accumulate the number of pixel points at each grayscale level from 0 to the grayscale level of the point to obtain the total number of pixel points, divide by the total number of pixel points in the low luminance region, and multiply by the adjustment parameter of the low luminance region to obtain the cumulative histogram equalization grayscale value of the pixel point; for the high luminance region pixel point, accumulate the number of pixel points at each grayscale level from the grayscale level of the point to 255 to obtain the total number of pixel points, divide by the total number of pixel points in the high luminance region, and multiply by the adjustment parameter of the high luminance region to obtain the cumulative histogram equalization grayscale value of the pixel point; according to the cumulative histogram equalization grayscale value of the pixel points of each sub-image block, perform weighted linear interpolation to obtain the luminance information of the entire image after equalization processing, as shown in Figure 3 .

[0054] 2) The image processed by the data enhancement module is sent to the parameter prediction module, and the following operations are performed:

[0055] The pixels in the input image are normalized, and a reference point c is defined by (x, y), where x represents the horizontal coordinate of the reference point and y represents the vertical coordinate of the reference point. The parameter prediction network locates the K basic reference points by regression, so 2K outputs are needed. The overall reference point coordinate set is where is a set of real numbers, is a matrix of size 2xK, and c1, c2,..., c K represents the use of a normalized coordinate system from the 1st to the Kth reference point, with the origin of the coordinate system being the center of the input image, and the reference point (x, y) value in the interval range [-1, 1].

[0056] The parameter prediction module is composed of a parameter prediction network localization network. The network is constructed by using a convolutional neural network CNN, and the pytorch library is used to build the network, as shown in Figure 5 The structure of the convolutional neural network CNN includes Conv traditional convolutional layers, Maxpool maximum pooling layers, and FC fully connected layers. The entire network is mainly composed of four convolutional layers, one maximum pooling layer, and two fully connected layers. This structure can efficiently perform parameter prediction, and the design features of the convolutional neural network CNN are considered in the construction process.

[0057] For the four convolutional layers, the first convolutional layer has 64 filters, the filter size is set to 3*3, the padding is set to 1, and the stride is set to 1. The filter number of the second, third, and fourth convolutional layers is multiplied by 2 in turn, i.e. the filter number of the next three convolutional layers is twice that of the previous convolutional layer, and their filter size, padding, and stride are the same as the first convolutional layer. The filter number of the maximum pooling layer is set to 512, the filter size is set to 2*2, the padding is set to 1, and the stride is set to 1. The fully connected layer uses tanh as the activation function, so that the output vector value is in the range [-1, 1]. The above design makes the network more efficient in parameter prediction and improves the prediction accuracy.

[0058] 3) Based on the K reference points obtained by the parameter prediction module, the reference points are input into the coordinate transformation module, and the following operations are performed:

[0059] Step 1: Define the basic reference points. Before performing the thin template interpolation TPS transformation, first define another set of reference points, called the basic reference points. Let C′ represent the set of basic reference point coordinates, c1′, c2′, ..., c K C′ represents the first to the Kth basic reference points, which are evenly distributed along the top and bottom edges of the corrected image. It should be noted that since K is a constant and the coordinate system has been normalized, C′ is also always a constant.

[0060] Step 2: Calculate the transformation matrix. After obtaining the global reference point coordinate set C and the basic reference point coordinate set C′ from the parameter prediction network localization network, the transformation matrix... The value can be determined:

[0061]

[0062] In the above formula, T represents the transformation matrix, and T represents the matrix transpose. -1 Represents finding the inversion of a matrix. Represents a 2×(K+3) matrix, Δ C′ The intermediate matrix is ​​calculated from C′:

[0063]

[0064] In the above formula, 1 K×1 It is a vector matrix of size K×1, where all values ​​are 1. 1×K It is a 1×K vector matrix containing only 1 values. It is a product of r i,j The resulting K×K vector matrix, where:

[0065]

[0066] d i,j =Euclidean(c i ′,c j ′)

[0067] In the above formula, r i,j d represents the element in the i-th row and j-th column of matrix R. i,j Represents the basic reference point c i ′ and c i The Euclidean distance between two points.

[0068] Step 3: As Figure 6 As shown, a feature sampling pane is generated, and the feature sampling pane on the corrected image is used... It means that pi = [x' i , y' i ] T represents the i-th pixel on the correction network, x' i , y' represent the horizontal and vertical coordinates respectively, and N is the number of pixels. Each point p i ' on the feature sampling window of the corrected image can be found by the thin-plate spline (TPS) transformation from the corresponding pixel point p i on the input image feature sampling window as follows: i = [x i , y i ] T represents the i-th pixel on the input image feature sampling window, x i , y i represent the horizontal and vertical coordinates respectively:

[0069]

[0070]

[0071]

[0072] In the above formula, r' i,k is the squared Euclidean distance between p' i and the k-th basic reference point c' k , r' i,K is the squared Euclidean distance between p' i and the k-th basic reference point c' K , d i,k is the Euclidean distance between p' i and the k-th basic reference point c' k , is the fitting matrix of the i-th reference point, which is generated by traversing all points in the input image feature sampling window . The feature sampling window is differentiable, so the gradient can be back-propagated.

[0073] 4) The feature sampling window and the image processed by the data augmentation module in step 1) are sent to the pixel collection module, and the following operations are performed:

[0074] The pixel value of p' i is bilinearly interpolated from the nearby pixel points of p i on the input image I. It can interpolate the values of the four neighboring pixel points around the target pixel point to obtain the value of the target pixel point. By setting all pixel values, the corrected image I' is obtained:

[0075]

[0076] Wherein V represents a bilinear sampler, which is also a differentiable module, the thin plate spline interpolation TPS transformation allows it to be flexible according to the geometry and characteristics of the input image, so as to realize more flexible and accurate image correction. Through the thin plate spline interpolation TPS transformation, we can correct the defect images in different situations, so as to obtain more accurate and reliable image results, such as Figure 7 The corrected image is shown.

[0077] The above-described embodiments are only the preferred embodiments of the present application, and are not intended to limit the scope of the present application. Any changes made in shape and principle according to the present application should be covered within the scope of protection of the present application.

Claims

1. A door defect image correction method based on an improved spatial transformation network (STN), characterized in that, The improved space transformation network STN is composed of a data enhancement module, a parameter prediction module, a coordinate transformation module and a pixel acquisition module, and the improvement lies in that the data enhancement module and the coordinate transformation module are optimized, for the data enhancement module, the limited contrast adaptive histogram equalization is added to preprocess the image to improve the quality of the image, and for the coordinate transformation module, the thin-plate spline interpolation TPS transformation is used instead of affine transformation as the coordinate transformation method to better adapt to the correction requirements of the door defect image; The specific implementation of the door defect image correction method includes the following steps: 1) Collect the door defect image through the camera, input the image into the improved space transformation network STN which has been trained in advance, perform limited contrast adaptive histogram equalization through the data enhancement module to improve the definition and contrast of the door defect image, and improve the uneven image illumination problem caused by excessive camera exposure and poor focusing to enhance the image quality; 2) Send the image processed by the data enhancement module into the CNN of the parameter prediction module to obtain K reference points required for coordinate transformation; wherein the parameter prediction module is composed of a parameter prediction network localization network, and the convolutional neural network CNN is used to construct the parameter prediction network; 3) Based on the K reference points obtained by the parameter prediction module, input the reference points into the coordinate transformation module for thin-plate spline interpolation TPS transformation, and the TPS transformation can perform pixel-level operation on the image to generate the feature sampling window required by the pixel acquisition module; 4) Send the feature sampling window and the image processed by the data enhancement module in step 1) into the pixel acquisition module to correct the input image and fill the image blank interval caused by the thin-plate spline interpolation TPS transformation to obtain the final corrected image. 2.The vehicle door defect image correction method based on an improved spatial transformation network (STN) according to claim 1, wherein, The data enhancement module applies a limited contrast adaptive histogram equalization method, which first divides the image into multiple sub-image blocks, then finds the middle boundary gray value in each sub-image block; according to the boundary gray value, the image is divided into high brightness area and low brightness area, and the total number of pixels in each area is counted; the gray adjustment parameters of the high and low brightness areas are calculated through the number of pixels in the high and low brightness areas, and the high and low brightness areas are histogram equalized respectively, effectively solving the problem of uneven image illumination caused by excessive camera exposure or poor focusing. 3.The vehicle door defect image correction method based on the improved spatial transformation network (STN) according to claim 2, characterized in that, The parameter prediction module performs the following operations: The pixels in the input image are normalized, a reference point c is defined by (x, y), x represents the horizontal coordinate of the reference point, and y represents the vertical coordinate of the reference point. The parameter prediction network is positioned by regressing K basic reference points, so 2K outputs are needed, and the overall reference point coordinate set is wherein is a real number set, is a matrix with a size of 2xK, c1, c2,..., c K is used from the 1st to the Kth reference point, using a normalized coordinate system, the coordinate system origin is the center of the input image, and the reference point (x, y) value is in the interval range of [-1, 1]. The parameter prediction module is composed of a parameter prediction network localization network, which is constructed by using a convolutional neural network CNN, and the pytorch library is used to build the network, the convolutional neural network CNN includes a traditional convolution layer Conv, a maximum pooling layer Maxpool and a full connection layer FC, the entire network structure is composed of four convolution layers, one maximum pooling layer and two full connection layers, this structure can efficiently predict parameters, and the design characteristics of the convolutional neural network CNN are considered in the construction process; For the four convolutional layers, the first convolutional layer is set to have 64 filters, a filter size of 3*3, a padding of 1, and a stride of 1; the filter number of the second, third and fourth convolutional layers is multiplied by 2 in turn, that is, the filter number of the next three convolutional layers is twice that of the previous convolutional layer, and the filter size, padding and stride of the three convolutional layers are the same as those of the first convolutional layer; the filter number of the max pooling layer is set to 512, the filter size is set to 2*2, the padding is set to 1, and the stride is set to 1; the fully connected layer uses tanh as the activation function, so that the value of the output vector is in the range of [-1, 1]; the above design enables the network to more efficiently predict parameters and improve the accuracy of prediction.

4. The vehicle door defect image correction method based on an improved spatial transformation network (STN) according to claim 3, characterized in that, The coordinate transformation module performs the following operations: First step: define basic fiducials, before performing the thin-plate spline (TPS) transformation, another set of fiducials, called basic fiducials, is defined as C' = {c1', c2',..., cK'}. C' represents the basic fiducial coordinate set, c1', c2',..., cK' represents the first to the Kth basic fiducial. The basic fiducials are evenly distributed along the top and bottom edges of the rectified image. Since K is a constant and the coordinate system is normalized, C' is also a constant. K C' = {c1', c2',..., cK'}. C' represents the basic fiducial coordinate set, c1', c2',..., cK' represents the first to the Kth basic fiducial. The basic fiducials are evenly distributed along the top and bottom edges of the rectified image. Since K is a constant and the coordinate system is normalized, C' is also a constant. Second step: calculate the conversion matrix, after the overall reference point coordinate set C and the basic reference point coordinate set C' are obtained from the parameter prediction network localization network, the value of the conversion matrix can be determined: in the above formulae, denotes the conversion matrix, T denotes the matrix transpose, -1 denotes the matrix inversion, denotes a 2 x (K+3) matrix, Δ C′ is an intermediate matrix calculated from C' : In the above equation, 1 K×1 is a K x 1 vector matrix of all ones, 1 1×K is a 1 x K vector matrix of all ones, is a K x K vector matrix composed of r i,j wherein: d i,j = Euclidean(c i ′, c j ′) In the above formula, r i,j represents the element of the i-th row and j-th column of matrix R, d i,j represents the Euclidean distance between the two points c i and c j ′. Step 3: Generate the feature sampling pane. The feature sampling pane on the corrected image is then used... It means that p′ i =[x′ i ,y′ i ] T x′ represents the i-th pixel on the correction network. i y′ represents the x-coordinate and y-coordinate respectively, N is the number of pixels, and each point p in the feature sampling pane on the corrected image... i ′, can all be found by the following thin template interpolation TPS transform to find the pixel point p corresponding to the feature sampling pane of the input image. i p i =[x i ,y i ] T x represents the i-th pixel in the feature sampling pane of the input image. i ,y i These represent the x-axis and y-axis respectively: In the above equation, r' is the squared Euclidean distance between p' and the kth basic reference point c' i,k i k In the above equation, r' is the squared Euclidean distance between p' and the kth basic reference point c' i,K i K In the above equation, d is the Euclidean distance between p' and the kth basic reference point c' i,k i k is the fitting matrix of the ith reference point, which is generated by traversing all the points in the feature sampling window of the input image is the feature sampling window of the input image, which is differentiable so that the gradient can be back-propagated.​​​​​​​ 5. The method for vehicle door defect image correction based on improved spatial transformation network (STN) according to claim 4, characterized in that, The pixel acquisition module performs the following operations: p′ i The pixel values ​​from the input image I p i Bilinear interpolation is performed on the pixels near the target pixel. It can perform interpolation calculations on the four neighboring pixels around the target pixel to obtain the value of the target pixel. By setting all pixel values, the corrected image I′ is obtained. In the formula, V represents a bilinear sampler, which is also a differentiable module; the thin plate spline interpolation TPS transformation allows it to flexibly change the image according to the geometry and characteristics of the input image, thereby achieving more flexible and accurate image correction; through the thin plate spline interpolation TPS transformation, defect images in different situations can be corrected, thereby obtaining more accurate and reliable image results.

Images