An image rectification method and system based on target intercepting and vanishing point hybrid calculation

Abstract

The application discloses a kind of computer vision and image processing technical field's based on target interception and vanishing point hybrid calculation image rectification method and system, to solve the problem that cannot quickly find vanishing point in the image rectification of prior art, and it is more complex to remove pseudo vanishing point etc., it includes collecting image data, intercepting target image, positioning and linear fitting to target image edge to obtain vanishing point vector data, by vanishing point hybrid calculation model processing to obtain the final predicted vanishing point coordinate, finally carries out image rectification.The application first uses target detection model to detect the position where main target is located, then intercepts, reduces the interference of background information, then the vanishing point hybrid calculation model is fused and calculated to the vanishing point obtained by fitting, to obtain real vanishing point coordinate, reduce vanishing point screening complexity.

Description

TECHNICAL FIELD

[0001] The application relates to an image rectification method and system based on target interception and vanishing point hybrid calculation, and belongs to the technical field of computer vision and image processing. BACKGROUND

[0002] In recent years, image rectification technology has been widely applied in the field of computer vision. The purpose of image rectification is to eliminate the distortion caused by perspective projection in the image, so that the geometric shape in the image is consistent with the actual scene. The development of image rectification technology benefits from the progress in the field of computer vision and image processing, including the application of feature extraction and description, geometric transformation, machine learning and other technologies, so that image rectification technology is widely used in document processing, image recognition, computer vision navigation and other fields, and the accuracy and efficiency of image processing are continuously improved.

[0003] Among them, the vanishing point technology is one of the important background technologies in image rectification. One of the commonly used technologies in traditional image rectification methods is the correction method based on vanishing points. The vanishing point refers to the point where parallel lines intersect in the image under perspective projection. The correction method based on vanishing points detects the parallel lines in the image, calculates the intersection point as the vanishing point, and then rectifies the image.

[0004] In the image rectification method, how to quickly find the vanishing point is crucial. The traditional method of finding the vanishing point mainly detects straight lines, detects the lines in the image, then calculates the intersection points of all straight lines, and finally removes the false vanishing points. However, this method is not good in the case of complex image and many lines, and it is difficult to calculate the intersection points of straight lines and remove false vanishing points. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art, and to provide an image rectification method and system based on target interception and vanishing point hybrid calculation. The position of the main target is detected using a target detection model, the background information is reduced by interception, and the real vanishing point coordinates are obtained by fusing and calculating the fitted vanishing points using a vanishing point hybrid calculation model, thereby reducing the complexity of vanishing point screening.

[0006] To achieve the above purpose, the technical scheme of the present application is as follows:

[0007] On the one hand, the present application provides an image rectification method based on target interception and vanishing point hybrid calculation, comprising the following steps:

[0008] Collecting image data under each shooting angle, and pre-processing the image data to obtain pre-processed image data.

[0009] Using a target detection model to intercept target images from the pre-processed image data.

[0010] The target image is processed by Gaussian filtering. The gradient magnitude and direction of each pixel in the target image after Gaussian filtering are calculated to locate the edge of the target image and obtain the edge detection output image. The edge detection output image is then fitted with a straight line, and the coordinates of the intersection of the fitted lines are calculated, which are the vanishing points. All vanishing point coordinates are collected to obtain the vanishing point vector data.

[0011] One image is selected from the preprocessed image data. Its features are extracted using the vanishing point hybrid calculation model and then mixed with the vanishing point vector data to obtain the final predicted vanishing point coordinates.

[0012] Based on the final predicted vanishing point coordinates and the length, width, and corner coordinates of the target image, the parameters in the perspective transformation matrix are calculated, and image correction is performed.

[0013] Furthermore, the preprocessing includes standardization and zero-mean normalization, with the standardization expression as follows:

[0014]

[0015] Where x is the image data sample, μ is the sample mean, and σ is the sample standard deviation. * These are standardized image data samples.

[0016] Zero-mean normalization of data involves subtracting the mean of each dimension of an image data sample from its data. This centers all dimensions of the image data sample to 0. The expression is as follows:

[0017]

[0018] Where N represents the total number of image data samples, x i This represents the value of the i-th dimension of the image data sample. Let x' represent the value of the i-th dimension of the j-th image data sample. i This represents the value of the i-th dimension of the image data sample after zero-mean processing.

[0019] Furthermore, the target image is extracted from the image using an object detection model, including:

[0020] Feature extraction is performed on the preprocessed image data using a feature extraction network to obtain feature map C1.

[0021] Feature prediction is performed on feature map C1 using a multi-scale feature prediction network to obtain feature prediction layer C2. Prior boxes of different sizes are formed on feature prediction layer C2, and the prior box with the minimum loss is selected to extract the target image.

[0022] Furthermore, the target image is subjected to Gaussian filtering, including:

[0023] A Gaussian function is used as the filter kernel, and a convolution operation is performed with the target image in the spatial domain. The expression of the Gaussian function is as follows:

[0024]

[0025] Where G(i,j) is the weight value of the coordinate position of the Gaussian filter kernel (i,j), σ is the standard deviation of the Gaussian function, and k is the radius of the Gaussian filter kernel.

[0026] For each pixel in the image, calculate its weighted average with respect to the Gaussian kernel, as shown in the following expression:

[0027]

[0028] Where I'(x,y) is the weighted average of pixel (x,y) and Gaussian filter kernel in the image, G(m,n) is the weight value of the coordinate position of Gaussian filter kernel (m,n), and I(x+m,y+n) is the pixel value of pixel (x+m,y+n) in the image.

[0029] Furthermore, the gradient magnitude and direction of each pixel in the target image after Gaussian filtering are calculated to locate the edges of the target image, resulting in an edge detection output image, including:

[0030] By performing convolution operations in the horizontal and vertical directions using the Sobel operator, the gradient magnitude and direction of each pixel can be obtained.

[0031] For each pixel, check its two adjacent points in the gradient direction. If the gradient magnitude of the current pixel is greater than the gradient magnitudes of its two adjacent points in the gradient direction, then retain the pixel; otherwise, suppress the pixel by setting its gradient magnitude to 0.

[0032] Set a high threshold and a low threshold for the gradient magnitude. Compare the gradient magnitude of each pixel with the threshold. Pixels with a gradient magnitude greater than the high threshold are considered strong edge points, those with a gradient magnitude less than the low threshold are considered non-edge points, and those in between are considered weak edge points.

[0033] For non-edge points, they are directly suppressed, that is, the gradient magnitude of the pixel is set to 0.

[0034] For weak edge points, first check if there are strong edge points in their neighboring points. If there are, then the point is considered a strong edge point; otherwise, the pixel is suppressed.

[0035] The edge detection output image is obtained by sequentially connecting strong edge points, and the length and starting coordinate information of the straight line segments formed during the connection process are retained.

[0036] Furthermore, a straight line is fitted to the edge detection output image, and the coordinates of the intersection points of the fitted lines are calculated, which are the vanishing points. All vanishing point coordinates are collected to obtain vanishing point vector data, including:

[0037] Set the length of the line segment, keep the line segments with a length greater than the set value, and remove the line segments that are too short. Perform line fitting on the set of retained line segments to obtain the slope and intercept parameters of each line.

[0038] For a fitted straight line, calculate the coordinates of its intersection points pairwise. The intersection points are the vanishing points, and all the intersection point information is used to form the vanishing point vector data.

[0039] Furthermore, from the preprocessed image data, one image is selected, and its features are extracted using a vanishing point fusion calculation model. This extracted features are then fused with the vanishing point vector data to obtain the final predicted vanishing point coordinates, including:

[0040] Randomly select one image data D from the preprocessed image data.

[0041] Feature map D2 is obtained by extracting features from image data D using a multi-angle feature extraction network.

[0042] The vanishing point weight fusion network is used to fuse the vanishing points of feature map D2 to obtain feature map D3. Feature map D3 is then output by the vanishing point hybrid calculation model to obtain the final predicted vanishing point coordinates.

[0043] Furthermore, based on the final predicted vanishing point coordinates and the length, width, and corner coordinates of the target image, the parameters in the perspective transformation matrix are calculated, and the expression for image correction is as follows:

[0044]

[0045] x′=m11*x+m12*y+m13

[0046] y′=m21*x+m22*y+m23

[0047] in, Let (x, y) be the perspective transformation matrix, (x, y) be the coordinates of the original image, and (x', y') be the corrected coordinates.

[0048] On the other hand, the present invention also provides an image correction system based on target cropping and vanishing point hybrid calculation, for implementing the image correction method based on target cropping and vanishing point hybrid calculation as described in any of the above claims, comprising:

[0049] Image acquisition and processing module: Used to acquire image data from various shooting angles and preprocess it.

[0050] Target image cropping module: Used to crop target images from preprocessed image data using a target detection model.

[0051] Vanishing point set calculation module: This module performs Gaussian filtering on the target image, calculates the gradient magnitude and direction of each pixel in the target image after Gaussian filtering, locates the edges of the target image, and obtains the edge detection output image. It then performs line fitting on the edge detection output image, calculates the coordinates of the intersection of the fitted lines, which are the vanishing points, and sets all the vanishing point coordinates to obtain the vanishing point vector data.

[0052] Vanishing point prediction module: This module selects an image from the preprocessed image data, extracts its features using a vanishing point hybrid calculation model, and then performs a hybrid calculation with the vanishing point vector data to obtain the final predicted vanishing point coordinates.

[0053] Image correction module: Used to calculate the parameters in the perspective transformation matrix based on the final predicted vanishing point coordinates and the length, width, and corner coordinates of the target image, and to perform image correction.

[0054] Optionally, the target detection model includes a feature extraction network and a multi-scale feature prediction network.

[0055] The feature extraction network includes a convolutional layer that increases the number of channels and three convolutional units. The convolutional layer that increases the number of channels is used to increase the number of channels in the image data to 64. The convolutional unit consists of a dilated convolutional layer and a normal convolutional layer. The dilated convolutional layer has a dilation rate of 2, which is used to increase the receptive field and reduce the number of parameters. The normal convolutional layer has a stride of 2, and the number of its convolutional kernels is twice the number of channels in the output feature map of the dilated convolutional layer in that unit.

[0056] The multi-scale feature prediction network includes a convolutional layer that increases the number of channels, a first downsampling convolutional layer, and a second downsampling convolutional layer. The convolutional layer that increases the number of channels is used to increase the number of channels in the feature map to 1024, thus obtaining the first feature prediction layer.

[0057] The first downsampling convolutional layer has a kernel size of 3 and a stride of 2, and the number of kernels is half the number of channels in the first feature prediction layer, thus obtaining the second feature prediction layer.

[0058] The number of convolution kernels in the second downsampling convolutional layer is half the number of channels in the second feature prediction layer, resulting in the third feature prediction layer, which is feature prediction layer C2.

[0059] Optionally, the vanishing point calculation model includes a multi-angle feature extraction network and a vanishing point weight fusion network. The multi-angle feature extraction network sequentially includes three downsampling convolutional layers, one convolutional layer that increases the number of channels, and three parallel convolutional layers with different kernel sizes. It is used to adjust the number of channels of the image data to 256 to obtain feature map D1, and then the output is stitched together after processing by the three parallel convolutional layers to form feature map D2.

[0060] The vanishing point weight fusion network is used to perform global average pooling and softmax operations on feature map D2 to obtain a channel weight vector l1 with a length of 256. The length of the vanishing point vector data is expanded to 256, and zeros are padded to the insufficient part to obtain the vanishing point vector l2.

[0061] Add the channel weight vector l1 and the vanishing point vector l2 by multiplying them at the same position to obtain vector l. Multiply l by feature map D2 and apply it to the feature map to obtain feature map D3.

[0062] The vanishing point calculation model also includes a convolutional layer and a fully connected layer. The feature map D3 is downsampled and convolutional, and then output through a fully connected layer to obtain the final predicted vanishing point coordinates.

[0063] Optionally, during the training phase, the vanishing point fusion calculation model needs to calculate the loss between the predicted vanishing point coordinates and the true vanishing point coordinates of the image, and update the various parameters in the vanishing point fusion calculation model through backpropagation to reduce the loss between the true and predicted values. The expression is as follows:

[0064]

[0065] Where loss is the difference between the true value and the predicted value, Q1 and Q2 are the final predicted vanishing point coordinates, and Q'1 and Q'2 are the true vanishing point coordinates of the image data.

[0066] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0067] This invention first uses a target detection model to detect the location of the main target. Subsequent operations are only performed on the extracted main image block, which reduces the interference of lines in the background information, reduces the computational load of the subsequent edge detection model and line fitting steps, and improves the speed of obtaining the vanishing point set.

[0068] This invention uses a vanishing point hybrid calculation model, which inputs multiple vanishing points and image features into the model to obtain the coordinates of the final true vanishing points, simplifying the complexity of selecting vanishing points and improving image correction efficiency. Attached Figure Description

[0069] Figure 1 This is a flowchart illustrating an image correction method based on a hybrid calculation of target cropping and vanishing point in one embodiment of the present invention.

[0070] Figure 2 This is a schematic diagram of the feature extraction network structure of an image correction method based on target cropping and vanishing point hybrid calculation in one embodiment of the present invention;

[0071] Figure 3 This is a schematic diagram of the structure of a multi-scale feature prediction network for an image correction method based on a hybrid calculation of target cropping and vanishing point, according to one embodiment of the present invention.

[0072] Figure 4 This is a schematic diagram of the vanishing point hybrid calculation model of an image correction method based on target cropping and vanishing point hybrid calculation in one embodiment of the present invention. Detailed Implementation

[0073] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0074] Example 1:

[0075] like Figure 1 As shown in the figure, an image correction method based on target cropping and vanishing point hybrid calculation provided by an embodiment of the present invention includes the following steps:

[0076] S1. Collect image data from various shooting angles and perform data preprocessing, including standardization and zero-mean normalization.

[0077] The expression for S1.1 data standardization is as follows:

[0078]

[0079] Where x is the image data sample, μ is the sample mean, and σ is the sample standard deviation. * These are standardized image data samples.

[0080] S1.2 Zero-mean data centering: For each image data sample, subtract the mean of the data in each dimension from the mean of that dimension. This centers the data in each dimension of the image data sample to 0. The expression is as follows:

[0081]

[0082] Where N represents the total number of image data samples, x i This represents the value of the i-th dimension of the image data sample. Let x' represent the value of the i-th dimension of the j-th image data sample.i This represents the value of the i-th dimension of the image data sample after zero-mean processing.

[0083] S2. The target image is extracted using an object detection model to reduce background interference and lower the computational load of subsequent steps. The object detection model used is an improvement on the SSD algorithm, incorporating dilated convolution in the feature extraction part. The object detection model includes a feature extraction network based on dilated convolution and a multi-scale feature prediction network.

[0084] S2.1 uses a feature extraction network to extract features from the preprocessed image data. First, a convolutional layer is used to increase the number of channels in the feature map to 64.

[0085] Next are three convolutional units consisting of dilated convolutional layers and regular convolutional layers. Each convolutional unit first contains a dilated convolutional layer with an expansion rate of 2, which helps to increase the receptive field and reduce the number of parameters. Then there is a convolutional layer with a stride of 2. The number of convolutional kernels is twice the number of channels in the output feature map of the dilated convolutional layer in this unit, and finally the feature map C1 is obtained.

[0086] S2.2 employs a multi-scale feature prediction network to predict features in feature map C1. First, it passes through a convolutional layer that increases the number of channels to 1024, resulting in the first feature prediction layer. Then, it passes through a downsampling convolutional layer with a kernel size of 3, a stride of 2, and half the number of channels as the first feature prediction layer, resulting in the second feature prediction layer. Finally, it passes through another downsampling convolutional layer with half the number of kernels as the previous layer, resulting in the third feature prediction layer, C2.

[0087] Prior boxes of varying sizes are formed on the feature prediction layer C2. For each prior box, a classification score and a position offset are calculated. The classification score predicts whether the prior box contains an object, while the position offset is used to fine-tune the position of the prior box to more closely approximate the actual object's bounding box. Finally, the loss is calculated, expressed as follows.

[0088] L = L conf +L loc

[0089] Where L is the loss, L conf For classification scoring, L loc This is the displacement deviation.

[0090] The prior bounding box with the minimum loss is selected to extract the target image, which is then used to obtain the vanishing point vector data in subsequent steps.

[0091] S3. Perform Gaussian filtering on the target image, then perform edge detection by calculating the gradient magnitude, and finally calculate the vanishing point set by a straight line fitting method.

[0092] S3.1 performs Gaussian filtering on the target image. Using Gaussian filtering to process image data can smooth the image and reduce noise. Gaussian filtering uses a Gaussian function as the filter kernel, which is convolved with the image in the spatial domain. The expression for the Gaussian function is as follows:

[0093]

[0094] Where G(i,j) is the weight value of the coordinate position of the Gaussian filter kernel (i,j), σ is the standard deviation of the Gaussian function, and k is the radius of the Gaussian filter kernel.

[0095] When performing Gaussian filtering on an image, for each pixel in the image, its weighted average with the Gaussian kernel is calculated, and the expression is as follows:

[0096]

[0097] Where I'(x,y) is the weighted average of pixel (x,y) and Gaussian filter kernel in the image, G(m,n) is the weight value of the coordinate position of Gaussian filter kernel (m,n), and I(x+m,y+n) is the pixel value of pixel (x+m,y+n) in the image.

[0098] S3.2 determines the edge intensity and direction information in the image by calculating the gradient magnitude and direction of each pixel on the image data after Gaussian filtering.

[0099] The calculation method used is the Sobel operator, which performs convolution operations in the horizontal and vertical directions to obtain the gradient magnitude and direction of the pixel.

[0100] After calculating the gradient magnitude and direction, non-maximum suppression (NMS) is applied to the image to accurately locate edges. The NMS process is as follows: for each pixel, its two neighboring points along the gradient direction are checked. If the gradient magnitude of the current pixel is greater than the gradient magnitudes of its two neighboring points along the gradient direction, then the pixel is retained. Otherwise, the pixel is suppressed by setting its gradient magnitude to 0, thus preserving local maxima on the edges.

[0101] Two thresholds are set, a high threshold and a low threshold. The gradient magnitude of each pixel is compared with the threshold. Pixels with a gradient magnitude greater than the high threshold are considered strong edges, those with a gradient magnitude less than the low threshold are considered non-edges, and those in between are considered weak edges. Non-edges are directly suppressed by setting their gradient magnitude to 0. For weak edges, it is first checked whether there are strong edges among its neighbors. If so, it is considered a strong edge; otherwise, it is suppressed. The strong edge points are then connected sequentially to obtain the edge detection output image, retaining the length and starting coordinate information of the line segments formed during the connection process.

[0102] S3.3 Set the length of the line segment, retain line segments with a length greater than the set value, and remove line segments that are too short. Perform line fitting on the retained line segment data set to obtain the slope and intercept parameters of each line. For the fitted lines, calculate the coordinates of their intersection points pairwise. The intersection points are considered as vanishing points. Save all intersection point information to obtain vanishing point vector data.

[0103] It should be noted that some of these are false vanishing points, which need to be filtered and processed by the subsequent model to calculate the true vanishing points.

[0104] S4. Extract features from the preprocessed image data obtained in step S1, and then perform mixed calculations with the vanishing point vector data using the vanishing point hybrid calculation model to obtain the predicted vanishing point coordinates.

[0105] S4.1 Randomly select an image data D from the preprocessed image data obtained from a certain angle. First, perform three consecutive downsampling convolution operations on it, followed by a convolutional layer to increase the number of channels. After adjusting the number of channels to 256, the feature map D1 is obtained.

[0106] Then, multi-angle feature fusion is performed. Three parallel convolutional layers with different kernel sizes are used to process feature map D1. Then, the outputs of the three convolutional layers are concatenated into a feature map D2 by averaging the elements at the same pixel position.

[0107] S4.2 After performing global average pooling and softmax operations on feature map D2, a channel weight vector l1 with a length of 256 is obtained. The length of the vanishing point vector data is extended to 256, and zeros are padded to the insufficient part to obtain the vanishing point vector l2.

[0108] Add the channel weight vector l1 and the vanishing point vector l2 by multiplying them at the same position to obtain vector l. Multiply l by feature map D2 and apply it to the feature map to obtain feature map D3.

[0109] S4.3 passes the feature map D3 through a downsampling convolution operation and then through a fully connected layer to output two predicted vanishing point coordinates, Q1 and Q2.

[0110] In addition, to improve the accuracy of the model during the training phase, the loss needs to be calculated by combining the predicted vanishing point coordinates with the true vanishing point coordinates of the image. This loss is then used to update the parameters in the vanishing point fusion calculation model through backpropagation, thereby reducing the loss between the true and predicted values. The expression is as follows:

[0111]

[0112] Here, loss is the difference between the true value and the predicted value, and Q'1 and Q'2 are the true vanishing point coordinates of the image data.

[0113] S5. Set the width, height, and corner coordinates of the corrected target image. Calculate the parameters in the perspective transformation matrix based on the final predicted vanishing point coordinates and the corner coordinates of the target image, and perform image correction. The expression is as follows:

[0114]

[0115] x′=m11*x+m12*y+m13

[0116] y′=m21*x+m22*y+m23

[0117] Where x, y are the coordinates of the original image, and x', y' are the corrected coordinates.

[0118] Example 2:

[0119] This embodiment provides an image correction system based on a hybrid calculation of target cropping and vanishing point, used to implement the correction method described in Embodiment 1, which includes:

[0120] Image acquisition and processing module: Used to acquire image data from various shooting angles and preprocess it.

[0121] Target image cropping module: Used to crop target images from preprocessed image data using a target detection model.

[0122] The object detection model includes a feature extraction network and a multi-scale feature prediction network.

[0123] like Figure 2 As shown, the feature extraction network includes a convolutional layer that increases the number of channels and three convolutional units. The convolutional layer that increases the number of channels is used to increase the number of channels in the image data to 64. The convolutional unit consists of a dilated convolutional layer and a normal convolutional layer. The dilated convolutional layer has a dilation rate of 2, which is used to increase the receptive field and reduce the number of parameters. The normal convolutional layer has a stride of 2, and the number of its convolutional kernels is twice the number of channels in the output feature map of the dilated convolutional layer in that unit.

[0124] CombinationFigure 3 The multi-scale feature prediction network includes a convolutional layer that increases the number of channels, a first downsampling convolutional layer, and a second downsampling convolutional layer. The convolutional layer that increases the number of channels is used to increase the number of channels in the feature map to 1024, resulting in the first feature prediction layer. The first downsampling convolutional layer has a kernel size of 3, a stride of 2, and the number of kernels is half the number of channels in the first feature prediction layer, resulting in the second feature prediction layer. The second downsampling convolutional layer has the same number of kernels as the second feature prediction layer, resulting in the third feature prediction layer, or feature prediction layer C2.

[0125] Vanishing point set calculation module: This module performs Gaussian filtering on the target image, calculates the gradient magnitude and direction of each pixel in the target image after Gaussian filtering, locates the edges of the target image, and obtains the edge detection output image. It then performs line fitting on the edge detection output image, calculates the coordinates of the intersection of the fitted lines, which are the vanishing points, and sets all the vanishing point coordinates to obtain the vanishing point vector data.

[0126] Vanishing point prediction module: This module selects an image from the preprocessed image data, extracts its features using a vanishing point hybrid calculation model, and then performs a hybrid calculation with the vanishing point vector data to obtain the final predicted vanishing point coordinates.

[0127] Combination Figure 4 The vanishing point calculation model includes a multi-angle feature extraction network and a vanishing point weight fusion network.

[0128] The multi-angle feature extraction network consists of three downsampling convolutional layers, one convolutional layer that increases the number of channels, and three parallel convolutional layers with different kernel sizes. It is used to adjust the number of channels of the image data to 256 to obtain feature map D1, and then the output is spliced ​​into feature map D2 after being processed by the three parallel convolutional layers.

[0129] The vanishing point weight fusion network is used to perform global average pooling and softmax operations on the feature map D2 to obtain a channel weight vector l1 with a length of 256. The length of the vanishing point vector data is then expanded to 256, and zeros are padded to the shortest part to obtain the vanishing point vector l2.

[0130] Add the channel weight vector l1 and the vanishing point vector l2 by multiplying them at the same position to obtain vector l. Multiply l by feature map D2 and apply it to the feature map to obtain feature map D3.

[0131] The vanishing point calculation model also includes a convolutional layer and a fully connected layer. The feature map D3 is downsampled and convolutional, and then output through a fully connected layer to obtain the final predicted vanishing point coordinates.

[0132] During the training phase, the vanishing point fusion calculation model needs to calculate the loss by combining the predicted vanishing point coordinates with the true vanishing point coordinates of the image. This loss is then updated through backpropagation to update the various parameters in the vanishing point fusion calculation model, reducing the difference between the true and predicted values. The expression is as follows:

[0133]

[0134] Where loss is the difference between the true value and the predicted value, Q1 and Q2 are the final predicted vanishing point coordinates, and Q'1 and Q'2 are the true vanishing point coordinates of the image data.

[0135] Image correction module: Used to calculate the parameters in the perspective transformation matrix based on the final predicted vanishing point coordinates and the length, width, and corner coordinates of the target image, and to perform image correction.

[0136] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An image correction method based on a hybrid calculation of target cropping and vanishing point, characterized in that, Includes the following steps: Image data from various shooting angles is collected and preprocessed to obtain preprocessed image data; The target image is extracted from the preprocessed image data using an object detection model; The target image is processed by Gaussian filtering. The gradient magnitude and direction of each pixel in the target image after Gaussian filtering are calculated to locate the edge of the target image and obtain the edge detection output image. The edge detection output image is then fitted with a straight line. The coordinates of the intersection of the fitted lines are calculated, which are the vanishing points. All vanishing point coordinates are collected to obtain the vanishing point vector data. Select one image from the preprocessed image data, extract its features using the vanishing point fusion calculation model, and then perform a fusion calculation with the vanishing point vector data to obtain the final predicted vanishing point coordinates. Based on the final predicted vanishing point coordinates and the length, width, and corner coordinates of the target image, the parameters in the perspective transformation matrix are calculated, and image correction is performed. The vanishing point hybrid calculation model includes a multi-angle feature extraction network and a vanishing point weight fusion network; The multi-angle feature extraction network consists of three downsampling convolutional layers, one convolutional layer that increases the number of channels, and three parallel convolutional layers with different kernel sizes. This network is used to adjust the number of channels in the image data to 256 to obtain the feature map. Then, after processing by three parallel convolutional layers, the outputs are concatenated to form a feature map. ; The vanishing point weight fusion network is used to integrate feature maps. After performing global average pooling and softmax operations, a channel weight vector of length 256 is obtained. The length of the vanishing point vector data is expanded to 256, and zeros are padded to fill any gaps, resulting in the vanishing point vector. ; Channel weight vector Sum of vanishing vectors The vector is obtained by performing addition using the method of multiplying at the same position. ,Will and feature map Multiply them and apply the result to the feature map to obtain the feature map. ; The vanishing point calculation model also includes a downsampling convolutional layer and a fully connected layer, and a feature map. After a downsampling convolution operation, the output is obtained through a fully connected layer to get the final predicted vanishing point coordinates.

2. The image correction method based on target cropping and vanishing point hybrid calculation according to claim 1, characterized in that, The preprocessing includes standardization and zero-mean normalization. The standardization expression is as follows: ； in, For image data samples, The average value of the sample. Standard deviation, These are standardized image data samples; Zero-mean normalization of data involves subtracting the mean of each dimension of an image data sample from its data. This centers all dimensions of the image data sample to 0. The expression is as follows: ； in, This represents the total number of image data samples. The first image data sample represents the... Dimensional value, Indicates the first The first image data sample Dimensional value, This represents the first image data sample after zero-mean processing. Dimensional value.

3. The image correction method based on target cropping and vanishing point hybrid calculation according to claim 1, characterized in that, Extracting target images from images using object detection models includes: Feature extraction networks are used to extract features from preprocessed image data to obtain feature maps. ; Feature maps are predicted using a multi-scale feature prediction network. Perform feature prediction to obtain the feature prediction layer. In the feature prediction layer Prior bounding boxes of different sizes are formed on the image, and the prior bounding box with the least loss is selected to extract the target image.

4. The image correction method based on target cropping and vanishing point hybrid calculation according to claim 1, characterized in that, Gaussian filtering is applied to the target image, including: A Gaussian function is used as the filter kernel, and a convolution operation is performed with the target image in the spatial domain. The expression of the Gaussian function is as follows: ； in, Gaussian filter kernel The weight value of the coordinate position, Standard deviation, The radius of the Gaussian filter kernel; For each pixel in the image, calculate its weighted average with respect to the Gaussian kernel, as shown in the following expression: ； in, The coordinates in the image are The weighted average of the pixels and the Gaussian filter kernel. Gaussian filter kernel The weight value of the coordinate position, For pixels in the image The pixel value at the coordinate position.

5. The image correction method based on target cropping and vanishing point hybrid calculation according to claim 1, characterized in that, The gradient magnitude and direction of each pixel in the target image after Gaussian filtering are calculated to locate the edges of the target image, resulting in an edge detection output image, including: By performing convolution operations in the horizontal and vertical directions using the Sobel operator, the gradient magnitude and direction of each pixel can be obtained. For each pixel, check its two adjacent points in the gradient direction. If the gradient magnitude of the current pixel is greater than the gradient magnitude of its two adjacent points in the gradient direction, then keep the pixel; otherwise, suppress the pixel by setting its gradient magnitude to 0. Set a high threshold and a low threshold for the gradient magnitude, compare the gradient magnitude of each pixel with the threshold, and identify pixels with gradient magnitude greater than the high threshold as strong edge points, pixels with gradient magnitude less than the low threshold as non-edge points, and pixels with gradient magnitude in between as weak edge points. For non-edge points, they are directly suppressed, that is, the gradient magnitude of the pixel is set to 0. For weak edge points, first check if there are strong edge points in their neighboring points. If there are, they are considered strong edge points. If not, the pixel is suppressed. The edge detection output image is obtained by sequentially connecting strong edge points, and the length and starting coordinate information of the straight line segments formed during the connection process are retained.

6. The image correction method based on target cropping and vanishing point hybrid calculation according to claim 1, characterized in that, A straight line is fitted to the edge detection output image, and the coordinates of the intersection points of the fitted lines are calculated. These are the vanishing points. All vanishing point coordinates are collected to obtain the vanishing point vector data, including: Set the length of the line segment, keep the line segments with a length greater than the set value, and remove the line segments that are too short. Perform line fitting on the set of retained line segments to obtain the slope and intercept parameters of each line. For a fitted straight line, calculate the coordinates of its intersection points pairwise. The intersection points are the vanishing points, and all the intersection point information is used to form the vanishing point vector data.

7. The image correction method based on target cropping and vanishing point hybrid calculation according to claim 1, characterized in that, Based on the final predicted vanishing point coordinates and the length, width, and corner coordinates of the target image, the parameters in the perspective transformation matrix are calculated, and the expression for image correction is as follows: ； in, This is the perspective transformation matrix. The coordinates of the original image. These are the corrected coordinates.

8. An image correction system based on a hybrid calculation of target interception and vanishing point, characterized in that, An image correction method based on target cropping and vanishing point hybrid calculation as described in any one of claims 1 to 7, comprising: Image acquisition and processing module: used to acquire image data from various shooting angles and preprocess it; Target image cropping module: Used to crop target images from preprocessed image data using a target detection model; Vanishing point set calculation module: This module performs Gaussian filtering on the target image, calculates the gradient magnitude and direction of each pixel in the target image after Gaussian filtering, locates the edges of the target image, and obtains the edge detection output image. It then performs line fitting on the edge detection output image, calculates the coordinates of the intersection of the fitted lines, which are the vanishing points, and collects all the vanishing point coordinates to obtain the vanishing point vector data. Vanishing point prediction module: Selects an image from the preprocessed image data, extracts its features using the vanishing point hybrid calculation model, and then performs hybrid calculation with the vanishing point vector data to obtain the final predicted vanishing point coordinates; The vanishing point hybrid calculation model includes a multi-angle feature extraction network and a vanishing point weight fusion network; The multi-angle feature extraction network consists of three downsampling convolutional layers, one convolutional layer that increases the number of channels, and three parallel convolutional layers with different kernel sizes. This network is used to adjust the number of channels in the image data to 256 to obtain the feature map. Then, after processing by three parallel convolutional layers, the outputs are concatenated to form a feature map. ; The vanishing point weight fusion network is used to integrate feature maps. After performing global average pooling and softmax operations, a channel weight vector of length 256 is obtained. The length of the vanishing point vector data is expanded to 256, and zeros are padded to fill any gaps, resulting in the vanishing point vector. ; Channel weight vector Sum of vanishing vectors The vector is obtained by performing addition using the method of multiplying at the same position. ,Will and feature map Multiply them and apply the result to the feature map to obtain the feature map. ; The vanishing point calculation model also includes a downsampling convolutional layer and a fully connected layer, and a feature map. After a downsampling convolution operation, the output is obtained through a fully connected layer to obtain the final predicted vanishing point coordinates; Image correction module: Used to calculate the parameters in the perspective transformation matrix based on the final predicted vanishing point coordinates and the length, width, and corner coordinates of the target image, and to perform image correction.

9. The image correction system based on target cropping and vanishing point hybrid calculation according to claim 8, characterized in that, The target detection model includes a feature extraction network and a multi-scale feature prediction network; The feature extraction network includes a convolutional layer that increases the number of channels and three convolutional units. The convolutional layer that increases the number of channels is used to increase the number of channels in the image data to 64. The convolutional unit consists of a dilated convolutional layer and a normal convolutional layer. The dilated convolutional layer has a dilation rate of 2, which is used to increase the receptive field and reduce the number of parameters. The normal convolutional layer has a stride of 2, and the number of its convolutional kernels is twice the number of channels in the output feature map of the dilated convolutional layer in that unit. The multi-scale feature prediction network includes a convolutional layer that increases the number of channels, a first downsampling convolutional layer, and a second downsampling convolutional layer. The convolutional layer that increases the number of channels is used to increase the number of channels in the feature map to 1024 to obtain the first feature prediction layer. The first downsampling convolutional layer has a kernel size of 3 and a stride of 2, and the number of kernels is half the number of channels in the first feature prediction layer, thus obtaining the second feature prediction layer; The number of kernels in the second downsampling convolutional layer is half the number of channels in the second feature prediction layer, resulting in the third feature prediction layer. .

10. The image correction system based on target cropping and vanishing point hybrid calculation according to claim 8, characterized in that, The vanishing point fusion calculation model needs to calculate the loss by combining the predicted vanishing point coordinates and the true vanishing point coordinates of the image during the training phase. This loss is then updated through backpropagation to update the various parameters in the vanishing point fusion calculation model, reducing the loss between the true and predicted values. The expression is as follows: ； in, The loss between the actual value and the predicted value. and These are the final predicted vanishing point coordinates. and These are the true vanishing point coordinates of the image data.

Images