Camera preset position correction method, device, equipment and medium

Through feature point set matching and perspective transformation matrix calculation, combined with 360° rotation and template matching, the problem of failure under the camera preset calibration conditions is solved, and accurate offset calculation and correction are achieved.

CN118644429BActive Publication Date: 2024-11-19SICHUAN KINGSCHEME INFORMATION TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411087388.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-19
Estimated Expiration
2044-08-09

AI Technical Summary

Technical Problem

The existing camera presetting correction method based on image feature matching fails when the shooting conditions of the reference map and the detection map are too large, resulting in the correction failure.

Method used

By extracting and matching the set of feature points, the number of them is differentiatedly corrected. The perspective transformation matrix and enhanced template matching algorithm are used, combined with 360° rotation and multi-scale scaling, the horizontal and vertical offsets of the camera are calculated to ensure the accuracy of correction .

Benefits of technology

When the shooting conditions of the reference map and the detection map are large, the offset of the camera can be accurately calculated, the correct position restoration can be achieved, and error correction can be avoided. It is suitable for a large range of offset correction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118644429B_ABST
    Figure CN118644429B_ABST
Patent Text Reader

Abstract

The present invention discloses a camera preset position correction method, device, equipment and medium, and relates to the field of camera application technology. The preset position correction method comprises: extracting a feature point set of a detection image, matching it with a feature point set of a reference image, obtaining a feature point set pair, and performing a camera preset position correction process based on the number of first feature point set pairs. The present invention performs a differential correction process based on the difference in the number of feature point set pairs, thereby solving the problem of correction failure in traditional correction methods when the shooting position or focal length difference between the reference image and the detection image is too large.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of camera application technology, and in particular to a camera preset position correction method, device, equipment and medium. Background Art

[0002] During the application stage after the implementation of the remote intelligent inspection system for substations, with the application of real-time monitoring and other camera preset services, the camera preset positions may shift due to environmental changes, human errors, etc., resulting in errors in the inspection points of the substation and inaccurate judgment of defects at the inspection points. Therefore, the correction of camera preset positions is particularly important.

[0003] The current preset position correction solution based on feature matching between the reference image and the detection image mainly adopts different feature extraction and matching algorithms. However, when the shooting position or focal length (the zoom parameter of the preset position is adjusted too large) of the reference image and the detection image is too different, the feature extraction and matching correction method will fail and the camera cannot be correctly reset. Summary of the invention

[0004] The present invention provides a camera preset position correction method, device, equipment and medium to solve the problem of correction failure in the existing correction method based on image feature matching when the shooting conditions of the detection image and the reference image are too different.

[0005] The present invention is achieved through the following technical solutions:

[0006] In a first aspect, a camera preset position correction method is provided, the method comprising:

[0007] Extracting a first feature point set of a first detection image;

[0008] Matching the first feature point set with a second feature point set to obtain a first feature point set pair, wherein the second feature point set is extracted from a reference image;

[0009] The camera preset position correction process is performed based on the number of the first feature point set pairs.

[0010] The difference between the detection image and the reference image is represented by feature point pairs, and a differential correction process is performed according to the number of feature point pairs, thereby solving the problem that the traditional correction method fails when the shooting position or focal length difference between the reference image and the detection image is too large. When the number of feature point pairs is greater than the threshold, the correct correction parameters can be obtained through the traditional correction algorithm, and the offset can be calculated based on the current detection image and the reference image to achieve position restoration. However, when the number of feature point pairs is less than the threshold, if the same correction method is always used, an erroneous correction result will be obtained. Therefore, the present invention proposes to perform differential correction processing of the camera preset position based on the number of feature point pairs to avoid erroneous correction. It can be understood that the correction method is not limited to the existing correction method based on image features or manual correction method.

[0011] In one implementation, the correction processing of the camera preset position based on the number of the first feature point set pairs includes:

[0012] When the number of the first feature point set pairs is less than a threshold, the following correction method is performed:

[0013] The camera is rotated 360 degrees and images are captured at predetermined time intervals to obtain a plurality of second detection images;

[0014] Extracting a feature point set of each second detection image and matching it with the second feature point set to obtain a second feature point set pair of each second detection image and the reference image;

[0015] sorting the second detection images in descending order based on the number of the second feature point set pairs;

[0016] Taking at least one of the first second detection images in the sorting order, and calculating a first perspective transformation matrix between the second detection image and the reference image;

[0017] Calculate the horizontal offset and the vertical offset of the camera based on the first perspective transformation matrix;

[0018] When the number of the first feature point set pairs is greater than or equal to a threshold, the following correction method is performed:

[0019] Calculating a second perspective transformation matrix between the first detection image and the reference image;

[0020] The horizontal offset and the vertical offset of the camera are calculated based on the second perspective transformation matrix.

[0021] In one implementation, the correction processing of the camera preset position based on the number of the first feature point set pairs includes:

[0022] When the number of the feature point set pairs is less than the threshold, the following correction method is performed:

[0023] Performing multi-scale scaling on the reference image to obtain multiple templates;

[0024] Match each of the templates with the first detection image pixel by pixel to obtain a similarity matrix corresponding to each of the templates;

[0025] Obtaining the first element in each of the similarity matrices, determining the similarity matrix to which the maximum value in the first element belongs as a matching result matrix, and using the template corresponding to the matching result matrix as a matching template, wherein the first element is taken from the maximum value in each of the similarity matrices;

[0026] The horizontal offset and the vertical offset of the camera are calculated based on the central coordinates of the matching template and the central coordinates of the first detection image.

[0027] In one embodiment, before obtaining the first element in each of the similarity matrices, the method further includes:

[0028] The elements in each similarity matrix are filtered by a filtering threshold, and elements smaller than the filtering threshold are removed; wherein each similarity matrix corresponds to a filtering threshold, and the larger the scaling ratio of the template is, the smaller the filtering threshold is.

[0029] In one embodiment, before pixel-by-pixel matching of each template with the first detection image, the method further includes:

[0030] According to the scaling size of the template, black borders of the same size are filled around the first detection image.

[0031] In one embodiment, the method further includes: calculating the aspect ratio of the matching template and the first detection image to obtain a focal length scaling factor: ,in, , are the width and height of the matching template respectively, , are the width and height of the first detection image respectively;

[0032] The camera is pre-positioned and calibrated based on the horizontal offset, the vertical offset and the focal length zoom factor.

[0033] In one implementation, when the horizontal offset or the vertical offset is greater than a preset value, the method further includes:

[0034] The camera is rotated 360 degrees and images are captured at predetermined time intervals to obtain a plurality of second detection images;

[0035] Matching each of the second detection images with the second feature point set to obtain a second feature point set pair of each of the second detection images and the reference image;

[0036] sorting the second detection images in descending order based on the number of the second feature point set pairs;

[0037] Taking at least one of the first second detection images, and calculating the perspective transformation matrix between the second detection image and the reference image;

[0038] The horizontal offset and the vertical offset of the camera are calculated based on the perspective transformation matrix.

[0039] In a second aspect, the present invention provides a camera preset position correction device, comprising:

[0040] A feature point set extraction module, used to extract a first feature point set of a first detection image;

[0041] A feature point set pair extraction module, used for matching the first feature point set with a second feature point set to obtain a first feature point set pair, wherein the second feature point set is extracted from a reference image;

[0042] The preset position correction module is used to perform correction processing of the camera preset position based on the number of the first feature point set pairs.

[0043] According to a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the camera preset position correction method described in any one of the embodiments of the first aspect when executing the computer program.

[0044] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the camera preset position correction method described in any one of the embodiments of the first aspect is implemented.

[0045] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0046] When there is a large difference in shooting position or focal length between the reference image and the test image, differential correction processing is performed according to the number of feature point sets to solve the problem of failure of traditional correction algorithms.

[0047] A method is provided for selecting a detection image with a smaller difference from a reference image for correction when the offset is large based on multiple detection images taken. The offset calculation is mainly based on feature point matching and matrix perspective transformation. The calculation amount is small and accurate correction results can be obtained.

[0048] When correction can be performed based on the current detection image, the offset is directly calculated based on the perspective transformation matrix of the detection image and the reference image, which reduces the amount of calculation and ensures high accuracy.

[0049] Another correction method based on enhanced matching of the detection image is provided for when the offset is large. This method does not require driving the camera to rotate and shoot. The correction result is obtained based on matching the current detection image with the enhanced matching template. It is suitable for situations where there is a target image in the detection image but it is significantly different from the reference image. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:

[0051] Figure 1 is a flow chart of a camera preset position correction method according to an embodiment of the present invention;

[0052] Figure 2 It is a flow chart of a preset position correction method based on an enhanced template matching algorithm according to an embodiment of the present invention;

[0053] Figure 3 The figure shows a schematic diagram of pixel-by-pixel matching between the template and the detection image;

[0054] Figure 4 It is a flow chart of a preset position correction method based on an automatic registration mode according to an embodiment of the present invention;

[0055] Figure 5 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0057] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to or inherent to other steps or units of the device.

[0058] The terms used in the various embodiments of the application are only used to describe the purpose of specific embodiments and are not intended to limit the various embodiments of the application. As used herein, the singular form is intended to also include the plural form, unless the context clearly indicates otherwise. Unless otherwise limited, all terms used here (including technical terms and scientific terms) have the same meaning as the meanings commonly understood by ordinary technicians in the field of the various embodiments of the application. The terms (such as the terms defined in the dictionary generally used) will be interpreted as having the same meaning as the contextual meaning in the relevant technical field and will not be interpreted as having an idealized meaning or an overly formal meaning, unless clearly defined in the various embodiments of the application.

[0059] The embodiments of the present invention provide a camera preset position correction method, device, equipment and medium, which are applicable to camera preset position correction, can correct the preset position according to different offset conditions, and correct a larger range of offsets.

[0060] like Figure 1 The figure is a flow chart of a camera preset position correction method according to an embodiment of the present invention, which can perform preset position correction for different offset ranges. The method comprises the following steps:

[0061] Step S11, extracting a first feature point set from a first detection image.

[0062] The detection image can be understood as the image taken by the camera to be corrected, which is the on-site image taken at the current position of the camera and based on the current focal length. The present invention adopts a correction method based on feature extraction and matching, by extracting the feature point set of the detection image and matching it with the feature point set of the reference image, and correcting the error based on the difference between the two.

[0063] When performing the deflection correction, one or more detection images can be collected, and the deflection correction can be performed based on the comprehensive processing results of the multiple detection images. The processing process of each detection image in the multiple detection images is the same, and the specific processing process will be described in detail below. The reference image is an image taken when the camera is in a preset position (including a preset fixed position and a preset focal length). The image is taken when the preset position of the camera is initialized and serves as the basis for deflection correction. It can be understood that the preset position of the camera can be set according to the monitoring task, but in the process of the camera being used for monitoring, it may deviate from the set preset position due to non-subjective factors, causing the monitored area to overflow the monitoring area of ​​the camera. At this time, the camera needs to be reset and readjusted to the set monitoring position, that is, the camera preset position correction, also called preset position correction.

[0064] Step S12: Match the first feature point set with the second feature point set to obtain a first feature point set pair.

[0065] The first feature point set is obtained by performing feature point detection on the test image, and the second feature point set is obtained by performing feature point detection on the reference image. Feature point detection refers to finding points with unique attributes in the image. These points can be matched and identified between different images. Common feature point detection methods include corner point detection and pixel color gradient detection. The extraction method of the first feature point set and the second feature point set is the same, and feature extraction algorithms such as SIFT, SURF, and ORB can be used. In addition, since the reference image is pre-photographed, the second feature set of the reference image can also be extracted in advance and stored in the memory as a backup, so as to avoid repeating the same process each time the correction is performed. It can also be understood that before the feature point extraction, the image has been subjected to basic preprocessing such as size unification and grayscale conversion. Here, the size of the test image is mainly adjusted to unify the size of the reference image, and the size adjustment will not change the content expressed by the image.

[0066] The first feature point set and the second feature point set respectively contain unique attribute features in the detection image and the reference image. By matching the first feature point set and the second feature point set, the similarity between the detection image and the reference image can be detected. The feature point set matching method can adopt a matching method such as brute force matching (BFMatcher) and fast nearest neighbor search (FLANN). It can be understood that the matched feature point set pair is smaller than the set of the first feature point set and the second feature point set, and can be approximately considered as the intersection of the two, that is, the feature point set with similar feature areas. The more the number of feature point set pairs, the more similar the detection image and the reference image are, and the smaller the offset position of the camera from the preset position; conversely, the fewer the number of feature point set pairs, the greater the difference between the detection image and the reference image, and the greater the offset position of the camera from the preset position.

[0067] Step S13: Perform camera preset position correction processing based on the number of first feature point set pairs.

[0068] Based on the analysis of step S12, the number of feature point pairs reflects the degree of offset. In the traditional correction method based on feature matching, different feature extraction and matching algorithms are mainly used. However, when the focal length difference between the reference image and the detection image is too large (such as the zoom parameter of the preset position is adjusted too large), or when the horizontal and vertical parameters of the reference image and the detection image are greatly different, the correction method of feature extraction and matching will fail due to the large difference between the detection image and the reference image. Therefore, the traditional correction method can only solve the preset position correction of the camera within a small offset range.

[0069] In order to solve the problem of excessive offset of the focal length of the preset position of the camera in the remote patrol scene, or excessive horizontal and vertical offset, the present invention performs correction processing according to the number of feature point sets. When the number of feature point sets is greater than the threshold, the traditional correction algorithm can be used to calculate the horizontal and vertical positions of the camera offset and the zoom parameters relative to the preset position by matching the detection map and the reference map to achieve position restoration. When the number of feature point sets is less than the threshold, the traditional matching correction method fails. The present invention provides an enhanced matching correction method, which performs preset position correction based on the detection map and the reference map to solve the problem of preset position correction within different offset ranges.

[0070] In one implementation, performing camera preset position correction processing based on the number of first feature point set pairs includes:

[0071] When the number of the first feature point set pairs is greater than or equal to the threshold, the following correction method is performed:

[0072] S21, calculating the perspective transformation matrix of the first detection image and the reference image.

[0073] S22, calculating the horizontal offset and vertical offset of the camera according to the perspective transformation matrix.

[0074] S23, adjusting the position of the camera according to the horizontal offset and the vertical offset.

[0075] Furthermore, it also includes: calculating the focal length zoom value based on the hypotenuse ratio of the detection image and the reference image, and step S23 further includes: adjusting the position of the camera according to the horizontal offset, the vertical offset and the focal length zoom value.

[0076] In general, the focal length of a camera with a fixed focal length will not change at will, or it may be automatically zoomed, so most of the time only the horizontal or vertical position needs to be corrected, and focal length correction may be added when necessary. The perspective transformation matrix is ​​calculated using existing technology.

[0077] In one implementation, performing camera preset position correction processing based on the number of first feature point set pairs includes:

[0078] When the number of the first feature point set pairs is less than a threshold, an enhanced template matching algorithm is used to perform preset position correction.

[0079] The perspective transformation matrix requires at least 4 matching key points, which is a requirement of the perspective transformation algorithm at the mathematical level. When the number of feature point sets is less than a certain value, the correct perspective transformation matrix cannot be generated, that is, correction cannot be performed. Therefore, the present invention proposes an enhanced template matching algorithm to solve this problem. It can be seen that the threshold value should be an integer greater than or equal to 4 to meet the minimum requirement of the perspective transformation matrix. In an embodiment of the present invention, the threshold is preferably 30.

[0080] There are several cases where the number of feature point sets is less than the threshold.

[0081] Case 1: In the reference image, there is a target object O1. The feature points that can be extracted by feature extraction are K1 and N1 in number. Because the color of the target object is too smooth or the image is too blurred, there is not enough color gradient change or corner point. The feature points N1 that can be extracted are less than the threshold N_H (N_H is 4), and the number of point sets is less than the point set pairs required by the perspective transformation matrix. In the test image, whether there is a target object O2 corresponding to the target object O1, if the key points of the target object O2 that can be extracted are K2 and N2 in number, the perspective transformation matrix H12 from K1 to K2 cannot be generated. In this case, the correction information cannot be generated. It should be noted that the problem of not being able to extract enough feature points may occur in both the reference image and the test image, and both will lead to the inability to find the correction information, and the effect is the same (such as case 2).

[0082] Case 2: In the reference image, there is a target object O1, and the feature points that can be extracted are K1, and their number is N1; in the detection image, when the target object O2 is reduced to a certain extent by the scaling ratio Z2 and tends to be close to 1 pixel, the feature points that can be extracted from the target object O2 by means of feature point extraction are K2, and their number N2 is close to 0, and the perspective transformation matrix H12 from K1 to K2 cannot be generated. The scaling ratio Z2 at this time is called the scaling critical point, but the surrounding environment can still generate key points K2_1, and the number is N2_1. At this time, through the feature point matching technology, the key point pair C12 between K1 and K2 is matched, the number N12, N12 must be less than or equal to N2, and must be less than N_H, and the perspective transformation matrix H12 between the target objects O2 and O1 cannot be generated; at this time, K1 and K2_1 may still find a matching key point pair C12_1, but K2_1 is not the key point of object O2, and then generate an erroneous perspective transformation matrix H12_1, which is not the perspective transformation matrix between O2 and O1, thereby generating erroneous correction information.

[0083] Case 3: In the reference image, there is a target object O1, and the feature points that can be extracted by feature extraction are K1 and N1 in number; in the detection image, there is a target object O2, and there are multiple target objects that are extremely similar to the target features (for example, one key on a keyboard and multiple keys on a keyboard). Here, O2_2 and O2_3 are taken as examples. Correspondingly, the key points that can be extracted from O2, O2_2, and O2_3 are K2, K2_2, and K2_3, respectively, and the numbers are N2, N2_2, and N2_3, respectively; at this time, through the feature point matching technology, K1 may be matched with K2, K2_2, and K2_3 at the same time to generate key point pairs C12, C12_2, and C12_3, resulting in the perspective change matrix that should have been generated by C12 being involved and interfered by C12_2 and C12_3, and the perspective transformation matrix H12 finally generated is wrong, generating wrong correction information.

[0084] like Figure 2 The figure shows a flow chart of a preset position correction method based on an enhanced template matching algorithm, wherein the enhanced template matching algorithm includes the following steps:

[0085] S31, performing multi-scale scaling on the reference image to obtain multiple templates.

[0086] The purpose of scaling is to adapt to the situation where the target object in the reference image is scaled in the test image, ensuring that when each pixel in the reference image is compared, the target object of the same size can be found in the test image, and a high correlation is obtained for each pixel at the corresponding position. This overcomes the situation described in Case 2 where the target object in the test image is reduced to a size close to 1 pixel and key points cannot be found and matched.

[0087] The zoom range is determined by the camera's variable zoom factor. x and the current zoom factor The calculation shows that the scaling ratio can be , where the zoom factor can be changed x The value is at the minimum zoom factor and maximum zoom ratio If the zoom range is , and set the zoom change step size to x When it is greater than 1, it takes 1. x When it is less than 1, take 0.1, which constitutes ,……, , , , , ,……, For example, the reference image is collected when the camera is zoomed 2 times, that is, , need to be scaled to Zoom, the image's zoom factor should be ,Right now Zoom factor.

[0088] Traverse the scaling factor to scale the reference image at multiple scales to obtain multiple corresponding matching templates. The image, after After scaling, the width and height are Image. Image with a width and height of 1920*1080 pixels, scaling ratio The base image will be scaled down to 192*108 pixels.

[0089] S32, matching each template with the first detection image pixel by pixel to obtain a similarity matrix corresponding to each template.

[0090] In this step, the scaled reference image is used as a template to perform template matching with the detection image. Before each model matching, a black border with the same scaled size as the reference image is filled around the detection image according to the scaled size of the currently used template relative to the reference image, so that the detection image is "surrounded" by the black border.

[0091] When matching pixel by pixel, if the upper left corner alignment is used, for the right edge and the lower edge of the detection image, when the upper left corner of the template is aligned with it, there will be a part of overflow, and by moving the position of the template in the detection image pixel by pixel (from left to right, from top to bottom), all the pixels in the detection image can be completely matched with the template. However, for the left and upper edges of the detection image, the template cannot overflow to adapt to the situation where the target object overflows the screen of the detection image. Therefore, before pixel matching, the detection image is filled with black borders of the same size as the reference image after scaling each time according to the size of the reference image after scaling, that is, a circle of black borders is added around the detection image, the width of the left and right black borders is W1, and the height of the upper and lower black borders is H1, so that the detection image is "surrounded" by the black borders. Here, the template matching algorithm is mainly enhanced to be compatible with the overflow of the detection area. When there is a target object O1 in the reference image and a corresponding target object O2 in the detection image, but the target object O2 may overflow around the detection image, leaving only a part of the target object O2 at the edge of the detection image. At this time, by adding a black border, when matching pixel by pixel, the upper left corner of the template is first pixel-aligned with the black border area in the upper left corner of the detection image. At this time, the template overflows relative to the left and upper edges of the detection image, so that the overflowed part of the object in the reference image is placed in the black border of the detection image, and the remaining part of the object in the reference image can be matched with the remaining part of the object in the detection image. Taking the target object as a human body as an example, there is a complete human body in the reference image, and the camera in the detection image is offset downward, and the human body above the ankle overflows the top of the detection image, leaving only the feet and the ground in the upper part of the detection image. At this time, if the template can overflow relative to the left and upper edges of the detection image, the feet in the reference image can be matched with the feet in the detection image in the same area.

[0092] The template matching algorithm preferably uses normalized correlation coefficient matching. This matching method calculates the correlation by using the template and each pixel in the detection image without relying on key points. Even pure-color targets can be matched, which can overcome the problem in situation 1 that the image is too blurred and enough key points cannot be found.

[0093] Since the reference image is scaled, when the template is used to match the detection image, the detection image overflows relative to the template. Therefore, during matching, a pixel-by-pixel matching method is used to complete the matching of all pixels in the detection image with each template. Figure 3 The figure shows the pixel-by-pixel matching between the template and the detection image. Taking template 1 as an example, the pixel-by-pixel matching process is as follows:

[0094] S32-1, pixel-align the upper left corner point of template 1 with the upper left corner point of the first detection image, compare each pixel in the template with each pixel in the corresponding area of ​​the detection image, and obtain a similarity value including all corresponding pixels in the corresponding area.

[0095] This step mainly involves aligning the template and the test image before matching. Since the test image has overflow relative to template 1, all pixels in the test image need to be matched with template 1, so the method of aligning first and then matching pixel by pixel is adopted. The alignment method can be any method, and the alignment point can be the upper left corner point. In special cases, other corner points can also be used for alignment. After the first alignment and matching, pixel-by-pixel mobile matching is performed later.

[0096] S32-2, move the position of template 1 in the detection image pixel by pixel in the order from left to right and from top to bottom, repeat the matching steps in S32-1 until all pixels in the detection image are matched, and obtain the similarity values ​​of all corresponding areas. All similarity values ​​constitute a similarity matrix corresponding to template 1.

[0097] The similarity value is the similarity between the template and the corresponding regions of the detection image, and can be directly calculated by calling an existing similarity algorithm.

[0098] After matching each scaled template with the detection image pixel by pixel, a similarity matrix corresponding to the template can be obtained. The values ​​of the elements in the matrix represent the similarity between each area in the detection image corresponding to the template and the template. In the areas corresponding to these matrix elements, some areas are similar to the template, while others are not. For each scaled template, pixel by pixel matching is performed with the detection image in the above manner to obtain a similarity matrix corresponding to each template. The most relevant matrix is ​​screened out from all these similarity matrices, that is, the scaled template that best matches the detection image and the area in the detection image that is most similar to the reference image are obtained, so that the offset of the camera can be obtained by locating the position of the area in the detection image. The screening method is shown in step S33, and the offset calculation method is shown in step S34.

[0099] S33, obtaining the first element in each similarity matrix, taking the similarity matrix to which the maximum value of the first element belongs as the matching result matrix, and taking the template corresponding to the matching result matrix as the matching template, wherein the first element is taken from the maximum value in each similarity matrix.

[0100] S34, calculate the horizontal offset and vertical offset based on the center coordinates of the matching template and the center coordinates of the first detection image. In step S34, calculate the center coordinates of the matching template, and calculate the horizontal offset and vertical offset based on the center coordinates of the matching template and the center coordinates of the detection image. The template matched in the detection image is the scaled reference image. Matching the template in the detection image is equivalent to a part of the reference image appearing in the detection image. At this time, this part needs to be translated to the center of the detection image and scaled to the size of the detection image. Calculating the offset by the center coordinates of the matched template and the center coordinates of the detection image is equivalent to translating the matched reference image to the center of the detection image. The specific calculation process is as follows:

[0101] First, by detecting the center coordinates of the image The center coordinates of the matching template Calculate the difference, and the pixels that need to be horizontally shifted in the detection image are , the number of pixels that need to be vertically translated in the detection image is .

[0102] Furthermore, the focal length scaling value is calculated according to the aspect ratio of the matching template and the detection image.

[0103] According to the width and height of the detection image The width and height of the matching template , calculate the ratios and average them respectively, and get the multiples of the detection image that need to be scaled: .

[0104] Finally, the offset With focal length zoom value r Return to the correction execution module for preset position correction.

[0105] The template matching algorithm described above can solve the problem of being unable to establish a correct perspective transformation matrix, thereby correcting the position of a large offset.

[0106] In one implementation, before step S33, the method further includes: filtering the elements in each similarity matrix by using a filtering threshold.

[0107] Perform threshold filtering on the elements in the similarity matrix corresponding to each template, and remove the elements in the matrix set that are less than the filtering threshold. Set the filtering threshold to TH, and set the default value to TH0. However, each similarity matrix uses a different filtering threshold. As the template scaling increases, the filtering threshold will decrease based on the default value, that is, the larger the size of the template obtained after scaling the base image, the smaller the filtering threshold. times, the corresponding filtering threshold is If the zoom is greater than or equal to 1 (i.e. greater than or equal to the original reference image), the filter threshold is the default value. . Preferably, the default value .

[0108] For example, the base image size is 100px, after multi-scale scaling: Template 1 is 200px wide ( ), filter threshold 0.7 (take the lower limit of the threshold 0.7); Template 2 width 150px ( ), filter threshold 0.7 (lower limit); template 3 width 100px ( ), filter threshold 0.7 (lower limit); template 4 width 90px ( ), filter threshold 0.72 (increased by 0.2); template 5 width 80px ( ), filter threshold 0.74 (increased by 0.2).

[0109] Each element in the similarity matrix represents the degree of match between the detection image and the corresponding position of the template, and the matching range is limited to the size of the template, which can greatly exclude interference information outside the template range. At this time, the threshold filtering technology can overcome the problem in case 3 that one object in the reference image corresponds to multiple similar objects through key point matching and cannot be filtered.

[0110] For example, the reference image (width and height = 200*100px) is scaled by a scaling factor of 1 to obtain template 1 (width and height = 200*100px). After template 1 is matched with the detection image (width and height = 400*200px), a 400×200 similarity matrix 1 is obtained, that is, there are 400*200=80,000 element values, and each element value ranges from 0 to 1, representing the similarity value of the template at each pixel position in each detection image. Using a filter threshold of 0.7 to filter the elements of similarity matrix 1, N1 element values ​​greater than the filter threshold and their corresponding areas in the detection image can be extracted. For example, the coordinates of position 1 , element value (similarity) 0.8, position 2 coordinates , element value (similarity) 0.85, ... Then both position 1 and position 2 are greater than the filtering threshold.

[0111] If there are 10 templates in total, there are 10 similarity matrices corresponding to them. Finally, the matching area in the detection image corresponding to the maximum value in each similarity matrix is ​​taken, indicating that these areas in the detection image have the highest similarity with each template. Then, the maximum similarity value is selected from these areas, that is, the most matching template is selected as the matching template, and the offset is calculated as the correction reference. In this way, the area in the detection image that is closest to the target object in the reference image is gradually locked, and then the best matching template is determined for correction. In the case where the target object overflows the detection image, a more accurate correction result can still be obtained. For multiple objects in the detection image that are similar to the target object, by filtering the threshold and the maximum element, only the area with the highest similarity in the detection image is finally located, avoiding interference with the correction caused by multiple target objects.

[0112] In one embodiment, when the horizontal offset or vertical offset obtained by template matching is greater than a preset value, or no valid result is obtained by template matching, the present invention provides an automatic registration mode for recalibration. If no valid result is obtained by template matching, that is, after filtering by the filter threshold, the result obtained is empty, and all scaled templates cannot find similar areas in the detection image. At this time, there may be no target object in the detection image, or there is no corresponding target object in the detection image and the reference image. No matter how the reference image is scaled or translated, no similar area can be found in the detection image. In view of this, the present invention provides an automatic registration mode for recalibration. Figure 4 The figure shows a flow chart of the preset position correction method based on the automatic registration mode. In the automatic registration mode, the following correction methods are performed:

[0113] S41, rotating the camera 360 degrees, and taking images at predetermined time intervals to obtain a plurality of second detection images.

[0114] When no effective correction result is obtained through template matching, the automatic alignment mode will be automatically entered. The camera's zoom parameter can be set to 1 or directly restored to the initial value before rotation. The purpose is to prevent the current camera's zoom parameter from being too large or too small, and the collected detection image is difficult to match with the reference image, which is convenient for horizontal and vertical position correction. The shooting time interval can be flexibly set according to the selected speed, for example, one shot is taken every 500 milliseconds.

[0115] S42, extracting the feature point set of each second detection image collected in step S41, and matching it with the second feature point set of the reference image, to obtain a second feature point set pair of each detection image and the reference image (same steps as S11-S12).

[0116] S43, sorting the detection images in descending order according to the number of second feature point set pairs obtained in step S42, taking at least one of the first second detection images (for example, the first or first three) to execute the template matching algorithm, and calculating the horizontal offset and the vertical offset (same steps as S31-S35).

[0117] In this step, the perspective transformation matrix of the second detection image and the reference image may be calculated, and the offset may be calculated according to the perspective transformation matrix (the steps are the same as S21 to S23).

[0118] Combined with the control accuracy of the camera, 20 pixels (px) are used as the unit for accepting the correction effect. When the horizontal and vertical offsets of a single detection image are within ±20px, or the final offsets of the three detection images after correction are within ±20px, the camera is subjected to preset position correction according to the horizontal offset and vertical offset calculated in step S43.

[0119] Each image can be corrected 30 times. First, select the first image for correction. If the correction is successful within 30 times, stop. If it is still unsuccessful after more than 30 times, select the next image and repeat the correction process.

[0120] The camera moves in the direction of correction once for a fixed correction. Some cameras can only rotate according to time control, such as 10 milliseconds for a horizontal rotation. Due to the insufficient camera control accuracy, each correction action cannot ensure that the camera can be in place in one step, so multiple movements are required. You can take a big step first and then a small step, or you can always use a fixed moving distance to gradually approach the target position until the offset is less than the preset accuracy (the similarity between the detection image taken again after correction and the reference image is greater than the similarity threshold TH_SIM, which is a success. Otherwise, if the correction action is executed 30 times and still does not meet the correction accuracy, it is a failure.

[0121] The reason for the failure of correction may be that the reference image is outdated, the content of the reference image and the detection image is very different (for example, grass and mud, green water and tree stumps, white clouds and starry sky, etc.), neither the traditional key point detection algorithm nor the template matching algorithm adopted by the present invention can match. Measures that can be taken include manually collecting a new reference image and then performing correction, or storing multiple reference images with multiple perspectives and multiple positions, traversing each reference image to execute the offset detection algorithm, to ensure that the reference image and the detection image have the same target object, or have a certain similarity.

[0122] In one implementation, performing camera preset position correction processing based on the number of feature point set pairs includes:

[0123] When the feature point set is smaller than the threshold, the automatic registration mode is automatically entered (the steps are the same as S41-S43).

[0124] In this embodiment, when the camera preset position correction processing is performed based on the number of feature point sets in step S13, when the number of feature point sets is less than a threshold, the enhanced template matching algorithm can be skipped and the automatic registration mode can be directly entered for correction.

[0125] A second aspect of the present invention provides a camera preset position correction device, comprising:

[0126] A feature point set extraction module, used to extract a first feature point set of a first detection image;

[0127] The feature point set pair extraction module is used to match the first feature point set with the second feature point set to obtain a first feature point set pair, wherein the second feature point set is extracted from a reference image.

[0128] The preset position correction module performs correction processing of the camera preset position based on the number of the first feature point set pairs.

[0129] Further, in the preset position correction module, when the number of the first feature point set pairs is less than a threshold, the following correction method is performed:

[0130] Restoring the focal length of the camera to an initial setting value;

[0131] The camera is rotated 360 degrees and images are captured at predetermined time intervals to obtain a plurality of second detection images;

[0132] Transmitting a plurality of second detection images to a feature point set extraction module, extracting a feature point set of each second detection image by the feature point set extraction module, matching the feature point set of the second detection image with the second feature point set by a feature point set pair extraction module, and obtaining a second feature point set pair of each second detection image and the reference image;

[0133] sorting the second detection images in descending order based on the number of the second feature point set pairs;

[0134] Taking at least one of the first second detection images, and calculating the perspective transformation matrix between the second detection image and the reference image;

[0135] The horizontal offset and the vertical offset of the camera are calculated based on the perspective transformation matrix.

[0136] Furthermore, the preset position correction module is further used to calculate a first perspective transformation matrix of the first detection image and the reference image when the feature point set pair is greater than or equal to a threshold;

[0137] The horizontal offset and the vertical offset of the camera are calculated based on the perspective transformation matrix, and the preset position of the camera is corrected based on the horizontal offset and the vertical offset.

[0138] Furthermore, the preset position correction module is also used to execute the following correction method when the number of the feature point set pairs is less than a threshold:

[0139] Performing multi-scale scaling on the reference image to obtain multiple templates;

[0140] Match each of the templates with the first detection image pixel by pixel to obtain a similarity matrix corresponding to each of the templates;

[0141] Obtaining the first element in each of the similarity matrices, determining the similarity matrix to which the maximum value in the first element belongs as a matching result matrix, and using the template corresponding to the matching result matrix as a matching template, wherein the first element is taken from the maximum value in each of the similarity matrices;

[0142] The horizontal offset and the vertical offset are calculated based on the central coordinates of the matching template and the central coordinates of the first detection image.

[0143] Furthermore, the camera preset position correction device also includes a correction execution module, which is used to perform preset position correction on the camera according to the horizontal offset, the vertical offset or the focal length zoom value.

[0144] A third aspect of the present invention provides an electronic device, such as Figure 5 As shown, Figure 5 FIG. 4 is a schematic diagram of the structure of an electronic device of the present invention, wherein the electronic device comprises a processor 40, a memory 41, an input device 42, an output device 43 and a communication device 44; the number of processors 40 in the computer device can be one or more. Figure 5 The processor 40, the memory 41, the input device 42 and the output device 43 in the electronic device can be connected by a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0145] The memory 41 is a computer-readable storage medium that can be used to store software programs, computer executable programs, and modules. The processor 40 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 41 to implement the camera preset position correction method of any of the above embodiments of the present invention.

[0146] The memory 41 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 41 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 41 may further include a memory remotely arranged relative to the processor 40, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0147] The input device 42 may be used to receive an ID and password etc. input by a user; the output device 43 is used to output a network configuration page.

[0148] In a fourth aspect of the present invention, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the camera preset position correction method of any of the above embodiments of the present invention is implemented. The storage medium may be a ROM / RAM, a magnetic disk, an optical disk, etc.

[0149] According to a fifth aspect of the present invention, a computer program product is provided. When the computer program product is run on a computer, the computer is enabled to execute the camera preset position correction method according to any one of the above embodiments of the present invention.

[0150] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A camera preset position correction method, characterized in that: include: Extracting a first feature point set of a first detection image; Matching the first feature point set with a second feature point set to obtain a first feature point set pair, wherein the second feature point set is extracted from a reference image; Performing correction processing on the preset position of the camera based on the number of the first feature point set pairs; The correction processing of the camera preset position based on the number of the first feature point set pairs includes: When the number of the feature point set pairs is less than the threshold, the following correction method is performed: Performing multi-scale scaling on the reference image to obtain multiple templates; Match each of the templates with the first detection image pixel by pixel to obtain a similarity matrix corresponding to each of the templates; Obtaining the first element in each of the similarity matrices, determining the similarity matrix to which the maximum value in the first element belongs as a matching result matrix, and using the template corresponding to the matching result matrix as a matching template, wherein the first element is taken from the maximum value in each of the similarity matrices; The horizontal offset and the vertical offset of the camera are calculated based on the central coordinates of the matching template and the central coordinates of the first detection image.

2. The camera preset position correction method according to claim 1, characterized in that: The correction processing of the camera preset position based on the number of the first feature point set pairs includes: When the number of the first feature point set pairs is less than a threshold, the following correction method is performed: The camera is rotated 360 degrees and images are captured at predetermined time intervals to obtain a plurality of second detection images; Extracting a feature point set of each second detection image and matching it with the second feature point set to obtain a second feature point set pair of each second detection image and the reference image; sorting the second detection images in descending order based on the number of the second feature point set pairs; Taking at least one of the first second detection images in the sorting order, and calculating a first perspective transformation matrix between the second detection image and the reference image; Calculate the horizontal offset and the vertical offset of the camera based on the first perspective transformation matrix; When the number of the first feature point set pairs is greater than or equal to a threshold, the following correction method is performed: Calculating a second perspective transformation matrix between the first detection image and the reference image; The horizontal offset and the vertical offset of the camera are calculated based on the second perspective transformation matrix.

3. The camera preset position correction method according to claim 1, characterized in that: Before obtaining the first element in each of the similarity matrices, the method further includes: The elements in each similarity matrix are filtered by a filtering threshold, and elements smaller than the filtering threshold are removed; wherein each similarity matrix corresponds to a filtering threshold, and the larger the scaling ratio of the template is, the smaller the filtering threshold is.

4. The camera preset position correction method according to claim 1, characterized in that: Before matching each of the templates with the first detection image pixel by pixel, the method further includes: According to the scaling size of the template, black borders of the same size are filled around the first detection image.

5. The camera preset position correction method according to claim 1, characterized in that: The method further comprises: Calculate the aspect ratio of the matching template and the first detection image to obtain the focal length scaling factor: r = (w1 / w0+h1 / h0) / 2, where w1 and h1 are the width and height of the matching template, respectively, and w0 and h0 are the width and height of the first detection image, respectively; The camera is pre-positioned and calibrated based on the horizontal offset, the vertical offset and the focal length zoom factor.

6. The camera preset position correction method according to claim 1, characterized in that: When the horizontal offset or the vertical offset is greater than a preset value, the method further includes: The camera is rotated 360 degrees and images are captured at predetermined time intervals to obtain a plurality of second detection images; Matching each of the second detection images with the second feature point set to obtain a second feature point set pair of each of the second detection images and the reference image; sorting the second detection images in descending order based on the number of the second feature point set pairs; Taking at least one of the first second detection images, and calculating the perspective transformation matrix between the second detection image and the reference image; The horizontal offset and the vertical offset of the camera are calculated based on the perspective transformation matrix.

7. A camera preset position correction device, comprising: A feature point set extraction module, used to extract a first feature point set of a first detection image; A feature point set pair extraction module, used for matching the first feature point set with a second feature point set to obtain a first feature point set pair, wherein the second feature point set is extracted from a reference image; A preset position correction module, used for performing a correction process of the camera preset position based on the number of the first feature point set pairs; The correction processing of the camera preset position based on the number of the first feature point set pairs includes: When the number of the feature point set pairs is less than the threshold, the following correction method is performed: Performing multi-scale scaling on the reference image to obtain multiple templates; Match each of the templates with the first detection image pixel by pixel to obtain a similarity matrix corresponding to each of the templates; Obtaining the first element in each of the similarity matrices, determining the similarity matrix to which the maximum value in the first element belongs as a matching result matrix, and using the template corresponding to the matching result matrix as a matching template, wherein the first element is taken from the maximum value in each of the similarity matrices; The horizontal offset and the vertical offset of the camera are calculated based on the central coordinates of the matching template and the central coordinates of the first detection image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the camera preset position correction method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the camera preset position correction method according to any one of claims 1 to 6 is implemented.