Camera attitude self-adaptive calibration method based on feature matching and related equipment

Abstract

The invention provides a camera attitude self-adaptive calibration method based on feature matching and related equipment. The method comprises the following steps: acquiring an original picture under an original calibration view angle and current pictures in different time periods under multiple scenes after view angle change with the original calibration view angle; extracting feature points from the original picture and the current picture; performing feature point matching on the original picture and the current picture to obtain feature point pairs; obtaining a homographic transformation matrix between the images according to the matched feature point pairs; and judging whether the homography transformation matrix is correct or not according to the pixel variation before and after the homography transformation matrix is applied to the current picture and the spatial distribution rate of the successfully matched feature points in the current picture, and correcting the picture shot by the camera to the original calibration view angle according to the homography transformation matrix. Deployment personnel do not need to re-calibrate the field equipment at regular intervals and only need to pay attention to the automatic calibration result, the workload of equipment maintenance is reduced, and meanwhile the positioning precision of the field equipment detection target is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of vehicle control technology, and in particular to a camera attitude adaptive calibration method based on feature matching and related equipment. Background Art

[0002] The roadside perception system uses a variety of sensors such as visual sensors, millimeter-wave radars, and lidars, combined with edge computing devices, to monitor the current road condition data in real time. In addition, the roadside perception system can provide drivers with real-time information about the road environment and early warning information related to travel safety based on the current road condition data, and provide monitoring and prediction of the road traffic environment for relevant transportation departments.

[0003] Considering the high cost of LiDAR, laser vision has become the mainstream solution for roadside perception equipment. Roadside camera positioning generally selects a picture from the pictures collected by the device as a reference picture, and uses a calibration method to associate the pixel coordinates with the 3D world coordinates. Roadside perception equipment is generally installed on electric poles on the side of the road, but in actual use, due to various factors such as vibration and natural wind, it is found that the viewing angle of the camera equipment that was originally calibrated has changed compared to the original calibration, which makes the positioning accuracy worse, thus causing positioning errors. Summary of the invention

[0004] The present invention provides a camera attitude adaptive calibration method based on feature matching and related equipment, aiming to solve the problems existing in the prior art.

[0005] The purpose of the present invention is achieved by the following technical solutions:

[0006] In a first aspect, the present invention provides a camera attitude adaptive calibration method based on feature matching, comprising:

[0007] Acquire an original image at an original calibrated viewing angle and current images at different time periods in multiple scenes after a viewing angle change occurs compared to the original calibrated viewing angle;

[0008] Extracting feature points from the original image and the current image;

[0009] Perform feature point matching on the original image and the current image to obtain a feature point pair;

[0010] Obtaining a homography transformation matrix between images according to the feature point pairs obtained by matching;

[0011] Obtaining a pixel change amount before and after the homography transformation matrix is ​​applied to the current image, and determining whether the pixel change amount exceeds a preset first threshold, and obtaining a spatial distribution rate of successfully matched feature points in the current image, and determining whether the spatial distribution rate exceeds a preset second threshold;

[0012] If the pixel change does not exceed the first threshold and the spatial distribution rate exceeds the second threshold, outputting the homography transformation matrix, and correcting the picture taken by the camera to the original calibration viewing angle according to the homography transformation matrix;

[0013] If the pixel variation exceeds the first threshold or the spatial distribution rate does not exceed the second threshold, a viewing angle correction failure warning is output.

[0014] According to a camera attitude adaptive calibration method based on feature matching provided by the present invention, the step of extracting feature points from the original picture and the current picture includes:

[0015] Detecting feature points of the original image and the current image using a scale-invariant feature transformation algorithm to obtain scale information of the feature points;

[0016] Obtaining gradient information of a neighborhood range of the feature point according to the scale information of the feature point;

[0017] According to the gradient information of the neighborhood range, the corresponding feature point descriptor information is constructed to determine the main direction of the feature point, and then the feature point in the image is extracted.

[0018] According to a camera attitude adaptive calibration method based on feature matching provided by the present invention, before the step of extracting feature points from the original picture and the current picture, the method further includes: preprocessing the original picture and the current picture, specifically including:

[0019] Gaussian smoothing filtering is performed on the original image and the current image, and histogram equalization processing is performed on a local area of ​​the image.

[0020] According to a camera attitude adaptive calibration method based on feature matching provided by the present invention, the step of matching feature points of the original image and the current image to obtain feature point pairs includes:

[0021] In the feature point matching process, for the feature point of the original image, the feature point closest to the feature point and the feature point with the second closest distance to the feature point are searched from the current image, and the closest distance and the second closest distance are obtained;

[0022] Calculating the difference between the closest distance and the second closest distance, and determining whether the difference is less than a preset third threshold value, if it is less than the third threshold value, determining it as a feature point pair that matches successfully, if not, the feature point matching fails;

[0023] All feature points of the original image are traversed, and all matching feature points are found from the current image to obtain the successfully matched feature point pairs.

[0024] According to a camera attitude adaptive calibration method based on feature matching provided by the present invention, the step of obtaining a homography transformation matrix between images according to the feature point pairs obtained by matching comprises:

[0025] The RANSAC algorithm is used to iteratively calculate the optimal homography transformation matrix and output it, including:

[0026] In each iteration, four groups of non-collinear feature point pairs are randomly selected from all the successfully matched feature point pairs as seed points to calculate the initial homography transformation matrix;

[0027] Using the initial homography transformation matrix, calculating the projection error of other non-seed points, and if the projection error is less than a preset fourth threshold, adding the non-seed point as an inlier point to the inlier point set;

[0028] Compare whether the number of interior points in the current interior point set is greater than the number of interior points in the previous optimal interior point set. If so, take the current interior point set as the new optimal interior point set;

[0029] The iterative process is repeated until the set iteration end condition is reached, and the homography transformation matrix is ​​calculated using all the interior points in the interior point set with the largest number of interior points to be used as the optimal homography transformation matrix and output.

[0030] According to a feature matching-based camera attitude adaptive calibration method provided by the present invention, after outputting the homography transformation matrix, it also includes: superimposing the current image to which the homography transformation matrix is ​​applied with the original image to verify whether the current image is corrected to the original calibration viewing angle.

[0031] According to a camera attitude adaptive calibration method based on feature matching provided by the present invention, the step of obtaining the spatial distribution rate of the successfully matched feature points in the current picture includes:

[0032] The current image is divided into a plurality of grid areas, and based on the number of the grid areas and the number of successfully matched feature points in the current image, the number of feature points in each grid area is counted respectively to determine the spatial distribution rate of the successfully matched feature points in the current image.

[0033] In a second aspect, the present invention provides an electronic device, the electronic device comprising:

[0034] one or more processors;

[0035] and a memory storing computer program instructions, wherein when the computer program instructions are executed, the processor executes the steps of any of the above-mentioned methods for adaptively calibrating camera postures based on feature matching.

[0036] In a third aspect, the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of any of the above-described feature matching-based camera posture adaptive calibration methods.

[0037] In a fourth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of any of the above-described feature matching-based camera posture adaptive calibration methods.

[0038] The present invention provides a camera attitude adaptive calibration method based on feature matching and related equipment, which obtains an original picture under an original calibration viewing angle and a current picture of different time periods in multiple scenes after the viewing angle changes with the original calibration viewing angle; extracts feature points from the original picture and the current picture; performs feature point matching on the original picture and the current picture to obtain feature point pairs; obtains a homography transformation matrix between the images according to the matched feature point pairs; obtains the pixel change amount before and after the current picture is applied with the homography transformation matrix, determines whether the pixel change amount exceeds a preset first threshold, and obtains the spatial distribution rate of the successfully matched feature points in the current picture, determines whether the spatial distribution rate exceeds a preset second threshold; if the pixel change amount does not exceed the first threshold and the spatial distribution rate exceeds the second threshold, outputs the homography transformation matrix, and corrects the picture taken by the camera to the original calibration viewing angle according to the homography transformation matrix; if the pixel change amount exceeds the first threshold or the spatial distribution rate does not exceed the second threshold, outputs a warning of viewing angle correction failure. The present invention designs this solution to cope with the continuous change of the camera's viewing angle of the roadside perception equipment during actual use. A reasonable and correct homography transformation matrix is ​​used to perform original calibration viewing angle correction on the pictures taken by the camera after the viewing angle changes, thereby eliminating the positioning error. This makes it unnecessary for deployment personnel to recalibrate the on-site equipment regularly, and they only need to pay attention to the corrected automatic calibration results, thus reducing the workload of equipment maintenance and ensuring the positioning accuracy of the on-site equipment detection targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other implementation methods can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 It is a flowchart of a camera posture adaptive calibration method based on feature matching and related equipment provided by an embodiment of the present invention;

[0041] Figure 2 This is a distribution effect diagram of image feature points extracted from a picture provided by an embodiment of the present invention;

[0042] Figure 3 This is an effect diagram of feature matching of feature points provided by an embodiment of the present invention;

[0043] Figure 4 This is an effect diagram of the current image not being corrected and superimposed and compared with the original image provided by an embodiment of the present invention;

[0044] Figure 5 This is an effect diagram of the current image after correction and superimposition and comparison with the original image provided by an embodiment of the present invention;

[0045] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0046] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following specifically cites a preferred embodiment and describes it in detail with the accompanying drawings as follows.

[0047] It should be noted that it is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in the present invention can be combined with other embodiments without conflict. Unless otherwise defined, the technical terms or scientific terms involved in the present invention should be the usual meanings understood by people with ordinary skills in the technical field to which the present invention belongs. The words "one", "a", "a", "the" and the like involved in the present invention do not indicate a quantitative limitation and may represent the singular or plural. The terms "including", "comprising", "having" and any of their variations involved in the present invention are intended to cover non-exclusive inclusions; the terms "first", "second", "third", etc. involved in the present invention are merely to distinguish similar objects and do not represent a specific ordering of objects.

[0048] The scale-invariant feature transform algorithm, referred to as the SIFT algorithm, is a classic image feature extraction algorithm that can detect key point features (SIFT features) in an image. It is a local feature descriptor with the following characteristics:

[0049] ⑴ Scale invariance: It has good adaptability to the scale change of the image and can detect and describe the same feature points at different scales;

[0050] ⑵Rotation invariance: SIFT features are invariant to image rotation. Even if the image is rotated, the SIFT features of the same object can still be matched;

[0051] ⑶ Illumination invariance: SIFT features are robust to illumination changes and can handle illumination changes of images to a certain extent;

[0052] ⑷ Locality: SIFT features extract local feature points in the image. By describing the gradient information of the local area, it can accurately represent the local features of the object;

[0053] ⑸High-dimensional description: Each SIFT feature point is represented by a 128-dimensional vector, which contains rich local feature information, making the feature point more distinguishable;

[0054] ⑹ Robustness: SIFT features can still extract stable feature points and maintain good matching performance when facing a certain degree of noise, blur and other interference.

[0055] Due to the above advantages, the SIFT algorithm is widely used in target detection, image matching, image stitching and other fields.

[0056] The RANSAC algorithm is an iterative method for estimating mathematical model parameters from observed data containing outliers (or outliers), and is also used as an outlier detection method. Its basic assumption is that the data consists of correct data (inliers) and outliers, where inliers can be described by certain model parameters, while outliers do not conform to the model. The main steps of the RANSAC algorithm include:

[0057] Randomly select a subset from the original data set as a sample, which is usually the smallest subset, such as two points in a straight line fit and three points in a circle fit;

[0058] Use these samples to estimate the parameters of the model;

[0059] Iterate through the remaining data points in the dataset and mark a data point as an inlier if its distance from the model is within a given error range, otherwise it is an outlier.

[0060] Count the number of all inliers. If the given threshold condition is met, use these inliers to re-estimate the model parameters and end the algorithm.

[0061] If the number of inliers does not meet the threshold condition, the above steps are repeated until the maximum number of iterations is reached.

[0062] The advantages of the RANSAC algorithm include its robustness, its ability to ignore data points that do not conform to the model, and its relative simplicity and ease of understanding. In addition, it does not require any assumptions about the distribution of the data, and is suitable for estimating parameters in data sets with noise and outliers in the field of computer vision and image processing.

[0063] When it comes to method steps, the sequence illustrated in this document represents an exemplary solution, but does not mean a limitation on the sequence.

[0064] Reference Figure 1 As shown, an embodiment of the present invention provides a camera posture adaptive calibration method based on feature matching, comprising:

[0065] Step S1: obtaining an original image under an original calibrated viewing angle and current images at different time periods under multiple scenes after a viewing angle change occurs compared to the original calibrated viewing angle.

[0066] In this step, the obtained original image and current image are preprocessed, specifically including: performing Gaussian smoothing filtering on the original image and the current image and performing histogram equalization processing on the local area of ​​the image.

[0067] Specifically, the image data is preprocessed using the functions provided by OpenCV, the GaussianBlur function in the OpenCV function library is called to perform Gaussian smoothing filtering on the image, and the equalizeHist function is called to perform histogram equalization processing on the local area of ​​the image, which can effectively remove the noise in the image, and the blurring processing through Gaussian smoothing filtering makes the image details softer and reduces the sharp edges in the image, which is equivalent to blurring the details. At the same time, the histogram equalization processing is used to enhance the adaptability to lighting. The stability of the subsequent feature point detection in the image is improved by preprocessing the original image and the current image.

[0068] Step S2: Extract feature points from the original image and the current image.

[0069] In this embodiment, the SIFT algorithm is used to extract feature points from the original image and the current image. The extracted feature points are as follows: Figure 2 As shown, step S2 specifically includes:

[0070] Step S201: Detect feature points of the original image and the current image using a scale-invariant feature transformation algorithm to obtain scale information of the feature points;

[0071] Step S202: Obtaining gradient information of a neighborhood range of the feature point according to the scale information of the feature point;

[0072] Step S203: construct corresponding feature point descriptor information according to the gradient information of the neighborhood range to determine the main direction of the feature point, and then extract the feature point in the image.

[0073] Specifically, the SIFT algorithm is used to obtain the scale information of the image feature points, and the gradient information of the neighborhood range of the feature points is calculated based on the size information, including the gradient direction and gradient amplitude of the feature points. For example, the neighborhood range of the feature points is divided into 4×4 pixel blocks, and a feature vector is generated based on the gradient information of the neighborhood range. Each pixel block is represented by an 8-dimensional vector with a feature description. A histogram of the statistical gradient amplitude is generated in a square area, and 16 (4×4) histograms are generated in a 16×16 neighborhood range. The amplitude and gradient direction of the feature points in each area are calculated and counted into an 8-bin histogram. Therefore, in order to describe the characteristics of each feature point, a total of 4×4×8=128-dimensional feature vectors are used.

[0074] Step S3: Match feature points of the original image and the current image to obtain feature point pairs.

[0075] In this embodiment, step S3 specifically includes:

[0076] Step S301: In the feature point matching process, for a feature point of the original image, find a feature point that is closest to the feature point and a feature point that is the second closest distance from the feature point in the current image, and obtain the closest distance and the second closest distance;

[0077] Step S302: Calculate the difference between the closest distance and the second closest distance, and determine whether the difference is less than a preset third threshold. If so, determine that the feature point pair is successfully matched; if not, the feature point matching fails.

[0078] Step S203: traverse all feature points of the original image, and find all matching feature points from the current image to obtain successfully matched feature point pairs.

[0079] Specifically, find the same feature points among the many feature points of the two images, and call the knnMatch function of the BFMatcher class in the OpenCV function library for feature matching. The specific algorithm process of the knnMatch function is: for the feature points of the original image, traverse all the feature points in the current image. For the situation where there may be multiple feature points in a certain area of ​​the image whose features are similar to the feature points of the original image, there may be noise points at this time. In order to avoid the interference of noise points on the matching results. The nearest neighbor distance and the second nearest neighbor distance method are used to find the feature points with the largest feature difference, so that the nearest neighbor distance of the successfully matched feature point pair is much smaller than the second nearest neighbor distance. Specifically, find the feature points and the second nearest distance closest to the feature points of the original image from the current image, use the Euclidean distance or cosine distance and other measurement methods to obtain the nearest distance and the second nearest distance, calculate the difference between the nearest distance and the second nearest distance, and judge whether the difference is less than the preset third threshold. If it is less than the third threshold, it is determined to be a successfully matched feature point pair. If not, it is considered that the feature point matching fails. The matching results are as follows: Figure 3 As shown, the green lines in the figure are the pairs of feature points that are successfully matched in the two pictures.

[0080] Step S4: Obtain the homography transformation matrix between the images based on the feature point pairs obtained by matching.

[0081] In this embodiment, the RANSAC algorithm is used to iteratively calculate and output the optimal homography transformation matrix, which specifically includes:

[0082] Step S401: In each iteration process, four groups of non-collinear feature point pairs are randomly selected from all successfully matched feature point pairs as seed points to calculate the initial homography transformation matrix;

[0083] Step S402: using the initial homography transformation matrix, calculating the projection error of other non-seed points, if the projection error is less than a preset fourth threshold, adding the non-seed point as an inlier point to the inlier point set;

[0084] Step S403: comparing whether the number of inliers in the current inlier set is greater than the number of inliers in the previous optimal inlier set, if so, taking the current inlier set as the new optimal inlier set;

[0085] Step S404: Repeat the iteration process until the set iteration end condition is reached, and use all the inliers in the inlier set with the largest number of inliers to calculate the homography transformation matrix as the optimal homography transformation matrix and output it.

[0086] Specifically, based on the RANSAC algorithm, the findHomography function in the opencv function library is called to calculate the homography transformation matrix based on the successfully matched feature point pairs. During each iteration, non-collinear feature point pairs are randomly selected to calculate a new homography transformation matrix until the iteration reaches the set number of iterations or the projection error reaches the set threshold range. The iteration ends and the inlier set with the largest number of inliers is obtained. The homography transformation matrix is ​​calculated using all the inliers in the inlier set as the optimal homography transformation matrix and output.

[0087] Step S5: obtaining the pixel change amount before and after the homography transformation matrix is ​​applied to the current image, determining whether the pixel change amount exceeds a preset first threshold, and obtaining the spatial distribution rate of the successfully matched feature points in the current image, determining whether the spatial distribution rate exceeds a preset second threshold;

[0088] If the pixel change amount does not exceed the first threshold and the spatial distribution rate exceeds the second threshold, executing step S6: outputting the homography transformation matrix, and correcting the picture taken by the camera to the original calibration perspective according to the homography transformation matrix;

[0089] If the pixel variation exceeds the first threshold or the spatial distribution ratio does not exceed the second threshold, step S7 is executed: outputting a viewing angle correction failure warning.

[0090] Specifically, for some extreme scenes (such as pictures taken at night with poor visibility), there are too few feature points found by the algorithm, and the homography transformation matrix obtained by applying the output will cause image distortion and cannot be restored. Therefore, it is necessary to set a protection mechanism for the algorithm to ensure that the output homography transformation matrix is ​​correct and reasonable. Specifically, the two pictures before and after the current picture is transformed by the homography transformation matrix are divided into multiple grid areas, and the pixel changes between corresponding pixels in different grid areas are counted, and a first threshold is set. For the case of large changes in viewing angle, the judgment threshold of pixel change can be set larger. On the contrary, for the case of small changes in viewing angle, the threshold of pixel change can be set smaller according to the actual situation to adapt to the algorithm protection mechanism of different viewing angle changes. At the same time, the spatial distribution rate of successfully matched feature points in different grid areas is determined in the form of a regional distribution histogram. The larger the spatial distribution rate of feature points (indicating that the feature points are more evenly distributed), the higher the accuracy of setting the judgment threshold of the spatial distribution rate is, so as to ensure that the error rate of the output homography transformation matrix is ​​lower. When the pixel change exceeds the set first threshold or the spatial distribution rate of the feature points does not exceed the set second threshold (indicating that the feature points are too concentrated), the algorithm outputs a warning of failed view correction, informing the operator to confirm whether the recalibration result is reliable. For the output homography transformation matrix when the pixel change does not exceed the first threshold and the spatial distribution rate exceeds the second threshold, the image taken by the camera is corrected to the original calibration view according to the homography transformation matrix, which can eliminate the positioning error of the camera and ensure the accuracy of the camera's detection of the target.

[0091] In this embodiment, the current image to which the homography transformation matrix is ​​applied is superimposed with the original image to verify whether the current image is corrected to the original calibration perspective. Figure 4 As shown in the figure, after the current image without homography transformation matrix is ​​superimposed and compared with the original image, the zebra crossing in the red selected area in the figure cannot be overlapped, indicating that there is a problem of perspective change between the two images, which affects the positioning ability of the camera. Figure 5 As shown in the figure, the two pictures fit perfectly, indicating that the current picture has been corrected to the original calibration perspective, which does not affect the positioning accuracy of the camera.

[0092] Therefore, in order to cope with the continuous changes in the camera perspective of the roadside perception equipment during actual use, an embodiment of the present invention provides a camera posture adaptive calibration method based on feature matching, which uses a reasonable and correct homography transformation matrix to correct the original calibration perspective of the pictures taken by the camera after the perspective changes, thereby eliminating the positioning error. This makes it unnecessary for deployment personnel to recalibrate the on-site equipment regularly, and they only need to pay attention to the corrected automatic calibration results, thereby reducing the workload of equipment maintenance and ensuring the positioning accuracy of the on-site equipment detection targets.

[0093] In addition, some embodiments of the present application also provide an electronic device. The electronic device may be a digital computer in various forms, such as a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, etc. The electronic device may also be a mobile device in various forms, such as a personal digital processing, a cellular phone, a smart phone, a wearable device, and other similar computing devices.

[0094] The electronic device includes: one or more processors; and a memory storing computer program instructions, and when the computer program instructions are executed, the processor executes the steps of the camera posture adaptive calibration method based on feature matching as provided in any one or more of the above embodiments. Figure 6 An exemplary structural diagram of the electronic device is disclosed. Figure 6 As shown, the electronic device includes: one or more processors 1101, memory 1102, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Among them, the components shown in this article, their connections and relationships, and their functions are only used as examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0095] The electronic device may further include: an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0096] The input device 1103 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1104 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0097] To provide interaction with a user, the electronic device may be a computer. The computer has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0098] In an embodiment of the present application, a computer program / instruction is stored on a computer-readable medium, and when the computer program / instruction is executed by a processor, the steps of the camera posture adaptive calibration method based on feature matching provided in any one or more of the above embodiments are implemented. The computer-readable medium may be included in the electronic device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more computer-readable instructions.

[0099] The memory 1102 can be used as a non-transient computer-readable storage medium, which can be used to store non-transient software programs, non-transient computer executable programs and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1102, so as to implement the program instructions / modules corresponding to the method provided by any one or more embodiments in the embodiments of the present application.

[0100] The memory 1102 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1102 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1102 may optionally include a memory remotely arranged relative to the processor 1101, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0101] It should be noted that more specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0102] Computer readable storage media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0103] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0104] In the above-described embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. For example, it can be implemented by using an application specific integrated circuit (ASIC), a general-purpose computer or any other similar hardware device. In certain embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented by hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0105] The computer program product provided in the embodiment of the present application includes one or more computer programs / instructions, and when the computer program / instructions are executed by the processor, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0106] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0107] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily mention changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims, and the above embodiments should be regarded as exemplary and non-restrictive.

Claims

1. A camera attitude adaptive calibration method based on feature matching, characterized in that: include: Acquire an original image at an original calibrated viewing angle and current images at different time periods in multiple scenes after a viewing angle change occurs compared to the original calibrated viewing angle; Extracting feature points from the original image and the current image; Perform feature point matching on the original image and the current image to obtain a feature point pair; Obtaining a homography transformation matrix between images according to the feature point pairs obtained by matching; Obtaining a pixel change amount before and after the homography transformation matrix is ​​applied to the current image, and determining whether the pixel change amount exceeds a preset first threshold, and obtaining a spatial distribution rate of successfully matched feature points in the current image, and determining whether the spatial distribution rate exceeds a preset second threshold; If the pixel change does not exceed the first threshold and the spatial distribution rate exceeds the second threshold, outputting the homography transformation matrix, and correcting the picture taken by the camera to the original calibration viewing angle according to the homography transformation matrix; If the pixel variation exceeds the first threshold or the spatial distribution rate does not exceed the second threshold, a viewing angle correction failure warning is output.

2. The camera attitude adaptive calibration method based on feature matching according to claim 1, characterized in that: The step of extracting feature points from the original picture and the current picture includes: Detecting feature points of the original image and the current image using a scale-invariant feature transformation algorithm to obtain scale information of the feature points; Obtaining gradient information of a neighborhood range of the feature point according to the scale information of the feature point; According to the gradient information of the neighborhood range, the corresponding feature point descriptor information is constructed to determine the main direction of the feature point, and then the feature point in the image is extracted.

3. The camera attitude adaptive calibration method based on feature matching according to claim 1, characterized in that: Before the step of extracting feature points from the original picture and the current picture, the method further includes: preprocessing the original picture and the current picture, specifically including: Gaussian smoothing filtering is performed on the original image and the current image, and histogram equalization processing is performed on a local area of ​​the image.

4. The camera attitude adaptive calibration method based on feature matching according to claim 1, characterized in that: The step of matching the feature points of the original image and the current image to obtain a feature point pair includes: In the feature point matching process, for the feature point of the original image, the feature point closest to the feature point and the feature point with the second closest distance to the feature point are searched from the current image, and the closest distance and the second closest distance are obtained; Calculating the difference between the closest distance and the second closest distance, and determining whether the difference is less than a preset third threshold value, if it is less than the third threshold value, determining it as a feature point pair that matches successfully, if not, the feature point matching fails; All feature points of the original image are traversed, and all matching feature points are found from the current image to obtain the successfully matched feature point pairs.

5. The camera attitude adaptive calibration method based on feature matching according to claim 1, characterized in that: The step of obtaining a homography transformation matrix between images according to the feature point pairs obtained by matching comprises: The RANSAC algorithm is used to iteratively calculate the optimal homography transformation matrix and output it, including: In each iteration, four groups of non-collinear feature point pairs are randomly selected from all the successfully matched feature point pairs as seed points to calculate the initial homography transformation matrix; Using the initial homography transformation matrix, calculating the projection error of other non-seed points, and if the projection error is less than a preset fourth threshold, adding the non-seed point as an inlier point to the inlier point set; Compare whether the number of interior points in the current interior point set is greater than the number of interior points in the previous optimal interior point set. If so, take the current interior point set as the new optimal interior point set; The iterative process is repeated until the set iteration end condition is reached, and the homography transformation matrix is ​​calculated using all the interior points in the interior point set with the largest number of interior points to be used as the optimal homography transformation matrix and output.

6. The camera attitude adaptive calibration method based on feature matching according to claim 1, characterized in that: After outputting the homography transformation matrix, the method further includes: superimposing the current image to which the homography transformation matrix is ​​applied with the original image to verify whether the current image is corrected to the original calibrated viewing angle.

7. The camera attitude adaptive calibration method based on feature matching according to claim 1, characterized in that: The step of obtaining the spatial distribution rate of the successfully matched feature points in the current image includes: The current image is divided into a plurality of grid areas, and a spatial distribution rate of the feature points is obtained based on the number of the grid areas and the number of successfully matched feature points in the current image.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; And a memory storing computer program instructions, wherein when the computer program instructions are executed, the processor executes the steps of the camera posture adaptive calibration method based on feature matching as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the camera pose adaptive calibration method based on feature matching are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the camera posture adaptive calibration method based on feature matching as described in any one of claims 1 to 7 are implemented.

Images