On-line testing method, system and equipment for validity of dual-target calibration parameter and medium

Abstract

The invention discloses a dual-target calibration parameter validity online inspection method, system and device and a medium. The method comprises the following steps: acquiring a left image and a right image after epipolar correction; calculating the pixel difference of the matching points in the y direction on the left and right images after epipolar correction by using an optical flow estimation algorithm; abnormal matching points are eliminated, and normal matching points meeting the precision requirement are screened out; and carrying out statistical analysis on the pixel difference of the normal matching point in the y direction, calculating a parameter validity coefficient, and evaluating the validity of the dual-target calibration parameter in the current state through the parameter validity coefficient. And for the principal point deviation caused by only horizontal translation, the principal point deviation is evaluated by periodically and repeatedly measuring the fixed marker in the world coordinate system and monitoring the point volatility of the measurement coordinate of the fixed marker converted into the world coordinate system. According to the method, the feature point matching accuracy and density of weak texture and texture-free objects are improved by using a deep learning matching algorithm, the online verification precision is improved, and the operation complexity of the algorithm is reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method, system, device and medium for online verification of the validity of dual-target positioning parameters. Background Art

[0002] There are many existing binocular calibration methods, but in practical applications, the calibration parameters often become invalid due to factors such as poor calibration plate image quality, inappropriate calibration algorithm selection, or loose camera equipment, thereby reducing the overall accuracy of the stereo vision system. For example, in the manual calibration process, if the calibration plate image quality is not ideal, it is easy to introduce corner extraction errors; different calibration algorithms may produce inconsistent calibration results due to their respective characteristics; in actual use, such as in unmanned navigation systems, the visual system may be affected by bumps or collisions due to long-term operation, resulting in parameter failure. At the same time, changes in ambient temperature can also cause calibration parameters to drift, affecting the accuracy of the system. In addition, the principal point deviation caused by horizontal translation alone can also introduce calibration errors, but this deviation is difficult to identify in existing methods.

[0003] The commonly used parameter monitoring methods currently mainly evaluate the effectiveness of calibration parameters by counting the epipolar errors of significant feature points. However, such methods are weak in the number of feature points and matching accuracy, especially when dealing with objects with weak textures or no textures, the effect is more limited, thus affecting the accuracy of online verification of binocular calibration parameters. Summary of the invention

[0004] The purpose of the present invention is to provide a method, system, device and medium for online verification of the validity of binocular calibration parameters, aiming at the problem of failure of binocular calibration parameters caused by poor calibration plate image quality, improper calibration algorithm selection or loose camera equipment in the above-mentioned prior art. On the one hand, the accuracy and density of feature point matching of weak texture and textureless objects are improved by using a deep learning matching algorithm. On the other hand, for the binocular case, the error introduced by the epipolar line estimation step is simplified, and a new indicator is used to measure the accuracy of epipolar line correction, thereby improving the online verification accuracy of the validity of binocular calibration parameters and reducing the computational complexity of the algorithm.

[0005] In order to achieve the above object, the present invention has the following technical solutions: In a first aspect, a method for online verification of the validity of dual-target calibration parameters is provided, comprising: Get the left and right images after epipolar correction; The optical flow estimation algorithm is used on the left and right images after epipolar correction to calculate the pixel difference of the matching points in the y direction; Eliminate abnormal matching points and select normal matching points that meet the accuracy requirements; The pixel differences of the screened normal matching points in the y direction are statistically analyzed to calculate the parameter validity coefficient, and the validity of the binocular positioning parameters in the current state is evaluated by the parameter validity coefficient.

[0006] As a preferred solution, the online verification method for the validity of dual-target positioning parameters further includes: For principal point deviations caused by horizontal translation alone, the principal point deviation is evaluated by periodically repeating measurements of fixed markers in the world coordinate system and monitoring the point fluctuations of the conversion of the measured coordinates of the fixed markers to the world coordinate system.

[0007] As a preferred solution, the measured coordinates of the fixed marker are converted to the world coordinate system according to the following expression:

[0008] In the formula, represents the measured coordinates of a fixed marker, The image pixel coordinates representing the fixed landmark, represents the pixel coordinates of the principal point, represents the baseline distance of the binocular camera, The disparity of image pixels representing fixed landmarks, Represents the intrinsic focal length of the binocular camera; Set the world coordinates of a fixed marker in multiple periodic repeated measurements to , where Indicates The volatility measure is expressed by calculating the overall volatility of the corresponding measurement values ​​as follows:

[0009] In the formula, is the standard deviation of multiple measurement fluctuations on the three coordinate axes; The error caused by the principal point deviation is determined by measuring the volatility; if the volatility measurement exceeds the set range, it is determined that the position of the fixed marker has changed abnormally and a principal point deviation has occurred.

[0010] As a preferred solution, when using the optical flow estimation algorithm to calculate the pixel difference of the matching points in the y direction on the left and right images after epipolar correction, a two-stage optical flow estimation algorithm based on deep learning is used for each pair of matching points. , calculate the deviation of the matching point in the y direction .

[0011] As a preferred solution, the method of calculating the pixel difference of the matching points in the y direction using the optical flow estimation algorithm on the left and right images after epipolar correction includes the following steps: Based on the brightness consistency assumption, the following optical flow constraint equation expression is constructed:

[0012] In the formula, express Moment image at The pixel intensity at is the optical flow vector, which represents the movement of the pixel; Taylor expansion of the optical flow constraint equation and ignoring high-order terms yields the following expression:

[0013] The sparse optical flow method is used to perform the least squares solution of the optical flow constraint equation in the local window; The two-stage optical flow estimation algorithm based on deep learning adopts the RAFT algorithm, extracts features through the convolutional neural network CNN, and constructs a dense full-pixel pair correlation tensor as follows:

[0014] In the formula, and Represents the pixels in the source image and the target image respectively and pixels The characteristic vector of is the correlation score between the feature vectors; then, the RAFT algorithm iteratively updates the optical flow field through a recurrent neural network, each time updating the optical flow vector according to the following expression :

[0015] The updated optical flow vector Get the left view pixel Corresponding right view pixel .

[0016] As a preferred solution, outliers are removed through threshold screening and consistency detection; The method of eliminating abnormal points by threshold screening includes: based on the collinear geometric characteristics of the left and right matching points after epipolar correction, setting a threshold to eliminate matching points whose y-direction differences exceed the threshold, so as to control the error within a reasonable range; The method of eliminating abnormal points through consistency detection includes: using a left-right consistency detection algorithm to verify whether the results of forward matching and reverse matching are consistent, and eliminating inconsistent matching points.

[0017] As a preferred solution, statistical analysis is performed on the pixel differences of the screened normal matching points in the y direction, and the parameter validity coefficient is calculated. The validity of the binocular positioning parameters in the current state is evaluated by the parameter validity coefficient, which includes: Calculate all normal matching points Mean pixel difference in y direction:

[0018] In the formula, Indicates Measurements, Indicates For the ordinate of the matching point in the right view, Indicates The ordinate of the matching point in the left view; The parameter validity coefficient is the ideal mean value of the pixel difference of the matching points in the y direction. , all normal matching points calculated The mean change rate of pixel differences in the y direction , the calculation expression is as follows:

[0019] The smaller it is, the closer the actual measurement result is to the ideal calibration, and the higher the calibration accuracy and stability.

[0020] In a second aspect, a dual-target calibration parameter validity online verification system is provided, comprising: A correction image acquisition module, used to obtain left and right images after epipolar correction; The optical flow estimation module is used to calculate the pixel difference of the matching points in the y direction using the optical flow estimation algorithm on the left and right images after epipolar correction; The matching point screening module is used to remove abnormal matching points and screen out normal matching points that meet the accuracy requirements; The parameter validity evaluation module is used to perform statistical analysis on the pixel differences of the screened normal matching points in the y direction, calculate the parameter validity coefficient, and evaluate the validity of the binocular positioning parameters in the current state through the parameter validity coefficient.

[0021] According to a third aspect, an electronic device is provided, including: A memory storing at least one instruction; and a processor executing the instruction stored in the memory to implement the online verification method for the validity of dual-target calibration parameters.

[0022] In a fourth aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the computer-readable storage medium, and the at least one instruction is executed by a processor in an electronic device to implement the online verification method for the validity of dual-target calibration parameters.

[0023] Compared with the prior art, the present invention has at least the following beneficial effects: By using the optical flow estimation algorithm to calculate the pixel difference of the matching points in the y direction on the left and right images after epipolar correction, the verification accuracy of the binocular positioning parameter validity test method in complex scenes (weak texture, no texture) is improved. Traditional methods usually rely on local significant features (such as corners or edges) for matching, but these features are difficult to extract or distinguish in weak texture or no texture areas (such as smooth surfaces or repeated textures). In a binocular vision system, the calculation of the epipolar line can be simplified. Ideally, the epipolar lines of the right image and the pixels of the left image should be in the same row, and the matching points of the same name should also be in the same row. By calculating the distance from the matching point of the right image to the epipolar line to which it belongs, it can be simplified to statistically calculate the pixel difference of the points of the same name in the y direction. The present invention uses an optical flow estimation algorithm to calculate the pixel difference of the matching points in the y direction, and the deep learning method extracts multi-scale features through a convolutional neural network (CNN), which can obtain global context information and generate more distinguishing feature representations in weak feature areas. And dynamically adjust the matching relationship during the iterative optimization process, effectively improving the number and accuracy of matching point pairs. Therefore, the present invention can obtain more accurate and richer feature matching results, and indirectly improve the online verification accuracy of parameter validity. In addition, the present invention statistically analyzes the pixel differences of the screened normal matching points in the y direction, calculates the parameter validity coefficient, and evaluates the validity of the dual-target positioning parameters in the current state through the parameter validity coefficient, which greatly simplifies the calculation complexity.

[0024] Furthermore, the method of the present invention also includes evaluating the principal point deviation caused by only horizontal translation by periodically repeating the measurement of fixed markers in the world coordinate system and monitoring the point fluctuation of the measurement coordinates of the fixed markers converted to the world coordinate system, thereby expanding the field of parameter validity verification. The present invention proposes a corresponding verification method for the principal point deviation problem caused only by horizontal translation, which can effectively calculate and correct the principal point deviation caused by translation error, expand the application scenarios of binocular positioning parameter validity verification, and can more comprehensively guarantee the accuracy and stability of the stereo vision system. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 Flow chart of the online verification method for the validity of dual-target calibration parameters according to an embodiment of the present invention; Figure 2 A structural block diagram of an online verification system for the validity of dual-target calibration parameters according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, ordinary technicians in this field can also obtain other embodiments without making creative work.

[0028] See also Figure 1 The embodiment of the present invention provides an online verification method for the validity of dual-target calibration parameters, comprising: S1. Obtain left and right images after epipolar correction; S2. Calculate the pixel difference of the matching points in the y direction using the optical flow estimation algorithm on the left and right images after epipolar correction; S3. Eliminate abnormal matching points and select normal matching points that meet the accuracy requirements; S4. Perform statistical analysis on the pixel differences of the screened normal matching points in the y direction, calculate the parameter validity coefficient, and evaluate the validity of the binocular positioning parameters in the current state through the parameter validity coefficient.

[0029] S5. For principal point deviations caused by horizontal translation alone, the principal point deviation is evaluated by periodically repeating measurements of fixed markers in the world coordinate system and monitoring the point fluctuations of the conversion of the measured coordinates of the fixed markers to the world coordinate system.

[0030] Regarding the effectiveness detection of dual-target calibration parameters, the general method measures the effectiveness of calibration parameters by statistically analyzing the epipolar offset error of matching feature points. The calculation expression is as follows:

[0031] in, , yes In the collection Neighbors in yes In the collection The nearest neighbors in The smaller the value, the higher the effectiveness.

[0032] These methods usually rely on significant features of local areas of the image (such as corners or edges) when calculating matching points. These features are difficult to extract or distinguish for areas with weak textures or no textures (such as smooth surfaces or repeated textures). At the same time, the number of feature points extracted by such methods is very sparse, and the number is even smaller after matching and screening, resulting in insufficient data for measuring the effectiveness of parameters, which is prone to large numerical fluctuations and cannot reliably measure the effectiveness of parameters.

[0033] Therefore, the embodiment of the present invention periodically captures the left and right images after epipolar correction to ensure good image quality. The optical flow estimation algorithm is used on the left and right images after epipolar correction to calculate the pixel difference of the matching points in the y direction. The optical flow estimation algorithm is selected in the embodiment of the present invention because it can obtain denser feature matching. In addition, the optical flow estimation algorithm combined with deep learning technology (such as the RAFT algorithm) can overcome the problem of difficult feature point extraction in weak texture and textureless scenes.

[0034] Optical flow is the velocity vector of brightness change caused by the relative motion between an object and an observer. The vector pointing from the point before the motion to the point after the motion is the optical flow, which reflects the motion of pixels in the image over time. Traditional optical flow algorithms are based on the assumption of constant light intensity, that is, when an object moves, the light intensity of its pixels remains unchanged. This assumption is the basis of the optical flow estimation algorithm. Optical flow estimation algorithms are generally divided into sparse optical flow algorithms and dense optical flow algorithms. Sparse optical flow algorithms track corners, edges, and pixels with obvious features. This method usually selects some key points in the image for optical flow estimation, and the amount of calculation is relatively small. The dense optical flow algorithm estimates the optical flow of all pixels in the image. This method can obtain the optical flow field of the entire image, but the amount of calculation is large. The RAFT (Recurrent Appearance Flow) algorithm is an efficient optical flow estimation algorithm proposed in recent years. The RAFT algorithm is an optical flow estimation framework based on recursive neural networks. It recursively updates the optical flow frame by frame and achieves accurate pixel-level matching by iteratively updating the correspondence. This design enables the model to gradually refine and correct the prediction results, thereby improving accuracy. In order to handle motions of different scales, the RAFT algorithm introduces a multi-layer feature pyramid structure. This structure can capture motion information of different sizes at different resolutions, enhancing the robustness of the model. By merging previous optical flow estimates at each iteration, the model can effectively utilize historical information and enhance the understanding of long-term dependencies. This helps to improve the accuracy and stability of optical flow estimation. The entire RAFT model can be trained as a complete end-to-end system, which allows all components to be optimized to maximize overall performance. The RAFT algorithm outperforms other modern optical flow estimation methods on multiple benchmark test sets, achieving high-precision optical flow estimation. Despite its powerful capabilities, the RAFT model is of moderate size, suitable for deployment on resource-limited devices, and easy to use.

[0035] In one possible implementation, when the optical flow estimation algorithm is used to calculate the pixel difference of the matching points in the y direction on the left and right images after epipolar correction in step S2, a two-stage optical flow estimation algorithm based on deep learning is used to calculate the pixel difference of the matching points in the y direction for each pair of matching points. , calculate the deviation of the matching point in the y direction .

[0036] Specifically, step S2 of the embodiment of the present invention includes the following steps: Based on the brightness consistency assumption, the following optical flow constraint equation expression is constructed:

[0037] In the formula, express Moment image at The pixel intensity at is the optical flow vector, which represents the movement of the pixel; Taylor expansion of the optical flow constraint equation and ignoring high-order terms yields the following expression:

[0038] The sparse optical flow method is used to perform the least squares solution of the optical flow constraint equation in the local window; The two-stage optical flow estimation algorithm based on deep learning described in the embodiment of the present invention adopts the RAFT algorithm, extracts features through a convolutional neural network CNN, and constructs a dense full-pixel pair correlation tensor according to the following formula:

[0039] In the formula, and Represents the pixels in the source image and the target image respectively and pixels The characteristic vector of is the correlation score between the feature vectors; then, the RAFT algorithm iteratively updates the optical flow field through a recurrent neural network, each time updating the optical flow vector according to the following expression :

[0040] The updated optical flow vector Get the left view pixel Corresponding right view pixel .

[0041] CNN (Convolutional Neural Network) is a deep learning algorithm model that is widely used in computer vision and image processing. Its core idea is to extract image features layer by layer through convolution operations, thereby achieving image classification, recognition and other tasks. The basic structure of CNN usually includes the following layers: Input layer: used to receive original image data.

[0042] Convolutional layer: extracts local features from the image through convolution operations. Convolutional layers are usually composed of multiple convolution kernels (also called filters), each of which performs a convolution operation with the input image to generate a feature map. These feature maps are then nonlinearly transformed through activation functions (such as ReLU) to enhance the expressive power of the model.

[0043] Pooling layer: used to downsample the feature maps output by the convolutional layer to reduce the dimension of the data and the amount of computation. Common pooling operations include Max Pooling and Average Pooling.

[0044] Fully connected layer: Usually located in the last few layers of the network, it is used to receive the features of the previous layers and output the final classification results. Each neuron in the fully connected layer is connected to all neurons in the previous layer, so it usually has a large number of parameters.

[0045] Output layer: used to output the final classification results or regression values. For classification tasks, the output layer usually uses a softmax function to convert the output value into a probability distribution.

[0046] In a possible implementation, in the matching results, due to noise, occlusion or algorithm error, there may be abnormal matching points, which affect the accuracy of subsequent processing. In step S3 of this embodiment, abnormal points are eliminated by two methods: threshold screening and consistency detection; the method of eliminating abnormal points by threshold screening includes: based on the colinear geometric characteristics of the left and right matching points after polar line correction, a threshold is set to eliminate matching points whose y-direction differences exceed the threshold to control the error within a reasonable range; the method of eliminating abnormal points by consistency detection includes: using the left and right consistency detection algorithm to verify whether the results of forward matching and reverse matching are consistent, and to eliminate inconsistent matching points (such as points in the occluded area or wrong matches). Through these processes, a normal matching point data set that meets the accuracy requirements can be retained, reducing the impact of abnormal points on subsequent calculations.

[0047] In a possible implementation manner, step S4 performs statistical analysis on the pixel differences of the screened normal matching points in the y direction, and the step of calculating the parameter validity coefficient includes: Calculate all normal matching points Mean pixel difference in y direction:

[0048] In the formula, Indicates Measurements, Indicates For the ordinate of the matching point in the right view, Indicates The ordinate of the matching point in the left view; The parameter validity coefficient is the ideal mean of the pixel differences of the matching points in the y direction. , all normal matching points calculated The mean change rate of pixel differences in the y direction , the calculation expression is as follows:

[0049] When evaluating the effectiveness of the dual-target calibration parameters in the current state through the parameter effectiveness coefficient, The smaller it is, the closer the actual measurement result is to the ideal calibration, and the higher the calibration accuracy and stability.

[0050] In a possible implementation, when evaluating the principal point deviation in step S5, the measured coordinates of the fixed marker are converted to the world coordinate system according to the following expression:

[0051] In the formula, represents the measured coordinates of a fixed marker, The image pixel coordinates representing the fixed landmark, represents the pixel coordinates of the principal point, represents the baseline distance of the binocular camera, The disparity of image pixels representing fixed landmarks, Represents the intrinsic focal length of the binocular camera; Set the world coordinates of a fixed marker in multiple periodic repeated measurements to , where Indicates The volatility measure is expressed by calculating the overall volatility of the corresponding measurement values ​​as follows:

[0052] In the formula, is the standard deviation of multiple measurement fluctuations on the three coordinate axes; The error caused by the principal point deviation is determined by measuring the volatility; if the volatility measurement exceeds the set range, it is determined that the position of the fixed marker has changed abnormally and a principal point deviation has occurred.

[0053] The embodiment of the present invention proposes a corresponding verification method for the problem of principal point deviation caused only by horizontal translation, which can effectively calculate and correct the principal point deviation caused by translation error, and expands the application scenarios of parameter validity verification.

[0054] In order to verify the effectiveness of the method of the present invention in different problem scenarios, the embodiment conducted multiple groups of simulation experiments to compare the performance of the method of the present invention under the problem of calibration algorithm accuracy and the problem of calibration parameter failure caused by camera looseness.

[0055] Experimental data: (1) Calibration algorithm accuracy issue As shown in Table 1, the embodiment of the present invention experimentally compares the performance of calibration algorithms of different precisions in the rate of change index. The second column of data represents the calibration results obtained by the checkerboard calibration algorithm (high-precision algorithm), and the third column of data represents the results obtained by the disc calibration algorithm (low-precision algorithm). According to the detection standard of the method of the embodiment of the present invention, if the rate of change is greater than 1, the calibration parameters are not effective and need to be recalibrated. The experimental data show that the rate of change obtained by the low-precision calibration algorithm (disc calibration) mostly exceeds 1, proving that its effect is not as good as the high-precision calibration algorithm (checkerboard calibration).

[0056] Table 1

[0057] (2) Calibration parameter failure caused by simulated camera looseness and temperature changes In practical applications, the camera may be loose, shifted or have temperature changes, which may affect the validity of the calibration parameters. The embodiments of the present invention conduct simulation experiments on these factors respectively: (2.1) Camera looseness and displacement By performing a translation operation on the original image as a whole, the physical displacement of the camera is simulated to detect the rate of change of the parameters after the physical position of the camera changes. As shown in Table 2, the experimental data show that the rate of change after the displacement operation is at least 3 times higher, indicating that when the camera is loose, the method of the present invention can successfully identify significant changes in the calibration parameters, indicating that the calibration may have failed.

[0058] Table 2

[0059] (2.2) Distortion failure caused by temperature change Temperature changes may cause slight deformation of the camera lens, thereby affecting the distortion coefficient of the image. In order to simulate the effect of temperature changes on the calibration parameters, the image was subjected to slight nonlinear distortion processing in the experiment of the embodiment of the present invention to observe its effect on the calibration parameters. As shown in Table 3, the results show that the rate of change after distortion processing increases significantly, indicating that the method of the embodiment of the present invention can detect the risk of calibration parameter failure caused by temperature changes.

[0060] Table 3

[0061] Another embodiment of the present invention periodically captures left and right images under the condition of camera looseness simulation, calculates the pixel difference of the matching point in the y direction, monitors the fluctuation of the difference, and detects the influence of camera looseness on the calibration parameters. The experimental steps are as follows: 1) Set up experimental conditions: A standard stereo camera was selected and the image resolution was set to 1920×1080.

[0062] Simulate camera looseness: simulate looseness by moving the right camera up and down, left and right (maximum displacement ±5mm) and rotating (maximum rotation angle ±2°).

[0063] 2) Image acquisition: The left and right image pairs were captured regularly, with the acquisition interval set to once every 30 seconds, and the image pairs were repeatedly acquired 10 times.

[0064] 3) Optical flow estimation algorithm processing: Use optical flow algorithms such as RAFT to perform dense point matching on binocular images and obtain the corresponding matching relationship of each pixel.

[0065] The pixel difference calculation range is set to [0,10] pixels, and outliers with a difference greater than 10 pixels are removed.

[0066] 4) Statistics and monitoring: Count the matching points and calculate the mean of the pixel differences in the y direction ( ) and the rate of change ( ).

[0067] The embodiment of the present invention monitors the fluctuation of the pixel difference in the y direction. If the change rate exceeds 2.0, it is determined that the looseness of the camera has a significant impact on the calibration parameters, the calibration parameters are invalid, and recalibration is required.

[0068] See also Figure 2 The embodiment of the present invention further provides a dual-target calibration parameter validity online verification system, comprising: A correction image acquisition module, used to obtain left and right images after epipolar correction; The optical flow estimation module is used to calculate the pixel difference of the matching points in the y direction using the optical flow estimation algorithm on the left and right images after epipolar correction; The matching point screening module is used to remove abnormal matching points and screen out normal matching points that meet the accuracy requirements; The parameter validity evaluation module is used to perform statistical analysis on the pixel differences of the screened normal matching points in the y direction, calculate the parameter validity coefficient, and evaluate the validity of the binocular positioning parameters in the current state through the parameter validity coefficient.

[0069] In a possible implementation, the system of the embodiment of the present invention also includes a principal point deviation evaluation module, which is used to evaluate the principal point deviation caused by only horizontal translation by periodically repeating measurements of fixed markers in the world coordinate system and monitoring the point fluctuation of converting the measured coordinates of the fixed markers into the world coordinate system.

[0070] An embodiment of the present invention further provides an electronic device, including: A memory storing at least one instruction; and a processor executing the instruction stored in the memory to implement the online verification method for the validity of dual-target calibration parameters.

[0071] An embodiment of the present invention further provides a computer-readable storage medium, wherein at least one instruction is stored in the computer-readable storage medium, and the at least one instruction is executed by a processor in an electronic device to implement the online verification method for the validity of dual-target calibration parameters.

[0072] Exemplarily, the instructions stored in the memory may be divided into one or more modules / units, which are stored in a computer-readable storage medium and executed by the processor to complete the online verification method for the validity of dual-target calibration parameters of the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the server.

[0073] The electronic device may be a computing device such as a smart phone, a notebook, a PDA, and a cloud server. The electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the electronic device may also include more or fewer components, or a combination of certain components, or different components, for example, the electronic device may also include an input / output device, a network access device, a bus, etc.

[0074] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0075] The memory may be an internal storage unit of the server, such as a hard disk or memory of the server. The memory may also be an external storage device of the server, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the server. Furthermore, the memory may include both an internal storage unit of the server and an external storage device. The memory is used to store the computer-readable instructions and other programs and data required by the server. The memory may also be used to temporarily store data that has been output or is to be output.

[0076] It should be noted that the information interaction, execution process and other contents between the above-mentioned module units are based on the same concept as the method embodiment. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0077] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0078] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0079] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0080] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for online verification of the validity of dual-target calibration parameters, characterized in that: include: Get the left and right images after epipolar correction; The optical flow estimation algorithm is used on the left and right images after epipolar correction to calculate the pixel difference of the matching points in the y direction; Eliminate abnormal matching points and select normal matching points that meet the accuracy requirements; The pixel differences of the screened normal matching points in the y direction are statistically analyzed to calculate the parameter validity coefficient, and the validity of the binocular positioning parameters in the current state is evaluated by the parameter validity coefficient.

2. The method for online verification of the validity of dual-target calibration parameters according to claim 1 is characterized in that: Also includes: For principal point deviations caused by horizontal translation alone, the principal point deviation is evaluated by periodically repeating measurements of fixed markers in the world coordinate system and monitoring the point fluctuations of the conversion of the measured coordinates of the fixed markers to the world coordinate system.

3. The method for online verification of the validity of dual-target calibration parameters according to claim 2 is characterized in that: The measured coordinates of the fixed marker are converted to the world coordinate system according to the following expression: In the formula, represents the measured coordinates of a fixed marker, The image pixel coordinates representing the fixed landmark, represents the pixel coordinates of the principal point, represents the baseline distance of the binocular camera, The disparity of image pixels representing fixed landmarks, Represents the intrinsic focal length of the binocular camera; Set the world coordinates of a fixed marker in multiple periodic repeated measurements to , where Indicates The volatility measure is expressed by calculating the overall volatility of the corresponding measurement values ​​as follows: In the formula, is the standard deviation of multiple measurement fluctuations on the three coordinate axes; The error caused by the principal point deviation is determined by measuring the volatility; if the volatility measurement exceeds the set range, it is determined that the position of the fixed marker has changed abnormally and a principal point deviation has occurred.

4. The method for online verification of the validity of dual-target calibration parameters according to claim 1 is characterized in that: When using the optical flow estimation algorithm to calculate the pixel difference of the matching points in the y direction on the left and right images after epipolar correction, a two-stage optical flow estimation algorithm based on deep learning is used to calculate the pixel difference of the matching points in the y direction for each pair of matching points. , calculate the deviation of the matching point in the y direction .

5. The method for online verification of the validity of dual-target calibration parameters according to claim 4 is characterized in that: The method of calculating the pixel difference of the matching points in the y direction using the optical flow estimation algorithm on the left and right images after epipolar correction comprises the following steps: Based on the brightness consistency assumption, the following optical flow constraint equation expression is constructed: In the formula, express Moment image at The pixel intensity at is the optical flow vector, which represents the movement of the pixel; Taylor expansion of the optical flow constraint equation and ignoring high-order terms yields the following expression: The sparse optical flow method is used to perform the least squares solution of the optical flow constraint equation in the local window; The two-stage optical flow estimation algorithm based on deep learning adopts the RAFT algorithm, extracts features through the convolutional neural network CNN, and constructs a dense full-pixel pair correlation tensor as follows: In the formula, and Represents the pixels in the source image and the target image respectively and pixels The characteristic vector of is the correlation score between the feature vectors; then, the RAFT algorithm iteratively updates the optical flow field through a recurrent neural network, each time updating the optical flow vector according to the following expression : The updated optical flow vector Get the left view pixel Corresponding right view pixel .

6. The method for online verification of the validity of dual-target calibration parameters according to claim 1 is characterized in that: Outliers are removed through threshold screening and consistency detection; The method of eliminating abnormal points by threshold screening includes: based on the collinear geometric characteristics of the left and right matching points after epipolar correction, setting a threshold to eliminate matching points whose y-direction differences exceed the threshold, so as to control the error within a reasonable range; The method of eliminating abnormal points through consistency detection includes: using a left-right consistency detection algorithm to verify whether the results of forward matching and reverse matching are consistent, and eliminating inconsistent matching points.

7. The method for online verification of the validity of dual-target calibration parameters according to claim 1 is characterized in that: The statistical analysis of the pixel differences of the screened normal matching points in the y direction is performed to calculate the parameter validity coefficient, and the validity of the binocular positioning parameters in the current state is evaluated by the parameter validity coefficient, including: Calculate all normal matching points Mean pixel difference in y direction: In the formula, Indicates Measurements, Indicates For the ordinate of the matching point in the right view, Indicates The ordinate of the matching point in the left view; The parameter validity coefficient is the ideal mean value of the pixel difference of the matching points in the y direction. , all normal matching points calculated The mean change rate of pixel differences in the y direction , the calculation expression is as follows: The smaller it is, the closer the actual measurement result is to the ideal calibration, and the higher the calibration accuracy and stability.

8. A dual-target calibration parameter validity online verification system, characterized in that: include: A correction image acquisition module, used to obtain left and right images after epipolar correction; The optical flow estimation module is used to calculate the pixel difference of the matching points in the y direction using the optical flow estimation algorithm on the left and right images after epipolar correction; The matching point screening module is used to remove abnormal matching points and screen out normal matching points that meet the accuracy requirements; The parameter validity evaluation module is used to perform statistical analysis on the pixel differences of the screened normal matching points in the y direction, calculate the parameter validity coefficient, and evaluate the validity of the binocular positioning parameters in the current state through the parameter validity coefficient.

9. An electronic device, characterized in that: include: A memory storing at least one instruction; and A processor executes instructions stored in the memory to implement the online verification method for the validity of dual-target positioning parameters as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the online verification method for the validity of dual-target positioning parameters as described in any one of claims 1 to 7.

Images