Intelligent automatic deviation rectifying method for equipment

Abstract

The invention discloses an intelligent automatic deviation correction method for equipment, which belongs to the field of image processing, adopts an automatic deviation correction scheme based on multiple sensors, preferentially performs automatic deviation correction based on image data through a deep learning technology, and when an image cannot be effectively subjected to feature point extraction and matching, the automatic deviation correction is performed. And when the image data cannot be automatically corrected due to the fact that the image data cannot be automatically corrected, an automatic correction scheme based on the point cloud data is adopted to achieve the purpose of automatic correction, so that the stability and reliability of the whole correction system are ensured, and meanwhile, the automatic correction efficiency of the whole correction system is greatly improved. Besides, the fixed target in the scene is extracted based on deep learning, the feature point extraction and matching speed can be greatly improved, meanwhile, the influence of other variable scenes on feature point detection and matching in the image is avoided, and the deviation correction speed and accuracy are improved.

Description

Technical Field

[0001] The invention relates to the field of image processing, and in particular to an intelligent automatic deviation correction method for equipment. Background Art

[0002] When smart sensor equipment (such as various sensors in the power field) is fixedly installed on a designated carrier, the fixing device ages and its seismic resistance deteriorates during use, resulting in insufficient ability of the equipment to withstand dynamic loads and impact loads after installation, causing equipment offset and tilt, etc., seriously affecting the accuracy of the smart sensor equipment in obtaining the required data, so the equipment needs to be corrected.

[0003] The existing equipment correction mostly relies on manual reinstallation and fixation. Some of them can be based on images to compare the differences between images before and after to assist in correction, but there are problems such as poor scene adaptability, poor robustness and inaccurate correction. Summary of the invention

[0004] The present invention provides an intelligent automatic deviation correction method for equipment, which ensures the stability and reliability of the deviation correction and improves the efficiency of the deviation correction.

[0005] The present invention provides the following technical solutions:

[0006] A method for intelligent automatic deviation correction of a device, wherein the device is fixedly mounted on a pan / tilt platform and includes a camera, and the method includes:

[0007] S10: After receiving the correction instruction, real-time image data of the scene is acquired through the camera;

[0008] S20: Performing fixed target detection on the real-time image data based on deep learning technology to extract at least one fixed target in the scene;

[0009] S30: extracting and matching corresponding feature point pairs between the extracted fixed target and the fixed target preset in the pre-stored reference image data based on deep learning technology;

[0010] S40: When the matching is successful, the space vector of the matched corresponding feature point pair is calculated, and the offset angle is calculated;

[0011] S50: Determine whether the deviation angle is greater than a preset limit value. If so, the gimbal automatically corrects the deviation according to the deviation angle. Otherwise, no correction is performed.

[0012] Furthermore, when the number of matched corresponding feature point pairs is more than 3 pairs, and the matching confidence of each pair of corresponding feature point pairs is greater than 90%, the matching is determined to be successful, otherwise, the matching is determined to be unsuccessful.

[0013] Further, the S30 includes:

[0014] S31: extracting local features of the fixed target through a convolution layer and an attention layer;

[0015] S32: Perform coarse matching pixel by pixel at a large scale, and then perform iterative fine-tuning at a small scale to perform precise matching.

[0016] Furthermore, the device further includes a point cloud acquisition device, and the method further includes:

[0017] S41: when the matching fails, acquiring real-time point cloud data of the scene through the point cloud acquisition device;

[0018] S42: performing point cloud matching on the real-time point cloud data and pre-stored reference point cloud data;

[0019] S43: When the point cloud is matched successfully, the offset matrix between the real-time point cloud data and the reference point cloud data is calculated, the offset angle is calculated according to the offset matrix, and the process goes to S50.

[0020] Furthermore, the point cloud acquisition device includes a depth camera and a laser radar, the laser radar scans to obtain the real-time point cloud data, the depth camera captures a depth image, and the real-time point cloud data is calculated based on the depth image.

[0021] Further, the S42 includes:

[0022] S421: Setting a maximum distance threshold;

[0023] S422: traverse all points in the real-time point cloud data, calculate the distance to each point in the reference point cloud data, and when the distance is less than the maximum distance threshold, consider it as a corresponding point pair;

[0024] S423: When the number of all corresponding point pairs is higher than 80% of the total number of point clouds of the reference point cloud data, it is determined that the point cloud matching is successful.

[0025] Further, the S40 includes:

[0026] S401: Query corresponding depth information according to the matched corresponding feature point pairs;

[0027] S402: Calculating the coordinates of the corresponding feature point pair in the camera coordinate system according to the depth information;

[0028]

[0029]

[0030]

[0031]

[0032] Among them, (u sd ,v sd ) and (x sd ,y sd ) is the point P of the corresponding feature point pair on the reference image data i The coordinates in the image coordinate system and the camera coordinate system, (u rt ,v rt ) and (x rt ,y rt ) is the point Q of the corresponding feature point pair on the real-time image data i The coordinates in the image coordinate system and the camera coordinate system, z sd and z rt are points P of the corresponding feature point pair on the reference image data i and point Q on the real-time image data i , f is the focal length of the camera, u0 and v0 represent half the width and half the height of the reference image data or the real-time image data, respectively;

[0033] S403: Express a space vector corresponding to a feature point pair as

[0034] S404: Calculating the offset angle θ according to the spatial vectors of the plurality of corresponding feature point pairs;

[0035]

[0036] Where n is the number of matching corresponding feature point pairs.

[0037] Further, the S43 includes:

[0038] S431: Initialize the offset matrix R and set the loss function f(R);

[0039]

[0040]

[0041] Wherein, k=1,2,…,N, j=1,2,…,N, N is the total number of point clouds of the real-time point cloud data or the reference point cloud data, p sk represents the kth point of the real-time point cloud data, p tj represents the j-th point of the reference point cloud data;

[0042] S432: solving the offset matrix by minimizing the loss function and performing iterative calculations;

[0043] S433: Calculate a horizontal offset angle and a vertical offset angle according to the solved offset matrix;

[0044] α=atan(m 13 , m 33 )

[0045] β=arcsin(-m 23 )

[0046] Wherein, α and β are the horizontal offset angle and the vertical offset angle respectively.

[0047] The present invention has the following beneficial effects:

[0048] The matching method in the prior art directly extracts and matches feature points of the entire image, which has an extremely slow processing speed and requires scaling the entire image into a small-sized image for feature point extraction and matching, which results in poor feature point extraction and matching effects due to the loss of some image details. The present invention uses deep learning to extract fixed targets in the scene, which can greatly improve the speed of feature point extraction and matching, while also avoiding the influence of other variable scenes on feature point detection and matching in the image, thereby improving the speed and accuracy of correction.

[0049] The present invention adopts an automatic deviation correction scheme based on multiple sensors, and preferentially performs automatic deviation correction based on image data through deep learning technology. When the image cannot effectively perform feature point extraction and matching, resulting in the inability to achieve automatic deviation correction based on image data, an automatic deviation correction scheme based on point cloud data is adopted to achieve the purpose of automatic deviation correction, thereby ensuring the stability and reliability of the entire deviation correction system, and greatly improving the automatic deviation correction efficiency of the entire deviation correction system. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of the intelligent automatic deviation correction method of the equipment of the present invention;

[0051] Figure 2 A schematic diagram of each module of the device. DETAILED DESCRIPTION

[0052] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0053] The present invention provides an intelligent automatic deviation correction method for equipment, such as Figure 2 As shown, the device 1 is fixedly mounted on the gimbal, and the device includes a camera 2, such as Figure 1 As shown, the method includes:

[0054] S10: After receiving the correction instruction, the real-time image data of the scene is acquired through the camera, and the automatic correction processing flow based on the image data is firstly adopted.

[0055] S20: Perform fixed target detection on the real-time image data based on deep learning technology to extract at least one fixed target in the scene.

[0056] In general, fixed targets are mostly buildings, power equipment, etc. The extracted fixed targets can be one or multiple. The purpose of extracting multiple fixed targets is to ensure that the subsequent processing flow can be carried out even if a fixed target is not effectively extracted, thereby improving the applicability of the entire system.

[0057] S30: Extracting and matching corresponding feature point pairs between the extracted fixed target and the fixed target preset in the pre-stored reference image data based on deep learning technology.

[0058] The specific process of extraction matching can be: first extract the local features of the fixed target through the convolution layer and the attention layer; then perform coarse matching pixel by pixel at the large-scale level, and then continuously perform iterative fine-tuning at the small-scale level to perform precise matching. It can generate high-quality matching results between images of fixed target areas with weak textures and blurred pixels.

[0059] When the number of successfully matched corresponding feature point pairs is more than 3, and the matching confidence of each pair of corresponding feature point pairs is greater than 90%, the match is considered successful. When the number of successfully matched corresponding feature point pairs is less than 3, in order to ensure the robustness of the system, the match is considered failed.

[0060] S40: When the matching is successful, the space vectors of the matched corresponding feature point pairs are calculated, and the offset angle is calculated.

[0061] S50: Determine whether the deviation angle is greater than a preset limit value. If so, the gimbal automatically corrects the deviation according to the deviation angle. Otherwise, no correction is made and the system enters a state of waiting for the next correction instruction.

[0062] The matching method in the prior art directly extracts and matches feature points of the entire image, which has an extremely slow processing speed and requires scaling the entire image into a small-sized image for feature point extraction and matching, which results in poor feature point extraction and matching effects due to the loss of some image details. The present invention uses deep learning to extract fixed targets in the scene, which can greatly improve the speed of feature point extraction and matching, while also avoiding the influence of other variable scenes on feature point detection and matching in the image, thereby improving the speed and accuracy of correction.

[0063] As an improvement of the embodiment of the present invention, Figure 2As shown, the device 1 may also include a point cloud acquisition device 3, which may be a depth camera and a laser radar, or other devices. When the aforementioned image matching fails, an automatic correction process based on point cloud data is adopted. Accordingly, the method of the present invention also includes:

[0064] S41: When the matching fails, real-time point cloud data of the scene is acquired through a point cloud acquisition device.

[0065] When the point cloud acquisition device is a laser radar, the laser radar scans to obtain real-time point cloud data. When the point cloud acquisition device is a depth camera, the depth camera captures a depth image and calculates the real-time point cloud data based on the depth image.

[0066] S42: performing point cloud matching between the real-time point cloud data and the pre-stored reference point cloud data.

[0067] One implementation of this step includes:

[0068] S421: Manually set the maximum distance threshold.

[0069] S422: Traverse all points in the real-time point cloud data, calculate the distance to each point in the reference point cloud data, and when the distance is less than the maximum distance threshold, it is considered to be a corresponding point pair.

[0070] The present invention manually sets a maximum distance threshold for corresponding point pairs, and when the distance between points is less than the maximum distance threshold, the points are confirmed to be corresponding point pairs, which can greatly reduce the calculation complexity.

[0071] S423: When the number of all corresponding point pairs is higher than 80% of the total number of point clouds of the reference point cloud data, the point cloud matching is determined to be successful and S43 is executed; otherwise, it is determined to be a matching failure and a manual verification process is prompted.

[0072] S43: When the point cloud is successfully matched, the offset matrix between the real-time point cloud data and the reference point cloud data is calculated, and the offset angle is calculated according to the offset matrix. Then go to S50 to determine whether the offset angle is greater than the preset limit value. If so, the gimbal automatically corrects the deviation according to the offset angle. Otherwise, no correction is made, and the system enters the state of waiting for the next correction instruction.

[0073] As an improvement of the embodiment of the present invention, the method of calculating the space vector and the offset angle of the corresponding feature point pair in S40 includes:

[0074] S401: Query corresponding depth information according to the matched corresponding feature point pairs.

[0075] S402: Calculate coordinates of corresponding feature point pairs in a camera coordinate system according to the depth information.

[0076]

[0077]

[0078]

[0079]

[0080] Among them, (u sd ,v sd ) and (x sd ,y sd ) is the point P corresponding to the feature point pair on the reference image data i The coordinates in the image coordinate system and the camera coordinate system, (u rt ,v rt ) and (x rt ,y rt ) is the point Q corresponding to the feature point pair on the real-time image data i The coordinates in the image coordinate system and the camera coordinate system, z sd and z rt are the points P corresponding to the feature point pairs on the reference image data i and point Q on the real-time image data i f is the focal length of the camera, u0 and v0 represent half the width and half the height of the reference image data or the real-time image data, respectively.

[0081] S403: Express a space vector corresponding to a feature point pair as

[0082] S404: Calculate the offset angle θ according to the spatial vectors of the plurality of corresponding feature point pairs.

[0083]

[0084] Where n is the number of matching corresponding feature point pairs.

[0085] As another improvement of the embodiment of the present invention, the method of calculating the offset matrix and the offset angle in S43 includes:

[0086] S431: Initialize the offset matrix R and set the loss function f(R).

[0087]

[0088]

[0089] Where k = 1, 2, ..., N, j = 1, 2, ..., N, N is the total number of point clouds of real-time point cloud data or benchmark point cloud data, psk represents the kth point of the real-time point cloud data, p t j Represents the jth point of the reference point cloud data.

[0090] S432: By minimizing the loss function and continuously performing iterative calculations, the optimal offset matrix is ​​solved.

[0091] S433: Calculate the horizontal offset angle and the vertical offset angle according to the solved offset matrix.

[0092] α=atan(m 13 , m 33 )

[0093] β=arcsin(-m 23 )

[0094] Among them, α and β are the horizontal offset angle and the vertical offset angle respectively.

[0095] In order to solve the problem that the fixing devices of intelligent sensor equipment are aged and their seismic resistance deteriorates during use, resulting in insufficient ability of the equipment to withstand dynamic loads and impact loads after installation, causing equipment deviation and tilt, the present invention adopts an automatic correction solution based on multiple sensors (including but not limited to gimbal cameras, gimbal depth cameras, gimbal lidar, etc.), and automatically corrects the image data based on deep learning technology first. When the image cannot effectively extract and match feature points, resulting in the inability to achieve automatic correction based on image data, an automatic correction solution based on point cloud data is adopted to achieve the purpose of automatic correction, thereby ensuring the stability and reliability of the entire correction system and greatly improving the efficiency of automatic correction of the entire correction system.

[0096] Compared with the prior art that uses additional angle sensors and displacement sensors to calculate the displacement or jitter of the device at different angles, the present invention can directly calculate the displacement information through the most basic image and point cloud data of the device, without the need for additional sensors, thereby greatly reducing the cost of the product.

[0097] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for intelligent automatic deviation correction of equipment, characterized in that: The device is fixedly mounted on a gimbal, the device includes a camera, and the method includes: S10: After receiving the correction instruction, real-time image data of the scene is acquired through the camera; S20: Performing fixed target detection on the real-time image data based on deep learning technology to extract at least one fixed target in the scene; S30: extracting and matching corresponding feature point pairs between the extracted fixed target and the fixed target preset in the pre-stored reference image data based on deep learning technology; S40: When the matching is successful, the space vector of the matched corresponding feature point pair is calculated, and the offset angle is calculated; S50: Determine whether the deviation angle is greater than a preset limit value. If so, the gimbal automatically corrects the deviation according to the deviation angle. Otherwise, no correction is performed.

2. The intelligent automatic deviation correction method of equipment according to claim 1, characterized in that: When the number of matched corresponding feature point pairs is more than 3 pairs, and the matching confidence of each pair of corresponding feature point pairs is greater than 90%, the match is determined to be successful, otherwise, the match is determined to be unsuccessful.

3. The intelligent automatic deviation correction method of equipment according to claim 1, characterized in that: The S30 includes: S31: extracting local features of the fixed target through a convolution layer and an attention layer; S32: Perform coarse matching pixel by pixel at a large scale, and then perform iterative fine-tuning at a small scale to perform precise matching.

4. The intelligent automatic deviation correction method for equipment according to any one of claims 1 to 3, characterized in that: The device further includes a point cloud acquisition device, and the method further includes: S41: when the matching fails, acquiring real-time point cloud data of the scene through the point cloud acquisition device; S42: performing point cloud matching on the real-time point cloud data and pre-stored reference point cloud data; S43: When the point cloud is matched successfully, the offset matrix between the real-time point cloud data and the reference point cloud data is calculated, the offset angle is calculated according to the offset matrix, and the process goes to S50.

5. The intelligent automatic deviation correction method of equipment according to claim 4, characterized in that: The point cloud acquisition device includes a depth camera and a laser radar. The laser radar scans to obtain the real-time point cloud data. The depth camera captures a depth image, and the real-time point cloud data is calculated based on the depth image.

6. The intelligent automatic deviation correction method of equipment according to claim 4, characterized in that: The S42 includes: S421: Setting a maximum distance threshold; S422: traverse all points in the real-time point cloud data, calculate the distance to each point in the reference point cloud data, and when the distance is less than the maximum distance threshold, consider it as a corresponding point pair; S423: When the number of all corresponding point pairs is higher than 80% of the total number of point clouds of the reference point cloud data, it is determined that the point cloud matching is successful.

7. The intelligent automatic deviation correction method of equipment according to claim 4, characterized in that: The S40 includes: S401: Query corresponding depth information according to the matched corresponding feature point pairs; S402: Calculating the coordinates of the corresponding feature point pair in the camera coordinate system according to the depth information; Among them, (u sd ,v sd ) and (x sd ,y sd ) is the point P of the corresponding feature point pair on the reference image data i The coordinates in the image coordinate system and the camera coordinate system, (u rt ,v rt ) and (x rt ,y rt ) is the point Q of the corresponding feature point pair on the real-time image data i The coordinates in the image coordinate system and the camera coordinate system, z sd and z rt are points P of the corresponding feature point pair on the reference image data i and point Q on the real-time image data i , f is the focal length of the camera, u0 and v0 represent half the width and half the height of the reference image data or the real-time image data, respectively; S403: Express a space vector corresponding to a feature point pair as S404: Calculating the offset angle θ according to the spatial vectors of the plurality of corresponding feature point pairs; Where n is the number of matching corresponding feature point pairs.

8. The intelligent automatic deviation correction method for equipment according to claim 4, characterized in that: The S43 includes: S431: Initialize the offset matrix R and set the loss function f(R); Wherein, k=1,2,…,N, j=1,2,…,N, N is the total number of point clouds of the real-time point cloud data or the reference point cloud data, p sk represents the kth point of the real-time point cloud data, p tj represents the j-th point of the reference point cloud data; S432: solving the offset matrix by minimizing the loss function and performing iterative calculations; S433: Calculate a horizontal offset angle and a vertical offset angle according to the solved offset matrix; α=atan(m 13 ,m 33 ) β=arcsin(-m 23 ) Wherein, α and β are the horizontal offset angle and the vertical offset angle respectively.

Images