Binocular camera self-calibration method and device based on feature point structure and storage medium
Through the binocular camera self-calibration method based on feature point structure, the multi-branch classification network is used to classify feature points, and the problem of low self-calibration accuracy of binocular cameras in the prior art is solved, and higher self-calibration accuracy and stability are achieved.
Patent Information
- Application Number
- CN202510220539.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-27
- Publication Date
- 2025-03-28
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, the effect of the binocular camera self-calibration algorithm is uncertain, resulting in low accuracy of the output result, mainly due to the failure to effectively utilize the structural information of the feature points.
A binocular camera self-calibration method based on feature point structure is adopted, feature points are extracted through feature point detection algorithm, and input them into the CPE module to obtain multiple tiles centered on feature points. These tiles are input to the multi-branch classification network, output the classification results of the feature point structure, and then perform self-calibration of the binocular camera, estimate internal and external parameters, and select the parameters with the highest accuracy through evaluation indicators.
The accuracy of the self-calibration results of the binocular camera is improved, feature points that are not conducive to the effect of the self-calibration algorithm are avoided, and the stability and accuracy of the algorithm are enhanced.
Smart Images
Figure CN119722816A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of assisted driving technology, and in particular to a binocular camera self-calibration method, device and storage medium based on a feature point structure. Background Art
[0002] In most complex intelligent driving scenarios, including road preview, there are generally a large number of unstructured feature points, as well as structured feature points in the shape of L, Y, star, and centroid. After the feature points are extracted, they are usually used indiscriminately as input to feature-based vision algorithms such as the intrinsic and extrinsic parameter self-calibration of binocular cameras. The quality of the extracted feature points is crucial to the effectiveness of these algorithms, and the feature point structure is an important factor affecting the quality of feature points. However, in practical applications, this factor has not been explored and applied to a large extent.
[0003] Extracting features from images is an important task in computer vision. For feature-based visual algorithms such as binocular camera self-calibration, the effect of the extracted features will directly affect the accuracy of the algorithm output results. Current feature extraction algorithms use the entire image as the search area and do not distinguish the structure of feature points. Although a large number of feature points can be extracted, in intelligent driving scenarios, the structure of feature points has a greater impact on the binocular camera self-calibration algorithm.
[0004] In view of this, the present invention is proposed. Summary of the invention
[0005] The main purpose of the present invention is to disclose a binocular camera self-calibration method, device and storage medium based on feature point structure, which are used to solve the problem of low output result accuracy caused by uncertain effect of self-calibration algorithm in the prior art.
[0006] To achieve the above object, according to one aspect of the present invention, a binocular camera self-calibration method based on feature point structure is provided, and the following technical scheme is adopted:
[0007] A binocular camera self-calibration method based on feature point structure comprises: performing feature point detection on the entire input image based on a feature point detection algorithm; inputting each detected feature point into a CPE module, obtaining multiple blocks centered on the feature point, and then using the blocks as inputs of a multi-branch classification network to output classification results of the feature point structure; performing self-calibration on the binocular camera based on the classification results, estimating the intrinsic parameters and extrinsic parameters of the binocular camera; calculating intrinsic parameter and extrinsic parameter evaluation indicators, obtaining the feature point category corresponding to the parameter with the highest accuracy by comparing the indicators, and selecting the group of parameters as the binocular camera self-calibration parameters.
[0008] Furthermore, the detected feature points are input into the CPE module, multiple blocks centered on the feature point are obtained, and these blocks are used as inputs of the multi-branch classification network to output the classification results of the feature point structure, including: for a given feature point , with feature points As the center, in the polar coordinate system, M radii are R={ }, capture the intensity distribution around the feature point within the circular surface range; using the polar angle and radius { , } to construct feature points The circular surface of which the polar angle ranges from ; Based on the polar angle and the set surface radius and formula (1), calculate Points in the neighborhood The coordinates of , formula (1) is;
[0009]
[0010] According to formula (1), traverse The value range of the polar coordinate system is The corresponding domain point coordinates and intensity values in the domain are obtained, and then M different radii are traversed to obtain M circular surfaces with different radii; after obtaining the M circular surfaces, it is also necessary to map them into square blocks, and set the size of the square blocks to N×N, where , , the polar angle is calculated by formula (2):
[0011]
[0012] By traversing the set of rows L and the set of columns C, and combining equations (1) and (2), we can get The generated M square tiles.
[0013] Furthermore, the self-calibration of the binocular camera based on the classification result to estimate the intrinsic parameters and extrinsic parameters of the binocular camera includes: classifying the feature points according to their structures, constructing a multi-branch feature point classification network, with a total of M branches, each branch receiving a square block as input, and learning the feature points according to the different forms of the feature points with different structures. The intensity distribution characteristics of the The structure belongs to one of five categories: L-type, Y-type, star-type, centroid-type and unstructured type.
[0014] Furthermore, the multi-branch classification network includes: an initial module: constructing an initial feature extraction module by combining a lightweight inverse residual block; a combination layer: connecting the output features of the initial modules in multiple branches to combine multiple features together; a fusion module: further filtering the features input from the combination layer through a squeeze excitation mechanism, and after the mechanism assigns weights according to the importance of the features, using 1*1 convolution to fuse the original connected features with the features output by the mechanism, and then ending with a nonlinear activation function Hard-Swish function that reduces the computational cost; a self-attention module: in the self-attention module, in order to further extract higher-level features, two inverted residual blocks are superimposed; a prediction module: performing a Dropout operation on the output features of the self-attention module, and then using 1*1 convolution and global average pooling to pool the average value of the entire feature map, and then using 1*1 convolution and softmax function to calculate the probability value of the feature point belonging to each category, wherein the function form of softmax is shown in formula (3):
[0015]
[0016] Among them, K is the number of categories, Represents the score of the i-th category in the 1*1 convolution output. This function outputs the probability value of the object to be predicted belonging to each category, and selects the category with the largest probability value as the category that the feature point is finally determined to be.
[0017] According to another aspect of the present invention, a binocular camera self-calibration device based on a feature point structure is provided, and the following technical solution is adopted:
[0018] A binocular camera self-calibration device based on a feature point structure comprises: a detection unit, which is used to perform feature point detection on the entire input image based on a feature point detection algorithm; an input unit, which is used to input each detected feature point into a CPE module, obtain a plurality of blocks centered on the feature point, and then use the blocks as inputs of a multi-branch classification network to output a classification result of the feature point structure; a self-calibration unit, which is used to perform self-calibration on the binocular camera based on the classification result, and estimate the intrinsic parameters and extrinsic parameters of the binocular camera; and a calculation unit, which is used to calculate evaluation indicators of the intrinsic parameters and extrinsic parameters, obtain the feature point category corresponding to the parameter with the highest accuracy by comparing the indicators, and select the group of parameters as the binocular camera self-calibration parameters.
[0019] Furthermore, the detection unit is also used to: for a given feature point , with feature points As the center, in the polar coordinate system, M radii are R={ }, capture the intensity distribution around the feature point within the circular surface range; using the polar angle and radius { ,} to construct feature points The circular surface of which the polar angle ranges from ; Based on the polar angle and the set surface radius and formula (1), calculate Points in the neighborhood The coordinates of , formula (1) is;
[0020]
[0021] According to formula (1), traverse The value range of the polar coordinate system is The corresponding domain point coordinates and intensity values in the domain are obtained, and then M different radii are traversed to obtain M circular surfaces with different radii; after obtaining the M circular surfaces, it is also necessary to map them into square blocks, and set the size of the square blocks to N×N, where , , the polar angle is calculated by formula (2):
[0022]
[0023] By traversing the set of rows L and the set of columns C, and combining equations (1) and (2), we can get M square tiles are generated.
[0024] Furthermore, the self-calibration unit is also used to classify the feature points according to their structures, construct a multi-branch feature point classification network, with a total of M branches, each branch receiving a square block as input, and learning the feature points according to the different forms of different structural feature points. The intensity distribution characteristics of the The structure belongs to one of five categories: L-type, Y-type, star-type, centroid-type and unstructured type.
[0025] Furthermore, the multi-branch classification network includes: an initial module: constructing an initial feature extraction module by combining a lightweight inverse residual block; a combination layer: connecting the output features of the initial modules in multiple branches to combine multiple features together; a fusion module: further filtering the features input from the combination layer through a squeeze excitation mechanism, and after the mechanism assigns weights according to the importance of the features, using 1*1 convolution to fuse the original connected features with the features output by the mechanism, and then ending with a nonlinear activation function Hard-Swish function that reduces the computational cost; a self-attention module: in the self-attention module, in order to further extract higher-level features, two inverted residual blocks are superimposed; a prediction module: performing a Dropout operation on the output features of the self-attention module, and then using 1*1 convolution and global average pooling to pool the average value of the entire feature map, and then using 1*1 convolution and softmax function to calculate the probability value of the feature point belonging to each category, wherein the function form of softmax is shown in formula (3):
[0026]
[0027] Among them, K is the number of categories, Represents the score of the i-th category in the 1*1 convolution output. This function outputs the probability value of the object to be predicted belonging to each category, and selects the category with the largest probability value as the category that the feature point is finally determined to be.
[0028] According to yet another aspect of the present invention, a storage medium is provided, wherein the storage medium includes the above-mentioned binocular camera self-calibration device.
[0029] The present invention proposes a multi-branch feature point classification network based on circular patch extraction (CPE) to extract feature points as the input of the self-calibration algorithm. The network includes a CPE module and a multi-branch feature point classification network. Compared with the conventional process of performing self-calibration tasks without distinguishing feature point structures, this method can avoid feature points that are not conducive to the effect of the self-calibration algorithm, thereby improving the accuracy of the calibration results. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0031] Figure 1 This is a flow chart of a binocular camera self-calibration method based on feature point structure according to an embodiment of the present invention;
[0032] Figure 2A schematic diagram of a multi-branch feature point classification network framework according to an embodiment of the present invention;
[0033] Figure 3 1 is a structural diagram of each module described in an embodiment of the present invention, wherein (a) is an initial module, (b) is a fusion module, (c) is a self-attention module, and (d) is a prediction module;
[0034] Figure 4 A schematic diagram of the classification model training steps according to an embodiment of the present invention;
[0035] Figure 5 This is an overall flow chart of a binocular camera self-calibration method based on a feature point structure according to an embodiment of the present invention;
[0036] Figure 6 Schematic diagram of the structure of a binocular camera self-calibration device based on feature point structure according to an embodiment of the present invention. DETAILED DESCRIPTION
[0037] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.
[0038] Figure 1 This is a flow chart of a binocular camera self-calibration method based on feature point structure according to an embodiment of the present invention.
[0039] See also Figure 1 As shown, a binocular camera self-calibration method based on feature point structure includes:
[0040] S101: Perform feature point detection on the entire input image based on a feature point detection algorithm;
[0041] S103: Input each detected feature point into the CPE module, obtain multiple blocks centered on the feature point, and then use these blocks as input to the multi-branch classification network to output the classification result of the feature point structure;
[0042] S105: performing self-calibration on the binocular camera based on the classification result, and estimating the intrinsic parameters and extrinsic parameters of the binocular camera;
[0043] S107: Calculate the intrinsic parameter and extrinsic parameter evaluation indexes, obtain the feature point category corresponding to the parameter with the highest accuracy by comparing the indexes, and select this group of parameters as the binocular camera self-calibration parameters.
[0044] Specifically, the proposed method for binocular camera self-calibration based on feature point structure includes two parts: circular patches extraction (CPE) module and multi-branch feature point classification network. First, in step S101, the feature point detection algorithm detects the feature point set in the whole image range, and then in step S103, each feature point is input into the CPE module to obtain multiple blocks centered on the feature point, and then these blocks are used as the input of the multi-branch classification network to output the classification result of the feature point structure, and the feature points are divided into L-type, Y-type, star-type, centroid type and non-structural type according to their structural characteristics. Finally, in step S105, the feature points of each category are used to self-calibrate the binocular camera to estimate the internal and external parameters of the binocular camera. And in S107, the evaluation index is calculated for the parameters estimated from the feature points of each category, and the feature point category corresponding to the parameter with the highest accuracy can be known by comparing the index, and this group of parameters is selected as the final estimation result.
[0045] Generally, extracting features from images is an important task in computer vision. For feature-based visual algorithms such as binocular camera self-calibration, the effect of the extracted features will directly affect the accuracy of the algorithm output results. Current feature extraction algorithms use the entire image as the search area and do not distinguish the structure of feature points. Although a large number of feature points can be extracted, the structure of feature points has a greater impact on the binocular camera self-calibration algorithm in intelligent driving scenarios. In order to solve this problem, the present invention proposes a multi-branch feature point classification network based on circular patch extraction (CPE) to extract feature points as the input of the self-calibration algorithm, and the network includes a CPE module and a multi-branch feature point classification network. Compared with the conventional process of performing self-calibration tasks without distinguishing the structure of feature points, this method can avoid feature points that are not conducive to the effect of the self-calibration algorithm, thereby improving the accuracy of the calibration results.
[0046] Figure 2 Schematic diagram of a multi-branch feature point classification network framework according to an embodiment of the present invention.
[0047] See also Figure 2 As shown, the detected feature points are input into the CPE module, M blocks centered on the feature point are obtained, and these blocks are used as inputs of the multi-branch classification network to output the classification results of the feature point structure, including: for a given feature point , with feature points As the center, in the polar coordinate system, M radii are R={ }, capture the intensity distribution around the feature point within the circular surface range; using the polar angle and radius { , } to construct feature points The circular surface of which the polar angle ranges from ; Based on the polar angle and the set surface radius and formula (1), calculate Points in the neighborhood The coordinates of , formula (1) is;
[0048]
[0049] According to formula (1), traverse The value range of the polar coordinate system is The corresponding domain point coordinates and intensity values in the domain are obtained, and then M different radii are traversed to obtain M circular surfaces with different radii; after obtaining the M circular surfaces, it is also necessary to map them into square blocks, and set the size of the square blocks to N×N, where , , the polar angle is calculated by formula (2):
[0050]
[0051] By traversing the set of rows and the set of columns CL and combining equations (1) and (2), we can get M square tiles are generated.
[0052] Specifically, the self-calibration of the binocular camera based on the classification result and the estimation of the intrinsic parameters and extrinsic parameters of the binocular camera include: classifying the feature points according to the structure of the feature points, constructing a multi-branch feature point classification network, with a total of M branches, each branch receiving a square block as input, and learning the feature points according to the different forms of different structural feature points. The intensity distribution characteristics of the The structure belongs to one of five categories: L-type, Y-type, star-type, centroid-type and unstructured type.
[0053] Figure 3 This is a structural diagram of each module described in an embodiment of the present invention.
[0054] See also Figure 3 As shown in (a), the initial module includes input, convolution, maximum pooling and reverse residual block steps. Specifically, the initial module takes into account that the subsequent fusion module will fuse the input features of all branches, so the feature extraction process of each stream should be lightweight to reduce the amount of calculation and improve efficiency. Therefore, the initial feature extraction module is constructed by combining the lightweight reverse residual block. Combination layer: The output features of the initial modules in multiple branches are concatenated to combine multiple features.
[0055] In (b) the fusion module, including input, multiplication, 1*1 convolution and Hard swish, specifically, in the initial module, due to the large amount of information extracted and the complex scene, it is easy to extract noise features. In order to reduce the interference of noise on the classification network, a squeezing excitation mechanism is used in this module to further filter the features input from the combination layer. This mechanism gives higher weights to beneficial features, thereby reducing the importance of perturbation features and improving the stability of the network. After the mechanism assigns weights according to the importance of the features, we use 1*1 convolution to fuse the features of the original connection with the features output by the mechanism. Finally, it ends with a nonlinear activation function Hard-Swish function that reduces the computational cost.
[0056] In the (c) self-attention module, including input and inverted residual blocks, two inverted residual blocks are superimposed to further extract higher-level features.
[0057] In the (d) prediction module, as the last module of the network, we first perform a Dropout operation on the output features of the self-attention module, then use 1*1 convolution and global average pooling to pool the average value of the entire feature map, and then use 1*1 convolution and softmax function to calculate the probability value of the feature point belonging to each category. The function form of softmax is shown in formula (3):
[0058]
[0059] Among them, K is the number of categories, Represents the score of the i-th class in the 1*1 convolution output. This function outputs the probability value of the object to be predicted belonging to each class, and selects the class with the largest probability value as the final class of the feature point.
[0060] Figure 4 Schematic diagram of the classification model training steps described in an embodiment of the present invention.
[0061] exist Figure 4 A schematic diagram of the classification model training steps is given in Figure 4 shown.
[0062] Step 40: Take a picture to extract feature points and mark the feature point categories;
[0063] Step 41: Divide the data of each scene into a training set, a validation set and a test set;
[0064] Step 42: Design a multi-branch feature point classification network;
[0065] Step 43: Model group parameter adjustment and segmentation performance evaluation.
[0066] Figure 5This is an overall flow chart of the binocular camera self-calibration method based on feature point structure described in an embodiment of the present invention.
[0067] See also Figure 5 As shown in FIG. 1 , the overall process of the binocular camera self-calibration method based on the feature point structure includes:
[0068] Step 50: Input picture;
[0069] Step 51: Extract SIFT feature points;
[0070] Step 52: Using CPE, generate M square blocks for each feature point;
[0071] Step 53: training the classification model;
[0072] Step 54: Obtain feature point classification results;
[0073] Step 55: Use the feature points of each category to perform binocular camera self-calibration and calculate multiple sets of camera parameters;
[0074] Step 56: Calculate the evaluation index of each group of parameters;
[0075] Step 57: Compare the indicators to obtain the feature point category corresponding to the parameter with the highest accuracy. This group of parameters is the final estimation result.
[0076] In short, binocular camera self-calibration does not require other auxiliary equipment. It only needs to take pictures with the binocular camera to obtain feature information in the scene, and use this information to estimate the internal and external parameters of the binocular camera.
[0077] Feature point extraction: Use feature point extraction algorithms such as Harris, SIFT, and SURF to detect and extract the coordinates of all feature points from the image.
[0078] Feature point classification: First, the K feature points extracted are used to generate K×M square blocks using CPE. The M blocks corresponding to each feature point are used as the input of the multi-branch feature point classification network to determine the structural category of the current feature point.
[0079] Self-calibration: After obtaining the feature point sets of each category, use the feature point sets of each category as the input data of the binocular camera self-calibration algorithm to calculate multiple sets of binocular camera parameters. Calculate the evaluation index of each set of parameters, compare the index to obtain the feature point category corresponding to the parameter with the highest accuracy, and select this set of parameters as the final estimation result.
[0080] Figure 6 Schematic diagram of the structure of a binocular camera self-calibration device based on feature point structure according to an embodiment of the present invention.
[0081] See also Figure 6As shown, the binocular camera self-calibration device based on the feature point structure includes: a detection unit 60, which is used to perform feature point detection on the entire input image based on a feature point detection algorithm; an input unit 62, which is used to input each detected feature point into the CPE module, obtain multiple blocks centered on the feature point, and then use these blocks as inputs of a multi-branch classification network to output a classification result of the feature point structure; a self-calibration unit 64, which is used to perform self-calibration on the binocular camera based on the classification result, and estimate the intrinsic parameters and extrinsic parameters of the binocular camera; a calculation unit 66, which is used to calculate evaluation indicators of the intrinsic parameters and the extrinsic parameters, obtain the feature point category corresponding to the parameter with the highest accuracy by comparing the indicators, and select this group of parameters as the binocular camera self-calibration parameters.
[0082] Preferably, the detection unit 60 is further configured to: for a given feature point , with feature points As the center, in the polar coordinate system, M radii are R={ }, capture the intensity distribution around the feature point within the circular surface range; using the polar angle and radius { , } to construct feature points The circular surface of which the polar angle ranges from ; Based on the polar angle and the set surface radius and formula (1), calculate Points in the neighborhood The coordinates of , formula (1) is;
[0083]
[0084] According to formula (1), traverse The value range of the polar coordinate system is The corresponding domain point coordinates and intensity values in the domain are obtained, and then M different radii are traversed to obtain M circular surfaces with different radii; after obtaining the M circular surfaces, it is also necessary to map them into square blocks, and set the size of the square blocks to N×N, where , , the polar angle is calculated by formula (2):
[0085]
[0086] By traversing the set of rows L and the set of columns C, and combining equations (1) and (2), we can get M square tiles are generated.
[0087] Preferably, the self-calibration unit 64 is further used to: classify the feature points according to their structures, construct a multi-branch feature point classification network, with a total of M branches, each branch receiving a square block as input, and learning the feature points according to the different forms of different structural feature points. The intensity distribution characteristics of the The structure belongs to one of five categories: L-type, Y-type, star-type, centroid-type and unstructured type.
[0088] The storage medium provided by the present invention includes the above-mentioned binocular camera self-calibration device.
[0089] The present invention proposes a multi-branch feature point classification network based on circular patch extraction (CPE) to extract feature points as the input of the self-calibration algorithm. The network includes a CPE module and a multi-branch feature point classification network. Compared with the conventional process of performing self-calibration tasks without distinguishing feature point structures, this method can avoid feature points that are not conducive to the effect of the self-calibration algorithm, thereby improving the accuracy of the calibration results.
[0090] The above only describes some exemplary embodiments of the present embodiment by way of illustration. It is undoubted that, for those skilled in the art, the described embodiments can be modified in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.
Claims
1. A binocular camera self-calibration method based on feature point structure, characterized in that: include: Based on the feature point detection algorithm, the feature points of the input image are detected in the entire image range; Input each detected feature point into the CPE module, obtain multiple blocks centered on the feature point, and then use these blocks as input to the multi-branch classification network to output the classification result of the feature point structure; Based on the classification results, the stereo camera is self-calibrated to estimate the intrinsic and extrinsic parameters of the stereo camera; Calculate the evaluation indexes of the intrinsic and extrinsic parameters, obtain the feature point category corresponding to the parameter with the highest accuracy by comparing the indexes, and select this group of parameters as the binocular camera self-calibration parameters.
2. The binocular camera self-calibration method according to claim 1, characterized in that: The detected feature points are input into the CPE module, multiple blocks centered on the feature point are obtained, and the blocks are used as inputs of the multi-branch classification network to output the classification results of the feature point structure, including: For a given feature point , with feature points As the center, in the polar coordinate system, M radii are R={ }, capturing the intensity distribution around the feature point; Using polar angle and radius { , } to construct feature points The circular surface of which the polar angle ranges from ; Based on the polar angle, the set surface radius and formula (1), calculate Points in the neighborhood The coordinates of According to formula (1), traverse The value range of the polar coordinate system is The corresponding domain point coordinates and intensity values in the domain, and then traverse M different radii to obtain M circular surfaces with different radii; After obtaining M circular surfaces, they need to be mapped into square blocks, and the size of the square blocks is set to N×N, where , are the set of rows and columns of the block respectively, and the polar angle is calculated by formula (2): By traversing each row and column at all radius sizes and combining equations (1) and (2), we can get The generated M square tiles.
3. The binocular camera self-calibration method according to claim 2, characterized in that: The binocular camera is self-calibrated based on the classification result, and the intrinsic parameters and extrinsic parameters of the binocular camera are estimated, including: The feature points are classified according to their structure, and a multi-branch feature point classification network is constructed. There are M branches in total, each branch receives a square block as input, and learns feature points according to the different forms of different structural feature points. The intensity distribution characteristics of the The structure belongs to one of five categories: L-type, Y-type, star-type, centroid-type and unstructured type.
4. The binocular camera self-calibration method according to claim 3, characterized in that: The multi-branch classification network includes: Initial module: The initial feature extraction module is constructed by combining a lightweight reverse residual block; Combination layer: connects the output features of the initial modules in multiple branches to combine multiple features together; Fusion module: Through the squeeze excitation mechanism, the features input from the combined layer are further filtered. After the mechanism assigns weights according to the importance of the features, the original connected features are fused with the features output by the mechanism using 1*1 convolution, and then ended with a nonlinear activation function Hard-Swish function that reduces the computational cost; Self-attention module: In the self-attention module, two inverted residual blocks are superimposed to further extract higher-level features; Prediction module: Dropout operation is performed on the output features of the self-attention module, and then 1*1 convolution and global average pooling are used to pool the average value of the entire feature map. Then, 1*1 convolution and softmax function are used to calculate the probability value of the feature point belonging to each category. The function form of softmax is shown in Formula 3: Among them, K is the number of categories, Represents the score of the i-th category in the 1*1 convolution output. This function outputs the probability value of the object to be predicted belonging to each category, and selects the category with the largest probability value as the category that the feature point is finally determined to be.
5. A binocular camera self-calibration device based on feature point structure, characterized in that: include: A detection unit, used to perform feature point detection on the entire input image based on a feature point detection algorithm; An input unit is used to input each detected feature point into the CPE module, obtain multiple blocks centered on the feature point, and then use these blocks as input to the multi-branch classification network to output the classification result of the feature point structure; A self-calibration unit, used to perform self-calibration on the binocular camera based on the classification result, and estimate the intrinsic parameters and extrinsic parameters of the binocular camera; The calculation unit is used to calculate the evaluation indexes of the internal and external parameters, obtain the feature point category corresponding to the parameter with the highest accuracy by comparing the indexes, and select this group of parameters as the self-calibration parameters of the binocular camera.
6. The binocular camera self-calibration device according to claim 5, characterized in that: The detection unit is also used for: For a given feature point , with feature points As the center, in the polar coordinate system, M radii are R={ }, capturing the intensity distribution around the feature point; Using polar angle and radius { , } to construct feature points The circular surface of which the polar angle ranges from ; Based on the polar angle, the set surface radius and formula (1), calculate Points in the neighborhood The coordinates of , formula (1) is; According to formula (1), traverse The value range of the polar coordinate system is The corresponding domain point coordinates and intensity values in the domain, and then traverse M different radii to obtain M circular surfaces with different radii; After obtaining M circular surfaces, they need to be mapped into square blocks, and the size of the square blocks is set to N×N, where , , the polar angle is calculated by formula (2): By traversing the set of rows L and the set of columns C and combining equations (1) and (2), we can get M square tiles are generated.
7. The binocular camera self-calibration device according to claim 6, characterized in that: The self-calibration unit is also used for: The feature points are classified according to their structure, and a multi-branch feature point classification network is constructed. There are M branches in total, each branch receives a square block as input, and learns feature points according to the different forms of different structural feature points. The intensity distribution characteristics of the The structure belongs to one of five categories: L-type, Y-type, star-type, centroid-type and unstructured type.
8. The binocular camera self-calibration device according to claim 7, characterized in that: The multi-branch classification network includes: Initial module: The initial feature extraction module is constructed by combining a lightweight reverse residual block; Combination layer: connects the output features of the initial modules in multiple branches to combine multiple features together; Fusion module: Through the squeeze excitation mechanism, the features input from the combined layer are further filtered. After the mechanism assigns weights according to the importance of the features, the original connected features are fused with the features output by the mechanism using 1*1 convolution, and then ended with a nonlinear activation function Hard-Swish function that reduces the computational cost; Self-attention module: In the self-attention module, two inverted residual blocks are superimposed to further extract higher-level features; Prediction module: Dropout operation is performed on the output features of the self-attention module, and then 1*1 convolution and global average pooling are used to pool the average value of the entire feature map. Then, 1*1 convolution and softmax function are used to calculate the probability value of the feature point belonging to each category. The function form of softmax is shown in formula (3): Among them, K is the number of categories, Represents the score of the i-th category in the 1*1 convolution output. This function outputs the probability value of the object to be predicted belonging to each category, and selects the category with the largest probability value as the category that the feature point is finally determined to be.
9. A storage medium, characterized in that: The invention comprises a binocular camera self-calibration device as described in any one of claims 5 to 8.