Matching point pair purification method, electronic device, and storage medium

Abstract

Embodiments of the present application disclose a matching point pair purification method, an electronic device and a storage medium. The method comprises: obtaining a target matching point pair set, wherein the target matching point pair set comprises a plurality of different matching point pairs, each matching point pair comprising a first feature point belonging to a first image and a second feature point belonging to a second image; converting the matching point pairs of the target matching point pair set into a target input image; inputting the target input image into a target model for processing to obtain an output result, wherein the target model is a model obtained by training a lightweight network based on deep learning; and identifying correct matching point pairs in the target matching point pair set according to the output result.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for purifying matching point pairs, an electronic device, and a storage medium. Background Technology

[0002] Image feature matching is a frequently involved step in computer vision. By matching descriptors between images or between an image and a map, it reduces the processing burden on subsequent operations such as pose estimation and optimization. Currently, due to the local characteristics of image features, mismatches are widespread. To reduce mismatches, image feature matching point pair purification methods have been introduced to remove feature matching point pairs with large errors.

[0003] In existing technologies, based on the Random Sample Consensus (RANSAC) algorithm, a mathematical model is pre-trained. The matching point pairs that need to be purified are input into the mathematical model, and the mathematical model selects the matching point pairs that satisfy the model the most, while the rest are removed, so as to achieve the purpose of purification.

[0004] While mathematical models based on the RANSAC algorithm can purify matching point pairs to some extent, they also have some problems. For example, when dealing with scenarios with a high proportion of incorrect matching point pairs, the runtime is long, resulting in a high cost for purifying matching point pairs. Summary of the Invention

[0005] This application provides a method for purifying matching point pairs, an electronic device, and a storage medium to solve the technical problem of high cost of purifying matching point pairs in the prior art.

[0006] According to a first aspect of this application, a method for purifying matching point pairs is disclosed, the method comprising:

[0007] Obtain a target matching point pair set, wherein the target matching point pair set includes multiple different matching point pairs, and each matching point pair includes: a first feature point belonging to the first image and a second feature point belonging to the second image;

[0008] Convert the matching point pairs of the target matching point pair set into the target input image;

[0009] The target input image is input into the target model for processing to obtain the output result, wherein the target model is a model trained based on a lightweight deep learning network;

[0010] Based on the output, identify the correct matching point pairs in the target matching point pair set.

[0011] According to a second aspect of this application, a matching point pair purification apparatus is disclosed, the apparatus comprising:

[0012] The acquisition module is used to acquire a target matching point pair set, wherein the target matching point pair set includes multiple different matching point pairs, and each matching point pair includes: a first feature point belonging to the first image and a second feature point belonging to the second image;

[0013] A conversion module is used to convert the matching point pairs of the target matching point pair set into a target input image;

[0014] The processing module is used to input the target input image into the target model for processing and obtain the output result, wherein the target model is a model trained based on a lightweight deep learning network.

[0015] The identification module is used to identify the correct matching point pairs in the target matching point pair set based on the output results.

[0016] According to a third aspect of this application, an electronic device is disclosed, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the matching point pair purification method as described in the first aspect.

[0017] According to a fourth aspect of this application, a computer-readable storage medium is disclosed having a computer program / instructions stored thereon, which, when executed by a processor, implements the matching point pair purification method as described in the first aspect.

[0018] According to a fifth aspect of this application, a computer program product is disclosed, comprising a computer program / instructions that, when executed by a processor, implement the matching point pair purification method as described in the first aspect.

[0019] In this embodiment of the application, a target matching point pair set is obtained, wherein the target matching point pair set includes multiple different matching point pairs, each matching point pair including: a first feature point belonging to a first image and a second feature point belonging to a second image; the matching point pairs in the target matching point pair set are converted into a target input image; the target input image is input into a target model for processing to obtain an output result, wherein the target model is a model trained based on a lightweight deep learning network; and the correct matching point pairs in the target matching point pair set are identified according to the output result.

[0020] As can be seen, in this embodiment, the idea of ​​deep learning is used to replace the traditional RANSAC algorithm to train the target model. Since the target model is a model trained based on a lightweight deep learning network, and deep learning trains a prediction model based on a large number of ground truths, this prediction model is equivalent to having a large amount of prior knowledge. A large amount of prior knowledge can provide a relatively accurate prediction and inference result for the input. Furthermore, the lightweight network consumes less hardware and software resources and has a shorter runtime. Therefore, when it is necessary to purify the matching point pairs in the target matching point pair set, the matching point pairs in the target matching point pair set are converted into a target input image that meets the input requirements of the target model. The target model processes the target input image, and the matching point pairs in the target matching point pair set are purified according to the output result of the model. This can reduce the runtime of the purification process while ensuring the purification accuracy and robustness, thereby reducing the cost of matching point pair purification. Attached Figure Description

[0021] Figure 1 This is an example diagram of the feature matching point pair set provided in the embodiments of this application;

[0022] Figure 2 This is an example diagram showing the purification results of the feature matching point pair set provided in the embodiments of this application;

[0023] Figure 3 This is an example diagram of the epipolar constraint provided in the embodiments of this application;

[0024] Figure 4 This is a flowchart of a matching point pair purification method provided in an embodiment of this application;

[0025] Figure 5 This is a flowchart of one implementation of step 102 provided in the embodiments of this application;

[0026] Figure 6 This is a flowchart of one implementation of step 104 provided in the embodiments of this application;

[0027] Figure 7 This is a flowchart of a model training method provided in an embodiment of this application;

[0028] Figure 8 This is a flowchart illustrating the process of constructing a set of matching point pairs for each sample in the training set provided in this application embodiment;

[0029] Figure 9 This is a flowchart illustrating the construction process of the target loss function provided in the embodiments of this application;

[0030] Figure 10 This is a schematic diagram of the structure of a matching point-pair purification device provided in an embodiment of this application;

[0031] Figure 11 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0034] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beautification, makeup, medical aesthetics, and intelligent temperature measurement.

[0035] This application provides a method for purifying point pairs, an electronic device, and a storage medium.

[0036] To facilitate understanding, the application scenarios and some concepts involved in the embodiments of this application will be introduced first below.

[0037] Matching point pair refinement: For example, when matching image A with image B, first extract a set of feature points {a1, a2, ..., a...} from image A. n Extract feature points b1 that match point a1, b2 that match point a2, and b3 that match point a1 from image B. n Matched feature point b n This yields a set of matching point pairs {(a1,b1), (a2,b2), ..., (a...}. n ,b n To reduce false matches, it is necessary to separate {(a1,b1), (a2,b2), ..., (a...} into {(a1,b1), (a2,b2), ..., (a...}}. n ,b n )} Filter out and remove incorrect matching point pairs. For example Figure 1 As shown, this is a practical application of multiple feature matching point pairs between two images. Figure 1 As can be seen, some matching point pairs contain mismatches; in order to remove these erroneous matching point pairs, it is necessary to... Figure 1 The feature matching point pairs shown are purified to obtain Figure 2 The purification results are shown.

[0038] Stereo correction refers to performing a planar phototransformation on two images, so that corresponding epipolar lines in the two images are on the same horizontal direction, while corresponding poles are mapped to infinity. This reduces the parallax between the two images to one dimension, thus improving the matching speed. Stereo correction is an important method for improving the speed and accuracy of stereo matching.

[0039] Distortion correction is a mapping that projects distorted pixels onto the positions of corrected pixels.

[0040] Fundamental matrix: In computer vision, the fundamental matrix F is a 3×3 matrix that represents the correspondence between image points in a stereo image pair. When two cameras capture the same object from different positions, some overlapping areas will appear on the image planes of the two cameras, creating a certain correspondence. This correspondence is called epipolar geometry. Figure 3 The figure shows the ideal binocular epipolar constraint. Point P1 in the left image is matched with point P2 in the right image. The epipolar line corresponding to point P1 in the right image is l2, and point P2 is on the epipolar line l2. Here, l2 = F*P1, and F is the basic matrix.

[0041] The intrinsic parameter matrix is ​​used to transform 3D camera coordinates to 2D homogeneous image coordinates. The intrinsic parameter matrix reflects the camera's own properties, which are different for each camera. Calibration is required to know these parameters.

[0042] The extrinsic parameter matrix is ​​used to perform the transformation from the world coordinate system to the camera coordinate system.

[0043] The following section first introduces a point-matching purification method provided in an embodiment of this application.

[0044] Figure 4 This is a flowchart of a point-pair purification method provided in an embodiment of this application, such as... Figure 4 As shown, the method may include the following steps: step 401, step 402, step 403, and step 404, wherein,

[0045] In step 401, a target matching point pair set is obtained, wherein the target matching point pair set includes multiple different matching point pairs, and each matching point pair includes: a first feature point belonging to the first image and a second feature point belonging to the second image.

[0046] In this embodiment of the application, the target matching point pair set is the matching point pair set to be purified.

[0047] In one example, the target matching point pair set is {(a1,b1), (a2,b2), ..., (a...}}. n ,b n )}, where (a1,b1), (a2,b2), (a n ,b n For different matching point pairs, point a1 to point a2. n For the first feature points belonging to the first image, points b1 to b2 are... n These are the second feature points belonging to the second image. That is, points a1 to a2. n Let b1 be a feature point in the first image, and b2 be a feature point in the first image. n These are feature points in the second image.

[0048] In step 402, the matching point pairs of the target matching point pair set are converted into the target input image.

[0049] In this embodiment, when refining the matching point pairs in the target matching point pair set, a deep learning-based target model is used. Since the deep learning-based target model has certain requirements for its input data—requiring the input data to be in image format—all matching point pairs in the target matching point pair set need to be converted into a single target input image before being input into the target model. That is, the matching point pairs are transformed from individual coordinate points into a representation on an image. For detailed conversion processes, please refer to [link to relevant documentation]. Figure 5The example shown.

[0050] In one example, the target matching point pair set is {(a1,b1), (a2,b2), ..., (a...}}. n ,b n )}, where {(a1,b1), (a2,b2), ..., (a n ,b n The input image is converted into a target image, for example, into a 128×128×4 image, where 128 is the length and width of the image and 4 is the number of channels of the image.

[0051] In step 403, the target input image is input into the target model for processing to obtain the output result. The target model is a model trained based on a lightweight deep learning network.

[0052] In this embodiment, the target model can be either a classification model or a non-classification model. When the target model is a classification model, the target input image is input into the target model for processing, and the output result is the classification probability value of each matching point pair in the target matching point pair set. This classification probability value can be used to characterize whether the matching point pair belongs to the correct matching point pair or the incorrect matching point pair. When the target model is a non-classification model, the target input image is input into the target model for processing, and the output result is the predicted target basis matrix. Based on the predicted target basis matrix, it can be determined whether each matching point pair in the target matching point pair set belongs to the correct matching point pair or the incorrect matching point pair.

[0053] In some embodiments, when the target model is a classification model, a classifier can be used as the backbone network for training to obtain the target model.

[0054] In some embodiments, when the target model is a non-classification model, a lightweight network such as ShuffleNet or MobileNet can be used as the backbone network for training to obtain the target model.

[0055] In one example, when ShuffleNet is used as the backbone network for training, the target model requires an input of a 128×128×4 image and outputs a 1×1×9 vector, which can be reshaped into a 3×3 matrix, i.e., the prediction basis matrix.

[0056] In step 404, based on the output results, the correct matching point pairs in the target matching point pair set are identified.

[0057] In some embodiments, when the target model is a classification model, the output result is the classification probability value of each matching point pair in the target matching point pair set. For each classification probability value, it is compared with a certain threshold. If it is greater than the threshold, the matching point pair is considered to be a correct matching point pair; otherwise, the matching point pair is considered to be an incorrect matching point pair.

[0058] In some embodiments, when the target model is a non-classification model, the output is a predicted target basis matrix. Based on this predicted target basis matrix, it can be determined whether each matching point pair in the target matching point pair set is a correct matching point pair or an incorrect matching point pair. For detailed determination procedures, please refer to [link to documentation]. Figure 6 The example shown.

[0059] As can be seen from the above embodiments, in this embodiment, a target matching point pair set is obtained, wherein the target matching point pair set includes multiple different matching point pairs, each matching point pair including: a first feature point belonging to the first image and a second feature point belonging to the second image; the matching point pairs in the target matching point pair set are converted into a target input image; the target input image is input into a target model for processing to obtain an output result, wherein the target model is a model trained based on a lightweight deep learning network; and the correct matching point pairs in the target matching point pair set are identified according to the output result.

[0060] As can be seen, in this embodiment, the idea of ​​deep learning is used to replace the traditional RANSAC algorithm to train the target model. Since the target model is a model trained based on a lightweight deep learning network, and deep learning trains a prediction model based on a large number of ground truths, this prediction model is equivalent to having a large amount of prior knowledge. A large amount of prior knowledge can provide a relatively accurate prediction and inference result for the input. Furthermore, the lightweight network consumes less hardware and software resources and has a shorter runtime. Therefore, when it is necessary to purify the matching point pairs in the target matching point pair set, the matching point pairs in the target matching point pair set are converted into a target input image that meets the input requirements of the target model. The target model processes the target input image, and the matching point pairs in the target matching point pair set are purified according to the output result of the model. This can reduce the runtime of the purification process while ensuring the purification accuracy and robustness, thereby reducing the cost of matching point pair purification.

[0061] Figure 5 This is a flowchart of one implementation of step 102 provided in the embodiments of this application, as shown below. Figure 5 As shown, step 102 above may include the following steps: step 501 and step 502;

[0062] In step 501, the average coordinates of the first feature points are calculated based on the coordinates of all first feature points in the target matching point set; and the average coordinates of the second feature points are calculated based on the coordinates of all second feature points in the target matching point set.

[0063] In this embodiment of the application, the average coordinate value of the first feature point includes an x-coordinate value and a y-coordinate value, wherein the x-coordinate value of the average coordinate value of the first feature point is the average of the x-coordinate values ​​of all the first feature points, and the y-coordinate value of the average coordinate value of the first feature point is the average of the y-coordinate values ​​of all the first feature points.

[0064] In this embodiment of the application, the average coordinate value of the second feature point includes an x-coordinate value and a y-coordinate value, wherein the x-coordinate value of the average coordinate value of the second feature point is the average of the x-coordinate values ​​of all the second feature points, and the y-coordinate value of the average coordinate value of the second feature point is the average of the y-coordinate values ​​of all the second feature points.

[0065] In step 502, the first feature point is transformed based on the average coordinate of the first feature point, and the second feature point is transformed based on the average coordinate of the second feature point, and the target input image is generated based on the transformed feature points.

[0066] In this embodiment, all first feature points are transformed based on the average coordinates of the first feature points to obtain transformed feature points, and an intermediate image data is generated based on the transformed feature points; all second feature points are transformed based on the average coordinates of the second feature points to obtain transformed feature points, and another intermediate image data is generated based on the transformed feature points; the two intermediate image data are merged into one image to obtain the target input image.

[0067] In some embodiments, step 502 may include the following steps: step 5021, step 5022, step 5023, step 5024 and step 5025;

[0068] In step 5021, the first feature point is translated so that its centroid is located at the first origin, with the average coordinates of the first feature point as the first origin; and the second feature point is translated so that its centroid is located at the second origin, with the average coordinates of the second feature point as the second origin.

[0069] In this embodiment, the centroid refers to the center of the area of ​​the planar figure.

[0070] In this embodiment of the application, all first feature points are translated as a whole until the centroid of all first feature points is located at the average coordinate position of the first feature points. Similarly, all second feature points are translated as a whole until the centroid of all second feature points is located at the average coordinate position of the second feature points.

[0071] In step 5022, the translated first feature points are scaled until the distance from each first feature point to the first origin is a first value, and the translated second feature points are scaled until the distance from each second feature point to the second origin is a first value.

[0072] In some embodiments, the first value can be It can be achieved through the transformation matrix H t H t *The coordinates of the first feature point are used to scale the translated first feature point so that the average distance from each first feature point to the first origin is... Similarly, the translated second feature points are scaled so that the average distance from each second feature point to the second origin is...

[0073] in,

[0074] The x-coordinate value is the origin (i.e., the mean of the coordinates). The y-coordinate of the origin, u i v is the x-coordinate value of the feature point. i The value is the y-coordinate of the feature point, which is magnified by 80 times here, mainly for matching with an image of size 128×128.

[0075] In step 5023, a blank first mapping table and a blank second mapping table are created, wherein the size of the first mapping table and the second mapping table are the same as the size of the target input image.

[0076] In some embodiments, two mapping tables of 128×128×2 are created, denoted as the first mapping table and the second mapping table, where 128 is the length and width, and 2 is the number of channels, used to store the x-coordinate value and y-coordinate value of each feature point.

[0077] In step 5024, the coordinate values ​​of the scaled first feature point are stored in the first mapping table to obtain the third mapping table, and the coordinate values ​​of the scaled second feature point are stored in the second mapping table to obtain the fourth mapping table.

[0078] In some embodiments, the scaled coordinate values ​​of the first feature point are stored in the corresponding cells of the first mapping table, and the cells that do not contain the coordinate values ​​of the first feature point are padded with 0 to obtain the third mapping table; similarly, the scaled coordinate values ​​of the second feature point are stored in the corresponding cells of the second mapping table, and the cells that do not contain the coordinate values ​​of the second feature point are padded with 0 to obtain the fourth mapping table.

[0079] In step 5025, the target input image is generated based on the third and fourth mapping tables.

[0080] In some embodiments, the third mapping table of 128×128×2 and the fourth mapping table of 128×128×2 are merged into a 128×128×4 map group, which yields the target input image.

[0081] As can be seen, in the embodiments of this application, when converting the matching point pairs in the matching point pair set into an image, only simple mean operation, translation and matrix multiplication operation are required. The operation is relatively simple, occupies less computing resources, and can achieve fast conversion.

[0082] Figure 6 This is a flowchart of one implementation of step 104 provided in the embodiments of this application. When the output result in step 103 is the target fundamental matrix, such as... Figure 6 As shown, step 104 above may include the following steps: step 601, step 602 and step 603;

[0083] In step 601, for each first feature point, the first epipolar line of the first feature point in the second image is determined based on the target basis matrix and the first feature point.

[0084] In one example, for each first feature point Q1 in the first image, the second feature point it matches in the second image is point Q2, and the target basis matrix is ​​F. Then, according to L2 = F * Q1, the first epipolar line L2 of point Q1 in the second image can be calculated.

[0085] In step 602, the first distance from the second feature point corresponding to the first feature point to the first polar line is calculated.

[0086] In one example, calculate the first distance D1 from point Q2 to the first epipolar line L2.

[0087] In step 603, if the first distance is less than the second value, then the first feature point and its corresponding second feature point are determined to be a correct matching point pair.

[0088] In this embodiment of the application, if the first distance is not less than the second value, then the first feature point and its corresponding second feature point are determined to be an incorrect matching point pair.

[0089] Ideally, the first distance from point Q2 to the first polar line L2 should be 0, meaning Q2 is located on the first polar line L2.

[0090] To allow for some error tolerance, a small threshold, or second value, can be set. If the first distance D1 from point Q2 to the first epipolar line L2 is less than this threshold, it means that point Q2 is close to the first epipolar line L2, and the matching point pair (Q1, Q2) is considered a correct matching point pair. If the first distance D1 from point Q2 to the first epipolar line L2 is not less than this threshold, it means that point Q2 is far from the first epipolar line L2, and the matching point pair (Q1, Q2) is considered an incorrect matching point pair.

[0091] As can be seen, in this embodiment of the application, the correct and incorrect matching point pairs in the target matching point pair set can be identified based on the target basis matrix. The computational load is relatively small, resulting in lower purification costs and faster speed.

[0092] Figure 7 This is a flowchart of a model training method provided in an embodiment of this application, such as... Figure 7 As shown, the method may include the following steps: step 701, step 702, step 703 and step 704;

[0093] In step 701, a training set is obtained, wherein the training set includes multiple different sets of sample matching point pairs and the true basis matrix corresponding to each set of sample matching point pairs. Each set of sample matching point pairs includes multiple different sample matching point pairs, which include: incorrect sample matching point pairs and correct sample matching point pairs. Each sample matching point pair includes: a first sample feature point belonging to the first sample image and a second sample feature point belonging to the second sample image.

[0094] In this embodiment of the application, considering that the more sample data involved in model training, the better the effect of the target model obtained by training, the training set may include a massive set of sample matching point pairs.

[0095] In some embodiments, the set of matching point pairs for each sample in the training set can be a manually labeled dataset.

[0096] In some embodiments, considering that manual annotation is costly when a large amount of sample data is required, sample data can be automatically generated to reduce costs. Accordingly, step 701 above includes the following steps: step 7011 and step 7012.

[0097] In step 7011, a binocular simulation system is constructed, which includes a virtual parallel binocular system and a real binocular system. The virtual parallel binocular system includes a first virtual main camera and a first virtual secondary camera. The optical axes of the first virtual main camera and the first virtual secondary camera are parallel, their intrinsic parameter matrices are the same, and their distortions are both 0. The real binocular system includes a second virtual main camera and a second virtual secondary camera. The optical axes of the second virtual main camera and the second virtual secondary camera are not parallel, their intrinsic parameter matrices are different, their distortions are the same, and their distortions are not 0. The intrinsic parameter matrices of the second virtual main camera and the first virtual main camera are the same.

[0098] In one example, firstly, a virtual parallel binocular system is constructed, with the image resolution set to 480×360, and the intrinsic parameter matrices of the main camera (i.e., the first virtual main camera) and the secondary camera (i.e., the first virtual secondary camera) are defined. And distortion D =

[00000] , the extrinsic parameter matrix from the main camera to the sub-camera. and The R matrix can also be represented using Euler angles. Let R = cv2.Rodrigues(Angle).

[0099] Next, the simulation parameters of the virtual parallel binocular system are set: (B is the baseline of the binocular system, a random value within the range [10, 100], in millimeters). According to the formula x = B * f / d, assuming we only consider the range [30cm, ∝], then the maximum disparity of a point in the scene on the parallel binocular image is... Where f is the focal length and d is the distance.

[0100] Next, the simulation parameters of the real binocular system are set: the intrinsic parameter matrix of the main camera (i.e., the second virtual main camera). Distortion D of the main camera m = [k1k2p1p20] (where k1 and k2 are random values ​​in the range [-0.01, 0.01], and p1 and p2 are random values ​​in the range [-10E-4, 10E-4]). This is the intrinsic parameter matrix of the secondary camera (i.e., the second virtual secondary camera). (where f is a random value in the range [-200, 200], (ut, vt) is a random value in the range [-24, 24]), the distortion D of the secondary camera. s = [k1k2p1p20] (where k1 and k2 are random values ​​in the range [-0.01, 0.01], and p1 and p2 are random values ​​in the range [-10E-4, 10E-4]). External parameter matrix from main camera to secondary camera. ( (α, β, γ) are random values ​​within the range [-10π / 180, 10π / 180].

[0101] In step 7012, a set of matching point pairs for each sample and the corresponding true basis matrix are generated using a binocular simulation system.

[0102] In some embodiments, such as Figure 8 As shown, each sample matching point pair set and its corresponding true basis matrix are generated through the following steps: Step 801, Step 802, Step 803 and Step 804;

[0103] In step 801, a set of M parallel stereo matching point pairs of an image is generated by a virtual parallel stereo system, wherein each parallel stereo matching point pair satisfies the constraint that they are on the same row, and M≥8.

[0104] It should be noted that, since the minimum number of matching points for F (the fundamental matrix) is generally considered to be less than 8 points when it is considered a failure and F cannot be calculated, M≥8.

[0105] In one example, for ease of description, the image produced by the main camera will be referred to as the "left image," and the image produced by the secondary camera will be referred to as the "right image." First, M feature points P are randomly generated on the left image, which has a size of 480×360. L-0 Based on the constraint that the points matched by the parallel binocular coordinates are all on the same row, P is generated in the right figure. L-0 Matching point pair P R-0 P R-0 The following conditions must be met: First, P R-0 y coordinate and P L-0 The y-coordinates are equal; secondly, P R-0 x-coordinate = P L-0 x-coordinate - D disparity value (D is in the range [0, x) max (random values ​​within ]); then at this point, a set of ideal matching point pairs P of the image is obtained. L-0 and P R-0 That is, M parallel binocular matching point pairs.

[0106] In step 802, using a real binocular system, distortion of the second virtual main camera is added to the feature points corresponding to the first virtual main camera in each parallel binocular matching point pair; and for the feature points corresponding to the first virtual secondary camera in each parallel binocular matching point pair, the stereo correction inverse process of the second virtual secondary camera and the distortion of the second virtual secondary camera are sequentially performed, resulting in M ​​matching point pairs with added distortion.

[0107] In one example, first, the transformation matrix for secondary stereo correction is calculated. The transformation matrix from the secondary camera's stereo correction to the original image is then H. so =H s -1Next, the transformation process of remapping the matching points of the stereo correction result image to the coordinates of the original image is performed: the main camera only needs to perform the transformation of P in the ideal matching point pair. L-0 Add distortion D m (Based on the distortion correction formula) r 2 =x 2 +y 2 x and y are the coordinates of the ideal matching point. The above formula is performed in the physical coordinate system. The image coordinates are converted to physical coordinates through the intrinsic parameter matrix P. 物理 =K -1 *P 图像 P was obtained L-2 The secondary camera requires two steps: first, processing P in the ideal matching point pair... R-0 The inverse process of stereo correction, i.e., P R-1 =H so *P R-0 Then for P R-1 Add distortion D s Get P R-2 .

[0108] In step 803, random noise is added to the M distorted matching point pairs, and random error is added to N matching point pairs among the M distorted matching point pairs to obtain a sample matching point pair set, where N≥1.

[0109] In one example, first, noise is added to the coordinate points: for image matching point pairs P L-2 and P R-2 Add random noise in the range [-0.5, 0.5] to obtain the image matching point pair P. L-3 and P R-3 Next, generate mismatched point pairs: for M image point pairs P L-3 and P R-3 Classify, randomly select N[0, 0.6*(M-8)] matching point pairs and add random errors ranging from [-5, -1] & [1, 5]. Then, number them P. L-3N and P R-3N The remaining point pairs are correct point pairs, numbered P. L-3T and P R-3T The already numbered matching point pairs P L-3 and P R-3 (P L-3N and P R-3N P L-3T and P R-3T ), which is a set of sample matching point pairs.

[0110] In step 804, the true basis matrix corresponding to the sample matching point pair set is calculated based on the intrinsic parameter matrix of the first virtual main camera, the extrinsic parameter matrix from the first virtual main camera to the first virtual secondary camera, the intrinsic parameter matrix of the second virtual main camera, and the intrinsic parameter matrix of the second virtual secondary camera.

[0111] In this embodiment of the application, the ground truth matrix is ​​the model's ground truth (GT).

[0112] In one example, P L-3N and P R-3N The corresponding true fundamental matrix

[0113] In practical applications, steps 801 to 804 can be repeated, for example, 100,000 times, to obtain a sufficient set of sample matching points for model training.

[0114] As can be seen, in this embodiment of the application, massive amounts of high-quality sample data can be automatically generated through the cooperation of virtual parallel binocular systems and real binocular systems for model training. This results in the target model having rich and realistic prior experience, and achieving high accuracy when using the target model for point pair purification.

[0115] In step 702, the sample matching point pairs of each sample matching point pair set are converted into images to obtain the sample input images corresponding to each sample matching point pair set.

[0116] In one example, step 701 yields the sample matching point pair set P. L-3 and P R-3 (P L-3N and P R-3N P L-3T and P R-3T Since the model requires a 128×128×4 image as input, transformation processing is necessary.

[0117] First, translate the point so that its centroid is located at the origin, which is the initial matching point pair P. L-3 and P R-3 Given the mean of the coordinates, then:

[0118] The origin on the left is The origin on the right is

[0119] Next, the point is scaled so that its average distance to the origin is... The transformation matrix is For P L-3 Transform to obtain P L-4 =H t-L *PL-3 , for P R-3 Transform to obtain P R-4 =H t-R *P R-3 Among them, P L-3-i ·x is the feature point P L-3-i The x-coordinate value, P L-3-i ·y represents feature point P L-3-i The y-coordinate value, P R-3-i ·x is the feature point P R-3-i The x-coordinate value, P R-3-i ·y represents feature point P R-3-i The y-coordinate value, The 80-fold increase here is primarily for matching with an image of size 128×128.

[0120] Finally, create two map tables, left and right, with a size of 128×128×2, denoted as m. L and m R Match point pair P L-4 and P R-4 Stored separately in mapping table m L and m R In the corresponding grid, then m L and m R The graphs are merged into a single 128×128×4 graph group, which is then used as the input to the model.

[0121] In step 703, the network to be trained and the corresponding target loss function are constructed.

[0122] In this embodiment of the application, the network to be trained can be a classifier, or a lightweight network such as ShuffleNet or MobileNet.

[0123] In some embodiments, when the network to be trained is a lightweight network such as ShuffleNet or MobileNet, such as Figure 9 As shown, the target loss function is constructed through the following steps: steps 901, 902, and 903;

[0124] In step 901, the first loss function is determined based on the true basis matrix and the predicted basis matrix corresponding to the sample matching point pair set.

[0125] In this embodiment of the application, the difference between the true basis matrix and the predicted basis matrix can be used as the first loss function.

[0126] In one example, the true fundamental matrix is ​​F, and the predicted fundamental matrix is ​​Ft. pred First loss function

[0127] In step 902, a second loss function is determined based on the second distance from the second sample feature point to the second epipolar line in the second sample image and the third distance from the second sample feature point to the third epipolar line in the second sample image. The second epipolar line is determined based on the first sample feature point and the true basis matrix, and the third epipolar line is determined based on the first sample feature point and the prediction basis matrix.

[0128] In this embodiment, the difference between the epipolar distance corresponding to the true basis matrix and the epipolar distance corresponding to the basis matrix predicted by the model can be used as the second loss function.

[0129] In one example, the true fundamental matrix is ​​F, and the predicted fundamental matrix is ​​Ft. pred Using the true fundamental matrix F to compare P L-3 Perform the transformation L = F * P L-3 The second epipolar line L is obtained; the prediction fundamental matrix F is used. pred For P L-3 Perform transformation L pred =F pred *P L-3 The third pole line L is obtained. pred ; Calculate P R-3 Distance d to the second polar line L L and calculation of P R-3 To the third pole line L pred distance d pred Second loss function

[0130] In step 903, a target loss function is generated based on the first loss function and the second loss function.

[0131] In one example, the first loss function is Loss1, the second loss function is Loss2, and the target loss function is Loss = Loss1 + Loss2.

[0132] As can be seen, in the embodiments of this application, when constructing the loss function of the model, the loss during the model training process can be considered from multiple dimensions to ensure the globality and accuracy of the model training.

[0133] In step 704, the sample input image is input into the network to be trained for processing to obtain the corresponding prediction basis matrix. Based on the true basis matrix, the prediction basis matrix and the target loss function, the parameters in the network to be trained are updated by backpropagation. The above iterative process is repeated until the model converges to obtain the target model.

[0134] In this embodiment, the model output is a 1x1x9 vector, which can be reshaped into a 3x3 F-shape. predAt this point, it is necessary to multiply by the point transformation matrix to obtain F. pred =H t-R *F pred H t-L .

[0135] In this embodiment, during model training, the network to be trained can be iterated multiple times. In each iteration, a portion of the sample input images are selected and input into the network to be trained for processing to obtain the corresponding prediction basis matrix. The loss value is calculated based on the prediction basis matrix, the true basis matrix, the sample matching point pair set, and the target loss function corresponding to the sample input image. Based on the calculated loss value, the parameters of the network to be trained are updated through backpropagation. This iterative training process is repeated multiple times until the network to be trained converges, thus obtaining the target model.

[0136] As can be seen, in this embodiment of the application, a lightweight deep learning-based network can be trained using massive amounts of training data to obtain a target model for refining matching point pairs. Since deep learning trains a prediction model based on a large number of ground truths, this prediction model is equivalent to having a large amount of prior knowledge. This large amount of prior knowledge can provide a relatively accurate prediction and inference result for the input. Furthermore, the lightweight network consumes less hardware and software resources and has a shorter runtime. Therefore, when it is necessary to refine the matching point pairs in the target matching point pair set, the matching point pairs in the target matching point pair set are converted into a target input image that meets the input requirements of the target model. The target model processes the target input image, and the matching point pairs in the target matching point pair set are refined based on the model's output. This can reduce the runtime of the refinement process while ensuring refinement accuracy and robustness, thereby reducing the cost of refining matching point pairs.

[0137] Figure 10 This is a schematic diagram of the structure of a matching point-to-pair purification device provided in an embodiment of this application, as shown below. Figure 10 As shown, the matching point pair purification device 1000 may include: an acquisition module 1001, a conversion module 1002, a processing module 1003, and an identification module 1004;

[0138] The acquisition module 1001 is used to acquire a target matching point pair set, wherein the target matching point pair set includes multiple different matching point pairs, and each matching point pair includes: a first feature point belonging to the first image and a second feature point belonging to the second image;

[0139] The conversion module 1002 is used to convert the matching point pairs of the target matching point pair set into a target input image;

[0140] The processing module 1003 is used to input the target input image into the target model for processing and obtain the output result, wherein the target model is a model trained based on a lightweight deep learning network.

[0141] The identification module 1004 is used to identify the correct matching point pairs in the target matching point pair set based on the output results.

[0142] As can be seen from the above embodiments, in this embodiment, a target matching point pair set is obtained, wherein the target matching point pair set includes multiple different matching point pairs, each matching point pair including: a first feature point belonging to the first image and a second feature point belonging to the second image; the matching point pairs in the target matching point pair set are converted into a target input image; the target input image is input into a target model for processing to obtain an output result, wherein the target model is a model trained based on a lightweight deep learning network; and the correct matching point pairs in the target matching point pair set are identified according to the output result.

[0143] As can be seen, in this embodiment, the idea of ​​deep learning is used to replace the traditional RANSAC algorithm to train the target model. Since the target model is a model trained based on a lightweight deep learning network, and deep learning trains a prediction model based on a large number of ground truths, this prediction model is equivalent to having a large amount of prior knowledge. A large amount of prior knowledge can provide a relatively accurate prediction and inference result for the input. Furthermore, the lightweight network consumes less hardware and software resources and has a shorter runtime. Therefore, when it is necessary to purify the matching point pairs in the target matching point pair set, the matching point pairs in the target matching point pair set are converted into a target input image that meets the input requirements of the target model. The target model processes the target input image, and the matching point pairs in the target matching point pair set are purified according to the output result of the model. This can reduce the runtime of the purification process while ensuring the purification accuracy and robustness, thereby reducing the cost of matching point pair purification.

[0144] Optionally, as an embodiment, the conversion module 1002 may include:

[0145] The first calculation submodule is used to calculate the average coordinate value of the first feature points based on the coordinate values ​​of all the first feature points in the target matching point set; and to calculate the average coordinate value of the second feature points based on the coordinate values ​​of all the second feature points in the target matching point set.

[0146] The generation submodule is used to transform the first feature point based on the average coordinate of the first feature point, and to transform the second feature point based on the average coordinate of the second feature point, and to generate a target input image based on the transformed feature points.

[0147] Optionally, as an embodiment, the generation submodule may include:

[0148] The translation unit is used to translate the first feature point with the average coordinates of the first feature point as the first origin so that its centroid is located at the first origin, and to translate the second feature point with the average coordinates of the second feature point as the second origin so that its centroid is located at the second origin.

[0149] The scaling unit is used to scale the translated first feature point until the distance from each first feature point to the first origin is a first value, and to scale the translated second feature point until the distance from each second feature point to the second origin is the first value.

[0150] A creation unit is used to create a blank first mapping table and a second mapping table, wherein the size of the first mapping table and the second mapping table are the same as the size of the target input image;

[0151] The storage unit is used to store the scaled coordinate values ​​of the first feature point into the first mapping table to obtain a third mapping table, and to store the scaled coordinate values ​​of the second feature point into the second mapping table to obtain a fourth mapping table;

[0152] The first generation unit is used to generate the target input image based on the third mapping table and the fourth mapping table.

[0153] Optionally, as an example, the output result is the target fundamental matrix;

[0154] The identification module 1004 may include:

[0155] The first determining submodule is used to determine, for each first feature point, the first epipolar line of the first feature point in the second image based on the target basis matrix and the first feature point;

[0156] The second calculation submodule is used to calculate the first distance from the second feature point corresponding to the first feature point to the first epipolar line;

[0157] The second determining submodule is used to determine that the first feature point and its corresponding second feature point are a correct matching point pair if the first distance is less than the second value.

[0158] Optionally, as an embodiment, the matching point pair purification device 1000 may further include: a training module; the training module includes:

[0159] The acquisition submodule is used to acquire a training set, wherein the training set includes multiple different sample matching point pair sets and the true basis matrix corresponding to each sample matching point pair set. Each sample matching point pair set includes multiple different sample matching point pairs, and the multiple different sample matching point pairs include: incorrect sample matching point pairs and correct sample matching point pairs. Each sample matching point pair includes: a first sample feature point belonging to a first sample image and a second sample feature point belonging to a second sample image.

[0160] The conversion submodule is used to convert the sample matching point pairs of each sample matching point pair set into images to obtain the sample input images corresponding to each sample matching point pair set.

[0161] The construction submodule is used to construct the network to be trained and the corresponding target loss function;

[0162] The training submodule is used to input the sample input image into the network to be trained for processing to obtain the corresponding prediction basis matrix. Based on the true basis matrix, the prediction basis matrix and the target loss function, the parameters in the network to be trained are updated by backpropagation. The above iterative process is repeated until the model converges to obtain the target model.

[0163] Optionally, as an embodiment, the acquisition submodule may include:

[0164] A construction unit is used to construct a binocular simulation system, wherein the binocular simulation system includes: a virtual parallel binocular system and a real binocular system. The virtual parallel binocular system includes a first virtual master camera and a first virtual slave camera. The optical axes of the first virtual master camera and the first virtual slave camera are parallel, their intrinsic parameter matrices are the same, and their distortions are both 0. The real binocular system includes a second virtual master camera and a second virtual slave camera. The optical axes of the second virtual master camera and the second virtual slave camera are not parallel, their intrinsic parameter matrices are different, their distortions are the same, and their distortions are not 0. The intrinsic parameter matrices of the second virtual master camera and the first virtual master camera are the same.

[0165] The second generation unit is used to generate each set of sample matching points and the corresponding real basis matrix through the binocular simulation system.

[0166] Optionally, as an embodiment, the second generating unit may include:

[0167] A generating subunit is used to generate M parallel binocular matching point pairs of a set of images through the virtual parallel binocular system, wherein each of the parallel binocular matching point pairs satisfies the constraint that they are on the same row, and M≥8;

[0168] The first addition subunit is used to add the distortion of the second virtual main camera to the feature points corresponding to the first virtual main camera in each of the parallel binocular matching point pairs through the real binocular system; and to sequentially perform the stereo correction inverse process of the second virtual secondary camera and add the distortion of the second virtual secondary camera to the feature points corresponding to the first virtual secondary camera in each of the parallel binocular matching point pairs, to obtain M matching point pairs after adding distortion.

[0169] The second addition subunit is used to add random noise to the M distorted matching point pairs and add random error to N matching point pairs in the M distorted matching point pairs to obtain a sample matching point pair set, where N≥1;

[0170] The calculation subunit is used to calculate the real basis matrix corresponding to the sample matching point pair set based on the intrinsic parameter matrix of the first virtual main camera, the extrinsic parameter matrix from the first virtual main camera to the first virtual secondary camera, the intrinsic parameter matrix of the second virtual main camera, and the intrinsic parameter matrix of the second virtual secondary camera.

[0171] Any step and specific operation within any step in the embodiments of the matching point pair purification method provided in this application can be performed by the corresponding module in the matching point pair purification apparatus. The process of the corresponding operation performed by each module in the matching point pair purification apparatus refers to the process of the corresponding operation described in the embodiments of the matching point pair purification method.

[0172] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0173] Figure 11 This is a structural block diagram of an electronic device provided in an embodiment of this application. The electronic device includes a processing component 1122, which further includes one or more processors, and memory resources represented by a memory 1132 for storing instructions executable by the processing component 1122, such as application programs. The application programs stored in the memory 1132 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1122 is configured to execute instructions to perform the methods described above.

[0174] The electronic device may also include a power supply component 1126 configured to perform power management of the electronic device, a wired or wireless network interface 1150 configured to connect the electronic device to a network, and an input / output (I / O) interface 1158. The electronic device may operate on an operating system stored in memory 1132, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™, or similar.

[0175] According to another embodiment of this application, this application also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps in the matching point pair purification method as described in any of the above embodiments.

[0176] According to yet another embodiment of this application, this application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps in the matching point pair purification method as described in any of the above embodiments.

[0177] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0178] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0179] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0180] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0181] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0182] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0183] The above provides a detailed description of the matching point pair purification method, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for purifying matching point pairs, characterized in that, The method includes: Obtain a target matching point pair set, wherein the target matching point pair set includes multiple different matching point pairs, and each matching point pair includes: a first feature point belonging to the first image and a second feature point belonging to the second image; Calculate the average coordinates of the first feature points based on the coordinates of all the first feature points in the target matching point set; and calculate the average coordinates of the second feature points based on the coordinates of all the second feature points in the target matching point set. The first feature point is transformed based on the average coordinates of the first feature point to obtain the transformed feature point, and first intermediate image data is generated based on the transformed feature point. The second feature point is transformed based on the average coordinates of the feature point to obtain the transformed feature point, and second intermediate image data is generated based on the transformed feature point. The first intermediate image data and the second intermediate image are merged to obtain the target input image. The target input image is input into the target model for processing to obtain the output result, wherein the target model is a model trained based on a lightweight deep learning network; the output result is the target basis matrix. For each of the first feature points, the first epipolar line of the first feature point in the second image is determined based on the target basis matrix and the first feature point; Calculate the first distance from the second feature point corresponding to the first feature point to the first epipolar line; If the first distance is less than the second value, then the first feature point and its corresponding second feature point are determined to be a correct matching point pair.

2. The method according to claim 1, characterized in that, The first feature point is transformed based on the average coordinate value of the first feature point to obtain the transformed feature point, and the first intermediate image data is generated based on the transformed feature point; the second feature point is transformed based on the average coordinate value of the feature point to obtain the transformed feature point, and the second intermediate image data is generated based on the transformed feature point. The target input image is obtained by merging the first intermediate image data with the second intermediate image, including: Using the average coordinates of the first feature point as the first origin, the first feature point is translated so that its centroid is located at the first origin; and using the average coordinates of the second feature point as the second origin, the second feature point is translated so that its centroid is located at the second origin. The translated first feature points are scaled until the distance from each first feature point to the first origin is a first value, and the translated second feature points are scaled until the distance from each second feature point to the second origin is the first value. Create a blank first mapping table and a blank second mapping table, wherein the size of the first mapping table and the second mapping table is the same as the size of the target input image; The scaled coordinate values ​​of the first feature point are stored in the first mapping table to obtain the third mapping table, and the scaled coordinate values ​​of the second feature point are stored in the second mapping table to obtain the fourth mapping table; The target input image is generated based on the third mapping table and the fourth mapping table.

3. The method according to any one of claims 1 to 2, characterized in that, The target model is trained through the following steps: Obtain a training set, wherein the training set includes multiple different sample matching point pair sets and the true basis matrix corresponding to each sample matching point pair set. Each sample matching point pair set includes multiple different sample matching point pairs, and the multiple different sample matching point pairs include: incorrect sample matching point pairs and correct sample matching point pairs. Each sample matching point pair includes: a first sample feature point belonging to a first sample image and a second sample feature point belonging to a second sample image. The sample matching point pairs of each sample matching point pair set are converted into images to obtain the sample input images corresponding to each sample matching point pair set; Construct the network to be trained and its corresponding target loss function; The sample input image is input into the network to be trained for processing to obtain the corresponding prediction basis matrix. Based on the true basis matrix, the prediction basis matrix and the target loss function, the parameters in the network to be trained are updated by backpropagation until the model converges to obtain the target model.

4. The method according to claim 3, characterized in that, The acquisition of the training set includes: A binocular simulation system is constructed, comprising: a virtual parallel binocular system and a real binocular system. The virtual parallel binocular system includes a first virtual master camera and a first virtual slave camera, wherein the optical axes of the first virtual master camera and the first virtual slave camera are parallel, their intrinsic parameter matrices are the same, and their distortions are both zero. The real binocular system includes a second virtual master camera and a second virtual slave camera, wherein the optical axes of the second virtual master camera and the second virtual slave camera are not parallel, their intrinsic parameter matrices are different, their distortions are the same, and their distortions are not zero. The intrinsic parameter matrices of the second virtual master camera and the first virtual master camera are the same. The binocular simulation system generates a set of matching point pairs for each sample and the corresponding real basis matrix.

5. The method according to claim 4, characterized in that, The step of generating each set of sample matching points and the corresponding true fundamental matrix through the binocular simulation system includes: The virtual parallel stereo system generates M parallel stereo matching point pairs for a set of images, wherein each of the parallel stereo matching point pairs satisfies the constraint that they are on the same row, and M≥8; Using the real binocular system, for each feature point corresponding to the first virtual main camera in the parallel binocular matching point pair, the distortion of the second virtual main camera is added; and for each feature point corresponding to the first virtual secondary camera in the parallel binocular matching point pair, the stereo correction inverse process of the second virtual secondary camera and the distortion of the second virtual secondary camera are sequentially performed to obtain M matching point pairs after distortion addition; Random noise is added to the M pairs of matched points after distortion, and random error is added to N pairs of matched points in the M pairs of matched points after adding random noise, to obtain a set of sample matched points, where N≥1; Based on the intrinsic parameter matrix of the first virtual main camera, the extrinsic parameter matrix from the first virtual main camera to the first virtual secondary camera, the intrinsic parameter matrix of the second virtual main camera, and the intrinsic parameter matrix of the second virtual secondary camera, calculate the true basis matrix corresponding to the sample matching point pair set.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1-5.

7. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1-5.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1-5.

Images