Image matching and its model training methods, autonomous driving equipment and its control methods and equipment

Abstract

This application provides an image matching and model training method, an autonomous driving device and its control method, and a device. The image matching method includes: determining the epipolar region corresponding to a target object in multiple images to be matched; generating an epipolar constraint mask for the multiple images to be matched based on the epipolar region; and performing local feature matching for the target object among the multiple images to be matched based on the epipolar constraint mask. This application makes the local feature matching process more efficient and the matching results more accurate.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to an image matching method, an autonomous driving device control method, an image matching model training method, an electronic device, an autonomous driving device, a computer storage medium, and a computer program product. Background Technology

[0002] Image matching is a technique that establishes spatial correspondences at the pixel, region, or feature level between two or more images, so that corresponding points / regions represent the same scene or have semantic consistency. It is the foundation of many 3D computer vision tasks. Among image matching techniques, local feature matching is one of the main implementation methods.

[0003] Local feature matching aims to extract and align feature representations with local supporting regions in two or more images to establish a correspondence across image space. Currently, local feature matching is mostly based on the overall feature maps of the corresponding images, resulting in high computational and storage costs; moreover, it is easily affected by interference from non-target objects such as backgrounds and dynamic objects, which affects the matching results. Summary of the Invention

[0004] In view of this, embodiments of this application provide an image matching and model training scheme to at least partially solve the above problems.

[0005] According to a first aspect of the embodiments of this application, an image matching method is provided, comprising: determining an epipolar band region corresponding to a target object in a plurality of images to be matched; generating an epipolar constraint mask for the plurality of images to be matched based on the epipolar band region; and performing local feature matching for the target object among the plurality of images to be matched based on the epipolar constraint mask.

[0006] According to a second aspect of the embodiments of this application, an autonomous driving device control method is provided, comprising: acquiring multiple images of the environment in which the autonomous driving device is located through multiple image acquisition devices installed on the autonomous driving device; sending the multiple images to an autonomous driving server for local feature matching, wherein the autonomous driving server performs local feature matching based on the image matching method described in the first aspect, and feeds back the results of the local feature matching to the autonomous driving device; and executing a preset autonomous driving task based on the received results of the local feature matching.

[0007] According to a third aspect of the embodiments of this application, an image matching model training method is provided, comprising: obtaining training samples, the training samples including multiple sample images and ground truth values ​​of matching relationships between the multiple sample images; determining the epipolar region corresponding to the sample object in the multiple sample images; generating an epipolar constraint mask for the multiple sample images based on the epipolar region using the image matching model to be trained; and performing local feature matching for the sample object between the multiple sample images based on the epipolar constraint mask; and training the image matching model based on the prediction results of the local feature matching and the ground truth values ​​of the matching relationships.

[0008] According to a fourth aspect of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described in the first or third aspect.

[0009] According to a fifth aspect of the embodiments of this application, an autonomous driving device is provided, comprising: a perception system including multiple image acquisition devices for acquiring images of the environment in which the autonomous driving device is located; a driving system including a steering subsystem, an acceleration subsystem, and a deceleration subsystem for controlling the driving behavior of the autonomous driving device; a communication system including a communication interface and a communication bus, wherein the communication interface is used to communicate with an autonomous driving server; and a control system including one or more control chips, wherein the control system is communicatively connected to the perception system and the driving system via the communication bus, and the control system is used to execute the autonomous driving device control method as described in the second aspect to obtain the result of local feature matching; and, based on the result of local feature matching, to send task instructions to the driving system to control the driving behavior of the autonomous driving device through the driving system.

[0010] According to a sixth aspect of the embodiments of this application, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first or third aspect.

[0011] According to a seventh aspect of the embodiments of this application, a computer program product is provided, including computer instructions that instruct a computing device to perform an operation corresponding to the method described in the first or third aspect.

[0012] According to the scheme provided in the embodiments of this application, during image matching, on the one hand, the epipolar region corresponding to the target object in multiple images to be matched is determined, and an epipolar constraint mask is generated for the multiple images to be matched based on the epipolar region. This epipolar constraint mask can provide important spatial geometric judgment basis for subsequent determination of the feature search space, making the candidate matching region determined for the target object more accurate and focused. On the other hand, for the local feature matching process, based on the epipolar constraint mask, local feature matching for the target object is performed among multiple images to be matched. Thus, when the system performs feature matching calculation, it can avoid indiscriminate global search based on the entire feature map, significantly reducing computation and storage overhead, and can fully shield the region outside the epipolar region by utilizing the epipolar constraint mask, making the matching calculation less susceptible to interference from non-target objects such as background and dynamic objects. Therefore, the establishment of image spatial correspondence can be guided by geometric constraints, focusing on the effective region corresponding to the epipolar constraint mask, accurately establishing the correspondence for the target object, making the local feature matching process more efficient and the matching result more accurate. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0014] Figure 1 A schematic diagram of an exemplary system to which the embodiments of this application are applicable; Figure 2 This is a flowchart illustrating the steps of an image matching method according to an embodiment of this application; Figure 3 This is a flowchart illustrating the steps of another image matching method according to an embodiment of this application; Figure 4A This is a flowchart illustrating the steps of a training method for an image matching model according to an embodiment of this application. Figure 4B for Figure 4A A schematic diagram of the structure of an image matching model in the embodiment shown; Figure 5 For based on Figure 4B The diagram illustrates an example of how an image matching model implements an image matching method. Figure 6 This is a flowchart illustrating the steps of an autonomous driving device control method according to an embodiment of this application; Figure 7This is a schematic diagram of the structure of an autonomous driving device according to an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0015] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0016] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0017] Figure 1 An exemplary system applicable to embodiments of this application is shown. For example... Figure 1 As shown, the system 100 may include a server 102, a communication network 104, and one or more autonomous driving devices 106. Figure 1 The example in the text shows multiple autonomous driving devices.

[0018] Server 102 can be any suitable device for storing information, data, programs, and any other suitable types of content, including but not limited to distributed storage system devices, server clusters, computing server clusters, etc. In some embodiments, server 102 can perform any suitable function. For example, in some embodiments, server 102 can be used to perform image matching tasks. As an optional example, in some embodiments, server 102 can be used to receive multiple images to be matched from autonomous driving device 106, determine the epipolar band region corresponding to the target object in the multiple images to be matched; generate an epipolar constraint mask for the multiple images to be matched based on the epipolar band region; and perform local feature matching for the target object among the multiple images to be matched based on the epipolar constraint mask. As an optional example, in other embodiments, server 102 can be used to receive imaging geometric parameters corresponding to the multiple images to be matched while receiving multiple images to be matched from autonomous driving device 106, and guide the determination of the epipolar band region and the entire image matching method based on the imaging geometric parameters. As an optional example, in some embodiments, the server 102 may also return the results of local feature matching to the autonomous driving device 106 for the autonomous driving device 106 to perform subsequent tasks, such as perception decision-making, 3D reconstruction, and visual positioning in autonomous driving. As an optional example, in other embodiments, the server 102 may also deploy an image matching model to perform the aforementioned image matching tasks based on the image matching model. As an optional example, in some embodiments, the server 102 may also be used to train the image matching model.

[0019] In some embodiments, the communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The autonomous driving device 106 can be connected to the communication network 104 via one or more communication links (e.g., communication link 112), and the communication network 104 can be linked to the server 102 via one or more communication links (e.g., communication link 114). The communication link can be any communication link suitable for transmitting data between the autonomous driving device 106 and the server 102, such as a network link, a dial-up link, a wireless link, a hardwired link, any other suitable communication link, or any suitable combination of such links.

[0020] The autonomous driving device 106 may include any device with autonomous driving capabilities, such as a device with semi-autonomous or partially autonomous driving capabilities, or a device with fully autonomous driving capabilities. Furthermore, in this embodiment, the autonomous driving device is not limited to land-based devices such as autonomous vehicles, but may also be water-based devices such as autonomous ships, or air-based devices such as autonomous aircraft.

[0021] In the autonomous driving scenario, the autonomous driving device 106 is equipped with a perception system, which includes at least multiple image acquisition devices (such as vehicle-mounted cameras, drone cameras, etc.). The images acquired by the image acquisition devices are uploaded to the server 102 via the communication network 104. The server 102 performs image matching and feeds back the matching results to the autonomous driving device 106 to support subsequent driving behavior processing.

[0022] Based on the above system, this application provides an image matching scheme, which will be described below through several embodiments.

[0023] Reference Figure 2 This document illustrates a flowchart of an image matching method according to an embodiment of the present application. The image matching method of this embodiment includes the following steps: Step S202: Determine the epipolar region corresponding to the target object in multiple images to be matched.

[0024] In autonomous driving scenarios, the ability to perceive the environment is fundamental for autonomous driving devices to make correct decisions and drive safely. Autonomous driving devices need to understand the three-dimensional spatial structure of their surroundings in real time and accurately while traveling at high speeds to support autonomous driving tasks such as path planning, obstacle avoidance, and precise positioning. To achieve three-dimensional environment perception, autonomous driving devices typically carry multiple image acquisition devices that continuously capture images of the environment from different perspectives, generating a large number of multi-view images. Image matching is a crucial means of accurately perceiving the three-dimensional environment by extracting reliable spatial correspondences from these multi-view images. Image matching aims to establish pixel-level, region-level, or feature-level spatial correspondences between two or more multi-view images taken at the same time, enabling corresponding points or regions in different images to be identified as the same physical location in the three-dimensional environment. This provides crucial geometric input for downstream tasks such as 3D reconstruction, visual odometry, and visual positioning. In this embodiment, image matching is achieved through local feature matching. However, existing local feature matching methods are mostly based on searching the entire feature map of an image, which not only generates a lot of unnecessary computational overhead and occupies a lot of storage resources, but is also easily affected by interference factors (such as cluttered backgrounds, interference from moving objects, etc.), leading to a decrease in matching accuracy.

[0025] Therefore, in this embodiment of the application, the epipolar region corresponding to the target object in multiple images to be matched is first determined, which serves as the geometric basis for subsequent processing.

[0026] In this application, the "target object" can be a pixel to be matched or a pixel region to be matched. As mentioned above, "multiple images to be matched" can be multiple images from different perspectives captured simultaneously by the image acquisition device in the autonomous driving device. In this application, unless otherwise specified, "multiple" means two or more. Furthermore, in several embodiments of this application, the target object is exemplified by the pixel to be matched; however, those skilled in the art should understand that the pixel region to be matched can be referenced to the pixel and, in an appropriate form (including but not limited to descriptors), image matching can be achieved by mimicking the method in this application.

[0027] After obtaining multiple images to be matched, the epipolar band region corresponding to the target object in each of the multiple images can be determined. In the epipolar geometry framework, for images (reference image and target image) acquired by two cameras with known relative poses, if a scene point is observed in the reference image, its corresponding point in the target image is not uniquely determined due to its unknown depth in three-dimensional space. Instead, it is distributed along a straight line on the target image, which is the epipolar line of the scene point. Conversely, any point in the target image also has a corresponding point in the reference image distributed along an epipolar line on the reference image. However, when there are errors or noise in the camera pose, the true corresponding point may deviate from the ideal epipolar line. Therefore, in this embodiment, the range corresponding to the scene point is expanded to a band-shaped region with the epipolar line as the central axis and the epipolar line bandwidth threshold as half-width, i.e., the epipolar band region, which is the set of all pixels whose geometric distance to the epipolar line is less than the epipolar line bandwidth threshold.

[0028] For two or more images captured by two or more cameras, optionally, any two images can be combined into an image pair, thereby forming multiple image pairs. After determining the epipolar band region corresponding to each image pair, these epipolar band regions can be merged to form the epipolar band region corresponding to the target object in the two or more images. However, this is not limited to this; alternatively, the epipolar band regions corresponding to one image on other images can be merged with the epipolar band regions corresponding to that image on other images to form the epipolar band region corresponding to the target object in the two or more images. In practical applications, those skilled in the art can also use other methods to determine the epipolar band region between two or more images. The embodiments of this application do not limit the specific method of determining the epipolar band region between two or more images.

[0029] In one alternative approach, determining the epipolar region corresponding to the target object in multiple images to be matched can be achieved by: determining the epipolar line corresponding to the target object based on the fundamental matrix corresponding to the multiple images to be matched; and determining the epipolar region corresponding to the target object in the multiple images to be matched based on the epipolar line and the obtained epipolar bandwidth threshold.

[0030] The fundamental matrix is ​​an algebraic expression of the polar geometry. For the standard pinhole camera model, it can be obtained from the camera intrinsic parameter matrix and the relative pose (rotation matrix R and translation vector T):

[0031] in, Represents the fundamental matrix. and Let represent the intrinsic parameter matrices of the reference camera corresponding to the reference image and the target camera corresponding to the target image, respectively. and This represents the relative rotation matrix and translation vector from the reference camera to the target camera. Represents the translation vector The corresponding antisymmetric matrix (cross product matrix).

[0032] Based on this, taking the target object as a pixel as an example, for a certain pixel in the reference image... Based on the aforementioned fundamental matrix, its epipolar line on the target image can be calculated, i.e.:

[0033] When the fundamental matrix between the reference image and the target image is At that time, the above That is .

[0034] Furthermore, the target image satisfies the epipolar line. All pixels whose geometric distance is less than the epipolar bandwidth threshold s are considered to form the aforementioned pixels. The corresponding polar zone region.

[0035] However, in this embodiment, the imaging geometric parameters are not limited to the rotation matrix and translation vector under the pinhole imaging model described above. For non-standard imaging models such as rational polynomial coefficients (RPCs) commonly used in satellite remote sensing images, equivalent epipolar relationships can be obtained through local linearization or numerical approximation methods, thereby applying this scheme to a wider range of imaging scenarios. Furthermore, for fisheye or wide-angle lenses, the epipolar lines can be extended to more general curve geometric constraints, while the generation principle of the epipolar band region remains unchanged. Those skilled in the art can flexibly select appropriate imaging geometric models according to actual needs, and this embodiment does not impose any limitations on this.

[0036] Based on this, in an optional embodiment of this application, before determining the epipolar line corresponding to the target object based on the fundamental matrices corresponding to multiple images to be matched, imaging geometric parameters corresponding to each of the multiple images to be matched can be obtained; based on the imaging geometric parameters, the fundamental matrices corresponding to the multiple images to be matched are obtained. The imaging geometric parameters may include, but are not limited to, camera intrinsic parameter matrices, relative rotation matrices and translation vectors between cameras, or other forms of equivalent imaging geometric descriptions. By first obtaining the imaging geometric parameters and then obtaining the fundamental matrices based on them, the solution of this application embodiment can flexibly adapt to camera pose information from different sources, including but not limited to visual odometry, inertial measurement unit (IMU) estimation, offline calibration results, etc. As long as some form of imaging geometric prior is provided, the subsequent epipolar region calculation process can be driven.

[0037] Furthermore, as mentioned earlier, the determination of the epipolar band region also needs to be based on the epipolar bandwidth threshold. The initial epipolar bandwidth threshold can be adaptively set according to the reliability of the camera pose prior. The less reliable the pose prior, the larger the initial epipolar bandwidth threshold can be set to tolerate the error of the prior pose and fully encompass the true corresponding points. For example, the initial epipolar bandwidth threshold... The initial epipolar bandwidth threshold can be set based on the signal-to-noise ratio estimation of the pose or empirically, optionally when the image is downsampled by 1 / 8. It can be set to 2-5 pixel units. Compared with the traditional method where the camera pose prior is used as a rigid geometric constraint for the entire image matching process, which is prone to matching errors, the solution in this application embodiment only uses it as an initial value and will be continuously updated thereafter. Therefore, based on the camera pose update, both the calculation of the epipolar bandwidth threshold and the determination of the fundamental matrix will also be continuously updated. The specific update method will be detailed below.

[0038] This step identifies geometrically feasible candidate matching regions for the target object in each image to be matched, based on the epipolar band region, providing a geometric basis for subsequent processing.

[0039] Step S204: Generate an epipolar constraint mask for multiple images to be matched based on the epipolar band region.

[0040] After determining the epipolar region corresponding to the target object in multiple images to be matched, the epipolar region can be further transformed into a structured representation that can be directly applied to local feature matching, namely, an epipolar constraint mask.

[0041] A epipolar constraint mask is a binary matrix used to explicitly label which pairs of pixel positions in the feature maps of two images have a geometrically feasible correspondence. In this embodiment, the epipolar constraint mask is generated based on epipolar band regions. For any pair of pixel positions (i, j) in an image pair, if pixel j falls within the epipolar band region corresponding to pixel i, the mask value at that position is 1, indicating that subsequent processing such as attention calculation is allowed between these positions; otherwise, the value is 0, indicating that subsequent processing between these positions is blocked. Formally, the epipolar constraint mask can be represented as a sparse binary matrix M. mask ∈{0,1} HхW , where H and W represent the height and width of the feature map, respectively.

[0042] By using this epipolar constraint mask, local feature matching can be limited to the region in the feature map corresponding to the epipolar constraint mask, while other irrelevant locations are masked, thereby achieving more accurate and efficient feature matching.

[0043] In one alternative approach, generating an epipolar constraint mask for multiple images to be matched can be achieved by: obtaining multiple epipolar band regions corresponding to the multiple images to be matched; merging the multiple epipolar band regions; and generating a unified epipolar constraint mask for the multiple images to be matched based on the merging result.

[0044] In real-world image matching scenarios, epipolar constraints between two images are bidirectional: each pixel in the reference image has a corresponding epipolar line in the target image, and vice versa. Using only unidirectional epipolar constraints may result in incomplete epipolar constraint mask coverage, missing some valid corresponding regions. Therefore, in this embodiment, by constructing and merging bidirectional epipolar band regions, a more complete and symmetrical epipolar constraint mask can be generated, ensuring that valid corresponding regions in both directions are included within the mask's effective range, thereby improving the mask's coverage integrity.

[0045] Alternatively, multiple epipolar regions can be merged, and a unified epipolar constraint mask can be generated for multiple images to be matched based on the merging result. This can be achieved using one of the following two methods or a combination thereof: The first approach involves generating multiple region masks for each epipolar region, then merging these masks (e.g., using a bitwise OR operation) to generate a unified epipolar constraint mask based on the merged result. This method allows for the parallel generation of epipolar regions in all directions, improving the efficiency of epipolar region generation.

[0046] The second approach involves first merging pixel regions from multiple epipolar regions (e.g., taking the union of pixel sets), and then generating a mask for the merged pixel region as a unified epipolar constraint mask for multiple images to be matched. This method allows merging to be completed in the spatial domain, avoiding the storage overhead of multiple mask matrices.

[0047] In practical applications, those skilled in the art can flexibly choose the above methods according to their needs and computing resource conditions, and the embodiments of this application do not impose any restrictions on this.

[0048] This step transforms the epipolar geometric constraints from geometric space into a mask representation that can be directly used in the local feature matching process, providing accurate geometric guidance information for subsequent feature matching processing.

[0049] Step S206: Based on the epipolar constraint mask, perform local feature matching for the target object among multiple images to be matched.

[0050] After obtaining the epipolar constraint mask, the mask can be used as a constraint to perform local feature matching for the target object among multiple images to be matched.

[0051] In this embodiment of the application, local feature matching may include three stages: feature extraction, feature interaction, and feature matching.

[0052] in: (1) Feature extraction stage In this stage, image features are extracted from multiple images to be matched to obtain corresponding feature maps. Each location in the feature map corresponds to a local receptive field in the image to be matched, and its feature vector encodes the local visual information at that location.

[0053] In one alternative approach, multi-scale feature extraction can be performed on multiple images to be matched, resulting in corresponding multi-scale feature maps. Among these multi-scale features, low-resolution coarse-scale feature maps (such as those extracted at downsampling rates of 1 / 32, 1 / 16, or 1 / 8) have a large receptive field, capable of capturing global structural and semantic information, and are suitable for preliminary coarse-grained feature matching. High-resolution fine-scale feature maps (such as those extracted at downsampling rates of 1 / 4 or 1 / 2) have finer spatial resolution, suitable for pixel-level fine-grained localization based on coarse-grained feature matching results, and are suitable for fine-grained feature matching. Through the coordinated use of multi-scale features, a progressive feature matching from coarse to fine can be achieved, balancing matching accuracy and computational efficiency.

[0054] Optionally, a feature extraction component can be built based on the MambaVision architecture for image feature extraction. MambaVision is a hybrid architecture combining the Mamba state-space model and the visual Transformer concept. It handles long-range dependencies through a linearly complex state-space model while retaining the efficient local feature extraction capability of convolutional neural networks. Compared to traditional convolutional backbone networks (such as ResNet), MambaVision achieves a better balance between computational efficiency and feature representation capability; compared to a pure Transformer structure, its linearly complex sequence modeling mechanism has better computational efficiency when processing high-resolution images. However, in practical applications, those skilled in the art can also choose other model structures to implement the feature extraction component according to actual needs and hardware performance, including but not limited to: convolutional neural networks (CNN), visual Transformers (ViT), Swing Transformers, or CNN-Transformer hybrid structures.

[0055] (2) Feature interaction stage After the feature extraction stage is completed, the feature maps of each image to be matched are in an independent state—the features of each image only encode the local visual information of that image itself, without containing any contextual information from other images. In order for the features of different images to perceive and enhance each other, feature interaction is required to provide a more discriminative feature representation for the subsequent establishment of cross-image correspondences.

[0056] In this stage of feature interaction, information is passed between feature maps of multiple images to be matched, enabling features at each location in each image to update and enhance their representations by referencing information from features at related locations in other images. This establishes cross-image context awareness, allowing each feature location to incorporate information from potential corresponding regions in other images, improving cross-viewpoint consistency. Furthermore, information fusion suppresses ambiguity; in areas with repetitive textures, weak textures, or significant viewpoint differences, cross-image contextual information eliminates the ambiguity of single-image features, making the similarity of true corresponding point pairs in the feature space more prominent, while suppressing the similarity of non-corresponding point pairs.

[0057] In one feasible implementation of feature interaction, an attention mechanism can be employed. For example, linear attention, bidirectional sparse attention, quadtree attention, etc., can be selected, still using an epipolar constraint mask to prune attention connections. However, in this embodiment, to better facilitate feature interaction and provide a more accurate data foundation for subsequent processing, a clustered attention mechanism under an epipolar constraint mask is adopted. Based on this, in one optional approach, local feature matching for the target object is performed between multiple images to be matched, based on the epipolar constraint mask. This includes: for the feature maps of multiple images to be matched, performing focused attention calculation in the mask region corresponding to the epipolar constraint mask to perform local feature matching for the target object. In this way, based on attention mechanisms such as multi-head cross-attention, the introduction of an epipolar constraint mask limits the spatial range of information transmission, allowing cross-image feature interaction to focus on geometrically feasible epipolar band regions. This achieves the basic function of feature interaction while further improving computational efficiency and reducing interference from non-target regions.

[0058] Optionally, when performing focused attention calculation on the feature maps of multiple images to be matched, within the mask region corresponding to the epipolar constraint mask, this can include: during the cross-attention calculation of the feature maps of multiple images to be matched, assigning weight values ​​approaching 0 to the regions outside the mask region corresponding to the epipolar constraint mask, so as to perform focused attention calculation on the mask region. This effectively achieves the transition from cross-attention to focused attention, improving the efficiency of attention calculation.

[0059] In one example, an epipolar constraint mask is introduced in cross-attention, including: using features of the reference image as the query and features of the target image as the key / value (or vice versa); for the location masked by the epipolar constraint mask (i.e., M... mask When (i,j)=0, a minimum value or -∞ is assigned when calculating attention logits, making the corresponding attention weights close to 0 after softmax. Then, after softmax normalization, attention is mainly focused on the geometrically feasible region, i.e., the candidate matching region falling within the epipolar band region. Through the epipolar constraint mask, cross-attention reduces the feature search range from the global feature map to a narrower geometric region, i.e., the epipolar band region, achieving "focused attention" under the "epochal band region" constraint. Further, alternatively, standard attention can be used on coarse-scale (e.g., 1 / 32 downsampling) feature maps, while linear attention or sparse attention can be used on fine-scale (e.g., 1 / 8 downsampling) maps to balance performance and efficiency.

[0060] (3) Feature matching stage After the focused attention calculation, the process moves to the feature matching stage. In this stage, matching results can be generated based on the features obtained after the focused attention calculation by calculating similarity, such as a similarity matrix, and selecting high-confidence point pairs.

[0061] In one alternative embodiment, the feature matching in this application includes sequentially performing coarse-grained feature matching for the target object among multiple images to be matched, and fine-grained feature matching based on the results of the coarse-grained feature matching. Through the synergistic effect of two-stage feature matching with different granularities, fine-grained matching, guided by the results of coarse-grained matching, can perform accurate matching only in the local area guided by the coarse-grained matching, thereby improving matching efficiency while enhancing overall matching accuracy and avoiding inference delay.

[0062] The following section provides a further explanation of the two-stage feature matching process.

[0063] (A) Coarse-grained feature matching stage Taking a 1 / 8 scale feature map as an example, the feature maps of the reference image and the target image are flattened into a two-dimensional sequence; then, the point-to-point similarity matrix is ​​calculated. For example, this can be expressed as a vector inner product multiplied by a temperature coefficient, which can be represented as:

[0064] in, Indicates the temperature coefficient. This represents the vector dot product operation. Indicates a point pair. These represent the first and second feature maps at a 1 / 8 scale of the reference image, respectively. Point, and the first The first point in the 1 / 8 scale feature map of the target image corresponding to the point point.

[0065] Furthermore, regarding the similarity matrix The rows and columns are respectively subjected to softmax normalization to obtain the bidirectional normalized matching probability matrix M, which can be expressed as:

[0066] Then, according to The size of the pair and whether it satisfies the Mutual Nearest Neighbor (MNN) condition are used to select the set of point pairs with higher confidence as the matching result of coarse-grained feature matching.

[0067] Since the epipolar constraint mask has eliminated most geometrically impossible point pairs, the resulting coarse-grained feature matching has high reliability in terms of geometric consistency.

[0068] Coarse-grained feature matching can yield preliminary pixel-level matching point pairs from downsampled feature maps at a coarse scale, such as the 1 / 8 scale mentioned above. However, the spatial positioning accuracy of coarse-grained feature matching is limited by the downsampling rate of the feature map—for example, with a 1 / 8 downsampling rate, the positioning error of the matching point pairs obtained by coarse-grained feature matching in the original image coordinate system can reach several pixels. This is insufficient for downstream tasks requiring high-precision spatial correspondences (such as visual odometry, 3D reconstruction, and precise calibration). Therefore, based on the matching results of coarse-grained feature matching, fine-grained feature matching can be further performed to improve matching and positioning accuracy.

[0069] (B) Fine-grained feature matching stage Since the matching point pairs obtained through coarse-grained feature matching have undergone geometric screening by the epipolar constraint mask and have high geometric consistency, fine-grained feature matching can be performed based on the matching results of coarse-grained feature matching without having to search again in the global feature map. This can effectively reduce the amount of computation and make full use of the detailed information of higher resolution feature maps to improve spatial positioning accuracy.

[0070] As mentioned earlier, since the matching range has been basically determined by coarse-grained feature matching, fine-grained feature matching can directly use this range as a benchmark to perform further precise matching within this range.

[0071] In this stage, fine-grained feature matching is performed directly based on the fine-scale feature map (e.g., the 1 / 2 scale feature map) from the aforementioned multi-scale feature maps, combined with the matching results of the coarse-grained feature matching. The fine-scale feature map, such as the 1 / 2 scale feature map, has high spatial resolution, preserving finer details such as local textures and edges in the image. The matching point pairs obtained from the coarse-grained feature matching provide the central location for the local search in fine-grained feature matching. Based on this, fine-grained feature matching uses the location indicated by the matching point pairs obtained from the coarse-grained feature matching as the center, and crops the corresponding local feature blocks on the fine-scale feature map (e.g., the 1 / 2 scale feature map), thus constraining the search range of fine-grained feature matching within the range of the aforementioned matching point pairs. This collaborative division of labor—coarse-grained feature matching locating candidate matching point pairs and fine-grained feature matching refining the corresponding locations—forms an effective complementarity in feature matching: coarse-grained feature matching guides the establishment of a preliminary, rough pixel-level correspondence globally using geometric constraints, while fine-grained feature matching achieves precise pixel-level localization within the local neighborhood defined by coarse-grained feature matching using high-resolution features.

[0072] In one example, the matching point pairs M obtained for each coarse-grained feature match c Using the matching point pair as the center, dense correlation calculations can be performed on the local feature blocks of the reference image and the target image cropped from the 1 / 2 scale feature map to obtain the local feature block correlation score matrix S. l In S l Perform a mutual nearest neighbor search and select the pixel location pair that satisfies the bidirectional nearest neighbor condition and has the highest relevance score as the pixel-level fine-grained feature matching result. Optionally, to control the overall number of output matching point pairs, the relevance scores can be sorted, and the matching result with the higher score, such as the first-ranked match, can be retained for each coarse-grained feature matching point pair.

[0073] Furthermore, correlation calculations can be performed within the smallest neighborhood of the matching result (such as within a 3×3 pixel block centered on the matching result) to obtain a fine-grained correlation distribution. The desired coordinates can then be calculated on this fine-grained correlation distribution to obtain a set of pixel-level matching point pairs, which serves as the matching result of the fine-grained feature matching.

[0074] However, this is not the only option. In practical applications, those skilled in the art may also choose other methods that use the matching results of coarse-grained feature matching as a guide and fine-grained feature maps as a basis for fine-grained feature matching. All of these methods are applicable to the embodiments of this application.

[0075] In this embodiment, during image matching, on the one hand, the epipolar region corresponding to the target object in multiple images to be matched is determined, and an epipolar constraint mask is generated for the multiple images to be matched based on this epipolar region. This epipolar constraint mask provides important spatial geometric criteria for subsequent determination of the feature search space, making the candidate matching regions determined for the target object more accurate and focused. On the other hand, for the local feature matching process, based on the epipolar constraint mask, local feature matching for the target object is performed among multiple images to be matched. Thus, when the system performs feature matching calculations, it can avoid indiscriminate global search based on the entire feature map, significantly reducing computational and storage overhead, and can also fully shield areas outside the epipolar region using the epipolar constraint mask, making the matching calculation less susceptible to interference from non-target objects such as backgrounds and dynamic objects. Therefore, the establishment of image spatial correspondences can be guided by geometric constraints, focusing on the effective region corresponding to the epipolar constraint mask, accurately establishing the correspondence for the target object, making the local feature matching process more efficient and the matching results more accurate.

[0076] Reference Figure 3 This document illustrates a flowchart of another image matching method according to an embodiment of the present application. The image matching method of this embodiment includes the following steps: Step S302: Determine the epipolar region corresponding to the target object in multiple images to be matched.

[0077] Step S304: Generate an epipolar constraint mask for multiple images to be matched based on the epipolar band region.

[0078] The specific implementation of steps S302-S304 can be referred to the aforementioned steps S202-S204, and will not be repeated here.

[0079] Step S306: Based on the epipolar constraint mask, perform coarse-grained feature matching for the target object among multiple images to be matched.

[0080] The specific implementation of this step can be referred to the description of the coarse-grained feature matching stage in step S206 above, and will not be repeated here.

[0081] Through steps S302-S306, a coarse-grained feature matching result can be obtained, and this result can be directly used for subsequent fine-grained feature matching. However, in actual image matching scenarios, the imaging geometric parameters upon which coarse-grained feature matching depends, and the epipolar band region that provides the data basis for the epipolar constraint mask, both have areas for further improvement. For example, imaging geometric parameters such as camera pose priors often inevitably contain noise or errors, leading to deviations in the calculation of the fundamental matrix, which in turn affects the accuracy of the epipolar band region, further impacting the accuracy of the epipolar constraint mask and even feature matching. To obtain high-precision matching results even when initial imaging geometric parameters such as camera pose priors are uncertain, the embodiments of this application, in the case of uncertainties, such as... Figure 2 Based on the illustrated embodiment, a scheme for iteratively correcting imaging geometric parameters and epipolar bandwidth thresholds based on coarse-grained feature matching results is further provided, including the following steps.

[0082] Step S308: Update the imaging geometric parameters based on the results of coarse-grained feature matching, and update the fundamental matrix based on the updated imaging geometric parameters; and update the epipolar region and even the epipolar constraint mask based on the updated fundamental matrix.

[0083] As previously described, step S302, determining the epipolar region corresponding to the target object in multiple images to be matched, may include: determining the epipolar line corresponding to the target object based on the fundamental matrix corresponding to the multiple images to be matched; and determining the epipolar region corresponding to the target object in the multiple images to be matched based on the epipolar line and the obtained epipolar bandwidth threshold. Specifically, before determining the epipolar line corresponding to the target object based on the fundamental matrix corresponding to the multiple images to be matched, the following may also be performed: obtaining the imaging geometric parameters corresponding to each of the multiple images to be matched; and obtaining the fundamental matrix corresponding to the multiple images to be matched based on the imaging geometric parameters.

[0084] Because feature matching (including coarse-grained and fine-grained feature matching) is performed iteratively, in this embodiment, each iteration of coarse-grained feature matching corrects the imaging geometric parameters and epipolar bandwidth threshold. As feature matching iterates, the correction of geometric parameters and epipolar bandwidth threshold also iterates until the number of iterations reaches a preset number; or, the imaging geometric parameters, such as camera pose, reach a preset parameter threshold; or, the epipolar bandwidth threshold reaches a preset bandwidth threshold; or, the feature matching result meets a preset standard, and so on. Through successive iterations, the imaging geometric parameters are continuously optimized based on the matching results of coarse-grained feature matching, progressively narrowing the epipolar bandwidth threshold and refining the fundamental matrix, thereby forming a mutually reinforcing positive feedback loop between geometric constraints and feature matching, achieving a gradual optimization from coarse to fine.

[0085] In one alternative approach, the imaging geometry parameters are updated based on the results of coarse-grained feature matching, which can be achieved through one or a combination of the following two methods: Method 1: Direct optimization based on epipolar error That is, based on the matching results of coarse-grained feature matching and the current base matrix, the corresponding epipolar error is obtained; based on the epipolar error, the imaging geometric parameters are updated. In this way, by directly utilizing the matching results of coarse-grained feature matching, the full utilization of existing data and data processing procedures can be effectively achieved, while reducing the complexity of the solution implementation.

[0086] For example, for each pair of matching points obtained from coarse-grained feature matching Calculate the epipolar error based on the current fundamental matrix F. It can be represented as:

[0087] in, and are the homogeneous coordinates of the k-th matching point in the reference image and the target image, respectively, where | represents the absolute value.

[0088] Polar error It measures the algebraic deviation between the matching point pair and the current epipolar constraint mask. The smaller the value, the more consistent the matching point pair is with the epipolar geometry described by the current fundamental matrix.

[0089] Based on this, an optimization problem with epipolar error as the objective function can be constructed based on the epipolar error of the matching point pairs obtained by matching all coarse-grained features. Imaging geometric parameters such as camera relative pose (R, T) can be updated through gradient descent or nonlinear least squares methods.

[0090] Furthermore, in the next iteration, the fundamental matrix can be calculated based on the updated camera relative pose (R, T), thereby realizing the correction and update of the fundamental matrix, and thus realizing the correction and update of subsequent processing.

[0091] Method 2: Re-estimate the fundamental matrix and then decompose. That is, the fundamental matrix is ​​re-estimated based on the matching results of coarse-grained feature matching; and the imaging geometric parameters are updated based on the re-estimated fundamental matrix. In this way, a reasonable balance can be achieved between robustness and computational efficiency.

[0092] For example, a new fundamental matrix F' can be re-estimated from the matching point pairs obtained from the current coarse-grained feature matching using RANSAC (random sample consensus) or differentiable estimation methods. Then, a new relative pose (R', T') can be obtained from F' by methods such as singular value decomposition (SVD), and the imaging geometric parameters can be updated accordingly.

[0093] The RANSAC method, through multiple rounds of random sampling and geometric consistency checks, can robustly estimate the fundamental matrix from noisy matching pairs and exhibits good resistance to outlier matching points. However, it is not limited to this; those skilled in the art can choose other methods to re-estimate the fundamental matrix according to actual needs, and the embodiments of this application do not impose such limitations.

[0094] Further optionally, after obtaining the epipolar error, the following can also be done: update the epipolar bandwidth threshold based on the epipolar error, and based on the updated epipolar bandwidth threshold, return the operation of determining the corresponding epipolar band region of the target object in multiple images to be matched based on the epipolar line and the obtained epipolar bandwidth threshold, so as to update the epipolar band region.

[0095] For example, the average epipolar error of all matching point pairs obtained from coarse feature matching in the current iteration round can be used as a reference. The epipolar bandwidth threshold is updated according to the following formula:

[0096] in, The epipolar bandwidth threshold used in the current iteration. The updated epipolar bandwidth threshold. A preset hyperparameter for controlling the epipolar bandwidth threshold shrinkage rate.

[0097] As can be seen from the above, when the average polar error A large value indicates that the current camera pose prior still has a significant deviation, and the epipolar bandwidth threshold should be kept relatively wide to encompass the true corresponding points; when the average epipolar error... As the threshold gradually decreases, it indicates that the current fundamental matrix is ​​already quite accurate, and the epipolar bandwidth threshold can be reduced accordingly. This allows for more focused feature searching in subsequent iterations, further improving matching accuracy. (Mean Epipolar Error) The smaller the value, the faster the epipolar bandwidth threshold shrinks. Therefore, the epipolar bandwidth threshold gradually tightens from wide to narrow during the iteration process, making the geometric constraints gradually more stringent as the pose estimation is optimized, thus achieving a precise iterative effect from coarse to fine.

[0098] Through the above process, the epipolar region calculation is driven by initial imaging geometric parameters such as camera pose prior, and the first round of coarse-grained feature matching is completed under the constraint of a relatively wide epipolar constraint mask. The matching results of the coarse-grained feature matching are used to estimate more accurate imaging geometric parameters such as camera pose, and the fundamental matrix is ​​updated. A more compact epipolar constraint mask is regenerated using the updated fundamental matrix and the shrunken epipolar bandwidth threshold. Focusing attention calculation is performed again under the constraint of the new epipolar constraint mask to obtain a more accurate second round of matching results. This process is repeated iteratively, improving the matching accuracy and geometric consistency round by round until convergence. Therefore, the scheme of this embodiment has good robustness and self-correction capability against fuzzy pose priors containing errors and noise. That is, the system does not require the pose prior to be completely accurate, but gradually corrects it through multiple rounds of iteration, so that the final matching result can still maintain high geometric accuracy even when there are large deviations in the pose prior. At the same time, the adaptive shrinking mechanism of the epipolar bandwidth threshold gradually reduces the computational load of each iteration as the pose accuracy improves, balancing accuracy and efficiency.

[0099] Step S310: Based on the updated epipolar constraint mask, the results of coarse-grained feature matching, and the fine-scale feature map, perform fine-grained feature matching for the target object.

[0100] and Figure 2 Unlike the illustrated embodiment, in this embodiment, the updated epipolar constraint mask is also considered during fine-grained feature matching. Introducing the updated epipolar constraint mask into fine-grained feature matching allows for further filtering of candidate matching positions using the latest geometric constraint information during the precise matching process of high-resolution local feature blocks. This filters out candidate positions that are geometrically infeasible under the constraints of the current refined epipolar constraint mask, thereby further improving the geometric consistency and matching reliability of the fine-grained feature matching results. In practical applications, those skilled in the art can flexibly decide whether to introduce the updated epipolar constraint mask into the fine-grained feature matching stage based on the credibility of the pose prior and the iterative convergence state in the actual scenario; this embodiment does not impose such limitations.

[0101] According to this embodiment, the two-stage feature matching method effectively improves the positioning accuracy of matching point pairs, providing a more reliable data foundation for downstream tasks requiring high-precision correspondence. Furthermore, with the introduction of an updated epipolar constraint mask, the latest geometric constraint information is further guided to the fine-grained feature matching stage, ensuring that accurate matching within high-resolution local feature blocks is also constrained by geometric consistency. This further reduces the probability of geometrically inconsistent mismatches in the fine-grained feature matching stage and improves the geometric reliability of the overall matching results. The fine-grained feature matching is directly guided by the matching results of coarse-grained feature matching, performing feature block pruning and accurate calculation within a small local neighborhood, without the need for large-scale feature transformations on the coarse-scale feature map. The calculation range of fine-grained feature matching is limited to the candidate neighborhood of the matching results obtained from coarse-grained feature matching, resulting in a smaller overall computational load compared to the coarse-grained matching stage. This allows for higher accuracy improvements without significantly increasing inference latency.

[0102] In one alternative approach, the image matching method described above can be implemented based on an image matching model. For ease of explanation, the training process of this image matching model will be described first, followed by the process of performing image matching based on the trained image matching model.

[0103] Reference Figure 4A The diagram illustrates a flowchart of the steps involved in training an image matching model according to an embodiment of this application. Figure 4A As shown, the training method for this image matching model includes the following steps: Step S402: Obtain training samples.

[0104] The training samples include multiple sample images and ground truth values ​​for matching relationships between these sample images. The multiple sample images are images from different perspectives containing the same target object, and the ground truth values ​​for matching relationships between the multiple sample images indicate the pixel-level correspondences between them.

[0105] In one alternative approach, the training samples may also include ground truth values ​​of imaging geometric parameters corresponding to multiple sample images, which can serve as supervision conditions for subsequent iterative corrections of imaging geometric parameters.

[0106] Training samples can be obtained by those skilled in the art in any appropriate manner, and the embodiments of this application do not limit the method of obtaining training samples.

[0107] Step S404: Determine the epipolar band region corresponding to the sample object in multiple sample images.

[0108] The sample object can be a sample point or a sample region.

[0109] During the training phase, the fundamental matrix corresponding to the sample image can be determined based on the imaging geometric parameter samples corresponding to the sample image, thereby identifying the epipolar region. To enable the trained model to effectively handle noisy imaging geometric parameters, such as camera pose priors, during actual inference, one alternative approach is to add noise to the ground truth imaging geometric parameters corresponding to the sample image, such as by applying artificial noise, to simulate the uncertainty of imaging geometric parameters in a real scene, thus obtaining imaging geometric parameter samples. For example, taking the ground truth imaging geometric parameters as the ground truth camera pose as an example, appropriate random perturbations can be added to the rotation matrix R and translation vector T of the ground truth camera pose, respectively, to generate blurred imaging geometric parameter samples. This allows the model to learn during training the ability to accurately match and correct imaging geometric parameters even under imprecise geometric constraints.

[0110] In the case where the training samples also include the ground truth values ​​of imaging geometric parameters corresponding to multiple sample images, one alternative approach to determine the epipolar region corresponding to the sample object in the multiple sample images may include: obtaining the basis matrix corresponding to the multiple sample images based on the imaging geometric parameter samples; and obtaining the epipolar region based on the basis matrix through an image matching model.

[0111] For ease of explanation, the following is combined with Figure 4B The image matching model in the embodiments of this application is illustrated by way of an example.

[0112] like Figure 4B As shown, the image matching model includes a feature extraction part, a coarse-grained feature matching part, a fine-grained feature matching part, and an epipolar constraint focusing part.

[0113] in: The feature extraction part is used to extract features from multiple sample images during the training phase to obtain corresponding feature maps; and during the inference phase, it is used to extract features from multiple images to be matched to obtain corresponding feature maps.

[0114] In this embodiment, the feature extraction section is used to perform multi-scale feature extraction to obtain corresponding multi-scale feature maps, providing a basis for subsequent coarse-to-fine two-stage feature matching. Further optionally, such as... Figure 4B As shown, a feature extraction part can be constructed based on the MambaVision structure to extract image features. However, it is not limited to this; Convolutional Neural Networks (CNN), Visual Transformer (ViT), Swing Transformer, or CNN-Transformer hybrid structures are also applicable to the schemes in the embodiments of this application. However, the MambaVision structure has a better balance between computational efficiency and feature representation capability, and has a better computational efficiency advantage when processing high-resolution images.

[0115] The epipolar constraint focusing component provides an epipolar constraint mask for the model during both the training and inference phases. In one optional embodiment, during training, the epipolar constraint focusing component is used to obtain the epipolar lines corresponding to the sample objects based on the fundamental matrix; based on these epipolar lines and an obtained epipolar bandwidth threshold, it obtains the epipolar band regions corresponding to the sample objects in multiple sample images; and, based on the epipolar band regions, it generates an epipolar constraint mask for the multiple sample images. During inference, it is used to obtain the epipolar lines corresponding to the target objects based on the fundamental matrix; based on these epipolar lines and an obtained epipolar bandwidth threshold, it obtains the epipolar band regions corresponding to the target objects in multiple images to be matched; and, based on the epipolar band regions, it generates an epipolar constraint mask for the multiple images to be matched.

[0116] For example, after receiving the fundamental matrix F (derived from the noisy imaging geometric parameter samples) and the current epipolar bandwidth threshold s, the epipolar constraint focusing part can, based on the epipolar geometry principle, traverse each discrete pixel position in the feature map of the reference image, calculate the corresponding epipolar line on the feature map of the target image, and collect all target image pixel positions within the range s of the epipolar line. These positions are marked as candidate matching regions, thus forming an epipolar band region from the reference image to the target image. Using the same logic, the epipolar band region from the target image to the reference image is calculated again. The bidirectional epipolar band regions are merged to obtain the epipolar band region used in this round, which is then encoded into a binary mask matrix for subsequent attention calculations.

[0117] Figure 4B In the process, the epipolar constraint focusing part is connected to the coarse-grained feature matching part. Initially, the epipolar constraint focusing part determines the epipolar band region (e.g., based on the initial fundamental matrix (i.e., based on the initial imaging geometry parameter samples (or initial imaging geometry parameters), such as the fundamental matrix determined by the camera pose prior) and the initial epipolar bandwidth threshold. Figure 4B The epipolar constraint focuses on the banded region of the middle image, and an initial epipolar constraint mask is generated based on this (e.g., Figure 4B (As shown in the left image of the epipolar constraint focusing part), it is passed to the coarse-grained feature matching part, such as, passed to... Figure 4B The diagram illustrates the multi-head cross-attention mechanism for coarse-grained feature matching, used to perform cross-attention calculations under the constraints of this mask. Optionally, during subsequent model training or inference, the epipolar constraint focusing mechanism is continuously updated as the imaging geometry parameter samples (or imaging geometry parameters) and the epipolar bandwidth threshold are continuously modified, thereby continuously modifying and updating the epipolar constraint mask.

[0118] The coarse-grained feature matching part is used to perform coarse-grained feature matching for sample objects (or target objects) among multiple sample images (or images to be matched) based on feature maps and epipolar constraint masks provided by the epipolar constraint focusing part.

[0119] For example, in this part, during the coarse feature matching process, cross-attention calculation can be performed with epipolar constraint mask constraint, focused attention feature interaction can be performed in the epipolar band region, and a similarity matrix can be calculated after feature interaction to filter out coarse matching point pairs. Figure 4B In the middle section, the coarse-grained feature matching mainly targets feature maps of larger scales, such as 1 / 4 scale, 1 / 8 scale, 1 / 16 scale, and 1 / 32 scale feature maps, and performs multi-head cross-attention calculation under epipolar constraint mask constraints. Figure 4B The diagram illustrates multiple "Cross-Attention" operations. In this part, on one hand, similarity is calculated based on the results of the cross-attention calculation, and a confidence matrix is ​​constructed based on the calculation results. Then, a double softmax operation is applied to this matrix to obtain the probability of the closest mutual match. By selecting matches with scores higher than a set threshold, a coarse correspondence is obtained, i.e., the matching point pairs of coarse-grained feature matching. Figure 4B The diagram illustrates multiple "double softmax and mutual consistency" operations. By using double softmax operations, discriminative features can be obtained. However, during the inference phase, a single softmax operation can also be used, or MNN matching can be used to replace the softmax operation to reduce runtime and shorten model inference latency.

[0120] As an alternative, this part can also, after obtaining the matching results of coarse-grained feature matching, i.e., the aforementioned matching point pairs, output parameters for updating the epipolar bandwidth threshold and updating the imaging geometric parameter samples (or imaging geometric parameters) corresponding to multiple sample images; wherein, the updated imaging geometric parameter samples are used to update the fundamental matrices corresponding to multiple sample images (or images to be matched). Figure 4B In this example, the coarse-grained feature matching part further uses the RANSAC algorithm to filter out matching point pairs (the parameter) and passes them to the epipolar constraint focusing part; and / or, calculates the epipolar error (the parameter) based on all matching point pairs obtained on the basis of the softmax operation or MNN matching, and the current basis matrix, and passes it to the epipolar constraint focusing part.

[0121] In this case, optionally, the epipolar constraint focusing part can be implemented as a differentiable geometric computation branch, so that the model can learn how to better utilize geometric information to complete accurate matching under noisy pose prior through backpropagation during the training phase, thereby achieving matching optimization.

[0122] Optionally, after receiving the above parameters, the epipolar constraint focusing part can also update the corresponding epipolar bandwidth threshold and imaging geometric parameter samples (or imaging geometric parameters) according to the above parameters; update the epipolar lines based on the updated epipolar bandwidth threshold and the updated imaging geometric parameter samples (or imaging geometric parameters); generate an updated epipolar constraint mask based on the updated epipolar lines; and pass it to the coarse-grained feature matching part so that the coarse-grained feature matching part can perform calculation optimization in the next iteration.

[0123] The fine-grained feature matching component performs fine-grained feature matching based on the feature map and the results of coarse-grained feature matching. Fine-grained feature matching uses the matching point pairs obtained from coarse-grained feature matching as a basis, performing pixel-level matching and localization on a higher-resolution, fine-scale feature map.

[0124] Figure 4B In the middle section, the fine-grained feature matching part mainly targets fine-scale feature maps. Figure 4B The diagram illustrates a 1 / 2 scale feature map. Matching is performed based on the matching results (matching point pairs) output from the coarse-grained feature matching part, using the MNN algorithm. Optionally, matching can also be performed based on the MNN algorithm, combining the updated epipolar constraint mask from the epipolar constraint focusing part with the matching results from the fine-scale feature map and the coarse-grained feature matching part.

[0125] Based on this image matching model structure, the aforementioned method obtains the fundamental matrix corresponding to multiple sample images based on imaging geometric parameter samples. Obtaining the epipolar region based on the fundamental matrix through the image matching model can be achieved as follows: obtaining the fundamental matrix corresponding to multiple sample images based on imaging geometric parameter samples; and obtaining the epipolar region based on the fundamental matrix through the epipolar constraint focusing part of the image matching model. By adding noise to the ground truth values ​​of imaging geometric parameters and calculating the epipolar region based on the image matching model during the training phase using the noisy parameter samples, the model will periodically encounter imaging geometric parameter samples with different noise levels, such as camera pose prior samples, during training. This allows the model to learn the ability to perform robust feature matching and adaptive pose correction under these conditions, which is more consistent with the situation where the confidence of the pose prior is uncertain in the actual inference stage, effectively improving the model's generalization ability.

[0126] Step S406: Using the image matching model to be trained, generate epipolar constraint masks for multiple sample images based on the epipolar band region; and perform local feature matching for the sample object among multiple sample images based on the epipolar constraint masks.

[0127] As mentioned earlier, the epipolar constraint focusing part of the image matching model can generate an epipolar constraint mask for multiple sample images based on the epipolar band region. Then, using the coarse-grained and fine-grained feature matching parts, local feature matching for the sample object is performed among the multiple sample images based on the epipolar constraint mask, obtaining the prediction result of the local feature matching (the result output by the fine-grained feature matching part).

[0128] Step S408: Train the image matching model based on the prediction results of local feature matching and the true values ​​of matching relationships.

[0129] Once the prediction results of local feature matching are obtained, the loss value can be calculated based on the preset loss function and the true value of the matching relationship of the training samples, and the image matching model can be trained based on the loss value.

[0130] In this embodiment of the application, the loss function may include a sub-loss function corresponding to the coarse-grained feature matching part and a sub-loss function corresponding to the fine-grained feature matching part.

[0131] In one example, the loss for the coarse-grained feature matching part. It can be based on the similarity matrix on the coarse-scale feature map. For the set of real matching point pairs Each real corresponding point pair in Minimize its corresponding negative log-likelihood, such as:

[0132] in, This represents the number of true coarse-grained feature matching point pairs. This loss function encourages the model to improve the similarity score of true corresponding point pairs, while suppressing the scores of non-corresponding point pairs through the softmax normalization mechanism, guiding the model to learn more discriminative feature representations.

[0133] Loss for fine-grained feature matching In one example, fine localization can be performed on fine-scale feature maps. Model training is supervised by minimizing the negative log-likelihood of the true fine-scale matching point pairs, which enables the model to achieve pixel-level precision localization at fine scales.

[0134] In this case, the overall loss function of the model can be expressed as: = +

[0135] However, this is not the only application. In practical applications, loss functions such as positive and negative sample cross-entropy loss function or matching position L2 loss function are also applicable to the solutions in the embodiments of this application.

[0136] Furthermore, based on the above loss function, geometric losses (such as rotation / translation angle errors, multi-point epipolar errors, etc.) can be further introduced according to actual needs to further constrain the model's learning in pose correction. The embodiments of this application do not limit the specific implementation of the above loss function.

[0137] The model training method in this embodiment enables the image matching model to learn multiple capabilities such as feature representation, geometric constraint utilization, and pose self-correction during end-to-end training, resulting in a mature model that can robustly perform high-precision local feature matching tasks under noisy pose prior conditions.

[0138] Based on the trained image matching model, the aforementioned image matching method can be implemented as follows: by using the epipolar constraint focusing part of the image matching model, the epipolar band region corresponding to the target object in multiple images to be matched is determined, and based on the epipolar band region, an epipolar constraint mask is generated for multiple images to be matched; by using the coarse-grained feature matching part and the fine-grained feature matching part of the image matching model, local feature matching for the target object is performed among multiple images to be matched based on the epipolar constraint mask.

[0139] The following is for reference Figure 5 This example illustrates the application of an image matching model.

[0140] In this example, the autonomous driving device takes an autonomous vehicle as an example. The autonomous vehicle is set to have a front-view camera and a side-view camera. The front-view camera and the side-view camera simultaneously acquire a frame of environmental image. The two cameras have a known calibration pose relationship (i.e., known intrinsic and extrinsic parameters). However, due to factors such as vibration and temperature changes, there may be some deviation in this pose relationship, which serves as a fuzzy pose prior.

[0141] Based on this, the image matching process for the autonomous vehicle includes the following steps: Step S502: Obtain the fuzzy pose prior.

[0142] This includes: obtaining reference and target images, as well as camera intrinsics. Based on the reference image, target image, and camera intrinsic parameters, the fuzzy relative pose prior from the reference camera corresponding to the reference image to the target camera corresponding to the target image is obtained. ).

[0143] Step S504: Calculate the fundamental matrix between the reference image and the target image.

[0144] Under the standard pinhole model, the fundamental matrix can be obtained based on the fuzzy relative pose prior. , is represented as:

[0145] Step S506: Initialize the epipolar bandwidth threshold.

[0146] For example, the initial epipolar bandwidth threshold can be set based on the signal-to-noise ratio estimate or experience from the fuzzy pose prior. The less reliable the fuzzy relative pose prior, the lower the initial epipolar bandwidth. The value can be set to a larger value to tolerate errors in the fuzzy relative pose prior.

[0147] Step S508: Construct an epipolar constraint mask.

[0148] Using a pixel in the reference image For example, the construction of a polar-constrained mask can include: (A) Calculate the pixel using the fundamental matrix. Epipolar lines on the target image:

[0149] (B) The set of all pixels whose geometric distance to the epipolar line is less than s is defined as the epipolar zone region.

[0150] Where, the initial value of s is Subsequent iterations and updates.

[0151] (C) For each All pixel locations in the target image that fall within its epipolar region are marked as "candidate matching locations," forming a mask from the reference image to the target image. Similarly, for each pixel in the target image, the epipolar region on the reference image is calculated, constructing a mask from the target image to the reference image. Combining these two-way masks can be represented as a sparse binary matrix. The value is 1 only within the polar zone region.

[0152] When this step is implemented using the aforementioned image matching model, it can be achieved by the epipolar constraint focusing part of the image matching model.

[0153] Step S510: Multi-scale feature extraction and fusion.

[0154] For example, multi-scale feature maps can be extracted from the reference image and the target image through the feature extraction part of the aforementioned image matching model.

[0155] For example, it can be used as For coarse-scale feature maps, For fine-scale feature maps, those skilled in the art can divide the scale into coarse and fine scales according to general rules in the art, but are not limited thereto. Those skilled in the art can also divide the scale into coarse and fine scales according to actual needs, for example, using... This is a fine-scale feature map; anything above this scale is considered a coarse-scale feature map.

[0156] Furthermore, features from feature maps of different scales can be fused through upsampling / downsampling and 1×1 convolution, enabling information exchange between feature maps of multiple scales.

[0157] Step S512: Focused attention calculation under epipolar constraint mask.

[0158] For example, this step can be achieved through the coarse-grained feature matching part of the aforementioned image matching model.

[0159] In the standard multi-head attention mechanism:

[0160] In this embodiment of the application, an epipolar constraint mask is introduced into the multi-head cross-attention mechanism, specifically: (A) Use the features of the reference image feature map as the Query and the features of the target image feature map as the Key / Value (or vice versa). (B) For the location shielded by the antipolar constraint mask (i.e.) When calculating attention logits, assign a minimum value or -∞, so that the corresponding attention weights are close to 0 after softmax; (C) After softmax normalization, attention is mainly focused on the geometrically feasible region corresponding to the epipolar constraint mask, that is, the candidate positions that fall within the epipolar band region. (D) Standard cross attention can be used on coarse-scale feature maps, while linear attention or sparse attention can be used on fine-scale feature maps to balance performance and efficiency.

[0161] By using epipolar constraint masks, cross-attention reduces the search scope from the global feature map to a narrow geometric region, achieving "focused attention" under the constraint of "epitaxy constraint masks".

[0162] Step S514: Obtain the similarity matrix and perform preliminary feature matching.

[0163] This step can still be achieved through the coarse-grained feature matching part of the aforementioned image matching model. After focused attention processing, taking a 1 / 8 scale feature map as an example, it can be: (A) Flatten the feature maps of the reference image and the target image into a two-dimensional sequence; (B) Calculate the similarity matrix of matching point pairs For example, using a vector inner product and multiplying by a temperature coefficient. :

[0164] (C) Similarity matrix Softmax is applied to both rows and columns to obtain bidirectional normalization:

[0165] (D) According to The size of the points and whether they meet the conditions of bidirectional nearest neighbor are used to select the set of point pairs with higher confidence as the matching result of coarse-grained feature matching.

[0166] Since the epipolar constraint mask has eliminated most geometrically impossible point pairs, the set of matching point pairs obtained by coarse-grained feature matching at this time has high geometric consistency.

[0167] Step S516: Polar error calculation.

[0168] This step can still be achieved through the coarse-grained feature matching part of the aforementioned image matching model. For each pair of matching points obtained from coarse-grained feature matching... Based on the current fundamental matrix Calculate polar error:

[0169] Step S518: Pose correction.

[0170] This step can be achieved by combining the coarse-grained feature matching part and the epipolar constraint focusing part of the aforementioned image matching model.

[0171] For example, one or a combination of the following two methods can be used: Method 1: Direct pose correction based on epipolar error.

[0172] For example, an optimization problem can be constructed with epipolar error as the objective function, and gradient descent or nonlinear least squares can be performed on the camera pose (R,T) to update the camera pose. This approach can be implemented through the epipolar constraint focusing part of the aforementioned image matching model.

[0173] Method 2: Re-estimate the fundamental matrix and then decompose it.

[0174] For example, a new fundamental matrix can be estimated from matching point pairs using RANSAC or differentiable estimation methods through the coarse-grained feature matching part of the image matching model. Then, it is passed to the epipolar constraint focusing part, and the fundamental matrix is ​​decomposed to obtain a new... .

[0175] Step S520: Adaptive update of epipolar bandwidth threshold.

[0176] This step can be achieved through the epipolar constraint focusing part of the aforementioned image matching model.

[0177] For example, the average epipolar error of the current iteration can be used as a basis. Update the epipolar bandwidth threshold s:

[0178] in, These are preset hyperparameters. The smaller the error, the faster the epipolar bandwidth threshold shrinks, and the narrower the search region in subsequent iterations.

[0179] Step S522: Iterative execution.

[0180] Using the updated (R,T) and epipolar bandwidth threshold s as new priors, repeat steps S508-S518. Iterate for several rounds until the preset number of iterations or geometric error convergence is reached, to obtain the final matching point pair and camera pose estimate.

[0181] This example achieves two key improvements. First, it significantly enhances the accuracy and reliability of local feature matching: by using an epipolar constraint mask, the search space for feature matching is limited to a geometrically feasible region, eliminating a large number of geometrically inconsistent mismatches in advance. Through iterative pose optimization based on the matching results and adaptive shrinkage of the epipolar bandwidth threshold, the geometric constraints gradually become more precise, and the matching accuracy improves significantly with iteration. It also demonstrates significant improvements over traditional methods in wide-viewpoint, weak-texture, multimodal, and dynamic scenes. Second, it reduces computational cost and interference while maintaining or improving accuracy: compared to global cross-attention, this example concentrates attention computation in a narrow-band region using an epipolar constraint mask, effectively reducing the effective dimensionality of the attention matrix. Experiments on multiple datasets show that this example achieves accuracy close to or even better than dense matching methods, while its inference time is close to that of efficient sparse methods, making it suitable for practical engineering deployments. Thirdly, it exhibits robustness and self-correction capability against blurred camera pose priors: This example does not assume that the pose prior is completely accurate, but rather gradually corrects (R,T) through a relatively wide initial epipolar bandwidth threshold and multiple rounds of iterative optimization; when the pose prior deviation is large, the system can still gradually correct the deviation through the matching results, without causing matching failure due to overly rigid initial geometric constraints. Fourthly, it deeply couples geometric priors and feature matching into a unified algorithm framework: Geometric constraints are not only used for post-processing filtering, but also directly enter feature interaction and attention calculation, becoming the core constraint for model forward propagation; it utilizes differentiable geometric computation branches to achieve joint learning of features and geometry, enabling the model to adaptively learn how to best use geometric priors under different noise levels and scenarios. Fifthly, it has good versatility and scalability in various practical scenarios: Whether it is urban street scenes, indoor scenes, vehicle-mounted multi-camera systems, or UAV low-altitude aerial photography and satellite remote sensing images, it can be applied as long as blurred camera pose priors are obtained; it does not depend on a specific sensor or calibration method and can be directly transferred to multiple industrial and scientific research scenarios.

[0182] Furthermore, it should be noted that the parameters in the same formula parts as in the previous embodiments can also refer to the descriptions of the corresponding parts in the previous embodiments.

[0183] Furthermore, embodiments of this application also provide a method for controlling an autonomous driving device. (Refer to...) Figure 6 The method includes the following steps: Step S602: Collect multiple images of the environment in which the autonomous driving equipment is located using multiple image acquisition devices installed on the autonomous driving equipment.

[0184] Autonomous driving devices (such as self-driving vehicles, drones, and unmanned vessels) are typically equipped with multiple image acquisition devices (such as front-view cameras, side-view cameras, rear-view cameras, and fisheye wide-angle cameras) to acquire images of the surrounding environment in real time from multiple perspectives, building a comprehensive perception of the surrounding environment. Multiple image acquisition devices can form image pairs for subsequent local feature matching calculations, thereby supporting various downstream tasks, such as 3D scene reconstruction, visual positioning, obstacle detection and tracking, and other autonomous driving perception tasks. Those skilled in the art can flexibly set the number, installation location, and imaging geometry parameters of the image acquisition devices according to the specific configuration of the autonomous driving device; this application does not impose any limitations on this.

[0185] Step S604: Send multiple images to the autonomous driving server for local feature matching.

[0186] The autonomous driving server performs local feature matching based on the aforementioned image matching method and feeds back the results to the autonomous driving device. The specific implementation of local feature matching by the autonomous driving server can be found in the descriptions of the preceding embodiments, and will not be repeated here.

[0187] Sending multiple images to the autonomous driving server for local feature matching processing, instead of executing it locally on the device, can fully utilize the server's large-scale computing resources to execute relatively complex image feature matching models and obtain more accurate matching results. On the other hand, the device's local computing resources can be focused on control decision-making tasks with higher real-time requirements, realizing a highly efficient cloud-edge collaborative computing architecture.

[0188] Step S606: Execute the preset autonomous driving task based on the received local feature matching results.

[0189] After receiving the local feature matching results, the autonomous driving device can use them for various autonomous driving perception and decision-making tasks, including but not limited to: visual odometry estimation based on the matching results to track the device's trajectory; 3D point cloud reconstruction based on the matching results to build a dense 3D map of the surrounding environment; visual positioning based on the matching results to accurately locate the device in a pre-built high-precision map; and supporting path planning and driving control decisions based on the above perception results, etc.

[0190] This embodiment injects precise image matching capabilities into the autonomous driving perception system, enabling autonomous driving devices to obtain stable and reliable visual perception results in complex real-world scenarios (such as weakly textured roads, wide-angle turns, dynamic pedestrian occlusion, etc.), providing a more solid perception foundation for the safe execution of autonomous driving tasks.

[0191] Furthermore, embodiments of this application also provide an autonomous driving device capable of executing the above-described autonomous driving device control method, such as... Figure 7 As shown, it includes: a perception system 702, a driving system 704, a communication system 706, and a control system 708.

[0192] in: The perception system 702 includes multiple image acquisition devices for acquiring images of the environment in which the autonomous driving device is located.

[0193] The driving system 704 includes a steering subsystem 7042, an acceleration subsystem 7044, and a deceleration subsystem 7046 to control the driving behavior of the autonomous driving device; The communication system 706 includes a communication interface 7062 and a communication bus 7064, wherein the communication interface 7062 is used to communicate with the autonomous driving server. The control system 708 includes one or more control chips. The control system 708 is communicatively connected to the perception system 702 and the driving system 704 through the communication bus 7064. The control system 708 is used to execute the method described in the above-described autonomous driving device control method embodiment to obtain the result of local feature matching; and to send task instructions to the driving system 704 based on the result of local feature matching, so as to control the driving behavior of the autonomous driving device through the driving system 704.

[0194] The autonomous driving device in this application embodiment is used to implement the above-described autonomous driving device control method. The specific implementation of each system can be referred to the relevant description in the foregoing method embodiment, and it has the same beneficial effects as the method embodiment, so it will not be repeated here.

[0195] Reference Figure 8 This document illustrates a schematic diagram of an electronic device according to an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0196] like Figure 8 As shown, the electronic device may include: a processor 802, a communications interface 804, a memory 806, and a communications bus 808.

[0197] in: The processor 802, communication interface 804, and memory 806 communicate with each other through communication bus 808.

[0198] Communication interface 804 is used to communicate with other electronic devices or autonomous driving devices.

[0199] The processor 802 is used to execute program 810, specifically to execute the relevant steps in the above-described multiple image matching method embodiments, and / or to execute the relevant steps in the above-described image matching model training method embodiments.

[0200] Specifically, program 810 may include program code that includes computer operation instructions.

[0201] The processor 802 may be a CPU, a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The electronic device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0202] Memory 806 is used to store program 810. Memory 806 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0203] Program 810 may include multiple computer instructions. Specifically, program 810 may use multiple computer instructions to cause processor 802 to execute the operations corresponding to any of the methods described in the foregoing multiple image matching method embodiments or image matching model training method embodiments.

[0204] The specific implementation of each step in program 810 can be found in the corresponding steps and units described in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0205] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in any of the foregoing method embodiments. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk.

[0206] This application also provides a computer program product, including computer instructions that instruct a computing device to perform an operation corresponding to any of the methods in the above-described multiple method embodiments.

[0207] Furthermore, it should be noted that the actions involved in the collection, storage, use, processing, transmission, provision, disclosure, and deletion of data in the embodiments of this application are all carried out in compliance with the relevant data protection laws and regulations of the country or region where the data is located, and with the full authorization of the corresponding data owner.

[0208] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0209] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Flash Memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0210] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0211] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. An image matching method, comprising: Identify the epipolar band region corresponding to the target object in multiple images to be matched; Based on the epipolar band region, an epipolar constraint mask is generated for multiple images to be matched; Based on the epipolar constraint mask, local feature matching for the target object is performed among multiple images to be matched.

2. The method according to claim 1, wherein, The step of generating an epipolar constraint mask for multiple images to be matched based on the epipolar band region includes: Obtain multiple epipolar band regions corresponding to multiple images to be matched; Multiple epipolar regions are merged, and a unified epipolar constraint mask is generated for multiple images to be matched based on the merging result.

3. The method according to claim 2, wherein, The step of merging multiple epipolar regions and generating a unified epipolar constraint mask for multiple images to be matched based on the merging result includes: Multiple region masks are generated for each of the multiple epipolar band regions; the multiple region masks are merged; and a unified epipolar constraint mask is generated for the multiple images to be matched based on the merging result. And / or, The pixel regions of the multiple epipolar band regions are merged; a mask is generated for the merged pixel regions as an epipolar constraint mask for the multiple images to be matched.

4. The method according to any one of claims 1-3, wherein, The step of performing local feature matching for the target object among multiple images to be matched based on the epipolar constraint mask includes: For the feature maps of multiple images to be matched, focus attention calculation is performed in the mask region corresponding to the epipolar constraint mask to perform local feature matching for the target object.

5. The method according to claim 4, wherein, The feature maps of the multiple images to be matched are used to perform focused attention calculations in the mask region corresponding to the epipolar constraint mask, including: During the cross-attention calculation of feature maps of multiple images to be matched, a weight value approaching 0 is assigned to the region outside the mask region corresponding to the epipolar constraint mask, so as to perform focused attention calculation on the mask region.

6. The method according to any one of claims 1-3, wherein, The determination of the epipolar band region corresponding to the target object in multiple images to be matched includes: Based on the fundamental matrices corresponding to the multiple images to be matched, determine the epipolar line corresponding to the target object; Based on the epipolar lines and the obtained epipolar bandwidth threshold, the epipolar band region corresponding to the target object in the multiple images to be matched is determined.

7. The method according to claim 6, wherein, Before determining the epipolar line corresponding to the target object based on the fundamental matrices corresponding to the plurality of images to be matched, the method further includes: Obtain the imaging geometric parameters corresponding to the multiple images to be matched; Based on the imaging geometric parameters, a plurality of fundamental matrices corresponding to the images to be matched are obtained.

8. The method according to claim 7, wherein, The step of performing local feature matching for the target object among multiple images to be matched includes: Coarse-grained feature matching for the target object is performed sequentially among the multiple images to be matched, and fine-grained feature matching is performed based on the results of the coarse-grained feature matching.

9. The method according to claim 8, wherein, Between the coarse-grained feature matching and the fine-grained feature matching, the method further includes: Based on the results of coarse-grained feature matching, the imaging geometric parameters are updated, and based on the updated imaging geometric parameters, the fundamental matrix is ​​updated.

10. The method according to claim 9, wherein, The step of updating the imaging geometric parameters based on the results of coarse-grained feature matching includes: Based on the matching results of coarse-grained feature matching and the current fundamental matrix, the corresponding epipolar error is obtained; based on the epipolar error, the imaging geometric parameters are updated. or, Based on the matching results of coarse-grained feature matching, the fundamental matrix is ​​re-estimated; and based on the re-estimated fundamental matrix, the imaging geometric parameters are updated.

11. The method according to claim 10, wherein, The method further includes: Based on the epipolar error, the epipolar bandwidth threshold is updated, and based on the updated epipolar bandwidth threshold, the operation of determining the epipolar band region corresponding to the target object in multiple images to be matched based on the epipolar line and the obtained epipolar bandwidth threshold is returned to update the epipolar band region.

12. The method according to claim 11, wherein, The step of sequentially performing coarse-grained feature matching for the target object among multiple images to be matched, and fine-grained feature matching based on the results of the coarse-grained feature matching, includes: Coarse-grained feature matching for the target object is performed sequentially among multiple images to be matched, and fine-grained feature matching is performed based on the results of the coarse-grained feature matching and the updated epipolar constraint mask; wherein the updated epipolar constraint mask is determined based on the updated epipolar band region.

13. The method according to claim 4, wherein, The image matching method is executed based on an image matching model; The image matching model includes: The feature extraction section is used to extract features from multiple images to be matched to obtain corresponding feature maps. The coarse-grained feature matching part is used to perform coarse-grained feature matching for the target object among multiple images to be matched based on the feature map and the epipolar constraint mask provided by the epipolar constraint focusing part; The fine-grained feature matching section is used to perform fine-grained feature matching based on the feature map and the result of the coarse-grained feature matching.

14. The method according to claim 13, wherein, The epipolar band region is determined based on the epipolar line and epipolar bandwidth threshold corresponding to the target object; The coarse-grained feature matching part is further configured to output parameters for updating the epipolar bandwidth threshold and imaging geometric parameters after obtaining the coarse-grained feature matching result; wherein, the updated imaging geometric parameters are used to update the basis matrices corresponding to multiple images to be matched, and the updated basis matrices are used to update the epipolar lines corresponding to the target object.

15. The method according to claim 14, wherein, The epipolar constraint focusing part is further configured to update the corresponding epipolar bandwidth threshold and the imaging geometry parameters according to the parameters; based on the updated epipolar bandwidth threshold and the epipolar line updated according to the imaging geometry parameters, an updated epipolar constraint mask is generated and passed to the coarse-grained feature matching part.

16. The method according to claim 13, wherein, The feature extraction section is used to perform multi-scale feature extraction on multiple images to be matched to obtain corresponding multi-scale feature maps.

17. The method according to claim 16, wherein, The feature extraction part is built based on the MambaVision architecture.

18. A method for controlling an autonomous driving device, comprising: Multiple images of the environment in which the autonomous driving equipment is located are collected by multiple image acquisition devices installed on the autonomous driving equipment; Multiple images are sent to an autonomous driving server for local feature matching, wherein the autonomous driving server performs the local feature matching based on any one of claims 1-15 as described above, and feeds back the result of the local feature matching to the autonomous driving device. Based on the results of the received local feature matching, the preset autonomous driving task is executed.

19. A method for training an image matching model, comprising: Obtain training samples, which include multiple sample images and ground truth values ​​of matching relationships between the multiple sample images; Determine the epipolar band region corresponding to the sample object in the multiple sample images; Using the image matching model to be trained, an epipolar constraint mask is generated for multiple sample images based on the epipolar band region; and based on the epipolar constraint mask, local feature matching for the sample object is performed among the multiple sample images. The image matching model is trained based on the prediction results of local feature matching and the ground truth of the matching relationship.

20. The method according to claim 19, wherein, The training samples also include ground values ​​of imaging geometric parameters corresponding to multiple sample images; Before determining the epipolar band region corresponding to the sample object in the multiple sample images, the method further includes: adding noise to the true values ​​of the imaging geometric parameters to obtain imaging geometric parameter samples; Determining the epipolar region corresponding to the sample object in the multiple sample images includes: obtaining a fundamental matrix corresponding to the multiple sample images based on the imaging geometric parameter samples; and obtaining the epipolar region based on the fundamental matrix using the image matching model.

21. The method according to claim 19 or 20, wherein, The image matching model includes: a feature extraction part, a coarse-grained feature matching part, a fine-grained feature matching part, and an epipolar constraint focusing part; The feature extraction section is used to extract features from multiple sample images to obtain corresponding feature maps. The coarse-grained feature matching part is used to perform coarse-grained feature matching for the sample object among multiple sample images based on the feature map and the epipolar constraint mask provided by the epipolar constraint focusing part; The fine-grained feature matching section is used to perform fine-grained feature matching based on the feature map and the result of the coarse-grained feature matching.

22. The method according to claim 21, wherein, The epipolar constraint focusing portion is used to obtain the epipolar line corresponding to the sample object based on the fundamental matrix; and to obtain the epipolar band region corresponding to the sample object in multiple sample images based on the epipolar line and the obtained epipolar bandwidth threshold. Furthermore, based on the epipolar band region, an epipolar constraint mask is generated for multiple sample images.

23. The method according to claim 22, wherein, The coarse-grained feature matching part is further used to output parameters for updating the epipolar bandwidth threshold and imaging geometric parameter samples after obtaining the coarse-grained feature matching result; wherein the updated imaging geometric parameter samples are used to update the basis matrix corresponding to multiple sample images.

24. The method according to claim 23, wherein, The epipolar constraint focusing part is further configured to update the corresponding epipolar bandwidth threshold and the imaging geometric parameter sample according to the parameters; update the epipolar line based on the updated epipolar bandwidth threshold and the updated imaging geometric parameter sample; generate an updated epipolar constraint mask based on the updated epipolar line and pass it to the coarse-grained feature matching part.

25. The method according to claim 19 or 20, wherein, The feature extraction section is used to perform multi-scale feature extraction on multiple sample images to obtain corresponding multi-scale feature maps.

26. The method of claim 25, wherein, The feature extraction part is built based on the MambaVision architecture.

27. An electronic device comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform an operation corresponding to the method as described in any one of claims 1-17; or, to perform an operation corresponding to the method as described in any one of claims 19-26.

28. An autonomous driving device, comprising: A perception system, comprising multiple image acquisition devices, which are used to acquire images of the environment in which the autonomous driving device is located; The driving system includes a steering subsystem, an acceleration subsystem, and a deceleration subsystem to control the driving behavior of the autonomous driving device; A communication system includes a communication interface and a communication bus, wherein the communication interface is used to communicate with an autonomous driving server. The control system includes one or more control chips. The control system is communicatively connected to the perception system and the driving system via the communication bus. The control system is used to execute the autonomous driving device control method as described in claim 18 to obtain the result of local feature matching; and to send task instructions to the driving system according to the result of local feature matching, so as to control the driving behavior of the autonomous driving device through the driving system.

29. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-17; or performs the method as described in any one of claims 19-26.

30. A computer program product comprising computer instructions that instruct a computing device to perform an operation corresponding to any one of the methods described in claims 1-17; or, to perform an operation corresponding to any one of the methods described in claims 19-26.

Images