Image matching method, computer equipment and program product

Abstract

The invention provides an image matching method, computer equipment and a program product, and the method comprises the steps: obtaining images shot by a plurality of cameras at different moments, and determining the key point coordinates and descriptors of a target key point in each image; for any moment, carrying out data packaging processing on key point coordinates and descriptors of images shot by the plurality of cameras at the moment to obtain key point set tensor data and description subset tensor data corresponding to the moment; wherein the packaged key point set tensor data and the packaged description subset tensor data take the number of corresponding packaged images as a feature dimension; and performing key point matching according to the key point set tensor data and the description subset tensor data corresponding to different moments by using a key point feature matching model, and determining key point matching results of images shot by each camera at different moments.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image matching method, a computer device, and a program product. Background Art

[0002] In the field of Simultaneous Localization and Mapping (SLAM), the method of feature extraction and matching of images is usually adopted to achieve feature point matching of different images. The initial feature extraction and matching method often requires manual feature point extraction, but with the development of deep learning technology, deep learning models are gradually being used to complete feature extraction and matching.

[0003] When deploying deep learning models on the end side (for example, on the car server side), in order to speed up the calculation process, it is usually necessary to use a graphics processing unit (GPU) or a neural network processing unit (NPU) for acceleration. However, when deep learning models are performing feature extraction and matching, there are often problems such as low utilization of computing resources (i.e. GPU resources or CPU resources) and slow matching speed, which are quite disadvantageous. Summary of the invention

[0004] The embodiments of the present disclosure at least provide an image matching method, a computer device, and a program product.

[0005] In a first aspect, an embodiment of the present disclosure provides an image matching method, including:

[0006] Acquire images taken by multiple cameras at different times, and determine key point coordinates and descriptors of target key points in each of the images;

[0007] For any of the moments, data packaging is performed on the key point coordinates of the images captured by the multiple cameras at the moment and the descriptors to obtain key point set tensor data and descriptor set tensor data corresponding to the moment; wherein the packaged key point set tensor data and the descriptor set tensor data use the number of images corresponding to the packaging as a feature dimension;

[0008] Using the key point feature matching model, key point matching is performed according to the key point set tensor data and the description subset tensor data corresponding to the different moments, and the key point matching results of the images taken by each camera at the different moments are determined; the key point matching results are used to characterize the matching between the target key points in the images taken by the camera at different moments.

[0009] In this implementation, since the number of images in the packaging can be used as a feature dimension when the data is packaged, and the number of images is directly related to the number of cameras, by screening out the multiple cameras that need to be used, it is possible to dynamically determine the number of images that need to be used in the packaging under different circumstances, thereby improving the flexibility of data packaging, and achieving flexible matching of different numbers of image features in different inferences, and completing the matching of images under various numbers of cameras. By using the key point feature matching model, the packaged key point set tensor data and the description subset tensor data are processed, and the simultaneous matching of the images taken by each camera at different times can be completed synchronously, and the full utilization of computing resources can be achieved while ensuring the calculation accuracy, and the resource utilization and matching efficiency can be improved.

[0010] In a second aspect, an optional implementation of the present disclosure further provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are performed.

[0011] In a third aspect, an optional implementation of the present disclosure further provides a computer program product, which, when executed, executes the steps of the above-mentioned first aspect, or any possible implementation of the first aspect.

[0012] For a description of the effects of the above-mentioned image matching apparatus, computer device, and computer program product, please refer to the description of the above-mentioned image matching method, which will not be repeated here.

[0013] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0015] Figure 1 A flow chart of an image matching method provided by an embodiment of the present disclosure is shown;

[0016] Figure 2A schematic diagram of a network structure of an image matching method provided by an embodiment of the present disclosure is shown;

[0017] Figure 3 A flowchart of a key point quantity adjustment process provided by an embodiment of the present disclosure is shown;

[0018] Figure 4 A schematic diagram showing a model structure and matching process of a key point feature matching model provided by an embodiment of the present disclosure is shown;

[0019] Figure 5 A schematic diagram of a calculation process of an attention feature provided by an embodiment of the present disclosure is shown;

[0020] Figure 6 A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present disclosure will be described below in conjunction with the accompanying drawings in the embodiments of the present disclosure.

[0022] In addition, the terms "first", "second", etc. in the description and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0023] The "multiple or several" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0024] Research has found that it is beneficial for deep learning models to complete feature extraction and matching, and can achieve higher accuracy and better adapt to various lighting conditions and scenes. When using GPU or NPU for computing acceleration, although it can bring a certain acceleration effect while ensuring accuracy, it is usually impossible to fully utilize GPU or NPU resources, resulting in surplus resources, affecting resource utilization and failing to achieve the best computing acceleration effect.

[0025] Based on the above research, the present disclosure provides an image matching method, computer device and program product. Since the number of images in the packaging can be used as a feature dimension when the data is packaged, and the number of images is directly related to the number of cameras, by screening out the multiple cameras to be used, it is possible to dynamically determine the number of images to be used in the packaging under different circumstances, thereby improving the flexibility of data packaging, and achieving flexible matching of different numbers of image features in different reasoning times, and completing the matching of images under various numbers of cameras. By using the key point feature matching model, the packaged key point set tensor data and the description subset tensor data are processed, and the simultaneous matching of the images taken by each camera at different times can be completed synchronously, and the full utilization of computing resources can be achieved while ensuring the calculation accuracy, and the resource utilization rate and matching efficiency can be improved.

[0026] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0027] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0028] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0029] It should be noted that the specific terms mentioned in the embodiments of the present disclosure include:

[0030] SuperPoint is a feature point detection and descriptor extraction method based on self-supervised training. It adopts a fully convolutional neural network (FCN) design and can predict the input image at the pixel level. It consists of two main parts: a key point detector and a descriptor generator. The detector part is a bidirectional recurrent neural network that generates a dense key point probability map; the descriptor generator extracts features around each detected key point to ensure the independence of each key point;

[0031] Quantization Aware Training (QAT) is a method to reduce the size of deep learning models and improve operating efficiency while minimizing the accuracy loss caused by quantization.

[0032] int8 Quantization: is the process of converting floating point numbers into 8-bit integers;

[0033] The Superglue model is a deep learning model for feature matching. Its main feature is that it combines the attention mechanism and the optimal matching layer to match feature points.

[0034] The LightGlue model is also a deep learning model used to match local features in images. Compared with SuperGlue, it is more efficient in terms of memory and computing, and its results are more accurate and easier to train.

[0035] To facilitate understanding of this embodiment, an image matching method disclosed in an embodiment of the present disclosure is first introduced in detail. The executor of the image matching method provided in the embodiment of the present disclosure is generally a terminal device or other processing device with certain computing capabilities, wherein the terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a personal digital assistant (PDA), a handheld device, a computer device, etc.; in some possible implementations, the image matching method may be implemented by a processor calling computer-readable instructions stored in a memory.

[0036] The image matching method provided by the embodiment of the present disclosure is described below by taking the execution subject as a server as an example.

[0037] like Figure 1 As shown, it is a flowchart of an image matching method provided by an embodiment of the present disclosure, which may include the following steps:

[0038] S101: Acquire images taken by multiple cameras at different times, and determine key point coordinates and descriptors of target key points in each image.

[0039] Here, the shooting angles and shooting positions of the multiple cameras may be different, and the image contents in the images shot by the multiple cameras may partially overlap or may be different. The different moments may be any two moments. For example, the different moments may be moment t0 and moment t1 adjacent to t0, or the different moments may be moment t0 and moment t1 with a preset time interval from t0.

[0040] The target key point is the key point identified from the image, and the target key point is often a point with specific features. For example, the target key point can be a point on the corner of the wall, a point on the edge of the window, a point on the corner of the building, etc. The key point coordinates are used to indicate the coordinate position of the target key point in the image, and the descriptor is used to describe the point features of the target key point. The point features may include color features, position features, texture features, brightness features, etc. A target key point may correspond to a key point coordinate and a descriptor.

[0041] In the embodiment of the present disclosure, multiple cameras may be selected. The number of multiple cameras may be different in different image matching processes. For example, when the image matching is performed in the current time, the number of multiple cameras may be 5, and when the image matching is performed in the next time, the number of multiple cameras may be 10. Specifically, in different shooting scenes, a first number of cameras may be pre-deployed, and then according to the requirements of image matching, a second number of cameras may be selected from the first number of cameras as the multiple cameras to be used. Among them, the first number may be less than or equal to the second number; the requirements of image matching may be, for example, that the images taken by the camera at different times must be complete in content and have a preset clarity, or the requirements of image matching may be determined according to the matching scene, wherein different matching scenes correspond to different numbers of cameras. For example, in the case where the matching scene is a rough calibration of camera pose, the number of multiple cameras to be used may be 2; in the case where the matching scene is a fine calibration of camera pose, the number of multiple cameras to be used may be the first number; in the case where the matching scene is a scene with rich key points, the number of multiple cameras to be used may be 2, and in the case where the matching scene is a scene with sparse key points, the number of multiple cameras to be used may be the first number.

[0042] For example, the images taken by each camera in cameras c0 to cx at time t0 and time t1 can be obtained, where image_c0_t0 represents the image taken by camera c0 at time t0, image_c0_t1 represents the image taken by camera c0 at time t1, image_cx_t0 represents the image taken by camera cx at time t0, and image_cx_t1 represents the image taken by camera cx at time t1. Then, the key point extraction model can be used to identify the key points of the images taken by each camera at each time, and the key point coordinates and descriptors of the target key points in each image can be obtained. Among them, kpts_c0_t0 can be used to represent the key point coordinates of each target key point in the image taken by camera c0 at time t0, and desc_c0_t0 can be used to represent the descriptor corresponding to each target key point in the image taken by camera c0 at time t0; kpts_c0_t1 can be used to represent the key point coordinates of each target key point in the image taken by camera c0 at time t1, and desc_c0_t1 can be used to represent the descriptor corresponding to each target key point in the image taken by camera c0 at time t1; kpts_cx_t0 can be used to represent the key point coordinates of each target key point in the image taken by camera cx at time t0, and desc_cx_t0 can be used to represent the descriptor corresponding to each target key point in the image taken by camera cx at time t0; kpts_cx_t1 can be used to represent the key point coordinates of each target key point in the image taken by camera cx at time t1, and desc_cx_t1 can be used to represent the descriptor corresponding to each target key point in the image taken by camera cx at time t1.

[0043] For the target key points in any image, the input shape of the key point coordinates corresponding to the image can be a matrix with three feature dimensions, represented by [1, point_nums, 2], where point_nums represents the number of target key points and 2 is used to indicate the two-dimensional coordinates of the target key points. Similarly, the descriptor corresponding to the image can also be a matrix with three feature dimensions, represented by [1, point_nums, 64], where 64 is the descriptor dimension, and the dimension 64 is taken as an example here.

[0044] For example, [1, point_nums00, 2] represents the input shape of kpts_c0_t0 corresponding to image_c0_t0, [1, point_nums00, 64] represents the input shape of desc_c0_t0 corresponding to image_c0_t0; [1, point_nums01, 2] represents the input shape of kpts_c0_t1 corresponding to image_c0_t1, [1, point_nums01, 64] represents the input shape of desc_c0_t1 corresponding to image_c0_t1 Shape; [1, point_nums10, 2] represents the input shape of kpts_c1_t0 corresponding to image_c1_t0, [1, point_nums10, 64] represents the input shape of desc_c1_t0 corresponding to image_c1_t0; [1, point_nums11, 2] represents the input shape of kpts_c1_t1 corresponding to image_c1_t1, [1, point_nums11, 64] represents the input shape of desc_c1_t1 corresponding to image_c1_t1. [1, point_numsx0, 2] represents the input shape of kpts_cx_t0 corresponding to image_cx_t0, [1, point_numsx0, 64] represents the input shape of desc_cx_t0 corresponding to image_cx_t0; [1, point_numsx1, 2] represents the input shape of kpts_cx_t1 corresponding to image_cx_t1, [1, point_numsx1, 64] represents the input shape of desc_cx_t1 corresponding to image_cx_t1.

[0045] S102: For any moment, data packaging is performed on the key point coordinates and descriptors of the images taken by multiple cameras at the moment to obtain the key point set tensor data and descriptor set tensor data corresponding to the moment; wherein the packaged key point set tensor data and descriptor set tensor data use the number of images corresponding to the packaging as a feature dimension.

[0046] Here, the key point set tensor data is data with three feature dimensions, one of which is a batch dimension, one is a key point number dimension, and one is a key point coordinate dimension, where the coordinates under the coordinate dimension are represented by (x, y). For example, the shape of the key point set tensor data can be represented by [batch, point_nums_tx, 2], where tx is t0 or t1, and 2 corresponds to the two-dimensional coordinates (x, y) of the key point. The key point set tensor data can include the key point information of the images taken by each camera at a certain moment. The description subset tensor data is also data with three feature dimensions, one of which is a batch dimension, one is a key point number dimension, and one is a description sub-dimension, where the description sub-dimension can be set according to experience, for example, the description sub-dimension can be 32, 64, 128, 256, etc. For example, the shape of the description subset tensor data can be represented by [batch, point_nums_tx, 64].

[0047] The value of batch can be determined according to the number of images used in packaging. Since each camera only takes one image at a time, the value of batch will also be determined according to the number of cameras. For example, when packaging the key point coordinates in the images taken by cameras c0~cx at time t0, there are x+1 cameras in total, so the value of batch can be x+1. In specific implementation, since the number of cameras used for each image matching is different, the batch will be dynamically determined each time the image is matched, so it can be determined using the following formula 1:

[0048] batch=cam_nums;(Formula 1)

[0049] Wherein, cam_nums is the number of cameras. Since the key points in the images taken by the same camera at different times are matched during image matching, the number of cameras is equal to the number of image pairs that need to be matched during one image reasoning. Therefore, in the disclosed embodiment, the value of batch is equal to the number of cameras, the number of image pairs that need to be matched, and the number of images used in packaging (i.e., the number of images corresponding to the packaging).

[0050] It should be noted that when performing a specific image match, once the batch value is determined according to the number of cameras, the batch value will not change during the match until the next image match. If the number of cameras changes, the batch value will change accordingly.

[0051] The purpose of image matching is to match the key points in the image taken by each camera at time t0 with the key points in the image taken by the camera at time t1. That is, the key points in the image image_c0_t0 taken by camera c0 at time t0 are matched with the key points in the image image_c0_t1 taken by camera c0 at time t1; the key points in the image image_cx_t0 taken by camera cx at time t0 are matched with the key points in the image image_cx_t1 taken by camera cx at time t1. In order to make full use of computing resources and improve the throughput of calculation, a solution for acceleration using batch is proposed. Specifically, the key point coordinates and descriptors in the images taken by all cameras at time t0 can be packaged to obtain an input corresponding to time t0, and the key point coordinates and descriptors in the images taken by all cameras at time t1 can be packaged to obtain an input corresponding to time t1. For example, the key point coordinates in all camera images taken at time t0 can be packaged into key point set tensor data kpts_t0 with a shape of [batch, point_nums_t0, 2], and the descriptors corresponding to the key points in all camera images taken at time t0 can be packaged into descriptor set tensor data desc_t0 with a shape of [batch, point_nums_t0, 64]. It should be noted that here only the descriptor dimension of 64 is used as an example for explanation. In specific implementation, the descriptor dimension can be set according to experience. Similarly, the key point coordinates in all camera images taken at time t1 can be packaged into kpts_t1 with a shape of [batch, point_nums_t1, 2], and the descriptors corresponding to the key points in all camera images taken at time t1 can be packaged into desc_t1 with a shape of [batch, point_nums_t1, 64].

[0052] S103: Using the key point feature matching model, key point matching is performed according to the key point set tensor data and the description subset tensor data corresponding to different moments, and the key point matching results of the images taken by each camera at different moments are determined; the key point matching results are used to characterize the matching between the target key points in the images taken by the camera at different moments.

[0053] Here, the key point feature matching model is a pre-trained neural network model that can be used to match key points in an image and output key point matching results. The key point matching results can be used to indicate the matching conditions between target key points in images captured by the camera at different times. For example, the matching conditions may be that target key point 1 does not have a matching target key point, target key point 1 matches target key point 2, etc.

[0054] Specifically, the key point matching result may include the matching results corresponding to each camera, and the matching result corresponding to any camera may indicate the matching conditions between the target key points in the images taken by the camera at different times. For example, when the cameras include c0 to cx, the key point matching result may indicate the matching conditions between the target key points in the images taken by camera c0 at time t0 and time t1, the matching conditions between the target key points in the images taken by camera c1 at time t0 and time t1, ..., the matching conditions between the target key points in the images taken by camera cx at time t0 and time t1.

[0055] In specific implementation, the packed key point set tensor data kpts_t0 (shape is [batch, point_nums_t0, 2]) and descriptor set tensor data desc_t0 (shape is [batch, point_nums_t0, 64]) at time t0, as well as the packed key point set tensor data kpts_t1 (shape is [batch, point_nums_t1, 2]) and descriptor set tensor data desc_t1 (shape is [batch, point_nums_t1, 64]) at time t1 can be input into the key point feature matching model. The key point feature matching model is used to perform feature recognition and key point matching processing on kpts_t0, desc_t0, kpts_t1 and desc_t1, and the key point matching results are output.

[0056] In addition, since the GPU or NPU is used for acceleration when matching using the key point feature matching model, when there are multiple input batches, the GPU or NPU can be used to synchronously process multiple batches of data in parallel, thereby fully utilizing the computing resources in the GPU or NPU and improving matching efficiency.

[0057] In this way, since the number of images in the packing can be used as a feature dimension when the key point coordinates and descriptors are packed, and the number of images is directly related to the number of cameras, by screening out the multiple cameras to be used, it is possible to dynamically determine the number of images to be used in the packing under different circumstances, that is, dynamically determine the batch to be used for each image matching, so as to improve the flexibility of data packing, realize flexible matching of different numbers of image features in different reasoning, and complete the matching of images under various numbers of cameras. Using the key point feature matching model, the packed key point set tensor data and descriptor set tensor data are processed, and the simultaneous matching of the images taken by each camera at different times can be completed synchronously, and the full utilization of computing resources can be achieved while ensuring the calculation accuracy, and the resource utilization and matching efficiency can be improved.

[0058] In one embodiment, the image matching method provided in the embodiment of the present disclosure may be applied to a scenario of determining a vehicle's posture. Therefore, the camera in the embodiment of the present disclosure may be a camera deployed on a vehicle.

[0059] For example, multiple cameras may be pre-deployed at different locations on the vehicle, and then according to the driving scene of the vehicle (e.g., intersection, village, building, highway, etc.) and the integrity and clarity of the image captured by each camera, a second number of cameras may be selected from the multiple cameras, and images captured by each camera in the second number of cameras at different times may be obtained, and the key point coordinates and descriptors of the target key points in each image may be determined. Then, the above S102 and S103 may be used to determine the key point matching result.

[0060] After determining the key point matching results, the vehicle's position information can be calculated according to the following steps:

[0061] Step 1: Use the key point matching results to determine each key point pair that is successfully matched.

[0062] Here, the key point pair may specifically include the successfully matched target key points in the two images taken by each camera, wherein the two target key points in the key point pair are located in different images but may correspond to the same position.

[0063] In specific implementation, after obtaining the key point matching result, the key point matching result can be used to filter out the key points that are successfully matched in the two images taken by each camera, and use the key points as key point pairs. In this way, using each key point pair corresponding to the two images taken by each camera, a key point pair set can be obtained, which includes each key point pair that is successfully matched corresponding to each camera.

[0064] Step 2: Use the key point coordinates of the target key points in each key point pair to determine the vehicle's position information.

[0065] Here, the vehicle's position and posture information can be used to characterize the vehicle's position and posture, which may specifically include but is not limited to the vehicle's position information, orientation information, pitch information, etc.

[0066] In specific implementation, after obtaining the set of key point pairs, the key point coordinates of the target key points in each key point pair can be used to perform an optimizer solution to obtain the vehicle's posture information. Alternatively, any posture solution method in the prior art can be used to solve the vehicle's posture information using the key point coordinates of the target key points in each key point pair. After obtaining the vehicle's posture information, the posture information can be used to perform vehicle positioning, repositioning, driving behavior decision-making, and other processing on the vehicle.

[0067] In this way, the two target key points in the key point pair that are successfully matched correspond to the same position in the two images, so the position and posture information of the vehicle during driving can be determined by using the key point coordinates corresponding to the target key points in each key point pair in different images. And because the image matching method provided by the embodiment of the present disclosure can quickly determine the key point matching results while fully utilizing computing resources, the calculation of the position and posture information using the key point matching results can improve the speed and real-time performance of the calculated position and posture information.

[0068] like Figure 2 As shown in FIG. 1 , a network structure diagram of an image matching method provided by an embodiment of the present disclosure is shown in FIG. Figure 2 The image matching method provided by the embodiment of the present disclosure is further described. In one embodiment, the target numbers of target key points in different images may be consistent or different. When the target numbers of target key points corresponding to the images taken by each camera at the same time are consistent, the key point coordinates of each target key point can be directly used for data packaging. At this time, since the target numbers corresponding to each image are consistent, the complexity of data packaging is low and the packaging efficiency is higher.

[0069] When the target numbers of target key points corresponding to the images taken by each camera at the same time are inconsistent, it is necessary to first align the target key points in the images taken by each camera in terms of number, and then perform packaging processing. At this time, the above S102 can be implemented according to the following steps:

[0070] S102-1: Determine the maximum number according to the number of target key points in each image corresponding to any moment, and use the minimum value between the maximum number and a preset number threshold as the target number corresponding to the moment.

[0071] Here, the preset number threshold may be a value preset based on experience, for example, 500, 1000, etc. The target number is used to indicate the number of target key points that need to be used in the images taken by each camera at a moment.

[0072] For example, for time t0 or time t1, you can use Figure 2 The dynamic batch and dynamic key point number modules shown in the figure first determine the number of target key points in the image captured by each camera at that moment. Then, the maximum number is determined from these numbers. Then, the minimum value between the maximum number and the preset number threshold can be used as the target number.

[0073] Taking time t0 as an example, the number of target key points in images image_c0_t0 to image_cx_t0 taken by cameras c0 to cx can be traversed, and the maximum number can be determined from these numbers. Then the minimum value between the maximum number and the preset number threshold can be used as the number of targets corresponding to time t0.

[0074] Exemplarily, the number of points to be aligned may be determined according to the following formula 2:

[0075]

[0076] Among them, point_nums 目标 represents the number of targets, i represents the camera identification (such as the camera number), kpts data[i] size() indicates the number of target key points in the image captured by camera i, and MAXIMUM_LIMIT indicates the preset number threshold.

[0077] In specific implementation, for any moment, the number of target key points in the images taken by each camera at that moment and the preset number threshold can be input into the above formula 2 to obtain the number of targets corresponding to that moment.

[0078] In this way, compared with selecting the minimum number of target key points for data alignment, the loss of valid point information in images with a large number of target key points can be reduced; compared with directly using a preset number threshold, for the case where the number of target key points in each image is far less than the preset number threshold, the number of key points that need to be filled in each image can be reduced, thereby reducing the amount of calculation in the feature matching process and improving calculation efficiency.

[0079] S102-2: Adjust the number of target key points of each of the images corresponding to any of the moments, obtain the key point coordinates and descriptors of the adjusted target key points of each of the images, and generate a mask matrix corresponding to each image; wherein the number of target key points of each image after adjustment is the target number, and the mask matrix is ​​used to indicate the type of target key points on each image.

[0080] Here, the target key point type may specifically include type 1 and type 2, where type 1 is a key point directly identified from the image, and type 2 is a filled key point. The mask matrix may specifically be a mask matrix, and the matrix feature in the matrix may be 0 or 1, wherein one value between 0 and 1 is used to indicate that the target key point is type 1, and the other value is used to indicate that the target key point is type 2.

[0081] The reason why the mask matrix needs to be generated is that for some images, the target number of target key points in the image is less than the target number, and in order to achieve the same number of key points in each image, the key points need to be filled. However, the filled key points will affect the matching results of the key point feature matching model, so the mask matrix needs to be used to eliminate the impact on the matching results.

[0082] At any moment, the number of adjusted target key points corresponding to each image at that moment is equal to the number of targets, the number of key point coordinates corresponding to each image at that moment is consistent, the number of descriptors is also consistent, and both are equal to the number of targets.

[0083] In specific implementation, for time t0 or t1, after determining the number of targets at that time, the dynamic batch and dynamic key point number modules can be used to adjust the number of target key points in each image to the target number, and the number of key point coordinates and descriptors are also adjusted to the target number accordingly. At the same time, the mask matrix corresponding to each image can be generated according to the adjusted coordinates of each target key point and the target key point type of each target key point.

[0084] Furthermore, the number of images and the number of targets can be used to package the key point coordinates and descriptors of the adjusted target key points corresponding to each camera at each time, so as to obtain the key point set tensor data and descriptor set tensor data corresponding to each time.

[0085] In specific implementation, for any moment, the dynamic batch and dynamic key point number modules can be used to determine the value of the batch according to the number of images, and then the value of the batch, the number of targets corresponding to each image at that moment, and the key point coordinates of each adjusted target key point corresponding to each image at that moment are used to perform data packaging processing to obtain the key point set tensor data corresponding to that moment. Similarly, the value of the batch, the descriptors of each adjusted target key point corresponding to each image at that moment, the number of targets, and the pre-set descriptor dimensions can be used to perform data packaging processing to obtain the descriptor set tensor data corresponding to that moment.

[0086] For example, in Figure 2In the example, kpts_c0_t0, desc_c0_t0, kpts_c0_t1, desc_c0_t1, ..., kpts_cx_t0, desc_cx_t0, kpts_cx_t1 and desc_cx_t1 are input to the dynamic batch and dynamic key point number modules to obtain kpts_t0 of shape [batch, point_nums_t0, 2], desc_t0 of shape [batch, point_nums_t0, 64], ..., kpts_t1 of shape [batch, point_nums_t1, 2] and desc_t1 of shape [batch, point_nums_t1, 64]. Among them, point_nums_t0 represents the number of targets at time t0, and point_nums_t1 represents the number of targets at time t1.

[0087] Furthermore, after obtaining the mask matrix corresponding to each image using the above S102-2, the mask matrix can be used in combination with the above S103 to determine the key point matching results. Specifically, the key point feature matching model can be used to perform multi-layer self-attention and cross-attention operations based on the key point set tensor data and description subset tensor data corresponding to different moments, and the mask matrix corresponding to the images at different moments, to determine the key point matching results of the images taken by each camera at different moments.

[0088] Exemplarily, the key point feature matching model may include multiple layers of attention calculation layers, and each layer of attention calculation layer may have a self-attention calculation function and a cross-attention calculation function. In implementation, the key point set tensor data and the description subset tensor data corresponding to different moments, as well as the mask matrix corresponding to the image at different moments, may be input into the key point feature matching model. Then, the multiple layers of attention layers in the key point feature matching model may be used to perform self-attention operations and cross-attention calculations on the key point set tensor data, the description subset tensor data, and the mask matrix corresponding to different moments in turn, thereby outputting the key point matching results of the images taken by each camera at different moments.

[0089] In this way, by adjusting the number of target key points using the determined number of targets, the key points and descriptors in the images taken by each camera at any time can be uniformly adjusted to the target number. By packing the key point coordinates and descriptors after the unified number, the uniformity of the data dimension in the packed data can be guaranteed, thereby ensuring the neatness of the packed data. At the same time, since the mask matrix can reflect whether the key points in the image are filled points, the use of the mask matrix for feature matching can eliminate the influence of the filled key points on the matching results and improve the accuracy of the key point matching results.

[0090] In one embodiment, the above S102-2 can be divided into the following three situations, and the quantity can be adjusted in different ways in different situations:

[0091] Case 1: For an image in which the number of target key points is less than the target number, the image is filled with target key points to obtain the key point coordinates and descriptors of the adjusted target key points.

[0092] Exemplarily, for any image in each image at any time, if the number of target key points in the image is less than the target number, in order to ensure that the number of key points of each image is the target number when data is packaged, the image can be filled with key point coordinates and descriptors, thereby achieving the filling of target key points. Among them, the number of key point coordinates and descriptors that need to be filled is determined according to the difference between the number of target key points in the image and the target number. And when filling, each filled target key point can be located behind the existing target key points in the image.

[0093] After the filling is completed, the adjusted target key points corresponding to the image, as well as the key point coordinates and descriptors of each target key point can be obtained. The key point coordinates at this time may include the coordinates of the key points identified from the image (i.e., the key point coordinates before filling) and the filled key point coordinates. After obtaining the adjusted target key points, the matrix eigenvalues ​​of the key point coordinates of the target key points before filling can be set to a value of 0 or 1, and the matrix eigenvalues ​​of the filled key point coordinates can be set to another value of 0 or 1, thereby obtaining the mask matrix corresponding to the image.

[0094] Exemplarily, when the image includes m target key points and the difference between the target number and m is k, it can be determined that k target key points need to be filled, and the key point coordinates of these k target key points are filled after the key point coordinates of the m target key points.

[0095] Case 2: For images where the number of target key points is greater than the target number, the target key points are removed from the image to obtain the adjusted key point coordinates and descriptors of the target key points.

[0096] For example, for any image in each image at any time, if the number of target key points in the image is greater than the target number, in order to ensure that the number of key points of each image is the target number when data is packaged, the target key points of the image can be removed. The number of target key points to be removed is determined according to the difference between the number of target key points in the image and the target number.

[0097] Specifically, according to the confidence level of the key point coordinates of each target key point, the key point coordinates with low confidence can be eliminated, and the descriptors corresponding to the eliminated key point coordinates can be eliminated, so as to obtain the key point coordinates and descriptors of each adjusted target key point.

[0098] Then, according to the key point coordinates of each target key point that has not been eliminated, a mask matrix corresponding to the image that is all 0 or all 1 can be generated.

[0099] Case 3: For an image in which the number of target key points is equal to the number of targets, the key point coordinates and descriptors of the target key points in the image are used as the adjusted key point coordinates and descriptors of the target key points.

[0100] Exemplarily, for any image in each image at any time, if the number of target key points in the image is equal to the number of targets, no processing is performed on the target key points in the image, and a mask matrix consistent with Case 2 is generated.

[0101] In this way, according to the size relationship between the number of targets and the number of target key points in each image, the number of points in each image can be aligned to reasonably unify the number of target key points in each image. During different matches, the number of key points corresponding to the image taken by each camera at any time is dynamically determined, which improves the flexibility of determining the number of key points.

[0102] like Figure 3 As shown, a flowchart of a key point quantity adjustment process provided by an embodiment of the present disclosure may include the following steps:

[0103] S301: At any moment, obtain the number of target key points in the image taken by each camera at that moment.

[0104] S302: Determine the maximum number from the numbers corresponding to the various images.

[0105] S303: Taking the minimum value between the maximum quantity and the preset quantity threshold as the target quantity.

[0106] S304: For any image, determine whether the number of target key points in the image is greater than the target number.

[0107] If yes, execute the following S305, if no, execute the following S306.

[0108] S305: Eliminate target key points from the image to obtain adjusted key point coordinates and descriptors of the target key points.

[0109] Here, after executing S305, the following S308 may be directly executed.

[0110] S306: Determine whether the number of target key points in the image is less than the target number.

[0111] If so, execute the following S307; if not, it means that the number of target key points is equal to the target number, the key point coordinates and descriptors of the target key points in the image are used as the adjusted key point coordinates and descriptors of the target key points, and execute the following S308.

[0112] S307: Fill the image with target key points to obtain the adjusted key point coordinates and descriptors of the target key points.

[0113] Here, after executing S307, the following S308 may be directly executed.

[0114] S308: Generate a mask matrix.

[0115] Regarding the process of generating the mask matrix, please refer to the specific description of the above cases 1 to 3, which will not be repeated here.

[0116] S309: Determine whether all images at that moment have been traversed.

[0117] If not, return to execute the above S304, and if so, execute the following S310.

[0118] S310: Packing into batch data.

[0119] Regarding the specific implementation process of the above S301 to S310, reference may be made to the above embodiments, which will not be described in detail here.

[0120] In one embodiment, the above S101 may be implemented according to the following steps:

[0121] S101 - 1: Acquire images taken by multiple cameras at different times.

[0122] Exemplarily, the different moments may be the above-mentioned moment t0 and moment t1. The multiple cameras may be a second number of cameras selected from the multiple cameras deployed on the vehicle. For each camera, an image taken by the camera at moment t0 and an image taken at moment t1 may be obtained.

[0123] S101-2: Using the key point extraction model, determine the key point data corresponding to each image; the key point data includes the key point coordinates, confidence and descriptor of each identified initial key point.

[0124] Here, the initial key points are the key points identified from the image, but whether these key points will be used as target key points for feature matching needs to be screened and determined using confidence. The confidence can be expressed in the form of a score, and the higher the score, the higher the confidence.

[0125] The key point extraction model can use the superpoint model for reasoning, and the superpoint model can be trained and deployed using the QAT int8 quantization method.

[0126] In specific implementation, after obtaining the images taken by each camera at different times, the Figure 2 The key point extraction model in is used to extract key points from each image, and the key point coordinates, confidence level and descriptor of each initial key point in each image are obtained, that is, the key point data corresponding to each image are obtained.

[0127] For example, in Figure 2 In , the images (image_c0_t0, image_c0_t1, ..., image_cx_t0 and image_cx_t1) taken by each camera among cameras c0~cx at time t0 and time t1 respectively are input into the key point extraction model to obtain the key point coordinates and descriptors (kpts_c0_t0, desc_c0_t0, kpts_c0_t1, desc_c0_t1, ..., kpts_cx_t0, desc_cx_t0, kpts_cx_t1, desc_cx_t1) of the initial key points corresponding to each image, as well as the confidence of each initial key point.

[0128] S101 - 3 : For any image, the confidence levels of the initial key points corresponding to the image are used to screen the initial key points to obtain screened key points.

[0129] For example, after obtaining the key point data corresponding to each image, you can use Figure 2 The key point post-processing module shown further screens the key point data corresponding to the image, thereby screening out reliable target key points. Specifically, for each image, the key point post-processing module can be used to perform confidence screening according to the confidence of each initial key point corresponding to the image, and each initial key point with a confidence greater than a preset confidence threshold is used as a screening key point.

[0130] S101-4: Using the key point coordinates and descriptors of each filtered key point as the key point coordinates and descriptors of the target key point in the image.

[0131] In this way, the confidence can characterize the accuracy of the key point data extracted by the key point extraction model. By using the confidence to screen the key points identified from the image, some key points with low confidence can be eliminated, and the target key points with higher accuracy can be retained. The target key points can then be used for feature matching to improve the matching accuracy.

[0132] In one embodiment, S101-3 may be implemented according to the following steps:

[0133] S101-3-1: Using a preset confidence threshold and the confidence of each initial key point corresponding to the image, each initial key point is screened to obtain each candidate key point.

[0134] In specific implementation, you can use Figure 2 The confidence screening module in the key point post-processing module screens out the initial key points whose confidence is greater than a preset confidence threshold as candidate key points.

[0135] S101-3-2: When the number of candidate key points is greater than a preset number, the confidence levels of the candidate key points are used to sort the candidate key points in descending order of confidence level to obtain a sorting order of the candidate key points.

[0136] Here, the higher the sorting order, the higher the confidence of the corresponding candidate key point.

[0137] In the specific implementation, only key points with confidence greater than the preset confidence threshold are credible, and the number of key points that meet the confidence requirements in images taken by different cameras is different, some are very few, and some are very many. In the case of a large number of key points, it is necessary to use sorting for secondary screening. At the same time, some car-machine servers can make full use of inference resources through batching to improve throughput, but because the number of key points in each image after screening is inconsistent, that is, the shape of the tensor corresponding to the key points is inconsistent, so it is necessary Figure 2 The dynamic batch and dynamic key point number modules in the data are used to further process the data to obtain key point set tensor data and descriptor subset tensor data with consistent shapes to ensure that the key point feature matching model can handle it. Among them, in the example of this application, the input shape of tensor is B*NUMS*DIMS, where B is the value of the batch, NUMS is the number of key points, and DIMS is the feature dimension. After the value of B is determined, NUMS needs to be determined. Since the number of target key points screened out in the images taken by each camera at the same time is inconsistent, it is necessary to use the dynamic batch and dynamic key point number modules to align the number.

[0138] In specific implementation, for any image, after screening out the candidate key points corresponding to the image, you can use Figure 2 The sorting module in the key point post-processing module determines whether the number of candidate key points is greater than the preset number. If not, each candidate key point can be directly used as a screening key point. If so, the sorting module can be used to sort each candidate key point in order from high to low confidence to obtain the sorting order of each candidate key point.

[0139] S101-3-3: Filter out a preset number of filter key points from the candidate key points using the sorting order.

[0140] Exemplarily, each candidate key point whose sorting order is less than or equal to a preset number may be used as a screening key point.

[0141] In this way, by using the confidence to screen the initial key points twice, the target key points with reliable confidence can be screened out while limiting the number of target key points in the image to a reasonable range, thereby reducing the complexity of the calculation and improving the operation efficiency.

[0142] For example, in Figure 2 In the example, after the key point extraction model outputs kpts_c0_t0, desc_c0_t0, kpts_c0_t1, desc_c0_t1, ..., kpts_cx_t0, desc_cx_t0, kpts_cx_t1, desc_cx_t1, the confidence screening module and the sorting module in the key point post-processing module can be used to screen the initial key points to obtain the key point coordinates and descriptors of the target key points corresponding to the cameras c0~cx at time t0 and time t1, that is, kpts_c0_t0 with a shape of [1, point_nums00, 2] and kpts_c0_t1 with a shape of [1, poi desc_c0_t0 of shape [1, point_nums00, 64], kpts_c0_t1 of shape [1, point_nums01, 2], desc_c0_t1 of shape [1, point_nums01, 64], ..., kpts_cx_t0 of shape [1, point_numsx0, 2], desc_cx_t0 of shape [1, point_numsx0, 64], kpts_cx_t1 of shape [1, point_numsx1, 2], desc_cx_t1 of shape [1, point_numsx1, 64].

[0143] In one embodiment, the above S103 may be implemented according to the following steps:

[0144] S103-1: Using the key point feature matching model, according to the key point set tensor data and the description subset tensor data corresponding to different moments, determine the sub-key point set tensor data and the sub-description subset tensor data corresponding to each camera at different moments.

[0145] Here, the key point feature matching model can specifically adopt deep learning models such as Superglue and Lightglue.

[0146] One image corresponds to one sub-keypoint set tensor data and one sub-description set tensor data. One keypoint set tensor data may include sub-keypoint set tensor data corresponding to the images taken by each camera. Similarly, one description subset tensor data may include sub-description subset tensor data corresponding to the images taken by each camera.

[0147] Taking the moment t0 as an example, the cameras include c0~cx. The shape of the key point set tensor data at the moment t0 can be [x+1, point_nums_t0, 2], and the shape of the description subset tensor data at the moment t0 can be [x+1, point_nums_t0, 64], where the value range of batch is (0, x+1]. Then when the value of batch is 1, the sub-key point set tensor data and sub-description subset tensor data corresponding to the camera c0 at the moment t0 can be obtained. When the value of batch is 2, the sub-key point set tensor data and sub-description subset tensor data corresponding to the camera c1 at the moment t0 can be obtained. ..., and so on. When the value of batch is x+1, the sub-key point set tensor data and sub-description subset tensor data corresponding to the camera cx at the moment t0 can be obtained.

[0148] In specific implementation, since image matching is to match images taken by the same camera at different times, after inputting the key point set tensor data and description subset tensor data at each time into the key point feature matching model, the key point feature matching model can be used to determine the sub-key point set tensor data and sub-description subset tensor data corresponding to each camera at different times, and the sub-key point set tensor data and sub-description subset tensor data corresponding to the same camera at different times are used as a group of data that need to be matched for image matching. Then, GPU or NPU can be used for acceleration to achieve synchronous matching of each group of data.

[0149] For example, the sub-key point set tensor data 1 and the sub-description subset tensor data 1 corresponding to camera c0 at time t0, as well as the sub-key point set tensor data 2 and the sub-description subset tensor data 2 corresponding to camera c0 at time t1 can be obtained, and the sub-key point set tensor data 1 and the sub-description subset tensor data 1 and the sub-key point set tensor data 2 and the sub-description subset tensor data 2 are used as a set of data corresponding to camera c0 that needs to be image matched.

[0150] S103-2: For any camera, use the first-level attention calculation layer to perform self-attention calculations on the sub-keypoint set tensor data and sub-description subset tensor data corresponding to the camera at different times, respectively, to obtain the self-attention features corresponding to the different times, and perform cross-attention calculations on the self-attention features corresponding to the different times, to obtain the cross-attention features corresponding to the different times.

[0151] like Figure 4 The figure is a schematic diagram of the model structure and matching process of a key point feature matching model provided by an embodiment of the present disclosure. Figure 4 The specific process of image matching is described below:

[0152] First, the key point feature matching model in the embodiment of the present disclosure may include: Figure 4 The multiple attention calculation layers shown in FIG. The specific number of attention layers can be set based on experience and is not specifically limited here. Figure 4 Only the first attention calculation layer (i.e., layer 1 (layer#1)) and the last attention calculation layer (i.e., layer n (layer#n)) are shown in the figure, and the remaining layers are marked with ellipsis, where n is a positive integer greater than 1. The attention layer may include two self-attention modules (SelfAttention modules) and one cross-attention module (Cross Attention module). The self-attention module is used to perform self-attention calculation on the features in an image, and the cross-attention module is used to perform cross-attention calculation on the features output by the two self-attention modules.

[0153] In specific implementation, for each camera, the sub-keypoint set tensor data and sub-description subset tensor data of the camera at time t0 can be input into a self-attention module in the first-level attention calculation layer, and the sub-keypoint set tensor data and sub-description subset tensor data of the camera at time t1 can be input into another self-attention module in the first-level attention calculation layer. Then, the two self-attention modules are used to perform self-attention processing on the input information respectively, and the self-attention features corresponding to the camera at time t0 and the self-attention features corresponding to the camera at time t1 are obtained. Then, the self-attention features corresponding to the two times are synchronously input into the cross-attention module for cross-attention processing, and the cross-attention features corresponding to the camera at time t0 and time t1 are obtained.

[0154] Among them, for any camera, when performing cross-attention calculation, when the image at the first moment is cross-attention calculated with respect to the image at the second moment, the cross-attention feature corresponding to the first moment can be obtained; when the image at the second moment is cross-attention calculated with respect to the image at the first moment, the cross-attention feature corresponding to the second moment can be obtained.

[0155] S103-3: Utilize the next attention calculation layer to perform self-attention calculations on the cross-attention features corresponding to different moments, respectively, to obtain new self-attention features corresponding to different moments, and perform cross-attention calculations on the new self-attention features corresponding to different moments, respectively, to obtain new cross-attention features corresponding to different moments.

[0156] Exemplarily, after obtaining the output of the first attention calculation layer, the cross-attention features output by the layer can be used as the input of the next attention calculation layer. Using the two self-attention modules in the next attention calculation layer, self-attention calculations are performed on the cross-attention features corresponding to the camera at different times, respectively, to obtain new self-attention features corresponding to the camera at different times. Then, using the cross-attention module, cross-attention processing is performed on the obtained new self-attention features to obtain new cross-attention features corresponding to the camera at time t0 and time t1, respectively.

[0157] S103-4: Return to the next attention calculation layer and perform self-attention calculation on the cross-attention features corresponding to different moments, until the new cross-attention features corresponding to different moments output by the last attention calculation layer are obtained;

[0158] For example, after obtaining a new cross-attention feature using S103-3, the obtained feature can be further input into the next attention calculation layer to perform self-attention calculation and cross-attention calculation again to obtain a new cross-attention feature, and then input into the next attention calculation layer for processing until it is processed in turn by each attention calculation layer in the key point feature matching model to obtain the new cross-attention features corresponding to the output of the last attention calculation layer at different times, and use it as the final cross-attention feature.

[0159] For example, in Figure 4 In the above, the sub-keypoint set tensor data and sub-description subset tensor data corresponding to any camera at different times are processed by layer#1 to layer#n in sequence to obtain the final cross-attention features corresponding to the camera at different times.

[0160] S103-5: Based on the new cross-attention features corresponding to different moments output by the last attention calculation layer, determine the key point matching results corresponding to the camera at different moments.

[0161] Exemplarily, feature similarity calculation may be performed on new cross-attention features corresponding to different moments output by the last attention calculation layer, and the key point matching result corresponding to the camera may be determined based on the calculated similarity.

[0162] In one embodiment, the above S103-5 can be implemented according to the following steps:

[0163] S103-5-1: Perform feature matching on the new cross-attention features corresponding to the output of the last attention calculation layer at different times to obtain the matching matrix, matching validity matrix, and matching score matrix corresponding to the camera; wherein the matching matrix is ​​used to indicate the matching relationship between the target key points in the images taken by the camera at different times, the matching validity matrix is ​​used to indicate whether the matching relationship of each target key point is valid; the matching score matrix is ​​used to indicate the confidence of the matching relationship of each target key point.

[0164] S103-5-2: Use the matching matrix, matching validity matrix and matching score matrix as the key point matching results corresponding to the camera at different times.

[0165] Here, the key point feature matching model may also include a feature matching module (Matching module), which is used to perform feature matching on the output of the last attention calculation layer and output the matching result.

[0166] Specifically, the key point matching results output by the Matching module may include the matching matrix (matches_idx), matching validity matrix (matches_valid), and matching score matrix (matches_score) corresponding to the camera, where idx represents the camera ID. The shapes of the matching matrix, matching validity matrix, and matching score matrix may all be [batch, point_numst0]. The matching matrix is ​​used to indicate the matching relationship between the key points in the images captured by the camera at time t0 and time t1. For example, the matching relationship may be that there is no matching key point for the target key point 1 in image_c0_t0, the target key point 2 in image_c0_t0 matches the target key point 5 in image_c0_t1, the target key point 3 in image_c0_t0 matches the target key point 1 in image_c0_t1, and the like. The matching validity matrix is ​​used to indicate whether the matching relationship of each target key point represented in the matching matrix is ​​valid. For example, the matching validity matrix can indicate that the matching relationship of target key point 1 in image_c0_t0 is valid, the matching relationship of target key point 2 in image_c0_t0 is invalid, and the matching relationship of target key point 3 in image_c0_t0 is valid. The matching score matrix is ​​used to indicate the confidence of the matching relationship of each target key point represented in the matching matrix. For example, the confidence of the matching relationship of target key point 1 in image_c0_t0 is 99%, the confidence of the matching relationship of target key point 2 in image_c0_t0 is 50%, and the confidence of the matching relationship of target key point 3 in image_c0_t0 is 95%.

[0167] In specific implementation, for any camera, after the last attention calculation layer in the key point feature matching model outputs the final cross-attention features corresponding to the camera at different times, the final cross-attention features can be input into the Matching module in the key point feature matching model for feature recognition and matching processing, thereby obtaining the matching matrix, matching validity matrix, and matching score matrix corresponding to the images taken by the camera at different times. Furthermore, these three matrices can be used as the key point matching results corresponding to the camera at different times.

[0168] Furthermore, after obtaining the key point matching results, the three matrices included in the key point matching results and the key point set tensor data corresponding to the camera at different times can be input into Figure 2The posture calculation module shown uses the posture calculation module to determine each key point pair that has successfully matched according to the key point matching results, and selects the key point coordinates of the target key point in each key point pair from the key point set tensor data, and uses these coordinates to calculate the vehicle's posture information.

[0169] In this way, by using multiple layers of attention calculation layers, multiple self-attention calculations are performed on the tensor data of the sub-keypoint set and the sub-description subset of the image, so that as the number of layers increases, the key points in the image gradually converge to themselves, thereby improving the accuracy of the calculated self-attention features; and by performing multiple cross-attention calculations on the self-attention features of images taken by the same camera at different times, it is possible to achieve that as the number of layers increases, two similar key points in two images gradually converge to each other, thereby achieving accurate matching of the two key points. The matching relationship between key points, as well as the reliability and effectiveness of the matching relationship can be characterized by using the matching matrix, matching validity matrix, and matching score matrix. By using these three matrices as the key point matching results, the target key points matched in different images can be accurately reflected.

[0170] In one embodiment, the step of performing self-attention calculation in S103-2 may be implemented as follows:

[0171] S103-2-1: Use the first attention calculation layer to positionally encode the key point coordinates in the sub-key point set tensor data corresponding to the camera at different times, and obtain the position feature matrix corresponding to different times.

[0172] For example, Figure 4 As shown, the first-level attention calculation layer may also include a position encoder module (Position encoder module) associated with each self-attention module. After the sub-keypoint set tensor data and the sub-description subset tensor data corresponding to the camera at time t0 and time t1 are input into the first-level attention calculation layer, the key point coordinates of each target key point in the image can be determined from the sub-keypoint set tensor data. Then, two Positionencoder modules are used to perform position encoding processing on the coordinates of each key point corresponding to time t0 and time t1, respectively, to obtain the position feature matrices corresponding to time t0 and time t1.

[0173] exist Figure 4In the figure, the part corresponding to the thinner solid arrow is a schematic part of processing the image taken by the camera at time t0 using the key point feature matching model; wherein d0 represents each descriptor corresponding to an image at time t0, and p0 represents each key point coordinate corresponding to an image at time t0. The part corresponding to the thicker solid arrow is a schematic part of processing the image taken by the camera at time t1 using the key point feature matching model; wherein d1 represents each descriptor corresponding to an image at time t1, and p1 represents each key point coordinate corresponding to an image at time t1.

[0174] S103-2-2: Fusing the position feature matrix and the descriptors in the sub-descriptor set tensor data to obtain fused feature matrices corresponding to different moments.

[0175] For example, in Figure 4 In the above, after obtaining the position feature matrix corresponding to a camera at time t0, the descriptors corresponding to the camera at time t0 can be screened out from the sub-descriptor set tensor data corresponding to the camera at time t0. Then, the first attention calculation layer can be used to perform feature fusion on the position feature matrix corresponding to the camera at time t0 and each descriptor to obtain the fused feature matrix corresponding to the camera at time t0.

[0176] The process of determining the fusion feature matrix corresponding to the camera at time t1 is similar to the process of determining the fusion feature matrix corresponding to t0 mentioned above, and will not be repeated here.

[0177] S103-2-3: Perform self-attention calculations on the fusion feature matrices corresponding to different moments respectively, to obtain the self-attention features corresponding to different moments respectively.

[0178] Exemplarily, for the image corresponding to the camera at time t0, after obtaining the fusion feature matrix corresponding to the image at that time, the self-attention module can be used to perform self-attention calculation processing on the fusion feature matrix to obtain the self-attention feature corresponding to the camera at time t0. Similarly, for the image corresponding to the camera at time t1, after obtaining the fusion feature matrix corresponding to the image at that time, the self-attention module can be used to perform self-attention calculation processing on the fusion feature matrix to obtain the self-attention feature corresponding to the camera at time t1.

[0179] It should be noted that, for the other attention calculation layers except the first attention calculation layer in the key point feature matching model, the structures of these attention calculation layers can be consistent with the first attention calculation layer, specifically including the Position encoder module. In this case, after obtaining the cross-attention features at different times output by the previous attention calculation layer, the Position encoder module can be used to position encode the cross-attention features at different times to obtain the position feature matrix. Then, for each moment, the position feature matrix at that moment can be feature fused with each descriptor at that moment to obtain the fused feature matrix corresponding to that moment; then the fused feature matrix at that moment can be self-attention calculated to obtain the self-attention feature corresponding to that moment. Alternatively, for the other attention calculation layers except the first attention calculation layer in the key point feature matching model, the structures of these attention calculation layers can be different from the first attention calculation layer, specifically, these attention calculation layers do not include the Position encoder module. In this case, after obtaining the cross-attention features at different times output by the previous attention calculation layer, the self-attention calculation can be directly performed on the cross-attention features at different times to obtain the cross-attention features corresponding to different moments.

[0180] In this way, by encoding the position of the key point coordinates, the feature vector can be used to represent the key point coordinates, and the position feature matrix representing the coordinates can be obtained. Then, by fusing the position feature matrix and the descriptor, a feature vector containing richer information can be obtained. The fused feature matrix composed of the feature vector can be used to accurately represent the key points. Finally, by performing self-attention calculation on the fused feature matrix, a relatively accurate self-attention feature can be obtained.

[0181] In one embodiment, the step of performing cross attention calculation in S103-3 may be implemented according to S1 to S4 as follows:

[0182] S1: Using the self-attention features corresponding to the first moment, determine the first query matrix, the first key matrix and the first value matrix corresponding to the first moment, and, using the mask matrix corresponding to the image at the first moment, determine the first mask query matrix and the first mask key matrix corresponding to the first moment.

[0183] S2: Determine the second query matrix, the second key matrix and the second value matrix corresponding to the second moment by using the self-attention features corresponding to the second moment, and determine the second mask query matrix and the second mask key matrix corresponding to the second moment by using the mask matrix corresponding to the image at the second moment; the first moment is one moment among the different moments, and the second moment is another moment among the different moments.

[0184] Here, the self-attention feature output by the self-attention module can be a self-attention matrix in matrix form. Each self-attention matrix can correspond to a query matrix (Query matrix), a key matrix (Key matrix) and a value matrix (Value matrix).

[0185] Different moments in the embodiments of the present disclosure may be the first moment and the second moment, for example, moment t0 is the first moment and moment t1 is the second moment. For the self-attention feature at the first moment, it may correspond to a first query matrix, a first key matrix, and a first value matrix. For the self-attention feature at the second moment, it may correspond to a second query matrix, a second key matrix, and a second value matrix.

[0186] Based on the above embodiments, it can also be known that a mask matrix (mask matrix) is generated corresponding to the image taken by the camera at each moment. For the mask matrix, each mask matrix can also correspond to a mask query matrix and a mask key matrix. For the first moment, the mask matrix corresponding to the image taken by the camera at the first moment can correspond to a first mask query matrix and a first mask key matrix. For the second moment, the mask matrix corresponding to the image taken by the camera at the second moment can correspond to a second mask query matrix and a second mask key matrix.

[0187] In specific implementation, for an image taken by any camera at the first moment, after obtaining the attention features output by the self-attention module, the first query matrix (Q1), the first key matrix (Key1 matrix) and the first value matrix (Value1 matrix) corresponding to the first moment can be generated according to the attention features, and, according to the mask matrix corresponding to the image, the first mask query matrix (mask_Q1 matrix) and the first mask key matrix (mask_Key1 matrix) corresponding to the first moment are generated. Similarly, for an image taken by any camera at the second moment, after obtaining the attention features output by the self-attention module, the second query matrix (Q2 matrix), the second key matrix (Key2 matrix) and the second value matrix (Value2 matrix) corresponding to the second moment can be generated according to the attention features, and, according to the mask matrix corresponding to the image, the second mask query matrix (mask_Q2 matrix) and the second mask key matrix (mask_Key2 matrix) corresponding to the second moment are generated.

[0188] S3: Add the product of the transposed matrix of the first query matrix and the second key matrix and the product of the transposed matrix of the first mask query matrix and the second mask key matrix, and determine the cross-attention feature corresponding to the first moment based on the first addition result and the second value matrix.

[0189] S4: Add the product of the second query matrix and the transposed matrix of the first key matrix and the product of the second mask query matrix and the transposed matrix of the first mask key matrix, and determine the cross-attention feature corresponding to the second moment based on the second addition result and the first value matrix.

[0190] When performing cross-attention calculation on the self-attention feature corresponding to the first moment, the cross-attention module can be used to calculate the product of the transposed matrix of the Q1 matrix and the Key2 matrix, and at the same time, the product of the transposed matrix of the mask_Q1 matrix and the mask_Key2 matrix can be calculated. After obtaining the two products, the first addition result can be obtained based on the addition of the two products, and the first addition result and the second value matrix are normalized to obtain the cross-attention feature corresponding to the first moment.

[0191] Similarly, when performing cross-attention calculation on the self-attention feature corresponding to the second moment, the cross-attention module can be used to calculate the product of the transposed matrix of the Q2 matrix and the Key1 matrix, and at the same time, the product of the transposed matrix of the mask_Q2 matrix and the mask_Key1 matrix can be calculated. After obtaining the two products, the second addition result can be obtained based on the addition of the two products, and the second addition result and the first value matrix are normalized to obtain the cross-attention feature corresponding to the second moment.

[0192] In this way, since the key point coordinates are filled when the data is packaged, when performing cross-attention calculation, by fusing the product of the mask query matrix and the mask key matrix, the influence of the filled key points on the attention calculation can be eliminated, thereby improving the accuracy of the calculation results.

[0193] In one embodiment, the step of “determining the cross-attention feature based on the first addition result” in S3 may be implemented as follows:

[0194] S3-1: Determine the square root of the feature dimension corresponding to the first addition result.

[0195] S3-2: Determine a ratio between the first addition result and the square root value, and normalize the ratio to obtain a normalized result.

[0196] S3-3: Determine the cross-attention feature corresponding to the first moment based on the normalization result and the second value matrix.

[0197] Exemplarily, the formula for attention calculation can be shown as follows:

[0198]

[0199] Among them, attention represents the attention feature, softmax represents the normalization process, Q represents the query matrix, K represents the key matrix, and K T represents the transposed matrix of the key matrix, V represents the value matrix, and d represents the feature dimension, which is represented by QK T The feature dimension corresponding to the product of is determined, k represents the number of key point coordinates filled in the image, K is greater than or equal to 0, Indicates the square root of the feature dimension.

[0200] In specific implementation, the cross-attention feature at any moment can be calculated based on the above formula 3. For example, taking the calculation of the cross-attention feature of an image taken by any camera at the first moment as an example, the first addition result at the first moment can be used as QK in the above formula 3. T , and the corresponding feature dimension d at the first moment k and the second value matrix (Value2 matrix), and substitute it into the above formula 3 to obtain the cross attention feature corresponding to the first moment. The probability matrix obtained can be used as the weight matrix of V.

[0201] Similarly, the step of “determining the cross attention feature based on the first addition result” in S4 is similar to the step of “determining the cross attention feature based on the first addition result”, and the second addition result, the first value matrix and the feature dimension d corresponding to the second moment can also be added together. k Substituting into the above formula 3, we can obtain the cross-attention feature corresponding to the second moment, which will not be repeated here.

[0202] In one embodiment, the step of generating the first mask query matrix and the first mask key matrix in S1 may also be implemented according to the following P1 and P2:

[0203] P1: When the key point coordinates corresponding to the image at the first moment include the filled key point coordinates, the feature values ​​of the first matrix features related to the filled key point coordinates in the mask matrix corresponding to the image at the first moment are set to preset negative values, and the feature values ​​of the second matrix features in the mask matrix except the first matrix features are retained to obtain the updated mask matrix corresponding to the image at the first moment.

[0204] P2: Generate a first mask query matrix and a first mask key matrix corresponding to the first moment by using the updated mask matrix.

[0205] Based on the above embodiments, it can be known that when packing the key point coordinates, the target key points may be filled (that is, the key point coordinates are filled) for some images to accelerate the operation, and the key point coordinates of the filled target key points do not belong to the image itself, so when performing attention calculations, in order to eliminate the influence of the filled target key points on the calculation results, it is also necessary to perform feature processing on the filled target key points. The processing method of this application is embodied in the use of the first mask query matrix and the first mask key matrix for processing.

[0206] For details, refer to the following Figure 5 As shown in FIG. 1 , a schematic diagram of a calculation process of an attention feature provided by an embodiment of the present disclosure is shown in FIG. Figure 5 The content explains why the filled key point coordinates affect the calculation results and how to eliminate the influence of such key point coordinates:

[0207] Take two images taken by any camera at time t0 and time t1 as image A and image B, and take image A performing cross-attention calculation on image B as an example. Image A includes m key points and image B includes n key points. The feature dimension corresponding to the key points is d. If the dynamic batch and dynamic key point number modules perform key point filling on both image A and image B, assume that image A is filled with j key point coordinates and image B is filled with k key point coordinates. Since the key point coordinates are filled after the existing key point coordinates, for image A, the j key point coordinates are filled at the tail of the mask matrix corresponding to image A. Similarly, for image B, the k key point coordinates are filled at the tail of the mask matrix corresponding to image B. Figure 5 In the example, the Q matrix corresponding to image A has a total of m+k rows, and the transposed matrix K of the K matrix corresponding to image B is T There are n+j columns in total. T Image A queries image B. After softmax is performed to obtain the probability value, the larger the probability value, the greater the matching correlation between the key points. In order to shield the interference of the filled key points, it is only necessary to mask them in terms of probability. Because the image features of each image are filled at the end, in QK T After that, we only need to set the probability of the last k rows and the last j columns to 0. Figure 5 In the example of image A querying image B, QK TThe sum of the probability values ​​in each row of the obtained matrix is ​​1 (the number of probability values ​​in a row is n+j), and if the last j elements are 0, the maximum probability value will only appear in the first n elements, that is, the existing key points in the image. In order to eliminate the influence of the last j element values, the method adopted by the embodiment of the present disclosure is to take the value to be masked as a preset large negative value, and then use these negative values ​​to transform the mask query matrix mask_Q and the transposed matrix of the mask key matrix (mask_K) corresponding to the mask matrix. T ) to obtain mask_Qmask_K T Then, with QK T After adding, softmax processing can be performed to make the eigenvalues ​​corresponding to the last j element values ​​close to 0, thereby shielding the influence of the filled key point coordinates on the attention calculation.

[0208] Exemplarily, taking an image taken by any camera at time t0 as an example, when performing attention calculation, it can be first determined whether the image has been filled with key point coordinates. If so, that is, when the key point coordinates of the image at t0 include the key point coordinates of the filled target key point, the feature value of the first matrix feature related to the filled key point coordinates in the mask matrix corresponding to the image (wherein the first matrix feature related to the filled key point coordinates is the matrix feature that matches the coordinate position of the filled key point) can be set to a preset negative value, which can be a large negative value set based on experience. At the same time, the feature values ​​of the second matrix features other than the first matrix features in the mask matrix are retained to obtain the updated mask matrix corresponding to the image at the first moment. For example, when the number of targets is 1000 and 100 key point coordinates are filled in the image, the feature values ​​of the first matrix features related to the 100 key point coordinates can be set to a preset negative value, and the feature values ​​of the second matrix features related to the remaining 900 key point coordinates can be retained as the original values. After obtaining the updated mask matrix corresponding to the image, the updated mask matrix can be used to generate the first mask query matrix and the first mask key matrix corresponding to the image at time t0.

[0209] If the image is not filled with key point coordinates, the mask matrix corresponding to the image can be used as an updated mask matrix (at this time, the matrix features in the mask matrix are all 0 or all 1), and the mask matrix is ​​used to generate the first mask query matrix and the first mask key matrix corresponding to the image at time t0.

[0210] At the same time, based on Figure 5 It can be seen that when image A performs cross attention calculation on image B, the QK corresponding to image A is obtained. T and mask_Qmask_K TAfter that, QK T With mask_Qmask_K T Add them together and use the result as QK in formula 3 T , and based on the value matrix (V matrix) corresponding to image B and the feature dimension d and formula 3, the cross attention feature of image A for image B is obtained. At the same time, based on Figure 5 It can also be known that if the influence of the filled key points is masked when calculating the probability using the key matrix of image B (that is, when calculating the weight of the V matrix of image B using the key matrix of image B), then in theory it will not affect the matching effect. Therefore, when generating the mask matrix, you only need to generate it according to the calculation format of the QK matrix.

[0211] In this way, because the key points filled in each image are generally filled at the end of the image, after obtaining the product of the transposed matrix of the query matrix and the key matrix, it is only necessary to set the probabilities corresponding to the filled key points to 0 to ensure that the determined matching key points appear in the original key points of the image. In this way, by setting the feature value of the first matrix feature related to the filled key point coordinates to a preset negative value, it is possible to set the probabilities corresponding to the filled key points to 0, thereby improving the accuracy of key point matching.

[0212] It is understandable that the steps of generating the second mask query matrix and the second mask key matrix in S2 are similar to the above-mentioned process of generating the first mask query matrix and the first mask key matrix based on P1 and P2, and will not be repeated here. That is, when the key point coordinates corresponding to the image at the second moment include the filled key point coordinates, the characteristic values ​​of the first matrix features related to the filled key point coordinates in the mask matrix corresponding to the image at the second moment are set to preset negative values, and the characteristic values ​​of the second matrix features other than the first matrix features in the mask matrix are retained to obtain the updated mask matrix corresponding to the image at the second moment; and then the updated mask matrix is ​​used to generate the second mask query matrix and the second mask key matrix corresponding to the second moment.

[0213] It is understandable that the above formula three will also be used in the process of calculating the self-attention of an image at any moment. Exemplarily, for an image at any moment, when the key point coordinates corresponding to the image include the filled key point coordinates, the feature values ​​of the first matrix features related to the filled key point coordinates in the mask matrix corresponding to the image at that moment can be set to a preset negative value, and the feature values ​​of the second matrix features in the mask matrix except the first matrix features are retained to obtain the updated mask matrix corresponding to the image at that moment, and the mask query matrix and mask key matrix of the image at that moment are generated according to the updated mask matrix. At the same time, the query matrix, key matrix and value matrix corresponding to the image can be determined based on the fusion feature matrix corresponding to the image at that moment. Determine the product of the transposed matrices of the query matrix and the key matrix corresponding to the image at that moment, and the product of the transposed matrices of the mask query matrix and the mask key matrix corresponding to the image at that moment, add the two products to obtain a third addition result, and then use the third addition result as QK in formula three. T , and based on the value matrix corresponding to the image at that moment, the feature dimension d and formula three, the self-attention feature corresponding to the image at that moment is obtained.

[0214] It can be understood that when self-attention processing is performed on images taken by any camera at different times, the mask query matrix and mask key matrix corresponding to each image can be obtained. To simplify the calculation steps, the mask query matrix and mask key matrix obtained at this time can be used as the mask query matrix and mask key matrix used in the cross-attention calculation process.

[0215] Based on the above embodiments, the image matching method provided by the embodiment of the present disclosure uses dynamic multi-batch and dynamic key point number to accelerate reasoning, thereby effectively improving the efficiency and resource utilization of image matching. In addition, when performing attention calculation, the mask matrix is ​​generated in combination with the filled key points, which can eliminate the influence of the filled key points on the calculation accuracy and ensure the calculation accuracy.

[0216] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0217] Based on the same inventive concept, an image matching device corresponding to the image matching method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned image matching method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0218] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Figure 6 FIG. 1 is a schematic diagram of a computer device provided in an embodiment of the present disclosure, including:

[0219] Processor 601, memory 602 and bus 603. The memory 602 stores machine-readable instructions executable by the processor 601, and the processor 601 is used to execute the machine-readable instructions stored in the memory 602. When the machine-readable instructions are executed by the processor 601, the processor 601 performs the following steps: S101: acquiring images taken by multiple cameras at different times, and determining the key point coordinates and descriptors of target key points in each image; S102: for any moment, performing data packaging processing on the key point coordinates and descriptors of the images taken by multiple cameras at the moment, and obtaining key point set tensor data and descriptor set tensor data corresponding to the moment; wherein the packaged key point set tensor data and descriptor set tensor data use the number of images corresponding to the packaging as a feature dimension; and S103: using the key point feature matching model, performing key point matching according to the key point set tensor data and descriptor set tensor data corresponding to different moments, and determining the key point matching results of the images taken by each camera at different moments; the key point matching results are used to characterize the matching between the target key points in the images taken by the camera at different moments.

[0220] The above-mentioned memory 602 includes a memory 6021 and an external memory 6022; the memory 6021 here is also called an internal memory, which is used to temporarily store the calculation data in the processor 601, as well as the data exchanged with the external memory 6022 such as a hard disk. The processor 601 exchanges data with the external memory 6022 through the memory 6021. When the computer device is running, the processor 601 communicates with the memory 602 through the bus 603, so that the processor 601 executes the execution instructions mentioned in the above method embodiment.

[0221] The present disclosure also provides a computer-readable storage medium, on which a computer program product is stored, and when the computer program product is executed by a processor, the steps of the image matching method described in the above method embodiment are executed. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0222] The computer program product of the image matching method provided in the embodiments of the present disclosure includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the steps of the image matching method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0223] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0224] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0225] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0226] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0227] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0228] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0229] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An image matching method, characterized in that: include: Acquire images taken by multiple cameras at different times, and determine key point coordinates and descriptors of target key points in each of the images; For any of the moments, data packaging is performed on the key point coordinates of the images captured by the multiple cameras at the moment and the descriptors to obtain key point set tensor data and descriptor set tensor data corresponding to the moment; wherein the packaged key point set tensor data and the descriptor set tensor data use the number of images corresponding to the packaging as a feature dimension; Using the key point feature matching model, key point matching is performed according to the key point set tensor data and the description subset tensor data corresponding to the different moments, and the key point matching results of the images taken by each camera at the different moments are determined; the key point matching results are used to characterize the matching between the target key points in the images taken by the camera at different moments.

2. The method according to claim 1, characterized in that The camera includes a camera deployed on a vehicle; After determining the key point matching results of the images taken by each camera at different times, the method further includes: Using the key point matching results, determining each key point pair that is successfully matched; The position and posture information of the vehicle is determined using the key point coordinates of the target key points in each of the key point pairs.

3. The method according to claim 1, characterized in that Data packaging processing is performed on the key point coordinates of the images captured by the multiple cameras at the time and the descriptors to obtain key point set tensor data and descriptor set tensor data corresponding to the time, including: Determine a maximum number according to the number of target key points in each of the images corresponding to any of the moments, and use the minimum value between the maximum number and a preset number threshold as the number of targets corresponding to the moment; The number of target key points of each of the images corresponding to any of the moments is adjusted to obtain key point coordinates and descriptors of the adjusted target key points of each of the images, and a mask matrix corresponding to each of the images is generated; wherein the number of target key points of each of the images after adjustment is the target number, and the mask matrix is ​​used to indicate the type of target key points on each of the images; The key point feature matching model is used to perform key point matching according to the key point set tensor data and the description subset tensor data corresponding to the different moments, and key point matching results of the images taken by each camera at the different moments are determined, including: Utilizing the key point feature matching model, multi-layer self-attention and cross-attention operations are performed according to the key point set tensor data and the description subset tensor data corresponding to the different moments, and the mask matrix corresponding to the images at the different moments, to determine the key point matching results of the images taken by each camera at the different moments.

4. The method according to claim 3, characterized in that: The step of adjusting the number of target key points of each image corresponding to any of the moments to obtain the key point coordinates and descriptors of the adjusted target key points of each image includes: For an image whose number of target key points is less than the target number, filling the image with target key points to obtain adjusted key point coordinates and descriptors of the target key points; For an image in which the number of target key points is greater than the target number, removing the target key points from the image to obtain adjusted key point coordinates and descriptors of the target key points; For an image in which the number of target key points is equal to the target number, the key point coordinates and descriptors of the target key points in the image are used as the adjusted key point coordinates and descriptors of the target key points.

5. The method according to claim 3, characterized in that: The key point feature matching model is used to perform multi-layer self-attention and cross-attention operations according to the key point set tensor data and the description subset tensor data corresponding to the different moments, and the mask matrix corresponding to the images at the different moments, to determine the key point matching results of the images taken by each camera at the different moments, including: Determine, by using the key point feature matching model, the sub-key point set tensor data and the sub-descriptor set tensor data corresponding to each camera at the different moments according to the key point set tensor data and the descriptor set tensor data corresponding to the different moments; For any camera, using the first attention calculation layer, self-attention calculations are performed on the sub-keypoint set tensor data and the sub-description subset tensor data corresponding to the camera at the different moments, respectively, to obtain the self-attention features corresponding to the different moments, and cross-attention calculations are performed on the self-attention features corresponding to the different moments, to obtain the cross-attention features corresponding to the different moments; Using the next attention calculation layer, the cross-attention features corresponding to the different moments are respectively self-attention calculated to obtain new self-attention features corresponding to the different moments, and the new self-attention features corresponding to the different moments are respectively cross-attention calculated to obtain new cross-attention features corresponding to the different moments; Returning to the next attention calculation layer, performing self-attention calculation steps on the cross-attention features corresponding to the different moments respectively, until obtaining new cross-attention features corresponding to the different moments respectively output by the last attention calculation layer; Performing feature matching on the new cross-attention features corresponding to the different moments output by the last attention calculation layer to obtain a matching matrix, a matching validity matrix, and a matching score matrix corresponding to the camera; wherein the matching matrix is ​​used to indicate the matching relationship between the target key points in the images taken by the camera at the different moments, the matching validity matrix is ​​used to indicate whether the matching relationship of each of the target key points is valid; and the matching score matrix is ​​used to indicate the confidence of the matching relationship of each of the target key points; The matching matrix, the matching validity matrix and the matching score matrix are used as key point matching results corresponding to the cameras at the different moments.

6. The method according to claim 5, characterized in that The first-layer attention calculation layer is used to perform self-attention calculations on the sub-keypoint set tensor data and the sub-description subset tensor data corresponding to the camera at the different moments, respectively, to obtain the self-attention features corresponding to the different moments, including: Using the first attention calculation layer, position encoding is performed on the key point coordinates in the sub-key point set tensor data corresponding to the camera at the different moments, so as to obtain the position feature matrices corresponding to the different moments; Performing feature fusion on the position feature matrix and the descriptors in the sub-descriptor set tensor data to obtain fused feature matrices corresponding to the different moments; Self-attention calculations are performed on the fusion feature matrices corresponding to the different moments respectively to obtain the self-attention features corresponding to the different moments respectively.

7. The method according to claim 5, characterized in that The cross-attention calculation is performed on the self-attention features corresponding to the different moments to obtain the cross-attention features corresponding to the different moments, including: Determine a first query matrix, a first key matrix, and a first value matrix corresponding to the first moment using the self-attention feature corresponding to the first moment, and determine a first mask query matrix and a first mask key matrix corresponding to the first moment using the mask matrix corresponding to the image at the first moment; Determine a second query matrix, a second key matrix, and a second value matrix corresponding to the second moment by using the self-attention feature corresponding to the second moment, and determine a second mask query matrix and a second mask key matrix corresponding to the second moment by using the mask matrix corresponding to the image at the second moment; the first moment is one of the different moments, and the second moment is another of the different moments; Adding a product of a transposed matrix of the first query matrix and the second key matrix and a product of a transposed matrix of the first mask query matrix and the second mask key matrix, and determining a cross-attention feature corresponding to the first moment based on the obtained first addition result and the second value matrix; Add the product of the second query matrix and the transposed matrix of the first key matrix and the product of the second mask query matrix and the transposed matrix of the first mask key matrix, and determine the cross-attention feature corresponding to the second moment based on the second addition result and the first value matrix.

8. The method according to claim 7, characterized in that The determining, using the mask matrix corresponding to the image at the first moment, a first mask query matrix and a first mask key matrix corresponding to the first moment includes: In a case where the key point coordinates corresponding to the image at the first moment include filled key point coordinates, setting feature values ​​of first matrix features related to the filled key point coordinates in the mask matrix corresponding to the image at the first moment to preset negative values, and retaining feature values ​​of second matrix features other than the first matrix features in the mask matrix, to obtain an updated mask matrix corresponding to the image at the first moment; The updated mask matrix is ​​used to generate a first mask query matrix and a first mask key matrix corresponding to the first moment.

9. A computer device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor performs the steps of the image matching method according to any one of claims 1 to 8.

10. A computer program product, characterized in that The computer program product stores a computer program. When the computer program product is executed by a computer device, the computer device executes the steps of the image matching method according to any one of claims 1 to 8.

Images