Neural network based cross camera image matching method and apparatus

Abstract

The application provides a neural network-based cross-camera image matching method and device, wherein the method comprises: constructing first and second image pyramids with N layers of gradually reduced resolution for first and second images respectively; extracting feature vectors of each layer of the second image pyramid using a deep neural network to obtain N reference feature vectors; performing global search on the Nth layer of the first image pyramid according to the Nth reference feature vector to determine an initial matching region; mapping the initial matching region to the N-1th layer of the first image pyramid to obtain a mapping region, and expanding the mapping region according to a preset expansion amount to form a restricted search region; performing local search on the restricted search region according to the N-1th reference feature vector to obtain a matching region of the current layer; repeating the mapping and local search steps until the first layer of the first image pyramid is reached, and outputting the matching result in the first image. The application does not require distortion correction and a large amount of training data.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, and in particular to a cross-camera image matching method and apparatus based on neural networks. Background Technology

[0002] Image matching is a fundamental technology in computer vision, widely used in tasks such as image registration, 3D reconstruction, and visual navigation. Traditional methods, based on feature point extraction and descriptors, perform stably on conventional perspective images. However, when the images to be matched come from cameras with different projection models, such as fisheye wide-angle images (with significant nonlinear distortion) and ordinary perspective images (with approximately no distortion), the performance of traditional methods drops sharply. This is because the feature distribution of fisheye wide-angle images deviates significantly from the perspective assumption, leading to unstable feature extraction and matching failures.

[0003] For matching fisheye wide-angle images, existing technologies are mainly divided into three categories: (1) Geometric distortion correction + traditional matching: First, the fisheye distortion model is used to reverse the distortion into a perspective image, and then the SIFT (Scale-Invariant Feature Transform) method is applied for matching. This type of method relies heavily on accurate calibration parameters, and the calibration error will be amplified. Different lenses need to be recalibrated, and the universality is poor. (2) Feature matching guided by geometric constraints: The feature points are projected onto a sphere or equidistant plane and matched in combination with geometric constraints. This type of method is usually limited to fisheye-fisheye lenses of the same origin and is difficult to be directly used for cross-model matching of fisheye wide-angle images and ordinary perspective images. (3) Feature extraction optimized for fisheye: such as Tri-SIFT, which modifies the feature detection strategy to adapt to distortion, but still relies on manually designed features and has limited robustness to large-scale differences and content changes across models.

[0004] Furthermore, while multi-resolution image pyramid structures are used in traditional vision to improve scale robustness, such as SIFT, they are typically used only within perspective images or between images of the same model, failing to build an effective coarse-to-fine search framework across projection models (fisheye-perspective).

[0005] In recent years, deep learning methods (such as SuperGlue) have adopted end-to-end networks to directly output matching points, which have strong expressive power, but require a large amount of cross-camera labeled data for training, which limits generalization ability and has high engineering costs.

[0006] Therefore, there is a lack of a fast cross-camera image matching method and device based on neural networks in the existing technology, so as to achieve robust matching between fisheye wide-angle images and ordinary perspective images without the need for precise distortion correction or relying on a large amount of training data. Summary of the Invention

[0007] Therefore, it is necessary to provide a cross-camera image matching method and apparatus based on neural networks to address the aforementioned technical problems.

[0008] In a first aspect, the present invention provides a cross-camera image matching method based on a neural network, comprising:

[0009] Acquire the first image and the second image;

[0010] Construct corresponding first image pyramids and second image pyramids for the first image and the second image, respectively. Each of the first image pyramid and the second image pyramid includes at least N layers of images with progressively decreasing resolution, where N is greater than or equal to 2. The first layer is the original resolution image, and the Nth layer is the lowest resolution image.

[0011] The feature vectors of each layer of the second image pyramid are extracted using a deep neural network to obtain N reference feature vectors;

[0012] Based on the Nth reference feature vector, a global search is performed on the Nth layer image of the first image pyramid to determine the initial matching region that matches the Nth layer image of the second image pyramid;

[0013] The initial matching region is mapped to the (N-1)th layer of the first image pyramid to obtain the corresponding mapped region. The mapped region is then expanded according to a preset expansion amount to form the corresponding restricted search region.

[0014] Based on the (N-1)th reference feature vector, a local search is performed on the restricted search region of the (N-1)th layer of the first image pyramid to obtain the matching region of the current layer;

[0015] Repeat the mapping and local search steps until the first layer of the first image pyramid is reached, and output the matching result in the first image.

[0016] Optionally, acquiring the first image and the second image includes:

[0017] Obtain a first image to be matched and a second image to be matched, wherein the field of view of the first image to be matched is greater than the field of view of the second image to be matched;

[0018] Compare the original sizes of the first image to be matched and the second image to be matched. Using the image with the larger original size as a reference, scale the other image to the same size as the reference to obtain a first aligned image and a second aligned image of the same size.

[0019] The first and second aligned images are converted to different color spaces to obtain the corresponding first and second images, both of which are grayscale images.

[0020] Optionally, constructing corresponding first image pyramids and second image pyramids for the first image and the second image respectively includes:

[0021] According to the preset downsampling ratio, the first image and the second image are downsampled at least once to obtain the corresponding first image pyramid and second image pyramid.

[0022] Optionally, feature vectors of each layer of the second image pyramid are extracted using a deep neural network to obtain N reference feature vectors, including:

[0023] Each layer of the second image pyramid is converted into a normalized tensor, resulting in N reference normalized tensors;

[0024] N reference normalized tensors are input into a pre-trained convolutional neural network to obtain N reference feature maps;

[0025] Pooling operations are performed on each of the N reference feature maps to obtain N reference feature vectors of fixed dimensions.

[0026] Optionally, based on the Nth reference feature vector, a global search is performed on the Nth layer image of the first image pyramid to determine the initial matching region that matches the Nth layer image of the second image pyramid, including:

[0027] Estimate the global scale factor between the first and second images;

[0028] The width and height of the first sliding window are determined based on the global scale factor and the width and height of the Nth layer of the second image pyramid.

[0029] With a first sliding step size, slide the first sliding window on the Nth layer of the first image pyramid. At each sliding position, crop out a first candidate image region with the same size as the first sliding window from the Nth layer of the first image pyramid.

[0030] Extract the first local feature vector corresponding to each first candidate image region;

[0031] Calculate the first similarity between the first local feature vector corresponding to each first candidate image region and the Nth reference feature vector, and determine the first candidate image region with the largest first similarity as the initial matching region.

[0032] Optionally, estimating the global scale factor between the first image and the second image includes:

[0033] First local feature points and second local feature points are extracted from the first image and the second image respectively, and the first local feature points and second local feature points are matched to obtain multiple matching point pairs, wherein each matching point pair includes a first local feature point in the first image and a second local feature point in the second image;

[0034] Multiple matching point pairs are grouped into multiple matching point pairs;

[0035] For each set of matching points, calculate the first distance between two first local feature points in the set and the second distance between two second local feature points in the set, and calculate the ratio of the first distance to the second distance.

[0036] The ratios calculated for all matching point pairs are statistically analyzed, and the average or median obtained from the statistics is used as the global scaling factor.

[0037] Optionally, extract the first local feature vector corresponding to each first candidate image region, including:

[0038] Each first candidate image region is converted into a normalized tensor to obtain the corresponding first normalized tensor;

[0039] Each first normalized tensor is input into a pre-trained convolutional neural network to obtain the corresponding first local feature map;

[0040] Pooling is performed on each first local feature map to obtain the corresponding first local feature vector of fixed dimension.

[0041] Optionally, the initial matching region is mapped to the (N-1)th layer of the first image pyramid to obtain the corresponding mapped region, and the mapped region is expanded according to a preset expansion amount to form the corresponding restricted search region, including:

[0042] Based on the scale ratio between adjacent layers of the first image pyramid, the position coordinates and size of the initial matching region in the Nth layer of the first image pyramid are mapped to the (N-1)th layer of the first image pyramid to obtain the mapped region;

[0043] Based on the mapping region and the preset expansion amount, the mapping region is expanded in all directions to form the restricted search region of the N-1th layer of the first image pyramid.

[0044] Optionally, based on the (N-1)th reference feature vector, a local search is performed on the restricted search region of the (N-1)th layer of the first image pyramid to obtain the matching region of the current layer, including:

[0045] Estimate the global scale factor between the first and second images;

[0046] The width and height of the second sliding window are determined based on the global scale factor and the width and height of the (N-1)th layer of the second image pyramid.

[0047] With a second sliding step size, a second sliding window is slid within a restricted search area of ​​the (N-1)th layer of the first image pyramid, wherein the second sliding step size is smaller than the first sliding step size;

[0048] At each sliding position, a second candidate image region with the same size as the second sliding window is cropped from the (N-1)th layer of the first image pyramid.

[0049] Extract the second local feature vector corresponding to each second candidate image region;

[0050] Calculate the second similarity between the second local feature vector corresponding to each second candidate image region and the (N-1)th reference feature vector, and determine the second candidate image region with the largest second similarity as the matching region of the current layer.

[0051] Optionally, a second local feature vector is extracted for each second candidate image region, including:

[0052] Each second candidate image region is converted into a normalized tensor to obtain the corresponding second normalized tensor;

[0053] Each second normalized tensor is input into a pre-trained convolutional neural network to obtain the corresponding second local feature map;

[0054] Pooling is performed on each second local feature map to obtain the corresponding fixed-dimensional second local feature vector.

[0055] Secondly, the present invention provides a cross-camera image matching device based on a neural network, comprising:

[0056] The acquisition module is used to acquire the first image and the second image;

[0057] The construction module, connected to the acquisition module, is used to construct corresponding first image pyramids and second image pyramids for the first image and the second image, respectively. The first image pyramid and the second image pyramid each include at least N layers of images with progressively decreasing resolution, where N is greater than or equal to 2. The first layer is the original resolution image, and the Nth layer is the lowest resolution image.

[0058] The feature extraction module, connected to the construction module, is used to extract feature vectors of each layer of the second image pyramid using a deep neural network, and obtain N reference feature vectors.

[0059] The global search module, connected to the feature extraction module, is used to perform a global search on the Nth layer of the first image pyramid based on the Nth reference feature vector to determine the initial matching region that matches the Nth layer of the second image pyramid.

[0060] The mapping extension module, connected to the global search module, is used to map the initial matching region to the N-1th layer of the first image pyramid to obtain the corresponding mapping region, and expand the mapping region to form the corresponding restricted search region according to the preset extension amount.

[0061] The local search module, which is connected to the feature extraction module and the mapping expansion module respectively, is used to perform a local search on the restricted search region of the N-1th layer of the first image pyramid based on the N-1th reference feature vector to obtain the matching region of the current layer.

[0062] The iterative control output module is connected to the mapping extension module and the local search module respectively. It is used to control the mapping extension module and the local search module to iterate layer by layer to the first layer of the first image pyramid, and output the matching result in the first image after the iteration is completed.

[0063] The present invention provides a neural network-based cross-camera image matching method and apparatus that eliminates the need for precise distortion correction by directly utilizing the semantic features of deep neural networks for matching. This avoids dependence on fisheye wide-angle lens calibration parameters and offers strong versatility. It does not rely on large amounts of labeled data, employing pre-trained deep neural networks to extract features, thus avoiding the high costs of cross-lens data acquisition and end-to-end training. It exhibits high robustness by using a multi-resolution image pyramid and a coarse-to-fine search strategy, combined with adaptive adjustment of the sliding window by a global scale factor, effectively overcoming the nonlinear distortion and cross-viewpoint scale differences of fisheye wide-angle images, achieving stable matching between fisheye wide-angle images and ordinary perspective images. The computational efficiency is significantly improved by performing a global search only at the lowest resolution layer and fine matching only within limited regions at high-resolution layers, greatly reducing the number of sliding windows and the number of forward propagations of the neural network, meeting the rapid matching requirements of high-resolution images. Attached Figure Description

[0064] Figure 1a A flowchart illustrating a cross-camera image matching method based on a neural network provided in an embodiment of the present invention;

[0065] Figure 1b Another flowchart illustrating the cross-camera image matching method based on neural networks provided in this embodiment of the invention;

[0066] Figure 1c This is a first image to be matched, captured using a fisheye camera;

[0067] Figure 1dThis is a second image to be matched, captured using a standard perspective camera.

[0068] Figure 1e This is another flowchart illustrating the cross-camera image matching method based on neural networks provided in this embodiment of the invention.

[0069] Figure 1f Another schematic diagram of the cross-camera image matching method based on neural networks provided in the embodiments of the present invention;

[0070] Figure 2 A schematic diagram of the circuit module structure of the neural network-based cross-camera image matching device provided in an embodiment of the present invention;

[0071] Figure 3 This is an internal structural diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0073] like Figure 1a As shown, this invention provides a cross-camera image matching method based on neural networks, comprising:

[0074] Step S10: Acquire the first image and the second image;

[0075] In one optional embodiment of the present invention, such as Figure 1b As shown, step S10 specifically includes:

[0076] Step S101: Obtain the first image to be matched and the second image to be matched, wherein the field of view of the first image to be matched is greater than the field of view of the second image to be matched;

[0077] Optionally, the first image to be matched is typically acquired by a fisheye wide-angle lens of a fisheye camera, with a field of view ranging from 135° to 180°, capable of covering an ultra-wide field of view, but exhibiting significant nonlinear distortion, such as... Figure 1c The first image to be matched is captured using a fisheye wide-angle lens of a fisheye camera; the second image to be matched is typically captured using a lens of a conventional perspective camera (such as a standard or telephoto lens), with a field of view of 15°–16°, which approximately conforms to the pinhole imaging model and has relatively small distortion. Figure 1dThis is a second image to be matched, captured using a lens from a standard perspective camera. Because the field of view of the first image to be matched is much larger than that of the second image, the pixel size of the same target object in the first image is smaller than that in the second image, resulting in a significant size difference between the two. Of course, those skilled in the art can choose other suitable images as the first and second images to be matched; this is not limited here.

[0078] Step S102: Compare the original sizes of the first image to be matched and the second image to be matched. Using the image with the larger original size as a reference, scale the other image to the same size as the reference to obtain a first aligned image and a second aligned image with the same size.

[0079] In this invention, the purpose of image scaling is to unify fisheye wide-angle images and ordinary perspective images from different sources and with different original sizes into the same size space, so that they can be used for subsequent construction of the first image pyramid and the second image pyramid, etc.

[0080] It should be noted that if the original size of the first image to be matched is larger than the original size of the second image to be matched, the original size of the first image to be matched is used as a reference, and the second image to be matched is scaled to the same size as the reference. In this case, the first image to be matched does not need to be scaled, and it is the first aligned image. If the original size of the first image to be matched is smaller than the original size of the second image to be matched, the original size of the second image to be matched is used as a reference, and the first image to be matched is scaled to the same size as the reference. In this case, the second image to be matched does not need to be scaled, and it is the second aligned image. If the original size of the first image to be matched is equal to the original size of the second image to be matched, neither the first nor the second image to be matched needs to be scaled. In this case, the first image to be matched is the first aligned image, and the second image to be matched is the second aligned image.

[0081] Step S103: Convert the first aligned image and the second aligned image to different color spaces to obtain the corresponding first image and second image, wherein both the first image and the second image are grayscale images.

[0082] The color space conversion can be performed using existing conversion methods, and this invention is not limited to this. It should be noted that if the first image to be matched and the second image to be matched are grayscale images, then step S103 is unnecessary.

[0083] Step S20: Construct corresponding first image pyramids and second image pyramids for the first image and the second image respectively. The first image pyramid and the second image pyramid each include at least N layers of images with progressively decreasing resolution, where N is greater than or equal to 2. The first layer is the original resolution image, and the Nth layer is the lowest resolution image.

[0084] In an optional embodiment of the present invention, the step S20 of constructing corresponding first image pyramids and second image pyramids for the first image and the second image respectively specifically includes: performing at least one step-by-step downsampling on the first image and the second image respectively according to a preset downsampling ratio to obtain corresponding first image pyramids and second image pyramids.

[0085] The preset downsampling ratio ranges from (0,1), and those skilled in the art can choose flexibly according to actual needs; no limitation is made here. Preferably, the preset downsampling ratio is 0.5, meaning that after each downsampling, the width and height of the image are reduced to half of the previous layer's image.

[0086] For the construction methods of the first and second image pyramids, those skilled in the art can choose existing image pyramid construction methods, such as the Gaussian pyramid construction method, according to actual needs, which will not be elaborated here.

[0087] Specifically, taking a first image with a size of 640×480 pixels as an example, a Gaussian pyramid with N=5 layers (i.e., the first image pyramid) is constructed: the first layer image is the first image, with a size of 640×480; Gaussian filtering is applied to the first layer image and sampling is performed at alternating rows and columns to obtain the second layer image, with a size of 320×240; the same operation is performed on the second layer image to obtain the third layer image, with a size of 160×120; further downsampling is performed to obtain the fourth layer image, with a size of 80×60; further downsampling is performed to obtain the fifth layer image, with a size of 40×30. The construction method of the second image pyramid is the same as that of the first image pyramid, and will not be repeated here.

[0088] Step S30: Use a deep neural network to extract feature vectors from each layer of the second image pyramid to obtain N reference feature vectors;

[0089] In one optional embodiment of the present invention, such as Figure 1e As shown, step S30 specifically includes:

[0090] Step S301: Convert each layer of the second image pyramid into a normalized tensor to obtain N reference normalized tensors;

[0091] Step S302: Input the N reference normalized tensors into the pre-trained convolutional neural network to obtain N reference feature maps;

[0092] Step S303: Perform pooling operations on the N reference feature maps respectively to obtain N reference feature vectors with fixed dimensions.

[0093] In this invention, the pre-trained convolutional neural network (i.e., deep neural network) can be an existing pre-trained ResNet-18. Those skilled in the art can choose flexibly according to actual needs, and no limitation is made here. ResNet (full name: Residual Network) is a deep neural network architecture that solves the gradient vanishing and representation bottleneck problems during deep neural network training by introducing residual connections. ResNet-18 is a lightweight model in the ResNet series.

[0094] For example, suppose the second image pyramid constructed in step S20 has 5 layers. The size of the grayscale image in layer 1 is 640×480, the size of the grayscale image in layer 2 is 320×240, the size of the grayscale image in layer 3 is 160×120, the size of the grayscale image in layer 4 is 80×60, and the size of the grayscale image in layer 5 is 40×30. A pre-trained ResNet-18 is used as the feature extraction network, and its input requirement is a 224×224 RGB normalized tensor, with pixel values ​​normalized to the [0,1] interval. The specific processes of steps S301 to S303 are as follows:

[0095] For the 40×30 grayscale image at level 5 of the second image pyramid, bilinear interpolation is first used to scale it to 224×224, and then the three channels are converted to RGB format. Dividing by 255 yields the first reference normalized tensor (size 1×3×224×224). Similarly, the 80×60 grayscale image at level 4 of the second image pyramid is scaled to 224×224, and the three channels are converted to RGB format. Dividing by 255 yields the second reference normalized tensor (size 1×3×224×224). The same operation is performed on the grayscale images at levels 3 (160×120), 2 (320×240), and 1 (640×480) of the second image pyramid, respectively. A total of five reference normalized tensors are obtained.

[0096] Five reference normalized tensors are sequentially input into the pre-trained ResNet-18, and the reference feature map output of the last convolutional layer is extracted (such as the layer4 output of ResNet-18, with 512 channels and a reference feature map size of 7×7), resulting in five reference feature maps (each reference feature map has a size of 512×7×7).

[0097] Global average pooling is performed on each reference feature map to obtain a 512-dimensional reference feature vector, which is a reference feature vector with 5 fixed dimensions.

[0098] In the above process, each layer of images is independently resized and standardized to ensure that the input convolutional neural network has a consistent size; global pooling is used to eliminate the size difference of the reference feature maps, so that the dimension of the reference feature vector output by each layer is fixed, which facilitates subsequent similarity calculation.

[0099] Step S40: Based on the Nth reference feature vector, perform a global search on the Nth layer of the first image pyramid to determine the initial matching region that matches the Nth layer of the second image pyramid;

[0100] In an optional embodiment of the present invention, step S40 specifically includes:

[0101] Step S401: Estimate the global scale factor between the first image and the second image;

[0102] In an optional embodiment of the present invention, step S401 specifically includes:

[0103] Step S4011: Extract first local feature points and second local feature points from the first image and the second image respectively, and match the first local feature points and the second local feature points to obtain multiple matching point pairs, wherein each matching point pair includes a first local feature point in the first image and a second local feature point in the second image;

[0104] For the extraction and matching methods of the first and second local feature points in step S4011, those skilled in the art can flexibly choose according to actual needs, and there are no limitations here. For example, the SIFT method, ORB (Oriented Fast and Rotated BRIEF) method, etc. can be used.

[0105] Step S4012: Combine multiple matching point pairs into groups of two to obtain multiple matching point pair groups;

[0106] Step S4013: For each set of matching points, calculate the first distance between two first local feature points in the set and the second distance between two second local feature points in the set, and calculate the ratio of the first distance to the second distance;

[0107] Step S4014: Statistically calculate the ratios of all matching point pairs and use the average or median as the global scaling factor.

[0108] For example, suppose that after step S4011, a total of 4 matching point pairs are obtained, denoted as P1 and Q1, P2 and Q2, P3 and Q3, and P4 and Q4, and the coordinates of each point are shown in Table 1:

[0109] Table 1

[0110]

[0111] Choose any two from the above four matching point pairs to form a group, and there are a total of The group, specifically:

[0112] Group 1: P1 and Q1, P2 and Q2; Group 2: P1 and Q1, P3 and Q3;

[0113] Group 3: P1 and Q1, P4 and Q4; Group 4: P2 and Q2, P3 and Q3;

[0114] Group 5: P2 and Q2, P4 and Q4; Group 6: P3 and Q3, P4 and Q4.

[0115] Taking the first group as an example:

[0116] In the first image, the Euclidean distance (i.e., the first distance) between P1(100,50) and P2(200,100) is: .

[0117] In the second image, the Euclidean distance (i.e., the second distance) between Q1(50,25) and Q2(100,50) is: .

[0118] Calculate the ratio of the first distance to the second distance: .

[0119] The same principle applies to the other groups, and will not be repeated here. The final calculated ratio of the first distance to the second distance for all other groups is 2.00.

[0120] The ratios calculated for all matched point pairs are statistically analyzed, and the average or median obtained from this statistical analysis is used as the global scaling factor. Taking the average as an example, the global scaling factor... Taking the median as an example, the global scaling factor .

[0121] Step S402: Determine the width and height of the first sliding window based on the global scale factor and the width and height of the Nth layer image of the second image pyramid;

[0122] Optionally, the width of the first sliding window on the Nth layer of the first image pyramid is calculated using the following formula: ,in, Let be the width of the first sliding window on the Nth layer of the first image pyramid. Let N be the width of the Nth layer of the second image pyramid. This is the global scale factor.

[0123] Optionally, the height of the first sliding window on the Nth layer of the first image pyramid is calculated using the following formula: ,in, The height of the first sliding window on the Nth layer of the first image pyramid. The height of the Nth layer of the second image pyramid. This is the global scale factor.

[0124] It should be noted that if the calculated width and height of the first sliding window are not integers, they will be rounded up to the nearest integer.

[0125] For example, if the number of layers in the second image pyramid is N=5, the width of the Nth layer (i.e., the 5th layer, the lowest resolution layer) of the second image pyramid... Pixels, height Pixels, the width of the first sliding window on the 5th layer of the first image pyramid. Pixel; the height of the first sliding window on the 5th layer of the first image pyramid. = Pixel.

[0126] It should be noted that the width and height of the first sliding window on the other layers of the first image pyramid are similar, and will not be repeated here.

[0127] Step S403: With a first sliding step size, slide the first sliding window on the Nth layer image of the first image pyramid. At each sliding position, crop out a first candidate image region with the same size as the first sliding window from the Nth layer image of the first image pyramid.

[0128] The first sliding step size is preferably 2 pixels, but those skilled in the art can choose flexibly according to actual needs, and it is not limited here. It should be noted that when the first sliding window slides on the Nth layer of the first image pyramid, it must be ensured that the first sliding window is completely within the Nth layer of the first image pyramid. If the first sliding window is not completely within the Nth layer of the first image pyramid, the sliding position is discarded. The other layers of the first image pyramid are analogous to this, and will not be described in detail here.

[0129] Continuing the previous example, on the 5th layer (40×30) of the first image pyramid, starting from the top-left corner coordinates (x, y), slide the first sliding window (20×15) with a first sliding step of 2 pixels. Since the first sliding window must be completely within the 5th layer of the first image pyramid, the range of x is 0 ≤ x ≤ 40 - 20 = 20, and the range of y is 0 ≤ y ≤ 30 - 15 = 15. That is, the actual last effective... There are 11 in total; the last one that actually came into effect was 11. There are 8 sliding positions in total, with a total of 11 × 8 = 88 sliding positions. The other layers of the first image pyramid follow the same pattern, and will not be described in detail here.

[0130] Step S404: Extract the first local feature vector corresponding to each first candidate image region;

[0131] In an optional embodiment of the present invention, step S404 specifically includes:

[0132] Step S4041: Convert each first candidate image region into a normalized tensor to obtain the corresponding first normalized tensor;

[0133] Step S4042: Input each first normalized tensor into a pre-trained convolutional neural network to obtain the corresponding first local feature map;

[0134] Step S4043: Perform pooling operation on each first local feature map to obtain the corresponding fixed-dimensional first local feature vector.

[0135] For the specific process of steps S4041 to S4043 of the present invention, please refer to the specific process of steps S301 to S303 of the present invention, which will not be repeated here.

[0136] Step S405: Calculate the first similarity between the first local feature vector corresponding to each first candidate image region and the Nth reference feature vector, and determine the first candidate image region with the largest first similarity as the initial matching region.

[0137] In this invention, the first similarity can be calculated using the existing cosine similarity calculation method, but those skilled in the art can also choose other similarity calculation methods (such as Euclidean distance, etc.) according to actual needs, and this invention does not limit this.

[0138] Step S50: Map the initial matching region to the (N-1)th layer of the first image pyramid to obtain the corresponding mapping region, and expand the mapping region according to the preset expansion amount to form the corresponding restricted search region;

[0139] In one optional embodiment of the present invention, such as Figure 1fAs shown, step S50 specifically includes:

[0140] Step S501: Based on the scale ratio between adjacent layers of the first image pyramid, map the position coordinates and size of the initial matching region in the Nth layer image of the first image pyramid to the (N-1)th layer image of the first image pyramid to obtain the mapped region;

[0141] The scale ratio between adjacent levels of the first image pyramid can be calculated using the following formula: ,in, This represents the scale ratio between adjacent levels of the pyramid in the first image. This is the preset downsampling ratio. At this point, the mapping relationship is: ,in, Let x be the x-coordinate of the initial matching region in the Nth layer of the first image pyramid. Let be the ordinate of the initial matching region in the Nth layer of the first image pyramid. This represents the scale ratio between adjacent levels of the pyramid in the first image. Let x be the x-coordinate mapped to the (N-1)th layer of the first image pyramid. Let be the ordinate mapped to the (N-1)th layer of the first image pyramid.

[0142] For example, if the scale ratio between adjacent levels of the first image pyramid... Furthermore, on the Nth layer (lowest resolution layer) of the first image pyramid, the coordinates of the top-left corner of the initial matching region are... ,width ,high After mapping to the (N-1)th layer of the first image pyramid, the coordinates of the top-left corner of the mapped region are... ,width ,high This mapped region is the corresponding region on the (N-1)th layer of the first image pyramid.

[0143] Step S502: Based on the mapping area and the preset expansion amount, expand the mapping area in all directions to form the restricted search area of ​​the N-1th layer of the first image pyramid.

[0144] The preset expansion amount can be flexibly selected by those skilled in the art according to actual needs, and is not limited here. Preferably, the value range of the preset expansion amount is [5, 20]. The restricted search region of the (N-1)th layer of the first image pyramid can be represented as... ,in, This represents the restricted search region on the (N-1)th layer of the first image pyramid; and Let x and y be the x and y coordinates of the top-left corner of the mapped region on the (N-1)th layer of the first image pyramid. and Let N be the width and height of the mapped region on the (N-1)th layer of the first image pyramid; This is the preset expansion amount. The other layers of the first image pyramid follow the same principle, and will not be elaborated further here.

[0145] Step S60: Based on the (N-1)th reference feature vector, perform a local search on the restricted search region of the (N-1)th layer of the first image pyramid to obtain the matching region of the current layer;

[0146] In an optional embodiment of the present invention, step S60 specifically includes:

[0147] Step S601: Estimate the global scale factor between the first image and the second image;

[0148] Step S602: Determine the width and height of the second sliding window based on the global scale factor and the width and height of the (N-1)th layer image of the second image pyramid;

[0149] The specific processes of steps S601 to S602 of the present invention can be referred to the specific processes of steps S401 to S402 of the present invention, and will not be repeated here. It should be noted that the global scale factor in step S401 and step S601 is the same global scale factor. In other words, step S602 can directly use the global scale factor calculated in step S401, that is, step S601 can be omitted at this time. In addition, if the calculated width and height of the second sliding window are not integers, they are adjusted to integers by rounding up.

[0150] Step S603: With a second sliding step size, slide the second sliding window within the restricted search area of ​​the (N-1)th layer of the first image pyramid, wherein the second sliding step size is smaller than the first sliding step size;

[0151] The second sliding step size is preferably 1 pixel, but those skilled in the art can choose flexibly according to actual needs, and it is not limited here. It should be noted that when the second sliding window slides on the restricted search area of ​​the (N-1)th layer of the first image pyramid, it must be ensured that the second sliding window is completely within the restricted search area of ​​the (N-1)th layer of the first image pyramid. If the second sliding window is not completely within the restricted search area of ​​the (N-1)th layer of the first image pyramid, the sliding position is discarded.

[0152] Step S604: At each sliding position, crop out a second candidate image region with the same size as the second sliding window from the (N-1)th layer image of the first image pyramid;

[0153] For the specific process of steps S603 and S604 of the present invention, please refer to the specific process of step S403 of the present invention, which will not be repeated here.

[0154] Step S605: Extract the second local feature vector corresponding to each second candidate image region;

[0155] In an optional embodiment of the present invention, step S605 specifically includes:

[0156] Step S6051: Convert each second candidate image region into a normalized tensor to obtain the corresponding second normalized tensor;

[0157] Step S6052: Input each second normalized tensor into the pre-trained convolutional neural network to obtain the corresponding second local feature map;

[0158] Step S6053: Perform pooling operation on each second local feature map to obtain the corresponding fixed-dimensional second local feature vector.

[0159] For the specific process of steps S6051 to S6053 of the present invention, please refer to the specific process of steps S301 to S303 of the present invention, which will not be repeated here.

[0160] Step S606: Calculate the second similarity between the second local feature vector corresponding to each second candidate image region and the (N-1)th reference feature vector, and determine the second candidate image region with the largest second similarity as the matching region of the current layer.

[0161] In this invention, the second similarity can be calculated using the cosine similarity method, but those skilled in the art can also choose other similarity calculation methods (such as Euclidean distance, etc.) according to actual needs, and this invention does not limit this.

[0162] Step S70: Repeat the mapping and local search steps until the first layer of the first image pyramid is reached, and output the matching results in the first image.

[0163] The matching result includes: the position coordinates, width, height, and matching confidence of the first layer matching region of the first image pyramid in the first image; wherein the matching confidence is the maximum similarity calculated when matching at the first layer.

[0164] It should be noted that steps S401 to S405, S4011 to S4014, S4041 to S4043, S601 to S606, and S6051 to S6053 are for ease of description only and are not shown in the figure.

[0165] The neural network-based cross-camera image matching method provided by this invention does not require precise distortion correction. It directly utilizes the semantic features of deep neural networks for matching, avoiding dependence on fisheye wide-angle lens calibration parameters and exhibiting strong versatility. It does not rely on a large amount of labeled data, but uses pre-trained deep neural networks to extract features, avoiding the high costs of cross-lens data acquisition and end-to-end training. It has high robustness, effectively overcoming the nonlinear distortion and cross-viewpoint scale differences of fisheye wide-angle images by using a multi-resolution image pyramid and a coarse-to-fine search strategy, combined with adaptive adjustment of the sliding window by a global scale factor, achieving stable matching between fisheye wide-angle images and ordinary perspective images. It significantly improves computational efficiency, performing global search only at the lowest resolution layer and fine matching only within limited areas at the high-resolution layer, greatly reducing the number of sliding windows and the number of forward propagations of the neural network, meeting the needs of rapid matching of high-resolution images.

[0166] Based on the same inventive concept, embodiments of the present invention also provide a neural network-based cross-camera image matching apparatus for implementing the aforementioned neural network-based cross-camera image matching method. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the neural network-based cross-camera image matching apparatus provided below can be found in the limitations of the neural network-based cross-camera image matching method described above, and will not be repeated here.

[0167] like Figure 2As shown, the present invention provides a cross-camera image matching device based on a neural network, comprising: an acquisition module 21, a construction module 22, a feature extraction module 23, a global search module 24, a mapping expansion module 25, a local search module 26, and an iterative control output module 27; wherein, the acquisition module 21 is used to acquire a first image and a second image; the construction module 22, connected to the acquisition module 21, is used to construct corresponding first image pyramids and second image pyramids for the first image and the second image, respectively, wherein both the first image pyramid and the second image pyramid include at least N layers of images with progressively decreasing resolution, N being greater than or equal to 2, the first layer being the original resolution image, and the Nth layer being the lowest resolution image; the feature extraction module 23, connected to the construction module 22, is used to extract feature vectors of each layer of the second image pyramid using a deep neural network to obtain N reference feature vectors; the global search module 24, connected to the feature extraction module 23, is used to... The Nth reference feature vector is used to perform a global search on the Nth layer of the first image pyramid to determine the initial matching region that matches the Nth layer of the second image pyramid. The mapping expansion module 25, connected to the global search module 24, is used to map the initial matching region to the (N-1)th layer of the first image pyramid to obtain the corresponding mapping region, and expand the mapping region to form the corresponding restricted search region according to the preset expansion amount. The local search module 26, connected to the feature extraction module 23 and the mapping expansion module 25 respectively, is used to perform a local search on the restricted search region of the (N-1)th layer of the first image pyramid according to the (N-1)th reference feature vector to obtain the matching region of the current layer. The iterative control output module 27, connected to the mapping expansion module 25 and the local search module 26 respectively, is used to control the mapping expansion module and the local search module to iterate layer by layer to the first layer of the first image pyramid, and output the matching result in the first image after the iteration is completed.

[0168] Optionally, the acquisition module 21 is specifically used for: acquiring a first image to be matched and a second image to be matched, wherein the field of view of the first image to be matched is greater than the field of view of the second image to be matched; comparing the original sizes of the first image to be matched and the second image to be matched, using the image with the larger original size as a reference, scaling the other image to the same size as the reference, to obtain a first aligned image and a second aligned image of the same size; and converting the first aligned image and the second aligned image to a color space to obtain a corresponding first image and a second image, wherein both the first image and the second image are grayscale images.

[0169] Optionally, the construction module 22 is specifically used to: perform at least one step-by-step downsampling on the first image and the second image according to a preset downsampling ratio to obtain the corresponding first image pyramid and second image pyramid.

[0170] Optionally, the feature extraction module 23 is specifically used to: convert each layer of the second image pyramid into a standardized tensor to obtain N reference standardized tensors; input the N reference standardized tensors into a pre-trained convolutional neural network to obtain N reference feature maps; and perform pooling operations on the N reference feature maps to obtain N fixed-dimensional reference feature vectors.

[0171] Optionally, the global search module 24 is specifically used for: estimating the global scale factor between the first image and the second image; determining the width and height of the first sliding window based on the global scale factor and the width and height of the Nth layer image of the second image pyramid; sliding the first sliding window on the Nth layer image of the first image pyramid with a first sliding step size, and at each sliding position, cropping a first candidate image region with the same size as the first sliding window from the Nth layer image of the first image pyramid; extracting the first local feature vector corresponding to each first candidate image region; calculating the first similarity between the first local feature vector corresponding to each first candidate image region and the Nth reference feature vector, and determining the first candidate image region with the largest first similarity as the initial matching region.

[0172] Optionally, estimating the global scale factor between the first image and the second image includes: extracting first local feature points and second local feature points in the first image and the second image respectively, and matching the first local feature points and the second local feature points to obtain multiple matching point pairs, wherein each matching point pair includes a first local feature point in the first image and a second local feature point in the second image; grouping the multiple matching point pairs into multiple matching point pair groups; for each matching point pair group, calculating a first distance between two first local feature points in the matching point pair group and a second distance between two second local feature points in the matching point pair group, and calculating the ratio of the first distance to the second distance; statistically analyzing the ratios calculated for all matching point pair groups, and using the statistically obtained average or median as the global scale factor.

[0173] Optionally, extracting the first local feature vector corresponding to each first candidate image region includes: converting each first candidate image region into a standardized tensor to obtain the corresponding first standardized tensor; inputting each first standardized tensor into a pre-trained convolutional neural network to obtain the corresponding first local feature map; and performing pooling operation on each first local feature map to obtain the corresponding fixed-dimensional first local feature vector.

[0174] Optionally, the mapping expansion module 25 is specifically used to: map the position coordinates and size of the initial matching region in the Nth layer image of the first image pyramid to the (N-1)th layer image of the first image pyramid according to the scale ratio between adjacent layers of the first image pyramid, to obtain the mapping region; and expand the mapping region in all directions based on the mapping region and a preset expansion amount to form a restricted search region of the (N-1)th layer image of the first image pyramid.

[0175] Optionally, the local search module 26 is specifically used for: estimating the global scale factor between the first image and the second image; determining the width and height of the second sliding window based on the global scale factor and the width and height of the (N-1)th layer image of the second image pyramid; sliding the second sliding window within the restricted search area of ​​the (N-1)th layer image of the first image pyramid with a second sliding step size, wherein the second sliding step size is smaller than the first sliding step size; at each sliding position, cropping a second candidate image region with the same size as the second sliding window from the (N-1)th layer image of the first image pyramid; extracting the second local feature vector corresponding to each second candidate image region; calculating the second similarity between the second local feature vector corresponding to each second candidate image region and the (N-1)th reference feature vector, and determining the second candidate image region with the largest second similarity as the matching region of the current layer.

[0176] Optionally, extracting the second local feature vector corresponding to each second candidate image region includes: converting each second candidate image region into a standardized tensor to obtain the corresponding second standardized tensor; inputting each second standardized tensor into a pre-trained convolutional neural network to obtain the corresponding second local feature map; and performing pooling operation on each second local feature map to obtain the corresponding fixed-dimensional second local feature vector.

[0177] Optionally, the matching result includes: the position coordinates, width, height, and matching confidence of the first layer matching region of the first image pyramid in the first image; wherein, the matching confidence is the maximum similarity calculated when matching at the first layer.

[0178] The neural network-based cross-camera image matching device provided by this invention does not require precise distortion correction. It directly utilizes the semantic features of deep neural networks for matching, avoiding dependence on fisheye wide-angle lens calibration parameters and exhibiting strong versatility. It does not rely on a large amount of labeled data, but uses pre-trained deep neural networks to extract features, avoiding the high costs of cross-lens data acquisition and end-to-end training. It has high robustness, effectively overcoming the nonlinear distortion and cross-viewpoint scale differences of fisheye wide-angle images by using a multi-resolution image pyramid and a coarse-to-fine search strategy, combined with adaptive adjustment of the sliding window by a global scale factor, achieving stable matching between fisheye wide-angle images and ordinary perspective images. It significantly improves computational efficiency by performing a global search only at the lowest resolution layer and fine matching only within the restricted area at the high-resolution layer, greatly reducing the number of sliding windows and the number of forward propagations of the neural network, thus meeting the needs of rapid matching of high-resolution images.

[0179] It should be noted that "multiple" in this invention includes two or more.

[0180] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0181] Each module in the devices of this invention can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0182] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data required for or generated by the aforementioned neural network-based cross-camera image matching method. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a neural network-based cross-camera image matching method.

[0183] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a neural network-based cross-camera image matching method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0184] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0185] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0186] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0187] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0188] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties.

[0189] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided by this invention may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided by this invention may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0190] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0191] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A cross-camera image matching method based on neural networks, characterized in that, include: Acquire the first image and the second image; Construct corresponding first image pyramids and second image pyramids for the first image and the second image, respectively. Each of the first image pyramid and the second image pyramid includes at least N layers of images with progressively decreasing resolution, where N is greater than or equal to 2. The first layer is the original resolution image, and the Nth layer is the lowest resolution image. The feature vectors of each layer of the second image pyramid are extracted using a deep neural network to obtain N reference feature vectors; Based on the Nth reference feature vector, a global search is performed on the Nth layer image of the first image pyramid to determine the initial matching region that matches the Nth layer image of the second image pyramid; The initial matching region is mapped to the (N-1)th layer of the first image pyramid to obtain the corresponding mapped region. The mapped region is then expanded according to a preset expansion amount to form the corresponding restricted search region. Based on the (N-1)th reference feature vector, a local search is performed on the restricted search region of the (N-1)th layer of the first image pyramid to obtain the matching region of the current layer; Repeat the mapping and local search steps until the first layer of the first image pyramid is reached, and output the matching result in the first image.

2. The method according to claim 1, characterized in that, The acquisition of the first image and the second image includes: Obtain a first image to be matched and a second image to be matched, wherein the field of view of the first image to be matched is greater than the field of view of the second image to be matched; Compare the original sizes of the first image to be matched and the second image to be matched. Using the image with the larger original size as a reference, scale the other image to the same size as the reference to obtain a first aligned image and a second aligned image of the same size. The first and second aligned images are converted to different color spaces to obtain the corresponding first and second images, both of which are grayscale images.

3. The method according to claim 1, characterized in that, The construction of corresponding first image pyramids and second image pyramids for the first image and the second image, respectively, includes: According to the preset downsampling ratio, the first image and the second image are downsampled at least once to obtain the corresponding first image pyramid and second image pyramid.

4. The method according to claim 1, characterized in that, The method of extracting feature vectors from each layer of the second image pyramid using a deep neural network yields N reference feature vectors, including: Each layer of the second image pyramid is converted into a normalized tensor, resulting in N reference normalized tensors; N reference normalized tensors are input into a pre-trained convolutional neural network to obtain N reference feature maps; Pooling operations are performed on each of the N reference feature maps to obtain N reference feature vectors of fixed dimensions.

5. The method according to claim 1, characterized in that, The step of performing a global search on the Nth layer of the first image pyramid based on the Nth reference feature vector to determine the initial matching region that matches the Nth layer of the second image pyramid includes: Estimate the global scale factor between the first and second images; The width and height of the first sliding window are determined based on the global scale factor and the width and height of the Nth layer of the second image pyramid. With a first sliding step size, slide the first sliding window on the Nth layer of the first image pyramid. At each sliding position, crop out a first candidate image region with the same size as the first sliding window from the Nth layer of the first image pyramid. Extract the first local feature vector corresponding to each first candidate image region; Calculate the first similarity between the first local feature vector corresponding to each first candidate image region and the Nth reference feature vector, and determine the first candidate image region with the largest first similarity as the initial matching region.

6. The method according to claim 5, characterized in that, The estimation of the global scale factor between the first image and the second image includes: First local feature points and second local feature points are extracted from the first image and the second image respectively, and the first local feature points and second local feature points are matched to obtain multiple matching point pairs, wherein each matching point pair includes a first local feature point in the first image and a second local feature point in the second image; Multiple matching point pairs are grouped into multiple matching point pairs; For each set of matching points, calculate the first distance between two first local feature points in the set and the second distance between two second local feature points in the set, and calculate the ratio of the first distance to the second distance. The ratios calculated for all matching point pairs are statistically analyzed, and the average or median obtained from the statistics is used as the global scaling factor.

7. The method according to claim 5, characterized in that, The step of extracting the first local feature vector corresponding to each first candidate image region includes: Each first candidate image region is converted into a normalized tensor to obtain the corresponding first normalized tensor; Each first normalized tensor is input into a pre-trained convolutional neural network to obtain the corresponding first local feature map; Pooling is performed on each first local feature map to obtain the corresponding first local feature vector of fixed dimension.

8. The method according to claim 1, characterized in that, The process of mapping the initial matching region to the (N-1)th layer of the first image pyramid to obtain the corresponding mapped region, and expanding the mapped region to form the corresponding restricted search region according to a preset expansion amount, includes: Based on the scale ratio between adjacent layers of the first image pyramid, the position coordinates and size of the initial matching region in the Nth layer of the first image pyramid are mapped to the (N-1)th layer of the first image pyramid to obtain the mapped region; Based on the mapping region and the preset expansion amount, the mapping region is expanded in all directions to form the restricted search region of the N-1th layer of the first image pyramid.

9. The method according to claim 1, characterized in that, The step of performing a local search on the restricted search region of the (N-1)th layer of the first image pyramid based on the (N-1)th reference feature vector to obtain the matching region of the current layer includes: Estimate the global scale factor between the first and second images; The width and height of the second sliding window are determined based on the global scale factor and the width and height of the (N-1)th layer of the second image pyramid. With a second sliding step size, a second sliding window is slid within a restricted search area of ​​the (N-1)th layer of the first image pyramid, wherein the second sliding step size is smaller than the first sliding step size; At each sliding position, a second candidate image region with the same size as the second sliding window is cropped from the (N-1)th layer of the first image pyramid. Extract the second local feature vector corresponding to each second candidate image region; Calculate the second similarity between the second local feature vector corresponding to each second candidate image region and the (N-1)th reference feature vector, and determine the second candidate image region with the largest second similarity as the matching region of the current layer.

10. The method according to claim 9, characterized in that, The step of extracting the second local feature vector corresponding to each second candidate image region includes: Each second candidate image region is converted into a normalized tensor to obtain the corresponding second normalized tensor; Each second normalized tensor is input into a pre-trained convolutional neural network to obtain the corresponding second local feature map; Pooling is performed on each second local feature map to obtain the corresponding fixed-dimensional second local feature vector.

11. A cross-camera image matching device based on a neural network, characterized in that, include: The acquisition module is used to acquire the first image and the second image; The construction module, connected to the acquisition module, is used to construct corresponding first image pyramids and second image pyramids for the first image and the second image, respectively. The first image pyramid and the second image pyramid each include at least N layers of images with progressively decreasing resolution, where N is greater than or equal to 2. The first layer is the original resolution image, and the Nth layer is the lowest resolution image. The feature extraction module, connected to the construction module, is used to extract feature vectors of each layer of the second image pyramid using a deep neural network, and obtain N reference feature vectors. The global search module, connected to the feature extraction module, is used to perform a global search on the Nth layer of the first image pyramid based on the Nth reference feature vector to determine the initial matching region that matches the Nth layer of the second image pyramid. The mapping extension module, connected to the global search module, is used to map the initial matching region to the N-1th layer of the first image pyramid to obtain the corresponding mapping region, and expand the mapping region to form the corresponding restricted search region according to the preset extension amount. The local search module, which is connected to the feature extraction module and the mapping expansion module respectively, is used to perform a local search on the restricted search region of the N-1th layer of the first image pyramid based on the N-1th reference feature vector to obtain the matching region of the current layer. The iterative control output module is connected to the mapping extension module and the local search module respectively. It is used to control the mapping extension module and the local search module to iterate layer by layer to the first layer of the first image pyramid, and output the matching result in the first image after the iteration is completed.

Images