Image feature matching method, electronic equipment, storage medium and program product

Abstract

The invention relates to an image feature matching method, electronic equipment, a storage medium and a program product, and the method is characterized in that the method comprises the steps: collecting a first image through a first camera of the electronic equipment; acquiring a second image through a second camera of the electronic equipment; wherein the view field area of the second camera at least covers the view field area of the first camera; cutting the second image to obtain a third image; and performing feature matching based on the first image and the third image to obtain a matched image. According to the embodiment of the invention, the image feature matching efficiency of the electronic equipment can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision, and in particular to an image feature matching method, an electronic device, a storage medium, and a program product. Background Art

[0002] In related technologies, electronic devices are usually equipped with multiple cameras working simultaneously, and it is necessary to obtain the relationship between images taken by different cameras through image feature extraction and matching operations. However, there are many differences between multiple camera images, such as field of view area and baseline, which makes the image feature extraction and matching results of low quality. In addition, the existing image feature matching methods are difficult to meet the requirements of real-time computing in terms of performance and power consumption of electronic devices. Summary of the Invention

[0003] To overcome the problems existing in the related art, the present disclosure provides an image feature matching method, electronic device, storage medium, and program product, which can improve the efficiency of image feature matching in electronic devices.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image feature matching method, comprising:

[0005] Capturing a first image through a first camera of the electronic device;

[0006] Capturing a second image through a second camera of the electronic device; wherein the field of view of the second camera at least covers the field of view of the first camera;

[0007] cropping the second image to obtain a third image;

[0008] Feature matching is performed based on the first image and the third image to obtain a matching image.

[0009] In some embodiments, cropping the second image to obtain the third image includes:

[0010] determining a target cropping ratio for cropping the second image based on preset calibration information of the first camera and the second camera;

[0011] determining a center offset value between a center point of the first image and a center point of the second image based on a size of the first image and a size of the second image;

[0012] determining a cropping area based on the target cropping ratio, the size of the second image, and the center offset value;

[0013] The image contained in the cropped area in the second image is scaled to obtain the third image.

[0014] In some embodiments, determining a target cropping ratio for cropping the second image based on preset calibration information of the first camera and the second camera includes:

[0015] Determining, based on the preset calibration information, a first cropping ratio of the second image in a first direction and a second cropping ratio in a second direction; wherein the first direction is perpendicular to the second direction;

[0016] The smallest cropping ratio between the first cropping ratio and the second cropping ratio is used as the target cropping ratio.

[0017] In some embodiments, determining a center offset value between a center point of the first image and a center point of the second image based on a size of the first image and a size of the second image includes:

[0018] determining homogeneous coordinates of a center point of the first image based on a size of the first image;

[0019] Determining the homogeneous coordinates of the center point of the second image based on the preset calibration information, the homogeneous coordinates of the center point of the first image, the preset spatial depth of the first image, and the preset spatial depth of the second image;

[0020] The center offset value is determined based on the homogeneous coordinates of the center point of the second image and the size of the second image.

[0021] In some embodiments, determining the cropping area based on the target cropping ratio, the size of the second image, and the center offset value includes:

[0022] determining a cropping width and a cropping height based on a size of the second image and the target cropping ratio;

[0023] Determining the coordinates of a cropping start point based on the cropping width, the cropping height, the center offset value, and the size of the second image;

[0024] A rectangle is drawn based on the cropping start point coordinates, the cropping width, and the cropping height to obtain the cropping area.

[0025] In some embodiments, performing feature matching based on the first image and the third image to obtain a matching image includes:

[0026] performing brightness correction processing on the third image to obtain a corrected image;

[0027] Matching the first image and the corrected image to obtain multiple pairs of initial feature points;

[0028] Correcting the multiple pairs of initial feature points to obtain multiple pairs of corrected feature points;

[0029] Error feature points are screened out from the multiple pairs of corrected feature points to obtain the matching image.

[0030] In some embodiments, the matching based on the first image and the third image to obtain multiple pairs of initial feature points includes:

[0031] Extracting a plurality of first feature points from the first image using a preset corner detection model;

[0032] Determining, in the rectified image, a plurality of second feature points that match the plurality of first feature points using a preset optical flow matching model;

[0033] Determining, in the first image, a plurality of third feature points that match the plurality of second feature points using the preset optical flow matching model;

[0034] When the Euclidean distance between the Kth first feature point and the Kth third feature point is less than a preset distance threshold, the Kth first feature point and the Kth second feature point are used as the Kth pair of initial feature points; wherein the Kth pair of initial feature points is any pair of initial feature points among the multiple pairs of initial feature points, and K is a positive integer.

[0035] In some embodiments, correcting the multiple pairs of initial feature points to obtain multiple pairs of corrected feature points includes:

[0036] Performing dedistortion processing on the multiple pairs of initial feature points to obtain multiple pairs of dedistorted feature points;

[0037] The multiple pairs of dedistorted feature points are rotated to obtain the multiple pairs of corrected feature points.

[0038] In some embodiments, the step of filtering out erroneous feature points from the multiple pairs of corrected feature points to obtain the matching image includes:

[0039] Selecting the first number of pairs of calibration feature points in the i-th cycle; wherein i is a positive integer;

[0040] Processing the first number of correction feature points of the i-th cycle using a preset row alignment model to obtain row alignment parameters of the i-th cycle;

[0041] The matching image is determined based on the row alignment parameters of the i-th cycle and the multiple pairs of correction feature points.

[0042] In some embodiments, determining the matching image based on the row alignment parameters of the i-th cycle and the multiple pairs of calibration feature points includes:

[0043] Determining a plurality of row alignment error values ​​of the i-th cycle based on the row alignment parameters of the i-th cycle and the plurality of pairs of correction feature points;

[0044] At least one pair of correction feature points corresponding to at least one row alignment error value within a preset error range is used as the target matching feature points of the i-th cycle;

[0045] Get the i-th cycle quantity threshold and the i-th cycle number threshold;

[0046] When the number of target matching feature points in the i-th cycle is greater than the i-th cycle number threshold, and i is greater than the i+1-th cycle number threshold, determining the matching image based on the target matching feature points in the i-th cycle; or

[0047] When the number of target matching feature points in the i-th cycle is less than or equal to the i-th cycle number threshold, and i is greater than the i-th cycle number threshold, the matching image is determined based on the target matching feature points in the i-th cycle.

[0048] In some embodiments, obtaining the i-th cycle quantity threshold and the i-th cycle number threshold includes:

[0049] When i is equal to 1, the preset number threshold is used as the i-th cycle number threshold;

[0050] When i is greater than 1, if the number of target matching feature points in the i-1th cycle is greater than the i-1th cycle quantity threshold, the number of target matching feature points in the i-1th cycle is used as the i-th cycle quantity threshold; if the number of target matching feature points in the i-1th cycle is less than or equal to the i-1th cycle quantity threshold, the i-1th cycle quantity threshold is used as the i-th cycle quantity threshold;

[0051] The i-th cycle number threshold is determined based on the i-th cycle number threshold, the total number of the multiple pairs of calibration feature points, and a preset probability.

[0052] According to a second aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0053] A first image acquisition unit, configured to acquire a first image through a first camera of the electronic device;

[0054] A second image acquisition unit is configured to acquire a second image through a second camera of the electronic device; wherein the field of view of the second camera at least covers the field of view of the first camera;

[0055] a cropping unit configured to crop the second image to obtain a third image;

[0056] The matching unit is configured to perform feature matching based on the first image and the third image to obtain a matching image.

[0057] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0058] processor;

[0059] memory for storing computer programs or instructions;

[0060] The processor executes the computer program or instruction to implement the steps of the image feature matching method described in any one of the first aspects above.

[0061] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, wherein the storage medium stores a computer program or instructions. When the computer program or instructions in the storage medium are executed by a processor, the steps of the image feature matching method described in any one of the first aspects are implemented.

[0062] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided. When the computer program or instruction is executed by a processor, the steps of the image feature matching method described in any one of the first aspects are implemented.

[0063] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0064] In the embodiment of the present disclosure, the field of view of the second camera at least covers the field of view of the first camera, that is, the image contained in the first image captured by the first camera and the image contained in the second image captured by the second camera have the same part, and cropping the second image can obtain a third image containing the same part of the image. In other words, the image feature matching method of the embodiment of the present disclosure does not require additional mapping calculations on the second image through the existing epipolar line correction. The third image can be obtained only by taking the value of the cropping of the second image, which reduces the computational complexity of the electronic device and improves the image matching efficiency. In addition, the embodiment of the present disclosure performs feature matching based on the first image and the cropped third image, which can reduce the features of image matching and further improve the image matching efficiency.

[0065] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0067] Figure 1a The figure is a comparison diagram of images captured by two cameras according to an exemplary embodiment.

[0068] Figure 1b The present invention is a flowchart illustrating a binocular image feature matching method according to an exemplary embodiment.

[0069] Figure 2 The figure is a flowchart of an image feature matching method according to an exemplary embodiment.

[0070] Figure 3 2 is an example diagram of a first image and a second image according to an exemplary embodiment.

[0071] Figure 4 is a schematic diagram of a logarithmic correction curve according to an exemplary embodiment.

[0072] Figure 5 FIG. 1 is a schematic diagram showing a first image and a corrected image according to an exemplary embodiment.

[0073] Figure 6 is a schematic diagram showing a rotated first image and a corrected image according to an exemplary embodiment.

[0074] Figure 7a The figure shows a flow chart of determining a matching image according to an exemplary embodiment.

[0075] Figure 7b FIG. 1 is a schematic diagram of a first image and a corrected image after erroneous feature points are screened out according to an exemplary embodiment.

[0076] Figure 8 FIG1 is a block diagram of an electronic device according to an exemplary embodiment.

[0077] Figure 9 A frame of an electronic device according to an exemplary embodiment is shown Figure 2 . DETAILED DESCRIPTION

[0078] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numbers in different drawings represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present disclosure. Rather, they are merely examples of image feature matching methods, electronic devices, storage media, and program products consistent with certain aspects of the present disclosure, as detailed in the appended claims.

[0079] In related technologies, Figure 1a : is a comparison diagram of images captured by two cameras according to an exemplary embodiment. Figure 1a As shown, Figure 1a (Left) has a field of view smaller than Figure 1a (Right) The field of view shows that images captured by electronic devices using different cameras differ. This makes it difficult for electronic devices to recognize and understand environmental information, and there is still much room for research and improvement in constructing more practical stereo vision systems. Feature extraction and matching between images captured by different cameras is a fundamental task in computer vision. In terms of feature extraction, existing methods include feature point detection based on local pixel grayscale differences, such as corner detection, edge detection, or feature point detection, as well as scale-differential feature point detection methods, such as scale-invariant feature transformation matching algorithms and accelerated robust feature algorithms. In terms of feature matching, there are direct matching methods based on multi-view geometric constraints, as well as indirect matching strategies based on feature descriptors. At the same time, deep learning is also making its mark in feature extraction and matching, with solutions such as feature point detection and descriptor extraction based on self-supervised training and local feature transformations emerging.

[0080] However, current feature extraction and matching solutions are usually only targeted at electronic devices with one camera, and lack the differences between multiple images obtained by multiple cameras. Figure 1b FIG. 1 is a flow chart showing a binocular image feature matching method according to an exemplary embodiment. Figure 1bAs shown, in existing binocular image processing methods, S11, binocular positioning parameters are obtained through the Zhang Zhengyou calibration method; S12, the binocular positioning results are used to perform epipolar correction on the image; S13, image features are extracted using a scale-invariant feature transformation matching algorithm to obtain initial matching results; S14, the initial matching results are randomly sampled based on the affine transformation matrix to screen for correct matching results; S15, accurate correct matches are screened based on the affine transformation matrix and the correct matching results. In existing binocular image feature matching methods, epipolar correction is performed on the image during image preprocessing, and a scale-invariant feature transformation matching algorithm is used to extract image features and match them. The performance and power consumption of electronic devices make it difficult to meet the real-time calculation requirements of the above steps. Moreover, the initial matching results are randomly sampled based on the affine transformation matrix model. In scenes with large depth, due to the presence of binocular parallax, the affine transformation matrix model cannot guarantee both the accuracy and efficiency of image matching for electronic devices.

[0081] Based on this, an embodiment of the present disclosure proposes an image feature matching method, which can reduce the calculation of electronic devices and improve the efficiency of image feature matching performed by electronic devices. Figure 2 FIG. 1 is a flow chart of an image feature matching method according to an exemplary embodiment. Figure 2 As shown, the image feature matching method mainly includes the following steps:

[0082] S101, capturing a first image through a first camera of an electronic device;

[0083] S102. Capturing a second image through a second camera of the electronic device; wherein the field of view of the second camera at least covers the field of view of the first camera;

[0084] S103, cropping the second image to obtain a third image;

[0085] S104: Perform feature matching based on the first image and the third image to obtain a matching image.

[0086] In the embodiments of the present disclosure, the electronic devices may include mobile terminal devices, smart home devices, or transportation devices, etc. The mobile terminal devices may include mobile phones or tablet computers; the smart home devices may include speakers or televisions; and the transportation devices may include cars, which are not limited in the embodiments of the present disclosure.

[0087] It should be noted that the image feature matching method is mainly used in electronic devices that are equipped with two or more cameras. For example, when a user uses a mobile phone to take a photo, he enters the camera application of the mobile phone and turns on the dual-camera mode of the camera application. Here, the dual-camera mode refers to calling any two cameras of the multiple cameras of the mobile phone, and the field of view area of ​​the second camera at least covers the field of view area of ​​the first camera. In this way, in the process of taking photos with a mobile phone, the image contained in the second image captured by the second camera has the same part as the image contained in the first image, and the image feature matching method can be applied to perform image feature matching on the first image captured by the first camera and the second image captured by the second camera. That is, the image feature matching method is applied to the pre-processing of the first image and the second image before the electronic device outputs the matching image.

[0088] Understandably, due to factors such as the phone's autofocus and optical image stabilization, camera performance and output results obtained solely based on calibration and adjustment of the phone's camera during production are inaccurate. Therefore, it is necessary to calibrate the camera's internal and external parameters in real time and correct them to line-aligned images. Real-time calibration is based on the feature points between the first and second images, so the accuracy of the image feature matching results must be guaranteed. Furthermore, due to the limited computing power of mobile phones, it is also necessary to ensure that users do not experience lag when taking photos with their phone's camera. Therefore, the efficiency of the phone's image feature matching must also be ensured.

[0089] In step S101, capturing a first image through a first camera of an electronic device includes: starting a camera application of the electronic device and ensuring that the first camera is in an activated state; the electronic device accesses the first camera by calling an interface of the first camera; setting parameters of the first camera, such as image resolution, exposure, and focus; and calling an acquisition function of the first camera interface to capture image data of the first image.

[0090] In the embodiment of the present disclosure, the electronic device may store a collected image database, and the first image may be stored in the collected image database.

[0091] In step S102, capturing a second image through the second camera of the electronic device includes: starting a camera application of the electronic device and ensuring that the second camera is in an activated state; the electronic device accesses the second camera by calling an interface of the second camera; setting parameters of the second camera, such as image resolution, exposure, and focus; and calling an acquisition function of the second camera interface to capture image data of the second image.

[0092] The field of view of the second camera at least covers the field of view of the first camera. Here, the field of view refers to the area that can be observed by the camera, that is, in the direction of incident light from the camera lens, there is a field of view consisting of the camera lens and the sensor of the electronic device that captures light. The scene within the field of view can be captured by the camera and converted into a digital image.

[0093] It should be noted that the field of view area of ​​the second camera at least covers the field of view area of ​​the first camera. By obtaining the range and boundaries of the field of view area of ​​the first camera and the range and boundaries of the field of view area of ​​the second camera, it can be determined whether the range and boundaries of the field of view area of ​​the first camera overlap with the range and boundaries of the field of view area of ​​the second camera. If there is an overlapping part, the field of view area of ​​the second camera at least covers the field of view area of ​​the first camera.

[0094] It is understood that if the second camera's field of view at least overlaps the first camera's field of view, the first image captured by the first camera will at least partially share the same portion of the captured image with the second image captured by the second camera. This ensures that when the second image is subsequently cropped, a third image at least partially identical to the first image is obtained, enabling feature matching between the first and third images.

[0095] In step S103, cropping the second image to obtain the third image includes: determining a cropping area in the second image; converting the second image into a processable data structure, such as an array or a matrix; cropping the second image based on a cropping function to obtain a third image; and saving the third image in an acquired image database.

[0096] It should be noted that the cropping area in the second image may be determined by cropping the second image based on the portion of the second image that is the same as the first image.

[0097] The shape and size of the cropping area can be set according to actual conditions, and the present disclosure does not limit this. For example, the cropping area can be rectangular, circular, triangular, or a special shape that is cropped according to the edge of the object in the second image.

[0098] It should also be noted that the third image is obtained by cropping the second image based on the cropping function. Here, the cropping function includes cropping parameters for cropping the second image, such as cropping starting point coordinates, cropping width, and cropping height, etc., which are not limited in the present embodiment.

[0099] In the disclosed embodiment, the second image can be cropped by acquiring the data structure of the second image and directly cropping the second image based only on cropping parameters, thereby reducing the computational complexity of the electronic device and improving image matching efficiency.

[0100] In step 104, feature matching based on the first image and the third image includes: extracting feature points in the first image and the third image; matching the feature points in the first image with the feature points in the third image to obtain matching results; and screening the matching results to eliminate some incorrectly matched feature points.

[0101] It should be noted that, in some embodiments, before extracting feature points from the first image and the third image, brightness correction may be performed on the third image to improve the accuracy of extracting feature points from the third image, thereby improving the accuracy of matching feature points in the third image with feature points in the first image; in other embodiments, before screening the matching results, the feature points in the matching results may be corrected to improve the accuracy of screening out incorrectly matched feature points.

[0102] In the disclosed embodiments, feature matching is performed on the first and third images to obtain a matched image. This matched image can be obtained by adding the pixels of the first image after feature matching to the pixels of the third image. Thus, the matched image obtained by feature matching the first and third images improves the quality of the obtained matched image.

[0103] In the embodiment of the present disclosure, the field of view of the second camera at least covers the field of view of the first camera, that is, the image contained in the first image captured by the first camera and the image contained in the second image captured by the second camera have the same part, and cropping the second image can obtain a third image containing the same part of the image. In other words, the image feature matching method of the embodiment of the present disclosure does not require additional mapping calculations on the second image through the existing epipolar line correction. The third image can be obtained only by taking the value of the cropping of the second image, which reduces the computational complexity of the electronic device and improves the image matching efficiency. In addition, the embodiment of the present disclosure performs feature matching based on the first image and the cropped third image, which can reduce the features of image matching and further improve the image matching efficiency.

[0104] In some embodiments, cropping the second image to obtain the third image includes:

[0105] determining a target cropping ratio for cropping the second image based on preset calibration information of the first camera and the second camera;

[0106] determining a center offset value between a center point of the first image and a center point of the second image based on a size of the first image and a size of the second image;

[0107] Determine a cropping area based on a target cropping ratio, a size of the second image, and a center offset value;

[0108] The image contained in the cropped area of ​​the second image is scaled to obtain a third image.

[0109] In the embodiment of the present disclosure, the preset calibration information of the first camera and the second camera includes internal parameters of the first camera, internal parameters of the second camera, and external parameters between the first camera and the second camera.

[0110] It should be noted that the internal parameters of the first camera include the internal coordinate system of the first camera, the focal length of the first camera on the x-axis of the internal coordinate system, the focal length of the first camera on the y-axis of the internal coordinate system, and the coordinates of the principal point of the first camera; the internal parameters of the second camera include the internal coordinate system of the second camera, the focal length of the second camera on the x-axis of the internal coordinate system, the focal length of the second camera on the y-axis of the internal coordinate system, and the coordinates of the principal point of the second camera; the external parameters between the first camera and the second camera are the rotation matrix and the translation vector.

[0111] Among them, the internal coordinate system is a plane rectangular coordinate system with the upper left corner of the rectangular image as the origin, extending to the right of the origin as the x-axis, and extending to the bottom of the origin as the y-axis. It is used to describe the planar geometric characteristics of the camera's image sensor. The principal point coordinates are the intersection of the perpendicular line from the camera center point to the plane where the image is located and the plane where the image is located.

[0112] Exemplarily, the internal parameter matrix of the first camera can be obtained by formula (1).

[0113]

[0114] Among them, K m is the internal parameter matrix of the first camera, f xm is the focal length of the first camera on the x-axis, f ym is the focal length of the first camera on the y-axis, c xm is the coordinate value of the principal point of the first camera on the x-axis, c ym is the coordinate value of the principal point of the first camera on the y-axis.

[0115] The internal parameter matrix of the second camera can also be obtained by formula (2).

[0116]

[0117] Among them, K s is the internal parameter matrix of the second camera, f xs is the focal length of the second camera on the x-axis, f ys is the focal length of the second camera on the y-axis, c xs is the coordinate value of the principal point of the second camera on the x-axis, c ys is the coordinate value of the principal point of the second camera on the y-axis.

[0118] The external parameters between the first camera and the second camera can also be obtained by formula (3) and formula (4).

[0119]

[0120] Among them, R is the rotation matrix, r 11 is the element in the first row and first column of the rotation matrix, r 12 is the element in the first row and second column of the rotation matrix, r 13 is the element in the first row and third column of the rotation matrix, r 21 is the element in the second row and first column of the rotation matrix, and so on, until r 33 The third row and third column of the rotation matrix represents the camera's rotation. This matrix accurately transforms image feature points from the internal coordinate system to the world coordinate system, or vice versa. The world coordinate system is the coordinate system used to represent the position of any object in the three-dimensional world.

[0121]

[0122] Where t is the translation vector, t x is the translation of the camera along the x-axis in the world coordinate system, t y is the translation of the camera along the x-axis in the world coordinate system, t z The translation amount of the camera along the z-axis in the world coordinate system. The translation vector is used to represent the position offset of the camera in the world coordinate system.

[0123] In some embodiments, determining a target cropping ratio for cropping the second image based on preset calibration information of the first camera and the second camera includes:

[0124] Determining, based on the preset calibration information, a first cropping ratio of the second image in a first direction and a second cropping ratio in a second direction; wherein the first direction is perpendicular to the second direction;

[0125] The smallest cropping ratio between the first cropping ratio and the second cropping ratio is used as the target cropping ratio.

[0126] The first cropping ratio of the second image in the first direction is the first cropping ratio of the second image in the x-axis direction in the internal coordinate system, and the second cropping ratio in the second direction is the first cropping ratio of the second image in the y-axis direction in the internal coordinate system.

[0127] In an embodiment of the present disclosure, determining a first cropping ratio of the second image in a first direction and a second cropping ratio in a second direction based on preset calibration information may include: obtaining a first product of the coordinate value of the principal point of the second camera on the x-axis and the focal length of the first camera on the x-axis, obtaining a second product of the coordinate value of the principal point of the first camera on the x-axis and the focal length of the second camera on the x-axis, and determining the first cropping ratio of the second image in the x-axis direction based on a ratio of the first product and the second product; obtaining a third product of the coordinate value of the principal point of the second camera on the y-axis and the focal length of the first camera on the y-axis, obtaining a fourth product of the coordinate value of the principal point of the first camera on the y-axis and the focal length of the second camera on the y-axis, and determining the second cropping ratio of the second image in the y-axis direction based on the ratio of the third product to the fourth product.

[0128] For example, the first cropping ratio of the second image in the x-axis direction can be obtained by formula (5).

[0129]

[0130] Among them, Scale x is the first cropping ratio of the second image in the x-axis direction.

[0131] For example, the second cropping ratio of the second image in the y-axis direction can be obtained by formula (6).

[0132]

[0133] Among them, Scale y is the second cropping ratio of the second image in the y-axis direction.

[0134] In the embodiment of the present disclosure, taking the smallest cropping ratio between the first cropping ratio and the second cropping ratio as the target cropping ratio includes: comparing the first cropping ratio and the second cropping ratio, and when the first cropping ratio is smaller than the second cropping ratio, taking the first cropping ratio as the target cropping ratio; or, when the first cropping ratio is larger than the second cropping ratio, taking the second cropping ratio as the target cropping ratio; or, when the first cropping ratio and the second cropping ratio are equal, arbitrarily selecting the first cropping ratio or the second cropping ratio as the target cropping ratio.

[0135] In the embodiment of the present disclosure, the second image is cropped based on the target cropping ratio, the size of the cropping area of ​​the second image can be adjusted, and the smallest cropping ratio between the first cropping ratio and the second cropping ratio is selected as the target cropping ratio, which can ensure that the third image obtained by cropping the second image contains more parts that are identical to the image contained in the first image, thereby laying a more accurate foundation for subsequent feature matching between the first image and the third image.

[0136] In some embodiments, determining a center offset value between a center point of the first image and a center point of the second image based on a size of the first image and a size of the second image includes:

[0137] Determining homogeneous coordinates of a center point of the first image based on a size of the first image;

[0138] Determining the homogeneous coordinates of the center point of the second image based on the preset calibration information, the homogeneous coordinates of the center point of the first image, the preset spatial depth of the first image, and the preset spatial depth of the second image;

[0139] A center offset value is determined based on the homogeneous coordinates of the center point of the second image and the size of the second image.

[0140] In an embodiment of the present disclosure, determining the homogeneous coordinates of the center point of the first image based on the size of the first image includes: when the first image is a rectangle, the size of the first image includes the width of the first image and the height of the first image, the coordinate value of the homogeneous coordinates of the center point of the first image on the x-axis is half the width of the first image, and the coordinate value of the homogeneous coordinates of the center point of the first image on the y-axis is half the height of the first image.

[0141] For example, the homogeneous coordinates of the center point of the first image can be obtained by formula (7).

[0142] p m =(W m / 2,H m / 2) (7)

[0143] Among them, p m is the homogeneous coordinate of the center point of the first image, W m is the width of the first image, H m is the height of the first image.

[0144] In an embodiment of the present disclosure, based on preset calibration information, the homogeneous coordinates of the center point of the first image, the preset spatial depth of the first image, and the preset spatial depth of the second image, determining the homogeneous coordinates of the center point of the second image includes: multiplying the product of the preset spatial depth of the first image, the rotation matrix, the inverse matrix of the internal parameters of the first image, and the homogeneous coordinates of the center point of the first image, adding it with the translation matrix of the external parameters, and then multiplying it with the internal parameters of the second image, and finally dividing it by the preset spatial depth of the second image to determine the homogeneous coordinates of the center point of the second image.

[0145] Exemplarily, the homogeneous coordinates of the center point of the second image can be obtained by formula (8).

[0146]

[0147] Among them, p sis the homogeneous coordinate of the center point of the second image, z m is the preset spatial depth of the first image, z s is the preset spatial depth of the second image.

[0148] In the embodiment of the present disclosure, determining the center offset value based on the homogeneous coordinates of the center point of the second image and the size of the second image includes: determining the center offset value between the first image and the second image on the x-axis based on the difference between the coordinate value of the homogeneous coordinates of the center point of the second image on the x-axis and half the width of the second image; determining the center offset value between the first image and the second image on the y-axis based on the difference between the coordinate value of the homogeneous coordinates of the center point of the second image on the y-axis and half the height of the second image.

[0149] For example, the center offset value between the first image and the second image on the x-axis can be obtained by formula (9).

[0150] Shift x =p sx -W s / 2 (9)

[0151] Among them, Shift x is the center offset value between the first image and the second image on the x-axis, p sx is the coordinate value of the homogeneous coordinate of the center point of the second image on the x-axis.

[0152] For example, the center offset value between the first image and the second image on the y-axis can be obtained by formula (10).

[0153] Shift y =ps y -H s / 2 (10)

[0154] Among them, Shift y is the center offset value between the first image and the second image on the y axis, p sy is the homogeneous coordinate value of the center point of the second image on the y-axis.

[0155] In the embodiment of the present disclosure, the above steps can accurately obtain the center offset value between the center point of the first image and the center point of the second image, and then when the second image is cropped based on the center offset value, it is ensured that the third image obtained after cropping the second image maintains the correct position alignment relationship with the first image.

[0156] In some embodiments, determining the cropping area based on the target cropping ratio, the size of the second image, and the center offset value includes:

[0157] Determining a crop width and a crop height based on a size of the second image and a target crop ratio;

[0158] Determine the coordinates of the cropping start point based on the cropping width, the cropping height, the center offset value, and the size of the second image;

[0159] Draw a rectangle based on the cropping start point coordinates, cropping width, and cropping height to get the cropping area.

[0160] In an embodiment of the present disclosure, determining the cropping width and cropping height based on the size of the second image and the target cropping ratio includes: determining the cropping width based on the ratio of the width of the second image and the target cropping ratio; determining the cropping height based on the ratio of the height of the second image and the target cropping ratio.

[0161] For example, the cutting width can be obtained by formula (11).

[0162] w=W s / Scale (11)

[0163] Among them, w is the cropping width and Scale is the target cropping ratio.

[0164] For example, the cutting height can be obtained by formula (12).

[0165] h=H s / Scale (12)

[0166] Where h is the cutting height.

[0167] In the embodiment of the present disclosure, determining the coordinates of the cropping starting point based on the cropping width, cropping height, center offset value and the size of the second image includes: determining the coordinate value of the cropping starting point on the x-axis based on half of the difference between the width of the second image and the cropping width and the sum of the center offset values ​​between the first image and the second image on the x-axis; determining the coordinate value of the cropping starting point on the y-axis based on half of the height of the second image and the cropping height and the sum of the center offset values ​​between the first image and the second image on the y-axis.

[0168] For example, the coordinates of the cropping starting point can be obtained by formula (13).

[0169] p=((W s -w) / 2+Shift x ,(H s -h) / 2+Shift y ) (13)

[0170] Among them, p is the cutting starting point, (W s -w) / 2+Shift x is the coordinate value of the cutting starting point on the x-axis, (Hs -h) / 2+Shift y The coordinate value of the cutting starting point on the y-axis.

[0171] Figure 3 : is an example diagram of a first image and a second image according to an exemplary embodiment. Figure 3 As shown, based on the size of the first image 110 and the size of the second image 120, the center offset value at a right angle between the center point of the first image and the center point of the second image is determined; based on the cropping width, cropping height, the center offset value and the size of the second image, the cropping starting point coordinate p is determined; based on the cropping starting point coordinate p, the cropping width and the cropping height, a rectangle is drawn to obtain the cropping area 130.

[0172] In the embodiment of the present disclosure, the cropping area is obtained by using the cropping starting point coordinates, the cropping width, and the cropping height, thereby improving the accuracy of cropping the second image.

[0173] Scaling the image contained in the cropped area of ​​the second image to obtain the third image includes: scaling the size of the image contained in the cropped area of ​​the second image according to a preset ratio to obtain the third image.

[0174] It should be noted that the preset ratio can be obtained according to the ratio of the width of the cropped area to the width of the first image, or according to the ratio of the height of the cropped area to the height of the first image, and the embodiment of the present disclosure does not limit this.

[0175] In the embodiment of the present disclosure, after scaling the size of the image included in the cropped area in the second image, a third image with the same size as the first image can be obtained, laying a more convenient and accurate image foundation for subsequent feature matching between the first image and the third image.

[0176] In some embodiments, performing feature matching based on the first image and the third image to obtain a matching image includes:

[0177] performing brightness correction processing on the third image to obtain a corrected image;

[0178] Matching is performed based on the first image and the rectified image to obtain multiple pairs of initial feature points;

[0179] Correcting multiple pairs of initial feature points to obtain multiple pairs of corrected feature points;

[0180] The wrong feature points are screened out from multiple pairs of corrected feature points to obtain a matching image.

[0181] In an embodiment of the present disclosure, brightness correction processing of the third image includes: obtaining a first average grayscale value of the first image, obtaining a second average grayscale value of the second image, and when the first average grayscale value is less than the second average grayscale value, performing gamma correction on the second image to obtain a third average grayscale value, and the third average grayscale value is equal to the first average grayscale value; when the first average grayscale value is greater than the second average grayscale value, performing logarithmic correction on the second image to obtain a fourth average grayscale value, and the fourth average grayscale value is equal to the first average grayscale value.

[0182] The performing gamma correction on the second image includes obtaining a third average grayscale value based on a product of a gamma value power of the second average grayscale value and a contrast constant.

[0183] Exemplarily, the third average grayscale value of the second image can be obtained by formula (14).

[0184] y1=c*x r (14)

[0185] Wherein, y1 is the third average grayscale value, c is the contrast constant, x is the second average grayscale value, and r is the gamma value.

[0186] It should be noted that the value range of c is a positive number, and the value range of r is greater than 1, which is not limited in the embodiment of the present disclosure.

[0187] In the embodiment of the present disclosure, performing logarithmic correction on the second image includes: obtaining a fourth average grayscale value based on a product of a logarithm of the second average grayscale value plus 1 with a gamma value as a base and a contrast constant.

[0188] For example, Figure 4 FIG. 1 is a schematic diagram of a logarithmic correction curve according to an exemplary embodiment. Figure 4 As shown, the fourth average grayscale value of the second image can be obtained by formula (15).

[0189] y2=c*log r (1+x) (15)

[0190] Wherein, y2 is the fourth average grayscale value.

[0191] In an embodiment of the present disclosure, matching is performed based on the first image and the corrected image to obtain multiple pairs of initial feature points, including: extracting multiple first feature points in the first image using a preset corner point detection model; determining multiple second feature points that match the multiple first feature points in the corrected image using a preset optical flow matching model; determining multiple third feature points that match the multiple second feature points in the first image using the preset optical flow matching model; and obtaining multiple fourth feature points that match the multiple second feature points in the second image based on the multiple second feature points.

[0192] It is understood that when the third image is obtained by cropping the second image and not scaling, multiple second feature points in the third image are mapped to multiple fourth feature points in the second image. In this way, matching the first feature points with the fourth feature points is equivalent to matching the first image and the second image, and ultimately obtaining a matched image is more convenient than obtaining a matched image based on the first image and the third image.

[0193] In other embodiments, Figure 5 FIG is a schematic diagram showing a first image and a corrected image according to an exemplary embodiment. Figure 5 As shown, the matching based on the first image and the rectified image to obtain multiple pairs of initial feature points may also include:

[0194] Extracting a plurality of first feature points p1 from the first image 110 using a preset corner detection model;

[0195] Determine, in the rectified image 140 , a plurality of second feature points p2 that match the plurality of first feature points p1 using a preset optical flow matching model;

[0196] Determine, in the first image 110 , a plurality of third feature points p3 that match the plurality of second feature points p2 using a preset optical flow matching model;

[0197] When the Euclidean distance between the Kth first feature point and the Kth third feature point is less than a preset distance threshold, the Kth first feature point and the Kth second feature point are used as the Kth pair of initial feature points; wherein the Kth pair of initial feature points is any pair of initial feature points among multiple pairs of initial feature points, and K is a positive integer.

[0198] In the embodiment of the present disclosure, the preset corner detection model includes Harris Corner Detection, Features from accelerated segment test (FAST) or Oriented FAST and Rotated BRIEF (ORB), etc., and the embodiment of the present disclosure is not limited to this.

[0199] The above-mentioned preset optical flow matching models include the pyramid optical flow algorithm (Lucas Kanade, LK), the dense optical flow algorithm (Farneback), etc., which are not limited in the embodiments of the present disclosure.

[0200] In an embodiment of the present disclosure, extracting multiple first feature points in a first image using a preset corner detection model includes: using the preset corner detection model to identify local areas with significant grayscale value changes in the first image, which are considered to be the positions of corner points; using multiple corner points in the first image as multiple first feature points, and extracting coordinates and descriptor information of the multiple first feature points; marking the multiple first feature points as circles in the first image, and saving the results as a new first image.

[0201] The above-mentioned method of determining multiple second feature points that match multiple first feature points in the corrected image using a preset optical flow matching model includes: extracting the coordinates of multiple first feature points in the first image; determining the coordinates of multiple second feature points corresponding to the multiple first feature points in the corrected image based on the preset optical flow matching model; extracting the coordinates and descriptor information of the multiple second feature points; marking the multiple second feature points as circles in the corrected image, and saving the result as a new corrected image.

[0202] The above-mentioned method of determining multiple third feature points that match multiple second feature points in the first image using a preset optical flow matching model includes: extracting the coordinates of multiple second feature points in the corrected image; determining the coordinates of multiple third feature points corresponding to the multiple second feature points in the first image based on the preset optical flow matching model; extracting the coordinates and descriptor information of the multiple third feature points; marking the multiple third feature points as circles in the first image, and saving the results as a new first image.

[0203] The Euclidean distance between the first feature point and the third feature point is obtained based on the coordinates of the first feature point in the first image and the coordinates of the third feature point in the first image. Here, the Euclidean distance between the first feature point and the third feature point is the straight-line distance between the first feature point and the second feature point in the first image.

[0204] It can be understood that since the third feature point is the feature point mapped onto the first image by the second feature point, the offset distance between the first feature point and the second feature point is proportional to the Euclidean distance between the first feature point and the third feature point. For example, the greater the offset distance between the first feature point and the second feature point, the greater the Euclidean distance between the first feature point and the third feature point; and the smaller the offset distance between the first feature point and the second feature point, the smaller the Euclidean distance between the first feature point and the third feature point.

[0205] In this way, when the Euclidean distance between the Kth first feature point and the Kth third feature point is less than the preset distance, using the Kth first feature point and the Kth second feature point as the Kth pair of initial feature points can reduce the deviation in matching the first feature point and the second feature point.

[0206] In the embodiment of the present disclosure, when the preset corner detection model is Harris corner detection and the preset optical flow matching model is the LK optical flow algorithm, the efficiency of real-time processing of the electronic device can be improved while ensuring the accuracy of matching the first image and the corrected image to obtain multiple pairs of initial feature points. At the same time, based on the two-way matching of the first image and the corrected image, the probability of mismatching can be reduced and the quality of obtaining initial feature points in the matching results can be improved.

[0207] In some embodiments, correcting the multiple pairs of initial feature points to obtain the multiple pairs of corrected feature points includes:

[0208] Performing dedistortion processing on multiple pairs of initial feature points to obtain multiple pairs of dedistorted feature points;

[0209] The multiple pairs of dedistorted feature points are rotated to obtain multiple pairs of corrected feature points.

[0210] In the disclosed embodiment, dedistorting multiple pairs of initial feature points to obtain multiple pairs of dedistorted feature points includes dedistorting the initial feature points based on a distortion model based on preset calibration information, so that the initial feature points conform to the pinhole imaging model. The distortion model employs a field of view distortion model, in which the distortion ratios of the initial feature point coordinates under different fields of view are obtained through table lookup and interpolation.

[0211] It should be noted that obtaining the deformation ratio of the initial feature point coordinate distortion includes: a ratio of the difference between the distance from the initial feature point to the center point and the distance from the dedistorted feature point to the image center point to the distance from the dedistorted feature point to the image center point.

[0212] For example, the deformation ratio of the initial feature point coordinate distortion can be obtained by formula (16).

[0213]

[0214] Among them, σ is the deformation ratio of the initial feature point coordinate distortion, r is the distance from the initial feature point to the center point of the image, r undis is the distance from the dedistorted feature point to the center of the image.

[0215] Correcting the coordinates of the initial feature points to the coordinates of the feature points after dedistortion includes: calculating the field of view corresponding to the initial feature points; performing table lookup and interpolation in the field of view distortion model to obtain the deformation ratio of the distortion under the field of view; and calculating the coordinates of the feature points after dedistortion.

[0216] The above calculation of the field of view corresponding to the initial feature point includes: calculating the distance between the initial feature point and the center point of the image and half the length of the diagonal line of the image.

[0217] For example, the field of view corresponding to the initial feature point can be obtained by formula (17).

[0218]

[0219] Among them, field is the field of view corresponding to the initial feature point, u is the coordinate value of the initial feature point on the x-axis, v is the coordinate value of the initial feature point on the y-axis, u0 is the coordinate value of the center point of the image on the x-axis, and v0 is the coordinate value of the center point of the image on the y-axis.

[0220] The above calculation of the coordinates of the feature point after dedistortion includes: adding 1 to the deformation ratio of the distortion and multiplying it by the difference between the coordinate value of the initial feature point on the x-axis and the coordinate value of the center point of the image on the x-axis, adding the coordinate value of the feature point on the x-axis after dedistortion, to determine the coordinate value of the feature point on the x-axis after dedistortion; adding 1 to the deformation ratio of the distortion and multiplying it by the difference between the coordinate value of the initial feature point on the y-axis and the coordinate value of the center point of the image on the y-axis, adding the coordinate value of the feature point on the y-axis after dedistortion, to determine the coordinate value of the feature point on the y-axis after dedistortion.

[0221] For example, the coordinate value of the feature point on the x-axis after dedistortion can be obtained by formula (18).

[0222] u′=u0+(1+σ)*(u-u0) (18)

[0223] Wherein, u′ is the coordinate value of the feature point on the x-axis after dedistortion.

[0224] For example, the coordinate value of the feature point on the y-axis after dedistortion can be obtained by formula (19).

[0225] v′=v0+(1+σ p )*(v-v0) (19)

[0226] Among them, v′ is the coordinate value of the feature point on the y-axis after dedistortion.

[0227] In an embodiment of the present disclosure, rotating multiple pairs of dedistorted feature points to obtain multiple pairs of corrected feature points includes: calculating a rotation matrix based on a translation matrix in preset calibration information to rotate the dedistorted feature points to obtain multiple pairs of corrected feature points; and making the corrected feature points conform to a row alignment model.

[0228] It should be noted that the rotation matrix is ​​calculated according to the translation matrix in the preset calibration information to rotate the dedistorted feature points to obtain multiple pairs of corrected feature points, including: obtaining the rotation angle; based on the rotation angle, obtaining the rotation matrix after the image with a width of w and a height of h is rotated around the center point at the rotation angle; based on the coordinates of the dedistorted feature points, the rotation matrix and the homogeneous coordinates of the center point, obtaining the homogeneous coordinates of the corrected feature points.

[0229] The obtaining of the rotation angle includes: obtaining the rotation angle based on an inverse tangent trigonometric function of a ratio of a translation amount of the camera along the y-axis in the world coordinate system to a translation amount of the camera along the z-axis in the world coordinate system.

[0230] For example, the rotation angle can be obtained by formula (20).

[0231] θ=arctan(ty / tx) (20)

[0232] Where θ is the rotation angle.

[0233] The above-mentioned method of obtaining a rotation matrix after the image with a width of w and a height of h is rotated around the center point at the rotation angle based on the rotation angle includes: obtaining elements in the rotation matrix based on the sine trigonometric function of the rotation angle, the cosine trigonometric function of the rotation angle, the width of the image, and the height of the image.

[0234] For example, the rotation matrix can be obtained by formula (21).

[0235]

[0236] Among them, H r The rotation matrix of the image after it is rotated around the center point by the rotation angle.

[0237] The obtaining of the homogeneous coordinates of the corrected feature points based on the coordinates of the dedistorted feature points, the rotation matrix and the homogeneous coordinates of the center point includes: obtaining the homogeneous coordinates of the corrected feature points based on the product of the homogeneous coordinates of the center point and the rotation matrix.

[0238] For example, the homogeneous coordinates of the correction feature points can be obtained by formula (22).

[0239]

[0240] Among them, (x, y, 1) is the homogeneous coordinate of the center point, and (x′, y′, 1) is the homogeneous coordinate of the calibration feature point.

[0241] Figure 6 FIG is a schematic diagram showing a rotated first image and a corrected image according to an exemplary embodiment. Figure 6 As shown, the correction feature points in the rotated first image 110 and the rotated correction image 140 conform to the pinhole imaging and row alignment model.

[0242] In the disclosed embodiments, the diffraction effect of light passing through the camera lens can result in an inverted or mirrored image. Correcting and dedistorting the initial feature points in the first and corrected images ensures that the rotated first and corrected images maintain their fundamental imaging principles, resulting in more accurate correction of the feature points in the first and corrected images.

[0243] Moreover, in the embodiment of the present disclosure, dedistortion and rotation operations are only performed on a limited number of initial feature points. Compared with performing dedistortion and rotation operations on all pixel points in the image, this reduces the electronic device's ineffective calculations on pixel points other than the initial feature points, thereby improving the electronic device's computing efficiency.

[0244] In some embodiments, Figure 7a FIG. 1 is a flow chart showing a matching image determination process according to an exemplary embodiment. Figure 7a As shown, the steps of obtaining the matching image may include the following:

[0245] S201, selecting a first number of pairs of correction feature points for the current cycle from multiple pairs of feature points;

[0246] S202, correcting feature points based on the first number of pairs in the current cycle to obtain row alignment parameters of the current cycle;

[0247] S203, determining target matching feature points of the current cycle based on the row alignment parameters of the current cycle and multiple pairs of correction feature points;

[0248] S204, determine whether the number of target matching feature points in the current cycle is greater than the current cycle number threshold; if yes, go to step S205; if not, go to step S206;

[0249] S205, determine whether the current number of cycles is greater than the next cycle number threshold; if yes, go to step S207; if not, go to step S201;

[0250] S206, determine whether the current number of cycles is greater than the current number of cycles threshold; if yes, go to step S207; if not, go to step S201;

[0251] S207 : Determine a matching image based on the target matching feature points of the current cycle.

[0252] The step of screening out erroneous feature points from the multiple pairs of correction feature points to obtain a matching image includes:

[0253] Selecting a first number of pairs of calibration feature points in the i-th cycle from the plurality of pairs of calibration feature points; wherein i is a positive integer;

[0254] Processing the first number of correction feature points of the i-th cycle using a preset row alignment model to obtain row alignment parameters of the i-th cycle;

[0255] Based on the row alignment parameters of the i-th cycle and multiple pairs of corrected feature points, a matching image is determined.

[0256] In the disclosed embodiment, selecting a first number of pairs of calibration feature points for the i-th cycle from the plurality of pairs of calibration feature points includes arbitrarily selecting a first number of pairs of calibration feature points for the i-th cycle from the plurality of pairs of calibration feature points. The first number is less than or equal to the total number of calibration feature points. For example, 5 pairs of calibration feature points for the i-th cycle may be arbitrarily selected from 20 pairs of calibration feature points.

[0257] It should be noted that the multiple pairs of calibration feature points include a set of multiple pairs of calibration feature points, and each element in the set of multiple pairs of calibration feature points is the coordinate of each calibration feature point.

[0258] Exemplarily, the set of correction feature points can be obtained by formula (23).

[0259] M={(x n ,y n ),(x′n,y′)} (23)

[0260] Among them, M is the set of correction feature points, x n is the coordinate of the nth pair of correction feature points in the first image on the x-axis, y n is the coordinate of the nth pair of calibration feature points in the first image on the y-axis, x′ n is the coordinate of the nth pair of correction feature points on the x-axis in the correction image, y′ n is the coordinate of the nth pair of correction feature points in the corrected image on the y-axis.

[0261] The above-mentioned processing of the first number of pairs of correction feature points in the i-th cycle using the preset row alignment model to obtain the row alignment parameters of the i-th cycle includes: obtaining the row alignment parameters based on the product of the inverse matrix of the arbitrarily selected first number of pairs of correction feature points and the transposed matrix of the coordinates of the first number of pairs of correction feature points on the y-axis in the first image.

[0262] Exemplarily, when the first number is 5, a matrix of 5 pairs of correction feature points can be obtained by formula (24).

[0263]

[0264] Among them, A is a matrix of 5 pairs of correction feature points.

[0265] Exemplarily, the transposed matrix of the coordinates of the five pairs of correction feature points on the y-axis in the first image can be obtained by formula (25).

[0266] b=[y1 y2 y3 y4 y5] T (25)

[0267] Wherein, b is the transposed matrix of the coordinates of the five pairs of calibration feature points on the y-axis in the first image.

[0268] For example, the row alignment parameter can be obtained by formula (26).

[0269] X=A -1 b=[h1 h2 h3 h4 h5] T (26)

[0270] Wherein, X is the row alignment parameter, and h1 to h5 are the row alignment parameters of the rows where the five pairs of calibration feature points are located.

[0271] In some embodiments, determining the matching image based on the row alignment parameters of the i-th cycle and the multiple pairs of calibration feature points includes:

[0272] Determining a plurality of row alignment error values ​​of the i-th cycle based on the row alignment parameters of the i-th cycle and the plurality of pairs of correction feature points;

[0273] At least one pair of correction feature points corresponding to at least one row alignment error value within a preset error range is used as the target matching feature points of the i-th cycle;

[0274] Get the i-th cycle quantity threshold and the i-th cycle number threshold;

[0275] When the number of target matching feature points in the i-th cycle is greater than the i-th cycle number threshold, and i is greater than the i+1-th cycle number threshold, a matching image is determined based on the target matching feature points in the i-th cycle; or

[0276] When the number of target matching feature points in the i-th cycle is less than or equal to the i-th cycle number threshold, and i is greater than the i-th cycle number threshold, a matching image is determined based on the target matching feature points in the i-th cycle.

[0277] In the embodiment of the present disclosure, based on the row alignment parameters and multiple pairs of correction feature points of the i-th cycle, determining multiple row alignment error values ​​of the i-th cycle includes: based on the product of the first row alignment parameter and the coordinates of 5 pairs of correction feature points in each pair of corrected images on the x-axis, plus the product of the second row alignment parameter and the coordinates of 5 pairs of correction feature points in each pair of corrected images on the y-axis, plus the first sum value of the third row alignment parameter; the product of the fourth row alignment parameter and the coordinates of the nth pair of correction feature points in each pair of corrected images on the x-axis, plus the second sum value of the product of the fifth row alignment parameter and the coordinates of the 5 pairs of correction feature points in each pair of corrected images on the y-axis plus 1; and determining 5 row alignment error values ​​by subtracting the coordinates of the 5 pairs of correction feature points in each pair of first images on the y-axis from the ratio of the first sum value to the second sum value.

[0278] For example, the row alignment error value can be obtained by formula (27).

[0279]

[0280] Among them, error is the row alignment error value.

[0281] In the disclosed embodiment, if the number of target matching feature points in the i-th cycle is greater than the i-th cycle number threshold, and the current cycle number i exceeds the i+1-th cycle number threshold, a matching image is determined based on the target matching feature points in the i-th cycle. Because the number of matching feature points has reached the number threshold and the number of cycles has also reached the cycle number threshold, the accuracy of the obtained target matching feature points can be ensured, that is, incorrectly matched matching feature points are screened out.

[0282] In some embodiments, obtaining the i-th cycle quantity threshold and the i-th cycle number threshold includes:

[0283] When i is equal to 1, the preset number threshold is used as the i-th cycle number threshold;

[0284] When i is greater than 1, if the number of target matching feature points in the i-1th cycle is greater than the i-1th cycle quantity threshold, the number of target matching feature points in the i-1th cycle is used as the i-th cycle quantity threshold; if the number of target matching feature points in the i-1th cycle is less than or equal to the i-1th cycle quantity threshold, the i-1th cycle quantity threshold is used as the i-th cycle quantity threshold;

[0285] The i-th cycle number threshold is determined based on the i-th cycle number threshold, the total number of the multiple pairs of calibration feature points, and the preset probability.

[0286] In the embodiment of the present disclosure, when i is equal to 1, the preset number threshold serves as the first iteration number threshold. Here, the preset number threshold can be set based on actual conditions. Because target matching feature points need to be output, the first iteration number threshold will be smaller than the number of target matching feature points. For example, the preset number threshold can have a value range of less than or equal to 0, which is not limited in the embodiment of the present disclosure.

[0287] In the embodiment of the present disclosure, since there must be one cycle in i cycles where the output target feature matching point meets the row alignment model parameters, the preset probability is the probability that the output target feature matching point meets the row alignment model parameters in i cycles.

[0288] Among them, obtaining the preset probability includes: obtaining a first difference based on the first power of the ratio of 1 minus the i-th cycle number threshold and the total number of multiple pairs of correction feature points; and determining the preset probability based on 1 minus the power of the i-th cycle number threshold of the first difference.

[0289] Exemplarily, the preset probability can be obtained by formula (28).

[0290]

[0291] Among them, P is the preset probability, bestn inlier is the threshold value of the number of cycles for the i-th time, n total is the total number of pairs of calibration feature points, m is the first number, and k is the i-th cycle number threshold.

[0292] When i is greater than 1, determining the i-th cycle number threshold based on the i-th cycle number threshold, the total number of multiple pairs of correction feature points and the preset probability includes: determining the i-th cycle number threshold based on the ratio between the logarithm of 1 minus the preset probability and the logarithm of the first power of the ratio of 1 minus the i-th cycle number threshold and the total number of multiple pairs of correction feature points.

[0293] Exemplarily, the i-th cycle number threshold can be obtained by formula (29).

[0294]

[0295] Figure 7b FIG. 1 is a schematic diagram of a first image and a corrected image after filtering out erroneous feature points according to an exemplary embodiment. Figure 7b As shown, the circles on the first image and the circles on the corrected image are target matching feature points obtained after error screening of multiple pairs of correction feature points after i cycles.

[0296] It should be noted that the number of cycles for obtaining matching images based on the affine transformation matrix model in the related art is greater than the number of cycles for obtaining matching images based on the row alignment model in the embodiment of the present disclosure. For example, the comparison table 1 is shown below:

[0297] Comparison Table 1

[0298] RANSAC method Affine transformation matrix model Row alignment model Get matching image - number of cycles 84 8 Get the matching image second cycle number 427 24 Get the matching image three times 243 10

[0299] In the embodiment of the present disclosure, by setting a loop count threshold, the maximum number of loops of the algorithm can be limited, avoiding the problem of the algorithm falling into an infinite loop, thereby enhancing the robustness of the algorithm; and by adjusting the loop count threshold and the loop count threshold of the next loop according to the result of the previous loop, dynamic adjustment of the loop count threshold and the loop count threshold is achieved, which can make the algorithm converge to a result that meets the conditions more stably, thereby improving the accuracy and reliability of obtaining matching images.

[0300] Figure 8 FIG1 is a block diagram of an electronic device according to an exemplary embodiment. Figure 8 As shown, the electronic device 1000 mainly includes:

[0301] The first image acquisition module 1001 is configured to acquire a first image through a first camera of the electronic device;

[0302] The second image acquisition module 1002 is configured to acquire a second image through a second camera of the electronic device; wherein the field of view of the second camera at least covers the field of view of the first camera;

[0303] A cropping module 1003 is configured to crop the second image to obtain a third image;

[0304] The matching module 1004 is configured to perform feature matching based on the first image and the third image to obtain a matching image.

[0305] In some embodiments, the cropping module is further configured to determine a target cropping ratio for cropping the second image based on preset calibration information of the first camera and the second camera; determine a center offset value between a center point of the first image and a center point of the second image based on the size of the first image and the size of the second image; determine a cropping area based on the target cropping ratio, the size of the second image and the center offset value; and scale the image contained in the cropping area in the second image to obtain a third image.

[0306] In some embodiments, the cropping module is further configured to determine a first cropping ratio of the second image in a first direction and a second cropping ratio in a second direction based on preset calibration information; wherein the first direction is perpendicular to the second direction; and the smallest cropping ratio between the first cropping ratio and the second cropping ratio is used as the target cropping ratio.

[0307] In some embodiments, the cropping module is further configured to determine the homogeneous coordinates of the center point of the first image based on the size of the first image; determine the homogeneous coordinates of the center point of the second image based on preset calibration information, the homogeneous coordinates of the center point of the first image, the preset spatial depth of the first image, and the preset spatial depth of the second image; and determine the center offset value based on the homogeneous coordinates of the center point of the second image and the size of the second image.

[0308] In some embodiments, the cropping module is further configured to determine the cropping width and cropping height based on the size of the second image and the target cropping ratio; determine the cropping starting point coordinates based on the cropping width, cropping height, center offset value and the size of the second image; draw a rectangle based on the cropping starting point coordinates, cropping width and cropping height to obtain a cropping area.

[0309] In some embodiments, the matching module is further configured to perform brightness correction processing on the third image to obtain a corrected image; perform matching based on the first image and the corrected image to obtain multiple pairs of initial feature points; correct the multiple pairs of initial feature points to obtain multiple pairs of corrected feature points; and screen out erroneous feature points from the multiple pairs of corrected feature points to obtain a matched image.

[0310] In some embodiments, the matching module is further configured to extract multiple first feature points in the first image using a preset corner detection model; determine multiple second feature points that match the multiple first feature points in the corrected image using a preset optical flow matching model; determine multiple third feature points that match the multiple second feature points in the first image using the preset optical flow matching model; when the Euclidean distance between the Kth first feature point and the Kth third feature point is less than a preset distance threshold, use the Kth first feature point and the Kth second feature point as the Kth pair of initial feature points; wherein the Kth pair of initial feature points is any pair of initial feature points among the multiple pairs of initial feature points, and K is a positive integer.

[0311] In some embodiments, the matching module is further configured to perform dedistortion processing on multiple pairs of initial feature points to obtain multiple pairs of dedistorted feature points; and rotate the multiple pairs of dedistorted feature points to obtain multiple pairs of corrected feature points.

[0312] In some embodiments, the matching module is further configured to select a first number of pairs of correction feature points in the i-th cycle from multiple pairs of correction feature points; where i is a positive integer; use a preset row alignment model to process the first number of pairs of correction feature points in the i-th cycle to obtain row alignment parameters for the i-th cycle; and determine a matching image based on the row alignment parameters of the i-th cycle and multiple pairs of correction feature points.

[0313] In some embodiments, the matching module is further configured to determine multiple row alignment error values ​​of the i-th cycle based on the row alignment parameters of the i-th cycle and multiple pairs of correction feature points; use at least one pair of correction feature points corresponding to at least one row alignment error value within a preset error range as the target matching feature points of the i-th cycle; obtain the i-th cycle quantity threshold and the i-th cycle number threshold; when the number of target matching feature points of the i-th cycle is greater than the i-th cycle quantity threshold, and i is greater than the i+1-th cycle number threshold, determine the matching image based on the target matching feature points of the i-th cycle; or, when the number of target matching feature points of the i-th cycle is less than or equal to the i-th cycle quantity threshold, and i is greater than the i-th cycle number threshold, determine the matching image based on the target matching feature points of the i-th cycle.

[0314] In some embodiments, the matching module is further configured to, when i is equal to 1, use a preset number threshold as the i-th cycle number threshold; when i is greater than 1, if the number of target matching feature points in the i-1th cycle is greater than the i-1th cycle number threshold, use the i-1th cycle number threshold as the i-th cycle number threshold; if the number of target matching feature points in the i-1th cycle is less than or equal to the i-1th cycle number threshold, use the i-1th cycle number threshold as the i-th cycle number threshold; determine the i-th cycle number threshold based on the i-th cycle number threshold, the total number of multiple pairs of correction feature points and the preset probability.

[0315] Regarding the electronic device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0316] Figure 9 A structural frame of an electronic device according to an exemplary embodiment is shown Figure 2 For example, the electronic device 900 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0317] Reference Figure 9, the electronic device 900 may include one or more of the following components: a processing component 902 , a memory 904 , a power component 906 , a multimedia component 908 , an audio component 910 , an input / output (I / O) interface 912 , a sensor component 914 , and a communication component 916 .

[0318] The processing component 902 generally controls the overall operation of the electronic device 900, such as operations associated with at least one of display, phone calls, data communications, camera operation, and recording operations. The processing component 902 may include one or more processors 920 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 902 may include one or more modules to facilitate interaction between the processing component 902 and other components. For example, the processing component 902 may include a multimedia module to facilitate interaction between the multimedia component 908 and the processing component 902.

[0319] The memory 904 is configured to store various types of data to support operations on the electronic device 900. Examples of such data include at least one of the following: instructions for any application or method operating on the electronic device 900, contact data, phone book data, messages, pictures, and videos. The memory 904 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0320] The power supply component 906 provides power to various components of the electronic device 900. The power supply component 906 may include at least one of the following: a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 900.

[0321] The multimedia component 908 includes a screen that provides an output interface between the electronic device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 908 includes a front camera and / or a rear camera. When the electronic device 900 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0322] The audio component 910 is configured to output and / or input audio signals. For example, the audio component 910 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 900 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 904 or transmitted via the communication component 916. In some embodiments, the audio component 910 also includes a speaker for outputting audio signals.

[0323] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as a keyboard, click wheel, and buttons. These buttons may include, but are not limited to, a home button, volume buttons, a start button, and a lock button.

[0324] The sensor assembly 914 includes one or more sensors for providing various aspects of the status assessment of the electronic device 900. For example, the sensor assembly 914 can detect the open / closed state of the electronic device 900, the relative positioning of components, such as the display and keypad of the electronic device 900. The sensor assembly 914 can also detect changes in the position of the electronic device 900 or a component thereof, the presence or absence of user contact with the electronic device 900, the orientation or acceleration / deceleration of the electronic device 900, and changes in the temperature of the electronic device 900. The sensor assembly 914 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 914 can also include an optical sensor, such as a complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 914 can also include, but is not limited to, at least one of the following: an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, and a temperature sensor.

[0325] The communication component 916 is configured to facilitate communication between the electronic device 900 and other devices in a wired or wireless manner. The electronic device 900 can access a wireless network based on a communication standard, such as Wi-Fi, 4G, 5G, or a combination thereof. In an exemplary embodiment, the communication component 916 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra wide band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0326] In an exemplary embodiment, the electronic device 900 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0327] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is further provided, such as a memory 904 including executable instructions or a computer program. The instructions or computer program can be executed by the processor 920 of the electronic device 900 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0328] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform any of the above-mentioned image feature matching methods of the embodiments of the present disclosure. For example, the image feature matching method includes: capturing a first image through a first camera of an electronic device; capturing a second image through a second camera of the electronic device; wherein the field of view of the second camera at least covers the field of view of the first camera; cropping the second image to obtain a third image; and performing feature matching based on the first image and the third image to obtain a matched image.

[0329] The present disclosure provides a computer program product comprising a computer program or executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, causing the computer device to perform any of the above-described image feature matching methods of the present disclosure.

[0330] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

[0331] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image feature matching method, characterized in that: include: Capturing a first image through a first camera of the electronic device; Capturing a second image through a second camera of the electronic device; wherein the field of view of the second camera at least covers the field of view of the first camera; cropping the second image to obtain a third image; Feature matching is performed based on the first image and the third image to obtain a matching image.

2. The method according to claim 1, characterized in that The step of cropping the second image to obtain a third image includes: determining a target cropping ratio for cropping the second image based on preset calibration information of the first camera and the second camera; determining a center offset value between a center point of the first image and a center point of the second image based on a size of the first image and a size of the second image; determining a cropping area based on the target cropping ratio, the size of the second image, and the center offset value; The image contained in the cropped area in the second image is scaled to obtain the third image.

3. The method according to claim 2, characterized in that The determining a target cropping ratio for cropping the second image based on preset calibration information of the first camera and the second camera includes: Determining, based on the preset calibration information, a first cropping ratio of the second image in a first direction and a second cropping ratio in a second direction; wherein the first direction is perpendicular to the second direction; The smallest cropping ratio between the first cropping ratio and the second cropping ratio is used as the target cropping ratio.

4. The method according to claim 2, characterized in that The determining, based on the size of the first image and the size of the second image, a center offset value between a center point of the first image and a center point of the second image includes: determining homogeneous coordinates of a center point of the first image based on a size of the first image; Determining the homogeneous coordinates of the center point of the second image based on the preset calibration information, the homogeneous coordinates of the center point of the first image, the preset spatial depth of the first image, and the preset spatial depth of the second image; The center offset value is determined based on the homogeneous coordinates of the center point of the second image and the size of the second image.

5. The method according to claim 2, characterized in that The determining the cropping area based on the target cropping ratio, the size of the second image, and the center offset value includes: determining a cropping width and a cropping height based on a size of the second image and the target cropping ratio; Determining the coordinates of a cropping start point based on the cropping width, the cropping height, the center offset value, and the size of the second image; A rectangle is drawn based on the cropping start point coordinates, the cropping width, and the cropping height to obtain the cropping area.

6. The method according to any one of claims 1 to 5, characterized in that The performing feature matching based on the first image and the third image to obtain a matching image includes: performing brightness correction processing on the third image to obtain a corrected image; Matching the first image and the corrected image to obtain multiple pairs of initial feature points; Correcting the multiple pairs of initial feature points to obtain multiple pairs of corrected feature points; Error feature points are screened out from the multiple pairs of corrected feature points to obtain the matching image.

7. The method according to claim 6, characterized in that The matching is performed based on the first image and the rectified image to obtain multiple pairs of initial feature points, including: Extracting a plurality of first feature points from the first image using a preset corner detection model; Determining, in the rectified image, a plurality of second feature points that match the plurality of first feature points using a preset optical flow matching model; Determining, in the first image, a plurality of third feature points that match the plurality of second feature points using the preset optical flow matching model; When the Euclidean distance between the Kth first feature point and the Kth third feature point is less than a preset distance threshold, the Kth first feature point and the Kth second feature point are used as the Kth pair of initial feature points; wherein the Kth pair of initial feature points is any pair of initial feature points among the multiple pairs of initial feature points, and K is a positive integer.

8. The method according to claim 6, characterized in that Correcting the multiple pairs of initial feature points to obtain multiple pairs of corrected feature points includes: Performing dedistortion processing on the multiple pairs of initial feature points to obtain multiple pairs of dedistorted feature points; The multiple pairs of dedistorted feature points are rotated to obtain the multiple pairs of corrected feature points.

9. The method according to claim 6, characterized in that The step of filtering out erroneous feature points from the plurality of pairs of corrected feature points to obtain the matching image includes: Selecting a first number of pairs of calibration feature points in the i-th cycle from the plurality of pairs of calibration feature points; wherein i is a positive integer; Processing the first number of correction feature points of the i-th cycle using a preset row alignment model to obtain row alignment parameters of the i-th cycle; The matching image is determined based on the row alignment parameters of the i-th cycle and the multiple pairs of correction feature points.

10. The method according to claim 9, characterized in that The determining the matching image based on the row alignment parameter of the i-th cycle and the multiple pairs of correction feature points includes: Determining a plurality of row alignment error values ​​of the i-th cycle based on the row alignment parameters of the i-th cycle and the plurality of pairs of correction feature points; Using at least one pair of correction feature points corresponding to at least one row alignment error value within a preset error range as target matching feature points for the i-th cycle; Get the i-th cycle quantity threshold and the i-th cycle number threshold; When the number of target matching feature points in the i-th cycle is greater than the i-th cycle number threshold, and i is greater than the i+1-th cycle number threshold, determining the matching image based on the target matching feature points in the i-th cycle; or When the number of target matching feature points in the i-th cycle is less than or equal to the i-th cycle number threshold, and i is greater than the i-th cycle number threshold, the matching image is determined based on the target matching feature points in the i-th cycle.

11. The method according to claim 10, characterized in that The obtaining of the i-th cycle quantity threshold and the i-th cycle number threshold includes: When i is equal to 1, the preset number threshold is used as the i-th cycle number threshold; When i is greater than 1, if the number of target matching feature points in the i-1th cycle is greater than the i-1th cycle quantity threshold, the number of target matching feature points in the i-1th cycle is used as the i-th cycle quantity threshold; if the number of target matching feature points in the i-1th cycle is less than or equal to the i-1th cycle quantity threshold, the i-1th cycle quantity threshold is used as the i-th cycle quantity threshold; The i-th cycle number threshold is determined based on the i-th cycle number threshold, the total number of the multiple pairs of calibration feature points, and a preset probability.

12. An electronic device, characterized in that: include: A first image acquisition module is configured to acquire a first image through a first camera of the electronic device; A second image acquisition module is configured to acquire a second image through a second camera of the electronic device; wherein the field of view of the second camera at least covers the field of view of the first camera; a cropping module, configured to crop the second image to obtain a third image; The matching module is configured to perform feature matching based on the first image and the third image to obtain a matching image.

13. An electronic device, characterized in that: include: processor; memory for storing computer programs or instructions; The processor executes the computer program or instructions to implement the steps of the image feature matching method according to any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium storing a computer program or instruction, characterized in that: When the computer program or instructions in the storage medium are executed by a processor, the steps of the image feature matching method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the image feature matching method according to any one of claims 1 to 11 are implemented.

Images