Image processing method and device, electronic equipment and storage medium

Abstract

This disclosure relates to an image processing method, apparatus, electronic device, and storage medium. The method includes: extracting a plurality of first feature points from a first image and a plurality of second feature points from a second image, wherein the first image is an image captured by a first camera and the second image is an image captured by a second camera; determining the incident direction of each first feature point and the incident direction of each second feature point, wherein the incident direction is the direction from the corresponding position of the feature point in the real world to the optical center of the camera; performing feature matching on the plurality of first feature points and the plurality of second feature points within each of a preset plurality of direction ranges, and performing preset processing on the first image and the second image based on the feature point matching results within each direction range.

Description

Technical Field

[0001] This disclosure relates to the field of data transmission technology, specifically to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In recent years, terminal devices have become increasingly feature-rich, with more advanced camera functions and higher image quality. The camera plays a decisive role in the quality of the camera; the more powerful the camera, the stronger the terminal device's camera capabilities and the higher the image quality. Camera capabilities are, to some extent, proportional to its size, but the installation space in terminal devices is often limited, making it difficult to improve shooting capabilities simply by enhancing the capabilities of a single camera. Therefore, current terminal devices utilize multiple cameras to improve shooting capabilities. This method requires aligning and merging images from multiple cameras to obtain the final output image. However, current technologies often suffer from poor alignment and merging effects, resulting in lower output image quality. Summary of the Invention

[0003] To overcome the problems existing in the related technologies, this disclosure provides an image processing method, apparatus, electronic device, and storage medium to solve the defects in the related technologies.

[0004] According to a first aspect of the present disclosure, an image processing method is provided, comprising:

[0005] Multiple first feature points of the first image and multiple second feature points of the second image are extracted respectively, wherein the first image is an image captured by the first camera and the second image is an image captured by the second camera;

[0006] The incident direction of each first feature point and the incident direction of each second feature point are determined respectively, wherein the incident direction is the direction from the corresponding position of the feature point in the real world to the optical center of the camera;

[0007] In each of the preset multiple directional ranges, feature matching is performed on multiple first feature points and multiple second feature points, and the first image and the second image are subjected to preset processing based on the feature point matching results in each directional range.

[0008] In one embodiment, determining the incident direction of each first feature point and the incident direction of each second feature point includes:

[0009] Based on the camera parameters of the first camera and the second camera, the incident direction of each first feature point and the incident direction of each second feature point are determined respectively.

[0010] In one embodiment, the camera parameters of both the first camera and the second camera include camera intrinsic parameters and distortion models;

[0011] The step of determining the incident direction of each first feature point and the incident direction of each second feature point based on the camera parameters of the first camera and the second camera includes:

[0012] Based on the camera intrinsic parameters and distortion model of the first camera and the coordinates of each first feature point in the coordinate system of the first camera, the direction vector of each first feature point is determined accordingly. Based on the camera intrinsic parameters and distortion model of the second camera and the coordinates of each second feature point in the coordinate system of the second camera, the direction vector of each second feature point is determined accordingly.

[0013] The incident direction of each first feature point is determined according to the direction vector of each first feature point, and the incident direction of each second feature point is determined according to the direction vector of each second feature point.

[0014] In one embodiment, determining the incident direction of each first feature point based on the direction vector of each first feature point, and determining the incident direction of each second feature point based on the direction vector of each second feature point, includes:

[0015] The direction vector of each first feature point is converted from the coordinate system of the first camera to the direction vector in the preset world coordinate system, and the direction vector of each second feature point is converted from the coordinate system of the second camera to the direction vector in the world coordinate system.

[0016] Within the world coordinate system, the incident direction of each first feature point is determined according to the direction vector of each first feature point, and the incident direction of each second feature point is determined according to the direction vector of each second feature point.

[0017] In one embodiment, the incident direction of the first feature point includes the pitch angle and yaw angle of the direction vector of the first feature point; and / or,

[0018] The incident direction of the second feature point includes the pitch angle and yaw angle of the direction vector of the second feature point.

[0019] In one embodiment, the world coordinate system is the coordinate system of the first camera; the camera parameters of both the first camera and the second camera include a rotation matrix;

[0020] The step of converting the direction vector of each first feature point from the coordinate system of the first camera to a direction vector in a preset world coordinate system, and converting the direction vector of each second feature point from the coordinate system of the second camera to a direction vector in the world coordinate system, includes:

[0021] Based on the rotation matrix of the first camera and the rotation matrix of the second camera, the direction vector of each second feature point is converted from the coordinate system of the second camera to the direction vector in the coordinate system of the first camera.

[0022] In one embodiment, the camera parameters of both the first camera and the second camera include a displacement vector;

[0023] Before determining the incident direction of each second feature point based on its corresponding direction vector, the process also includes:

[0024] Based on the displacement vectors of the first camera and the second camera, the direction vector of each second feature point is compensated by a preset angle.

[0025] In one embodiment, the origin of the world coordinate system is located on the baseline of the first camera and the second camera, and one of the coordinate axes coincides with the baseline;

[0026] The step of converting the direction vector of each first feature point from the coordinate system of the first camera to a direction vector in a preset world coordinate system, and converting the direction vector of each second feature point from the coordinate system of the second camera to a direction vector in the world coordinate system, includes:

[0027] Based on the displacement vectors of the first camera and the second camera, the direction vector of each first feature point is converted from the coordinate system of the first camera to a direction vector in a preset world coordinate system. Based on the rotation matrix and displacement vector of the first camera, and the rotation matrix and displacement vector of the second camera, the direction vector of each second feature point is converted from the coordinate system of the second camera to a direction vector in a preset world coordinate system.

[0028] In one embodiment, it also includes:

[0029] An angle compensation of a preset angle is performed on the yaw angle of the direction vector of each second feature point.

[0030] In one embodiment, it also includes:

[0031] The union of all incident directions is divided into multiple directional ranges, and the directional range to which the incident direction of each first feature point belongs is determined, as well as the directional range to which the incident direction of each second feature point belongs.

[0032] In one embodiment, dividing the union range of all incident directions into multiple direction ranges includes:

[0033] The union range of all pitch angles is divided into multiple pitch angle ranges, and the union range of all yaw angles is divided into multiple yaw angle ranges.

[0034] Each pitch angle range is combined with each yaw angle range, and each combination is defined as a directional range.

[0035] In one embodiment, the step of performing feature matching on multiple first feature points and multiple second feature points within each of a preset range of multiple directions, and performing preset processing on the first image and the second image based on the feature point matching results within each range of directions, includes:

[0036] Multiple first feature points and multiple second feature points within the directional range are matched to obtain multiple feature point pairs, wherein each feature point pair includes a first feature point and a second feature point.

[0037] Based on the coordinates of the first feature point and the second feature point in each feature point pair, the displacement vector of each feature point pair is determined accordingly.

[0038] Based on the displacement vector of each feature point pair, a reference vector is determined, and feature point pairs to which the displacement vectors of the multiple feature point pairs belong are deleted, wherein the difference value includes the magnitude of the vector difference or the magnitude of the vector inner product.

[0039] Based on the feature point pairs retained within each directional range, the first image and the second image are subjected to preset processing.

[0040] In one embodiment, it also includes:

[0041] Based on the displacement vectors of the feature point pairs within each angular range, the average vector for each angular range is determined accordingly.

[0042] Clustering is performed on each angle range based on the average vector of each angle range;

[0043] The image regions in the first image that correspond to multiple angle ranges within at least one category in the clustering results are segmented; and / or the image regions in the second image that correspond to multiple angle ranges within at least one category in the clustering results are segmented.

[0044] According to a second aspect of the present disclosure, an image processing apparatus is provided, comprising:

[0045] An extraction module is used to extract multiple first feature points from a first image and multiple second feature points from a second image, wherein the first image is an image captured by a first camera and the second image is an image captured by a second camera.

[0046] The orientation module is used to determine the incident direction of each first feature point and the incident direction of each second feature point, wherein the incident direction is the direction from the corresponding position of the feature point in the real world to the optical center of the camera.

[0047] The matching module is used to perform feature matching on multiple first feature points and multiple second feature points in each of the preset multiple directional ranges, and to perform preset processing on the first image and the second image based on the feature point matching results in each directional range.

[0048] In one embodiment, the direction module is used for:

[0049] Based on the camera parameters of the first camera and the second camera, the incident direction of each first feature point and the incident direction of each second feature point are determined respectively.

[0050] In one embodiment, the camera parameters of both the first camera and the second camera include camera intrinsic parameters and distortion models;

[0051] When the orientation module determines the incident direction of each first feature point and the incident direction of each second feature point based on the camera parameters of the first camera and the second camera, it is used for:

[0052] Based on the camera intrinsic parameters and distortion model of the first camera and the coordinates of each first feature point in the coordinate system of the first camera, the direction vector of each first feature point is determined accordingly. Based on the camera intrinsic parameters and distortion model of the second camera and the coordinates of each second feature point in the coordinate system of the second camera, the direction vector of each second feature point is determined accordingly.

[0053] The incident direction of each first feature point is determined according to the direction vector of each first feature point, and the incident direction of each second feature point is determined according to the direction vector of each second feature point.

[0054] In one embodiment, when the direction module is used to determine the incident direction of each first feature point based on the direction vector of each first feature point, and to determine the incident direction of each second feature point based on the direction vector of each second feature point, it is used to:

[0055] The direction vector of each first feature point is converted from the coordinate system of the first camera to the direction vector in the preset world coordinate system, and the direction vector of each second feature point is converted from the coordinate system of the second camera to the direction vector in the world coordinate system.

[0056] Within the world coordinate system, the incident direction of each first feature point is determined according to the direction vector of each first feature point, and the incident direction of each second feature point is determined according to the direction vector of each second feature point.

[0057] In one embodiment, the incident direction of the first feature point includes the pitch angle and yaw angle of the direction vector of the first feature point; and / or,

[0058] The incident direction of the second feature point includes the pitch angle and yaw angle of the direction vector of the second feature point.

[0059] In one embodiment, the world coordinate system is the coordinate system of the first camera; the camera parameters of both the first camera and the second camera include a rotation matrix;

[0060] The direction module is used to convert the direction vector of each first feature point from the coordinate system of the first camera to a direction vector in a preset world coordinate system, and to convert the direction vector of each second feature point from the coordinate system of the second camera to a direction vector in the world coordinate system, for the following purposes:

[0061] Based on the rotation matrix of the first camera and the rotation matrix of the second camera, the direction vector of each second feature point is converted from the coordinate system of the second camera to the direction vector in the coordinate system of the first camera.

[0062] In one embodiment, the camera parameters of both the first camera and the second camera include a displacement vector;

[0063] It also includes a first compensation module, which is used to perform angle compensation on the direction vector of each second feature point by a preset angle based on the displacement vector of the first camera and the displacement vector of the second camera before determining the incident direction of each second feature point according to the direction vector of each second feature point.

[0064] In one embodiment, the origin of the world coordinate system is located on the baseline of the first camera and the second camera, and one of the coordinate axes coincides with the baseline;

[0065] The direction module is used to convert the direction vector of each first feature point from the coordinate system of the first camera to a direction vector in a preset world coordinate system, and to convert the direction vector of each second feature point from the coordinate system of the second camera to a direction vector in the world coordinate system, for the following purposes:

[0066] Based on the displacement vectors of the first camera and the second camera, the direction vector of each first feature point is converted from the coordinate system of the first camera to a direction vector in a preset world coordinate system. Based on the rotation matrix and displacement vector of the first camera, and the rotation matrix and displacement vector of the second camera, the direction vector of each second feature point is converted from the coordinate system of the second camera to a direction vector in a preset world coordinate system.

[0067] In one embodiment, a second compensation module is further included, for:

[0068] An angle compensation of a preset angle is performed on the yaw angle of the direction vector of each second feature point.

[0069] In one embodiment, a partitioning module is further included, for:

[0070] The union of all incident directions is divided into multiple directional ranges, and the directional range to which the incident direction of each first feature point belongs is determined, as well as the directional range to which the incident direction of each second feature point belongs.

[0071] In one embodiment, the partitioning module is used for:

[0072] The union range of all pitch angles is divided into multiple pitch angle ranges, and the union range of all yaw angles is divided into multiple yaw angle ranges.

[0073] Each pitch angle range is combined with each yaw angle range, and each combination is defined as a directional range.

[0074] In one embodiment, the matching module is used for:

[0075] Multiple first feature points and multiple second feature points within the directional range are matched to obtain multiple feature point pairs, wherein each feature point pair includes a first feature point and a second feature point.

[0076] Based on the coordinates of the first feature point and the second feature point in each feature point pair, the displacement vector of each feature point pair is determined accordingly.

[0077] Based on the displacement vector of each feature point pair, a reference vector is determined, and feature point pairs to which the displacement vectors of the multiple feature point pairs belong are deleted, wherein the difference value includes the magnitude of the vector difference or the magnitude of the vector inner product.

[0078] Based on the feature point pairs retained within each directional range, the first image and the second image are subjected to preset processing.

[0079] In one embodiment, a segmentation module is further included, for:

[0080] Based on the displacement vectors of the feature point pairs within each angular range, the average vector for each angular range is determined accordingly.

[0081] Clustering is performed on each angle range based on the average vector of each angle range;

[0082] The image regions in the first image that correspond to multiple angle ranges within at least one category in the clustering results are segmented; and / or the image regions in the second image that correspond to multiple angle ranges within at least one category in the clustering results are segmented.

[0083] According to a third aspect of the present disclosure, an electronic device is provided, the electronic device including a memory and a processor, the memory being configured to store computer instructions executable on the processor, and the processor being configured to execute the computer instructions based on the image processing method described in the first aspect.

[0084] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0085] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0086] This disclosure extracts first feature points from the first image captured by the first camera and second feature points from the image captured by the second camera. This allows for the determination of the incident direction of each first feature point and each second feature point. Finally, feature matching is performed on multiple first and second feature points within each of a preset range of directions. Based on the feature point matching results within each direction range, preset processing is applied to the first and second images. Since the incident direction is the direction from the corresponding position of the feature point in the real world to the optical center of the camera, first and second feature points with incident directions within the same direction range correspond to the same area in the real world, resulting in a high probability of mutual matching. Therefore, feature matching of first and second feature points within each direction range is targeted, avoids false matching, and improves matching efficiency. This leads to faster and better feature matching between images, thereby improving the effectiveness of preset processing based on feature point matching results, such as improving image alignment and image fusion, and enhancing the quality of the output image obtained from alignment and fusion processing. Attached Figure Description

[0087] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0088] Figure 1 This is a flowchart illustrating an exemplary embodiment of the image processing method disclosed herein;

[0089] Figure 2 This is a flowchart illustrating the determination of the incident direction as shown in an exemplary embodiment of this disclosure;

[0090] Figure 3 This is a schematic diagram illustrating pitch and yaw angles in an exemplary embodiment of this disclosure;

[0091] Figure 4 This is a schematic diagram illustrating a world coordinate system according to an exemplary embodiment of this disclosure;

[0092] Figure 5 This is a schematic diagram illustrating another world coordinate system as shown in an exemplary embodiment of this disclosure;

[0093] Figure 6 This is a schematic diagram illustrating the displacement vectors of various feature point pairs as shown in an exemplary embodiment of this disclosure;

[0094] Figure 7 This is a schematic diagram of the structure of an image processing apparatus shown in an exemplary embodiment of the present disclosure;

[0095] Figure 8 This is a structural block diagram of an electronic device illustrated in an exemplary embodiment of the present disclosure. Detailed Implementation

[0096] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0097] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0098] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0099] Firstly, at least one embodiment of this disclosure provides an image processing method; please refer to the appendix. Figure 1 The diagram illustrates the process of the method, including steps S101 and S103.

[0100] This method can be applied to terminal devices with camera functionality. These terminal devices can be user equipment (UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistant (PDA) handheld devices, computing devices, in-vehicle devices, wearable devices, etc. The terminal device has multiple cameras, such as a first camera and a second camera. The parameters of each camera can be the same or different, and the types of each camera can be the same or different; for example, the first camera can be a color camera, and the second camera can be a depth camera.

[0101] This method can be applied to scenarios where terminal devices take pictures, that is, each camera captures an image, and then this method is used to process these images, such as alignment and fusion, to finally obtain the captured output image.

[0102] In step S101, multiple first feature points of the first image and multiple second feature points of the second image are extracted respectively, wherein the first image is an image captured by the first camera and the second image is an image captured by the second camera.

[0103] The first image and the second image can be images captured simultaneously by two cameras. The real-world regions corresponding to the first image and the second image are completely identical or partially overlap. Extracting the first feature points from the first image and extracting the second feature points from the second image can be performed simultaneously or sequentially. After extracting the first feature points, a description vector for each first feature point can be generated; similarly, after extracting the second feature points, a description vector for each second feature point can be generated. For example, algorithms such as SIFT, SURF, and ORB can be used to generate the description vectors for the first and second feature points.

[0104] In step S102, the incident direction of each first feature point and the incident direction of each second feature point are determined respectively, wherein the incident direction is the direction from the corresponding position of the feature point in the real world to the optical center of the camera.

[0105] The incident direction of the first feature point is the direction from its corresponding position in the real world to the optical center of the first camera, and the incident direction of the second feature point is the direction from its corresponding position in the real world to the optical center of the second camera. The incident direction of each feature point is related to its corresponding position in the real world; that is, first and second feature points with the same or similar incident directions correspond to the same location area in the real world. The incident direction may include at least one angle within the camera's coordinate system.

[0106] In one possible embodiment, the incident direction of each first feature point and the incident direction of each second feature point can be determined based on the camera parameters of the first camera and the second camera, respectively. The camera parameters may include camera intrinsic parameters K, distortion model D, rotation matrix R, and displacement vector t. The rotation matrix R and displacement vector t are the rotation matrix and displacement vector of the camera relative to a reference frame. Generally, one of multiple cameras can be chosen as the reference frame; in this case, the rotation matrix of that camera is an identity matrix, and the displacement vector is a zero vector. The rotation matrices of the other cameras are rotation matrices relative to that camera, and the displacement vectors of the other cameras are displacement vectors relative to that camera. In this embodiment, the first camera can be used as the reference frame; therefore, the rotation matrix of the first camera is an identity matrix, and the displacement vector is a zero vector. The rotation matrix of the second camera is its rotation matrix relative to the first camera, and the displacement vector of the second camera is its displacement vector relative to the first camera.

[0107] In step S103, feature matching is performed on multiple first feature points and multiple second feature points within each of the preset multiple directional ranges, and the first image and the second image are subjected to preset processing based on the feature point matching results within each directional range.

[0108] Specifically, the union of all incident directions can be pre-divided into multiple directional ranges, and the directional range to which the incident direction of each first feature point belongs, as well as the directional range to which the incident direction of each second feature point belongs, can be determined separately. The division of multiple directional ranges can be uniform or non-uniform.

[0109] Feature matching within each directional range allows for the traversal of each first feature point against each second feature point, thereby determining the matching second feature point for each first feature point. This approach, compared to matching all first and second feature points, improves targeting and accuracy while also increasing matching efficiency. For example, feature matching can be performed using the description vectors of the first and second feature points.

[0110] Preset processing can include image alignment, image fusion, etc. In the shooting scenario of the terminal device, the output image can be obtained based on the preset processing.

[0111] This disclosure extracts first feature points from the first image captured by the first camera and second feature points from the image captured by the second camera. It then determines the incident direction of each first feature point and each second feature point. Finally, feature matching is performed on multiple first and second feature points within each of a preset range of directions. Based on the feature point matching results within each direction range, preset processing is applied to the first and second images. Since the incident direction is the direction from the corresponding position of the feature point in the real world to the camera, first and second feature points with incident directions within the same range correspond to the same area in the real world, resulting in a high probability of mutual matching. Therefore, feature matching of first and second feature points within each direction range is targeted, avoids false matching, and improves matching efficiency. This leads to fast and accurate feature matching between images, thereby improving the effectiveness of preset processing based on feature point matching results, such as improving image alignment and image fusion, and enhancing the quality of the output image obtained from alignment and fusion processing.

[0112] In some embodiments of this disclosure, it can be carried out as follows Figure 2 The method shown determines the incident direction of each first feature point and the incident direction of each second feature point based on the camera parameters of the first camera and the second camera, including steps S201 to S202.

[0113] In step S201, based on the camera intrinsic parameters and distortion model of the first camera, and the coordinates of each first feature point in the coordinate system of the first camera, the direction vector of each first feature point is determined accordingly. Similarly, based on the camera intrinsic parameters and distortion model of the second camera, and the coordinates of each second feature point in the coordinate system of the second camera, the direction vector of each second feature point is determined accordingly. Here, the direction vector refers to the mathematical representation of the direction from the optical center of the camera to the corresponding position of the feature point in the real world in a reference coordinate system (e.g., the coordinate system of the first camera or the coordinate system of the second camera).

[0114] For example, the direction vectors of the first feature point and the second feature point can be determined according to the following formulas:

[0115]

[0116]

[0117] Wherein, v1 is the direction vector of the first feature point in the coordinate system of the first camera, p1 is the coordinate of the first feature point in the coordinate system of the first camera, F1 is the transformation function between the coordinates and direction vector of the first feature point, K1 is the camera intrinsic parameter of the first camera, and D1 is the distortion function of the first camera; v2 is the direction vector of the second feature point in the coordinate system of the second camera, p2 is the coordinate of the second feature point in the coordinate system of the second camera, F2 is the transformation function between the coordinates and direction vector of the second feature point, K2 is the camera intrinsic parameter of the second camera, and D2 is the distortion function of the second camera.

[0118] In step S202, the incident direction of each first feature point is determined according to the direction vector of each first feature point, and the incident direction of each second feature point is determined according to the direction vector of each second feature point.

[0119] Optionally, the direction vector of each first feature point is first converted from the coordinate system of the first camera to the direction vector in the preset world coordinate system, and the direction vector of each second feature point is converted from the coordinate system of the second camera to the direction vector in the world coordinate system; then, in the world coordinate system, the incident direction of each first feature point is determined according to the direction vector of each first feature point, and the incident direction of each second feature point is determined according to the direction vector of each second feature point.

[0120] Wherein, the incident direction of the first feature point includes the pitch angle and yaw angle of the direction vector of the first feature point; and / or, the incident direction of the second feature point includes the pitch angle and yaw angle of the direction vector of the second feature point. The yaw angle refers to the angle between the projection of the direction vector v onto the xz plane of the world coordinate system and the z-axis; for example, the yaw angle is positive when the direction vector v is located on the positive x-axis and negative when the direction vector v is located on the negative x-axis. The pitch angle refers to the angle between the direction vector v and the xz plane of the world coordinate system; for example, the pitch angle is positive when the direction vector v passes through the xz horizontal plane and is on the positive y-axis, and negative otherwise. See Appendix. Figure 3 It shows a schematic diagram of the pitch and yaw angles of the direction vector of a feature point in the world coordinate system, where v is the direction vector, pitch is the pitch angle, and yaw is the yaw angle. The pitch and yaw angles can be calculated using the following formulas:

[0121] v = [x v y v , z v ]

[0122]

[0123] yaw = atan2(x v , zv )

[0124] Wherein, the world coordinate system is the coordinate system of the first camera or the origin of the world coordinate system is located on the baseline of the first camera and the second camera, and one of the coordinate axes coincides with the baseline.

[0125] In one possible embodiment, the coordinate system of the first camera and the coordinate system of the second camera can be as follows: Figure 4 As shown, the world coordinate system is the coordinate system of the first camera. The direction vector of each first feature point can be converted from the coordinate system of the first camera to a direction vector within a preset world coordinate system in the following manner, and the direction vector of each second feature point can be converted from the coordinate system of the second camera to a direction vector within the world coordinate system: keeping the direction vector of the first feature point unchanged within the coordinate system of the first camera, and based on the rotation matrix of the first camera and the rotation matrix of the second camera, converting the direction vector of each second feature point from the coordinate system of the second camera to a direction vector within the coordinate system of the first camera.

[0126] For example, if the rotation matrix of the first camera is an identity matrix, and the rotation matrix of the second camera is its rotation matrix R relative to the first camera, then the direction vector of each second feature point can be converted from the coordinate system of the second camera to the coordinate system of the first camera using the following formula:

[0127]

[0128] Based on the world coordinate system in this embodiment, before determining the incident direction of each second feature point according to the direction vector of each second feature point, the direction vector of each second feature point can be compensated by a preset angle according to the displacement vector of the first camera and the displacement vector of the second camera.

[0129] For example, the displacement vector of the first camera is zero, and the displacement vector of the second camera is its displacement vector t relative to the first camera. Then, the displacement vector t of the second camera is the baseline of the two cameras. The rotation vector r can be calculated first using the following formula:

[0130]

[0131] Then, the rotation angle of the rotation vector r is set to a preset angle θ (for example, the preset angle is 2°, i.e., 0.035 rad), to obtain the compensated rotation vector r':

[0132] r′=θ·|r|=θ[r x r y r2]T

[0133] Then, the compensation rotation vector r' is transformed into a rotation matrix R' using the Rodrigues formula:

[0134]

[0135] Next, the direction vector of each second feature point is multiplied by the rotation matrix R' on the left, and the direction vector of the second feature point is compensated by a preset angle θ along the baseline direction:

[0136] v′2=R′v2

[0137] Finally, the pitch and yaw angles of the direction vector v1 of each first feature point in the world coordinate system can be determined, as well as the pitch and yaw angles of the direction vector v2' of each first feature point in the world coordinate system.

[0138] In another possible embodiment, the coordinate system of the first camera, the coordinate system of the second camera, and the world coordinate system are as follows: Figure 5 As shown, the origin of the world coordinate system is located on the baseline of the first camera and the second camera, and one of its coordinate axes coincides with the baseline. The direction vector of each first feature point can be converted from the coordinate system of the first camera to a direction vector within the preset world coordinate system, and the direction vector of each second feature point can be converted from the coordinate system of the second camera to a direction vector within the world coordinate system, as follows: based on the displacement vectors of the first camera and the second camera, the direction vector of each first feature point is converted from the coordinate system of the first camera to a direction vector within the preset world coordinate system; and based on the rotation matrix and displacement vector of the first camera, and the rotation matrix and displacement vector of the second camera, the direction vector of each second feature point is converted from the coordinate system of the second camera to a direction vector within the preset world coordinate system.

[0139] For example, the rotation matrix of the first camera is the identity matrix, the rotation matrix of the second camera is its rotation matrix R relative to the first camera, the displacement vector of the first camera is the zero vector, and the displacement vector of the second camera is its displacement vector t relative to the first camera. Then, the displacement vector t of the second camera is the baseline of the two cameras. The unit vector t' of the baseline can be calculated using the following formula:

[0140]

[0141] Then, calculate the rotation matrix R1 from the coordinate system of the first camera to the world coordinate system, and the rotation matrix R2 from the coordinate system of the second camera to the world coordinate system using the following formulas:

[0142]

[0143] R1 is any orthogonal matrix that satisfies the equation.

[0144] Finally, the direction vector of each first feature point is converted from the coordinate system of the first camera to a direction vector in the preset world coordinate system according to the following formula, and the direction vector of each second feature point is converted from the coordinate system of the second camera to a direction vector in the coordinate system of the first camera:

[0145]

[0146] Next, the pitch and yaw angles of the direction vector v1 of each first feature point in the world coordinate system can be determined, as well as the pitch and yaw angles of the direction vector v2' of each first feature point in the world coordinate system.

[0147] Based on the world coordinate system in this embodiment, the yaw angle of the direction vector of each second feature point can be compensated by a preset angle. For example, if the preset angle θ is 2°, or 0.035 rad, then the yaw angle of the direction vector of the second feature point is compensated according to the following formula:

[0148] yaw′2=yaw2+θ

[0149] Where yaw2 is the yaw angle before compensation, and yaw2' is the yaw angle after compensation.

[0150] In some embodiments of this disclosure, the incident direction of the first feature point includes the pitch angle and yaw angle of the direction vector of the first feature point; the incident direction of the second feature point includes the pitch angle and yaw angle of the direction vector of the second feature point. The union of all incident directions can be divided into multiple directional ranges as follows: First, the union of all pitch angles is divided into multiple pitch angle ranges, and the union of all yaw angles is divided into multiple yaw angle ranges; next, each pitch angle range is combined with each yaw angle range, and each combination is defined as a directional range. For example, the pitch angle and yaw angle dimensions can form a spherical polar space with a spherical surface structure. This space is divided from two directions by the pitch angle and yaw angle, thereby forming multiple grids on the spherical surface, each grid corresponding to a directional range.

[0151] This embodiment uses pitch angle and yaw angle as two dimensions to finely divide the incident direction, thereby ensuring that mismatches are reduced during feature point matching and improving matching speed.

[0152] In some embodiments of this disclosure, feature matching can be performed on multiple first feature points and multiple second feature points in each of a preset range of directions in the following manner, and the first image and the second image can be subjected to preset processing based on the feature point matching results in each range of directions:

[0153] First, feature matching is performed on multiple first feature points and multiple second feature points within the specified directional range to obtain multiple feature point pairs, where each feature point pair includes one first feature point and one second feature point. For example, the matching method uses L1 or L2 metrics, and the search method includes, but is not limited to, brute-force matching methods such as the FLANN algorithm and the KNN algorithm.

[0154] Next, based on the coordinates of the first feature point and the second feature point in each feature point pair, the displacement vector for each feature point pair is determined. For example, the displacement vector m is obtained by subtracting the coordinates p1 of the first feature point in the coordinate system of the first camera and the coordinates p2 of the second feature point in the coordinate system of the second camera, i.e., m = p1 - p2. It is understandable that the first and second images can be scaled before calculating the displacement vector.

[0155] Based on the displacement vector of each feature point pair, a reference vector is determined, and feature point pairs to which the displacement vectors of the multiple feature point pairs belong have a difference value greater than a preset threshold with respect to the reference vector. The difference value includes the magnitude of the vector difference or the magnitude of the vector inner product. For example, the x-values ​​of the displacement vectors of all feature point pairs are sorted, and then a certain percentage (e.g., 10%) of the first and last x-values ​​are removed. The median of the remaining x-values ​​is then used as the x-value of the reference vector. Similarly, the y-values ​​of the displacement vectors of all feature point pairs are sorted, and then a certain percentage (e.g., 10%) of the first and last y-values ​​are removed. The median of the remaining y-values ​​is then used as the y-value of the reference vector. Finally, displacement vectors that simultaneously satisfy the following two formulas are retained:

[0156]

[0157] Where α and β are both pre-set proportionality constants.

[0158] It is understandable that there is a displacement relationship between the first and second images, meaning the second image is displaced relative to the first image. Based on the consistency of displacement, the displacement vectors between the matched first and second feature points should be identical or substantially identical. Please refer to the appendix. Figure 6 It shows the displacement vectors of each feature point pair. The feature point pairs deleted in the above manner are the feature point pairs in the circle, that is, the direction may be different from other displacement vectors, or the magnitude may be different from other displacement vectors.

[0159] Finally, based on the feature point pairs retained in each directional range, the first image and the second image are subjected to preset processing.

[0160] In this embodiment, by filtering the consistency of feature point pairs, noise or interference items in the feature point pairs can be deleted, thereby improving the accuracy of feature point pairs, and thus improving the effect of preset processing and the quality of the obtained output image.

[0161] Based on the displacement vector of each feature point pair in the above embodiments, image segmentation can also be performed in the following manner:

[0162] First, based on the displacement vectors of the feature point pairs within each angular range, determine the average vector for each angular range. For example, take the average x-value of the displacement vectors of all feature point pairs as the x-value of the average vector, and take the average y-value of the displacement vectors of all feature point pairs as the y-value of the average vector.

[0163] Next, clustering is performed on each angular range based on the average vector of that range. The number of clusters can be preset, for example, determined based on the segmentation accuracy.

[0164] Finally, the image regions in the first image corresponding to multiple angle ranges within at least one category in the clustering results are segmented; and / or, the image regions in the second image corresponding to multiple angle ranges within at least one category in the clustering results are segmented.

[0165] In this embodiment, feature point pairs within a directional range are clustered using the average displacement vector of that range, thereby segmenting the image into image regions corresponding to each category. Alternatively, the image can be segmented in other ways, such as calculating the geometric transformation matrix (e.g., homography matrix, affine matrix) for each directional range based on the feature point pairs within that range, then clustering the directional ranges based on the set transformation matrices of each range, and segmenting the image regions in the first image corresponding to multiple directional ranges within at least one category in the clustering results; and / or, segmenting the image regions in the second image corresponding to multiple directional ranges within at least one category in the clustering results.

[0166] According to a second aspect of the embodiments of this disclosure, an image processing apparatus is provided; please refer to the appendix. Figure 7 The device includes:

[0167] The extraction module 701 is used to extract multiple first feature points of a first image and multiple second feature points of a second image, wherein the first image is an image captured by a first camera and the second image is an image captured by a second camera.

[0168] The orientation module 702 is used to determine the incident direction of each first feature point and the incident direction of each second feature point, wherein the incident direction is the direction from the corresponding position of the feature point in the real world to the camera.

[0169] The matching module 703 is used to perform feature matching on multiple first feature points and multiple second feature points in each of the preset multiple directional ranges, and to perform preset processing on the first image and the second image based on the feature point matching results in each directional range.

[0170] In some embodiments of this disclosure, the direction module is used for:

[0171] Based on the camera parameters of the first camera and the second camera, the incident direction of each first feature point and the incident direction of each second feature point are determined respectively.

[0172] In some embodiments of this disclosure, the camera parameters of both the first camera and the second camera include camera intrinsic parameters and distortion models;

[0173] When the orientation module determines the incident direction of each first feature point and the incident direction of each second feature point based on the camera parameters of the first camera and the second camera, it is used for:

[0174] Based on the camera intrinsic parameters and distortion model of the first camera and the coordinates of each first feature point in the coordinate system of the first camera, the direction vector of each first feature point is determined accordingly. Based on the camera intrinsic parameters and distortion model of the second camera and the coordinates of each second feature point in the coordinate system of the second camera, the direction vector of each second feature point is determined accordingly.

[0175] The incident direction of each first feature point is determined according to the direction vector of each first feature point, and the incident direction of each second feature point is determined according to the direction vector of each second feature point.

[0176] In some embodiments of this disclosure, when the direction module is used to determine the incident direction of each first feature point according to the direction vector of each first feature point, and to determine the incident direction of each second feature point according to the direction vector of each second feature point, it is used to:

[0177] The direction vector of each first feature point is converted from the coordinate system of the first camera to the direction vector in the preset world coordinate system, and the direction vector of each second feature point is converted from the coordinate system of the second camera to the direction vector in the world coordinate system.

[0178] Within the world coordinate system, the incident direction of each first feature point is determined according to the direction vector of each first feature point, and the incident direction of each second feature point is determined according to the direction vector of each second feature point.

[0179] In some embodiments of this disclosure, the incident direction of the first feature point includes the pitch angle and yaw angle of the direction vector of the first feature point; and / or,

[0180] The incident direction of the second feature point includes the pitch angle and yaw angle of the direction vector of the second feature point.

[0181] In some embodiments of this disclosure, the world coordinate system is the coordinate system of the first camera; the camera parameters of both the first camera and the second camera include a rotation matrix;

[0182] The direction module is used to convert the direction vector of each first feature point from the coordinate system of the first camera to a direction vector in a preset world coordinate system, and to convert the direction vector of each second feature point from the coordinate system of the second camera to a direction vector in the world coordinate system, for the following purposes:

[0183] Based on the rotation matrix of the first camera and the rotation matrix of the second camera, the direction vector of each second feature point is converted from the coordinate system of the second camera to the direction vector in the coordinate system of the first camera.

[0184] In some embodiments of this disclosure, the camera parameters of both the first camera and the second camera include displacement vectors;

[0185] It also includes a first compensation module, which is used to perform angle compensation on the direction vector of each second feature point by a preset angle based on the displacement vector of the first camera and the displacement vector of the second camera before determining the incident direction of each second feature point according to the direction vector of each second feature point.

[0186] In some embodiments of this disclosure, the origin of the world coordinate system is located on the baseline of the first camera and the second camera, and one of the coordinate axes coincides with the baseline;

[0187] The direction module is used to convert the direction vector of each first feature point from the coordinate system of the first camera to a direction vector in a preset world coordinate system, and to convert the direction vector of each second feature point from the coordinate system of the second camera to a direction vector in the world coordinate system, for the following purposes:

[0188] Based on the displacement vectors of the first camera and the second camera, the direction vector of each first feature point is converted from the coordinate system of the first camera to a direction vector in a preset world coordinate system. Based on the rotation matrix and displacement vector of the first camera, and the rotation matrix and displacement vector of the second camera, the direction vector of each second feature point is converted from the coordinate system of the second camera to a direction vector in a preset world coordinate system.

[0189] In some embodiments of this disclosure, a second compensation module is also included, for:

[0190] An angle compensation of a preset angle is performed on the yaw angle of the direction vector of each second feature point.

[0191] In some embodiments of this disclosure, a partitioning module is also included, for:

[0192] The union of all incident directions is divided into multiple directional ranges, and the directional range to which the incident direction of each first feature point belongs is determined, as well as the directional range to which the incident direction of each second feature point belongs.

[0193] In some embodiments of this disclosure, the partitioning module is used for:

[0194] The union range of all pitch angles is divided into multiple pitch angle ranges, and the union range of all yaw angles is divided into multiple yaw angle ranges.

[0195] Each pitch angle range is combined with each yaw angle range, and each combination is defined as a directional range.

[0196] In some embodiments of this disclosure, the matching module is used for:

[0197] Multiple first feature points and multiple second feature points within the directional range are matched to obtain multiple feature point pairs, wherein each feature point pair includes a first feature point and a second feature point.

[0198] Based on the coordinates of the first feature point and the second feature point in each feature point pair, the displacement vector of each feature point pair is determined accordingly.

[0199] Based on the displacement vector of each feature point pair, a reference vector is determined, and feature point pairs to which the displacement vectors of the multiple feature point pairs belong are deleted, wherein the difference value includes the magnitude of the vector difference or the magnitude of the vector inner product.

[0200] Based on the feature point pairs retained within each directional range, the first image and the second image are subjected to preset processing.

[0201] In some embodiments of this disclosure, a segmentation module is also included, for:

[0202] Based on the displacement vectors of the feature point pairs within each angular range, the average vector for each angular range is determined accordingly.

[0203] Clustering is performed on each angle range based on the average vector of each angle range;

[0204] The image regions in the first image that correspond to multiple angle ranges within at least one category in the clustering results are segmented; and / or the image regions in the second image that correspond to multiple angle ranges within at least one category in the clustering results are segmented.

[0205] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the method in the first aspect, and will not be elaborated upon here.

[0206] According to a third aspect of the embodiments of this disclosure, please refer to the appendix. Figure 8 The diagram illustrates, for example, a block diagram of an electronic device. For instance, device 800 could be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0207] Reference Figure 8 The device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0208] Processing component 802 typically controls the overall operation of device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0209] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of this data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0210] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 800.

[0211] Multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0212] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0213] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0214] Sensor assembly 814 includes one or more sensors for providing status assessments of various aspects of device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of device 800, changes in position of device 800 or a component of device 800, the presence or absence of user contact with device 800, orientation or acceleration / deceleration of device 800, and temperature changes of device 800. Sensor assembly 814 may also include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0215] Communication component 816 is configured to facilitate wired or wireless communication between device 800 and other devices. Device 800 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G or 5G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0216] In an exemplary embodiment, the device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the power supply method of the aforementioned electronic device.

[0217] Fourthly, in exemplary embodiments, this disclosure also provides a non-transitory computer-readable storage medium including instructions, such as a memory 804 including instructions, which can be executed by a processor 820 of the device 800 to complete the power supply method of the electronic device. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0218] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0219] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: Multiple first feature points from the first image and multiple second feature points from the second image are extracted respectively, wherein the first image is an image captured by the first camera and the second image is an image captured by the second camera, and the relative positions of the first camera and the second camera are fixed; Based on the camera parameters of the first camera and the second camera, the incident direction of each first feature point and the incident direction of each second feature point are determined respectively. The incident direction is the direction from the corresponding position of the feature point in the real world to the optical center of the camera. The camera parameters include camera intrinsic parameters and distortion model. In each of the preset multiple directional ranges, feature matching is performed on multiple first feature points and multiple second feature points, and the first image and the second image are subjected to preset processing based on the feature point matching results in each directional range; The method further includes: The union of all incident directions is divided into multiple directional ranges, and the directional range to which the incident direction of each first feature point belongs is determined, as well as the directional range to which the incident direction of each second feature point belongs, are determined. The incident directions include pitch angle and yaw angle.

2. The image processing method according to claim 1, characterized in that, The step of determining the incident direction of each first feature point and the incident direction of each second feature point based on the camera parameters of the first camera and the second camera includes: Based on the camera intrinsic parameters and distortion model of the first camera and the coordinates of each first feature point in the coordinate system of the first camera, the direction vector of each first feature point is determined accordingly. Based on the camera intrinsic parameters and distortion model of the second camera and the coordinates of each second feature point in the coordinate system of the second camera, the direction vector of each second feature point is determined accordingly. The incident direction of each first feature point is determined according to the direction vector of each first feature point, and the incident direction of each second feature point is determined according to the direction vector of each second feature point.

3. The image processing method according to claim 2, characterized in that, The step of determining the incident direction of each first feature point based on the direction vector of each first feature point, and determining the incident direction of each second feature point based on the direction vector of each second feature point, includes: The direction vector of each first feature point is converted from the coordinate system of the first camera to the direction vector in the preset world coordinate system, and the direction vector of each second feature point is converted from the coordinate system of the second camera to the direction vector in the world coordinate system. Within the world coordinate system, the incident direction of each first feature point is determined according to the direction vector of each first feature point, and the incident direction of each second feature point is determined according to the direction vector of each second feature point.

4. The image processing method according to claim 3, characterized in that, The world coordinate system is the coordinate system of the first camera; the camera parameters of both the first camera and the second camera include a rotation matrix; The step of converting the direction vector of each first feature point from the coordinate system of the first camera to a direction vector in a preset world coordinate system, and converting the direction vector of each second feature point from the coordinate system of the second camera to a direction vector in the world coordinate system, includes: Based on the rotation matrix of the first camera and the rotation matrix of the second camera, the direction vector of each second feature point is converted from the coordinate system of the second camera to the direction vector in the coordinate system of the first camera.

5. The image processing method according to claim 4, characterized in that, The camera parameters of both the first camera and the second camera include displacement vectors; Before determining the incident direction of each second feature point based on its corresponding direction vector, the process also includes: Based on the displacement vectors of the first camera and the second camera, the direction vector of each second feature point is compensated by a preset angle.

6. The image processing method according to claim 3, characterized in that, The origin of the world coordinate system is located on the baseline of the first camera and the second camera, and one of the coordinate axes coincides with the baseline. The step of converting the direction vector of each first feature point from the coordinate system of the first camera to a direction vector in a preset world coordinate system, and converting the direction vector of each second feature point from the coordinate system of the second camera to a direction vector in the world coordinate system, includes: Based on the displacement vectors of the first camera and the second camera, the direction vector of each first feature point is converted from the coordinate system of the first camera to a direction vector in a preset world coordinate system. Based on the rotation matrix and displacement vector of the first camera, and the rotation matrix and displacement vector of the second camera, the direction vector of each second feature point is converted from the coordinate system of the second camera to a direction vector in a preset world coordinate system.

7. The image processing method according to claim 6, characterized in that, Also includes: An angle compensation of a preset angle is performed on the yaw angle of the direction vector of each second feature point.

8. The image processing method according to claim 3, characterized in that, The division of the union range of all incident directions into multiple directional ranges includes: The union range of all pitch angles is divided into multiple pitch angle ranges, and the union range of all yaw angles is divided into multiple yaw angle ranges. Each pitch angle range is combined with each yaw angle range, and each combination is defined as a directional range.

9. The image processing method according to claim 1, characterized in that, The step of performing feature matching on multiple first feature points and multiple second feature points within each of a preset range of multiple directions, and performing preset processing on the first image and the second image based on the feature point matching results within each range of directions, includes: Multiple first feature points and multiple second feature points within the directional range are matched to obtain multiple feature point pairs, wherein each feature point pair includes a first feature point and a second feature point. Based on the coordinates of the first feature point and the second feature point in each feature point pair, the displacement vector of each feature point pair is determined accordingly. Based on the displacement vector of each feature point pair, a reference vector is determined, and feature point pairs to which the displacement vectors of the multiple feature point pairs belong are deleted, wherein the difference value includes the magnitude of the vector difference or the magnitude of the vector inner product. Based on the feature point pairs retained within each directional range, the first image and the second image are subjected to preset processing.

10. The image processing method according to claim 9, characterized in that, Also includes: Based on the displacement vectors of the feature point pairs within each angular range, the average vector for each angular range is determined accordingly. Clustering is performed on each angle range based on the average vector of each angle range; The image regions in the first image that correspond to multiple angle ranges within at least one category in the clustering results are segmented; and / or the image regions in the second image that correspond to multiple angle ranges within at least one category in the clustering results are segmented.

11. An image processing apparatus, characterized in that, include: An extraction module is used to extract multiple first feature points from a first image and multiple second feature points from a second image, wherein the first image is an image captured by a first camera, the second image is an image captured by a second camera, and the relative positions of the first camera and the second camera are fixed. The orientation module is used to determine the incident direction of each first feature point and the incident direction of each second feature point according to the camera parameters of the first camera and the second camera, respectively. The incident direction is the direction from the corresponding position of the feature point in the real world to the optical center of the camera. The camera parameters include camera intrinsic parameters and distortion model. The matching module is used to perform feature matching on multiple first feature points and multiple second feature points in each of the preset multiple directional ranges, and to perform preset processing on the first image and the second image based on the feature point matching results in each directional range. The partitioning module is used to divide the union range of all incident directions into multiple directional ranges, and to determine the directional range to which the incident direction of each first feature point belongs, and to determine the directional range to which the incident direction of each second feature point belongs.

12. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory is used to store computer instructions that can be executed on the processor. The processor is used to execute the computer instructions based on the image processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method of any one of claims 1 to 10.

Images