Positioning method and device, intelligent automobile and storage medium

Abstract

The present application relates to the technical field of computer vision, and discloses a positioning method and device, an intelligent automobile and a storage medium. The present application provides a positioning method, comprising: collecting a current frame image; determining a perspective transformation matrix between the current frame image and a previous frame image; identifying and screening out dynamic transformation feature points in the current frame image based on the perspective transformation matrix to obtain a static transformation feature point set of the current frame image; and determining a collection position of the current frame image based on a processing result of visual positioning processing on the static transformation feature point set. By screening out the dynamic transformation feature points in the current frame image, it is ensured that the static transformation feature point set in the current frame image is processed in the process of visual positioning processing, thereby effectively improving the positioning accuracy and making the obtained collection position of the current frame image more accurate.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to positioning methods, devices, intelligent vehicles, and storage media. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a method that uses a single camera to perform localization and map building simultaneously, enabling autonomous localization and mapping in unknown environments.

[0003] In related technologies, self-localization estimation is performed by utilizing the static environment and the camera's own motion changes within a series of images captured by the camera, thereby determining its acquisition position. However, in actual image acquisition processes, it is difficult to avoid the occurrence of mistakenly captured dynamic objects, which can affect the accuracy of subsequent self-localization estimation due to the presence of dynamic objects. Summary of the Invention

[0004] In view of this, the present invention provides a positioning method, device, intelligent vehicle, and storage medium to solve the problem of inaccurate positioning in a timely manner.

[0005] In a first aspect, the present invention provides a positioning method, the method comprising: acquiring a current frame image; determining a perspective transformation matrix between the current frame image and a previous frame image; identifying and filtering out dynamic transformation feature points in the current frame image based on the perspective transformation matrix to obtain a set of static transformation feature points of the current frame image; and determining the acquisition position of the current frame image based on the processing result of visual positioning processing of the set of static transformation feature points.

[0006] The positioning method provided in this embodiment of the invention can filter out dynamic transformation feature points in the current frame image based on the perspective transformation matrix between the current frame image and the previous frame image, so as to ensure that the visual positioning process is obtained by using the static transformation feature point set in the current frame image, thereby effectively improving the positioning accuracy and making the acquisition position of the current frame image more accurate.

[0007] In one optional implementation, determining the perspective transformation matrix between the current frame image and the previous frame image includes: dividing the current frame image into regions to obtain a first number of local region images; randomly extracting a first transformation feature point from the target local region image to obtain a second number of first transformation feature points, wherein the target local region image is a local region image randomly selected from the first number of local region images, and the second number is less than the first number; determining second transformation feature points that match each first transformation feature point from the previous frame image to obtain a second number of feature point matching pairs; constructing an initial transformation model based on the second number of feature point matching pairs; updating the parameters of the initial transformation model based on the projection position of each transformation feature point in the current frame image onto the previous frame image through the initial transformation model, and the actual position of each feature matching point in the previous frame image, to obtain the perspective transformation matrix, wherein the feature matching point is the transformation feature point in the previous frame image that matches the transformation feature point.

[0008] The positioning method provided in this embodiment determines the perspective transformation matrix between the current frame image and the previous frame image based on the matching relationship between each transformation feature point in the current frame image and each transformation feature point in the previous frame image. This can quickly clarify the relative motion relationship between the current frame image and the previous frame image, and thus effectively identify and filter out dynamic transformation feature points in the current frame image, thereby improving the accuracy of determining the acquisition position of the current frame image.

[0009] In one optional implementation, based on the projection position of each transformed feature point in the current frame image projected onto the previous frame image through the initial transformation model, and the actual position of each feature matching point in the previous frame image, the parameters of the initial transformation model are updated to obtain a perspective transformation matrix. This includes: determining transformed feature points in the current frame image whose projection positions overlap with their corresponding actual positions as valid feature points, obtaining a set of valid feature points; determining transformed feature points in the current frame image whose projection positions do not overlap with their corresponding actual positions as invalid feature points, obtaining a set of invalid feature points; determining a third number of valid regions based on the distribution positions of the valid feature point set in a first number of local region images, where the valid regions are local image regions containing valid feature points, and the third number is less than or equal to the first number; determining the distribution probability of the invalid feature point set under the initial transformation model based on the ratio of the number of valid feature points to invalid feature points in each valid region and the third number; if the distribution probability is less than a first preset threshold, then the target transformation model is determined as a perspective transformation matrix; if the distribution probability is greater than or equal to the first preset threshold, then a second number of first transformed feature points are re-determined until the distribution probability is less than the preset threshold.

[0010] In an optional implementation, before redetermining the second number of first change feature points, the method further includes: determining the covariance matrix of the effective feature point set based on the ratio of the number of effective feature points to invalid feature points in each effective region and the domain center position of the corresponding region; evaluating the value of the initial transformation model based on the covariance matrix, the image area of ​​the current frame image, and the ratio of the number of effective feature points to invalid feature points in each effective region to obtain the model evaluation value of the initial transformation model, so that when the model evaluation value is less than a second preset threshold, the step of redetermining the second number of first change feature points is performed.

[0011] In one optional implementation, based on the perspective transformation matrix, dynamic transformation feature points in the current frame image are identified and filtered out to obtain a set of static transformation feature points for the current frame image. This includes: constructing a motion estimation image of the previous frame image using the perspective transformation matrix; determining the optical flow value of each transformation feature point between the current frame image and the motion estimation image; identifying and filtering out transformation feature points with optical flow values ​​greater than a second preset threshold as dynamic transformation feature points; and identifying and filtering out transformation feature points with optical flow values ​​less than or equal to the second preset threshold as static transformation feature points to obtain a set of static transformation feature points.

[0012] The positioning method provided in this embodiment can effectively identify and filter out dynamic change feature points in the current frame image, thereby effectively reducing the impact of dynamic objects on visual positioning processing, and thus greatly improving the accuracy of determining the acquisition position of the current frame image.

[0013] In one optional implementation, filtering out dynamic transformation feature points includes: performing target detection processing on the current frame image to determine at least one object included in the current frame image; counting a fourth number of dynamic transformation feature points in the image of the region where the object is located; if the fourth number is less than a third preset threshold, filtering out all dynamic transformation feature points in the image of the region where the object is located; if the fourth number is greater than or equal to the third preset threshold, filtering out all transformation feature points in the image of the region where the object is located, wherein all transformation feature points include all dynamic transformation feature points and all static transformation feature points in the image of the region where the object is located.

[0014] In one alternative implementation, a preset semantic segmentation network model performs object detection processing on the current frame image.

[0015] Secondly, the present invention provides a positioning device, comprising: an acquisition module for acquiring a current frame image; a determination module for determining the perspective transformation matrix between the current frame image and the previous frame image; a filtering module for identifying and filtering out dynamic transformation feature points in the current frame image based on the perspective transformation matrix to obtain a set of static transformation feature points of the current frame image; and a positioning module for determining the acquisition position of the current frame image based on the processing result of visual positioning processing of the set of static transformation feature points.

[0016] Thirdly, the present invention provides an intelligent vehicle, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the positioning method described in the first aspect or any corresponding embodiment thereof; and a camera for acquiring the current frame image and / or the previous frame image.

[0017] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the positioning method of the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the positioning method according to an embodiment of the present invention;

[0020] Figure 2 This is a flowchart illustrating another positioning method according to an embodiment of the present invention;

[0021] Figure 3 This is a flowchart illustrating another positioning method according to an embodiment of the present invention;

[0022] Figure 4 This is a flowchart illustrating a repositioning method according to an embodiment of the present invention;

[0023] Figure 5 This is a structural block diagram of a positioning device according to an embodiment of the present invention;

[0024] Figure 6 This is a schematic diagram of the hardware structure of an intelligent vehicle according to an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] The positioning method provided by this invention is applied to scenarios where intelligent vehicles need to estimate their location based on their surrounding environment while driving. The intelligent vehicle is equipped with a camera capable of image acquisition. The intelligent vehicle can be an autonomous vehicle, an intelligent assisted driving vehicle, etc.

[0027] In related technologies, self-localization estimation is performed by utilizing the static environment and the camera's own motion changes within a series of images captured by the camera, thereby determining its acquisition position. However, in actual image acquisition processes, it is difficult to avoid the occurrence of mistakenly captured dynamic objects, which can affect the accuracy of subsequent self-localization estimation due to the presence of dynamic objects.

[0028] In view of this, embodiments of the present invention provide a positioning method that can filter out dynamic transformation feature points in the current frame image based on the perspective transformation matrix between the current frame image and the previous frame image, so as to ensure that the visual positioning process is obtained by using the static transformation feature point set in the current frame image, thereby effectively improving the positioning accuracy and making the acquisition position of the current frame image more accurate.

[0029] According to an embodiment of the present invention, a positioning method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0030] This embodiment provides a positioning method that can be used in intelligent vehicles equipped with cameras, including autonomous vehicles, intelligent assisted driving vehicles, etc. Figure 1 This is a flowchart of a positioning method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0031] Step S101: Acquire the current frame image.

[0032] In this embodiment of the invention, the current frame image can be understood as the image captured by the camera in the current environment at the current moment.

[0033] Step S102: Determine the perspective transformation matrix between the current frame image and the previous frame image.

[0034] In this embodiment of the invention, the current frame image and the previous frame image are multiple frames acquired consecutively. Since the camera's position may change during image acquisition, to determine the relative positional difference between the current and previous frame images, a perspective transformation matrix between the two images is estimated. This matrix is ​​then used to track the camera's acquisition position.

[0035] Step S103: Based on the perspective transformation matrix, identify and filter out dynamic transformation feature points in the current frame image to obtain the static transformation feature point set of the current frame image.

[0036] In this embodiment of the invention, the relative displacement changes of each transformation feature point in the current frame image can be clearly identified through the perspective transformation matrix. This allows for the selection of which transformation feature points in the current frame image are dynamic and which are static. Removing the dynamic transformation feature points from the current frame image yields the set of static transformation feature points, minimizing the impact of dynamic objects on localization. Dynamic objects can include pedestrians, moving vehicles, etc.

[0037] Step S104: Based on the processing result of visual positioning processing of the static transformation feature point set, determine the acquisition position of the current frame image.

[0038] In this embodiment of the invention, since the displacement change of the static transformed feature point set between two adjacent frames is relatively static, the acquisition position of the current frame image can be determined by processing the static transformed feature point set through visual localization. This ensures that the localization result is more accurate and more closely matches the actual acquisition position of the current frame image. Traditional visual SLAM algorithms can be used for visual localization processing of the static transformed feature point set. For example, any visual SLAM algorithm such as Parallel Tracking and Mapping (PTAM), Direct Sparse Odometry (DSO), Large-Scale Direct Monocular SLAM, or ORB-SLAM2 (a feature-point-based visual odometry algorithm) can be used for visual localization processing.

[0039] The positioning method provided in this embodiment can filter out dynamic transformation feature points in the current frame image based on the perspective transformation matrix between the current frame image and the previous frame image, so as to ensure that the visual positioning process is obtained by using the static transformation feature point set in the current frame image, thereby effectively improving the positioning accuracy and making the acquisition position of the current frame image more accurate.

[0040] This embodiment provides a positioning method that can be used in the aforementioned smart car equipped with a camera, wherein the smart car includes autonomous vehicles, intelligent assisted driving vehicles, etc. Figure 2 This is a flowchart of a positioning method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0041] Step S201: Acquire the image of the current frame. For details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0042] Step S202: Determine the perspective transformation matrix between the current frame image and the previous frame image.

[0043] Specifically, step S202 includes:

[0044] Step S2021: Divide the current frame image into regions to obtain a first number of local region images.

[0045] In this embodiment of the invention, to better capture details and local information in the current frame image, the current frame image is divided into regions to obtain a first number of local region images, thereby improving extraction efficiency during subsequent transformation feature extraction. For example, the current frame image can be divided into 8*8 local region images. Preferably, the current frame image can be evenly divided according to the first number, which helps to improve the accuracy of dynamic detection.

[0046] Step S2022: Randomly extract a first transformation feature point from the target local region image to obtain a second number of first transformation feature points.

[0047] In this embodiment of the invention, a second number of target local region images are randomly selected from a first number of local region images, and a first transformation feature point is randomly extracted from each target local region image. The first transformation feature point can be any transformation feature point in the corresponding local region image. For example, after dividing the current frame image into 8*8 local region images, four (the second number) local region images are randomly selected as target local region images, and then a transformation feature point is randomly extracted from each target local region image as the first transformation feature point, thus obtaining four first transformation feature points.

[0048] Step S2023: Determine the second transformation feature points that match each of the first transformation feature points from the previous frame image to obtain the second number of feature point matching pairs.

[0049] In an embodiment of the present invention, in order to predict the relative motion relationship between the current frame image and the previous frame image, after determining a second number of first transformation feature points, second transformation feature points that match each of the first transformation feature points are determined from the previous frame image, and a second number of feature point matching pairs are constructed, so as to use the second number of feature point matching pairs to construct an initial transformation model of the perspective transformation matrix.

[0050] In one example, to improve the efficiency of determining the second transformation feature points, the corresponding mapping positions in the previous frame image are determined according to the image positions of each target local region image in the current frame image, and then the second transformation feature points that match the first transformation feature points are selected from the mapping positions in the previous frame image.

[0051] Step S2024: Construct an initial transformation model based on the second number of feature point matching pairs.

[0052] In this embodiment of the invention, the initial transformation model can be understood as an initial estimation model used to determine the final perspective transformation matrix.

[0053] Step S2025: Based on the projection position of each transformed feature point in the current frame image onto the previous frame image through the initial transformation model, and the actual position of each feature matching point in the previous frame image, update the parameters of the initial transformation model to obtain the perspective transformation matrix.

[0054] In this embodiment of the invention, the feature matching point is the transformed feature point in the previous frame that matches the transformed feature point. Since the initial transformation model is the initial estimation model of the perspective transformation matrix, the projection position of the current transformed feature point onto the previous frame image through the initial transformation model may deviate from the actual position of the corresponding feature matching point in the previous frame image. Therefore, to obtain a perspective transformation matrix that effectively reflects the relative motion relationship between the current frame image and the previous frame image, an iterative approach is adopted. Based on the deviation between the projection position of each transformed feature point onto the previous frame image through the initial transformation model and the actual position of the corresponding feature matching point in the previous frame image, the parameters of the initial transformation model are adjusted to obtain the perspective transformation matrix.

[0055] In some optional implementations, step S2025 above includes:

[0056] Step a1: Determine the transformed feature points in the current frame image whose projected positions overlap with their corresponding actual positions as valid feature points, and obtain the set of valid feature points.

[0057] In this approach, when the projection position of the transformed feature point in the previous frame overlaps with the actual position of the corresponding feature matching point, it indicates that the initial transformation model can effectively express the relative motion relationship of the transformed feature point between two consecutive frames. Therefore, the transformed feature point can be called an effective feature point.

[0058] Step a2: Determine the transformed feature points in the current frame image whose projected positions do not overlap with their corresponding actual positions as invalid feature points, and obtain the invalid feature point set.

[0059] In this approach, when the projection position of the transformed feature point in the previous frame does not overlap with the actual position of the corresponding feature matching point, it indicates that the initial transformation model cannot effectively express the relative motion relationship of the transformed feature point between two consecutive frames. Therefore, the transformed feature point can be called an invalid feature point.

[0060] Step a3: Based on the distribution of the effective feature point set on the current frame image, evaluate the model evaluation value of the initial transformation model.

[0061] In this approach, to assess the reliability of the current initial transformation model, the initial transformation model is evaluated based on the distribution of the effective feature point set on the current frame image. Based on the model evaluation value of the obtained initial model, it is determined whether the parameters of the initial transformation model need to be adjusted, thereby ensuring that the final determined perspective transformation matrix is ​​as accurate as possible.

[0062] In some alternative implementations, step a3 above includes:

[0063] Step a31: Based on the distribution of the effective feature point set in the first number of local region images, determine the third number of effective regions.

[0064] Step a32: Based on the proportion of effective feature points in each effective region and the location of the corresponding region's center, determine the covariance matrix of the effective feature point set.

[0065] Step a33: Based on the covariance matrix, the image area of ​​the current frame image, and the proportion of effective feature points in each effective region, evaluate the value of the initial transformation model to obtain the model evaluation value of the initial transformation model.

[0066] Specifically, the local regions containing valid feature points are identified as valid regions. Then, based on the distribution of each valid feature point in the current frame image, the number of valid regions is determined, resulting in a third number of valid regions. This third number is less than or equal to the first number. That is, when the valid feature points are unevenly distributed and exist only in some local regions, the third number is less than the first number. When the valid feature points are relatively evenly distributed and exist in all local regions of the current frame image, the third number is equal to the first number.

[0067] An evaluation model is constructed to assess the value of the initial transformation model based on the covariance matrix of the effective feature point set, the image area of ​​the current frame image, and the proportion of effective feature points in each effective region.

[0068] In one example, the expression for evaluating the model is as follows:

[0069]

[0070]

[0071]

[0072] Where i represents the currently valid region; ε i This represents the percentage of valid feature points in the current valid region; x i Indicates the center location of the currently valid region; denoted by , where represents the average number of effective feature points in each effective region; A represents the image area of ​​the current frame; C represents the covariance matrix of the effective feature point set; and s represents the obtained model evaluation value.

[0073] Step a4: Adjust the parameters of the initial transformation matrix until the optimal model evaluation value is determined, and use the initial transformation matrix corresponding to the optimal model evaluation value as the target transformation matrix.

[0074] In this approach, the parameters of the initial transformation matrix are continuously adjusted to update them, ensuring that the updated initial transformation model can adequately represent the relative motion relationship between the current frame and the previous frame. After each parameter adjustment, the adjusted initial transformation matrix is ​​evaluated to obtain a model evaluation value. A higher model evaluation value indicates a more reliable initial transformation matrix. The model evaluation value with the highest score is then selected as the optimal model evaluation value, and the initial transformation matrix corresponding to the optimal model evaluation value is used as the target transformation matrix.

[0075] Step a5: Determine the distribution probability of the invalid feature point set under the target transformation model.

[0076] In this approach, based on the proportion of valid feature points within each valid region and the third quantity, the distribution probability of the invalid feature point set under the initial change model is determined using the following formula:

[0077] P=(1-(∑ i p i σ i ) m ) Ks ;

[0078] Where, σ i p represents the percentage of valid points in the region corresponding to the optimal model. i = 1 / N (N is the third number of effective regions), m is the second number of target local region images, and Ks is the number of update iterations of the initial transformation model.

[0079] Step a6: If the probability distribution is less than the first preset threshold, then the target transformation model is determined as the perspective transformation matrix.

[0080] In this method, a first preset threshold is set to stop the update iteration. If the probability distribution of the invalid feature point set under the target transformation model is less than the first preset threshold, it indicates that the currently determined target transformation model better fits the relative motion relationship between the current frame image and the previous frame image. Therefore, there is no need to adjust the parameters of the target transformation model, and the target transformation model can be directly determined as the perspective transformation matrix.

[0081] In one example, if the probability distribution is greater than or equal to the first preset threshold, it indicates that the currently determined target transformation model still has a certain error, and the parameters of the target transformation model need to be further adjusted until the probability distribution of the finally determined invalid feature point set under the target transformation model is less than the first preset threshold.

[0082] In some optional implementation scenarios, the specific implementation process for determining the perspective transformation matrix between the current frame image and the previous frame image can be as follows:

[0083] The input current frame image is divided into 8*8=64 (first number) local region images. From the 64 local region images, 4 local region images are randomly selected as target local region images, and a first transformation feature point is extracted from each target local region image to obtain 4 first transformation feature points.

[0084] From the previous frame image, determine the matching second transformation feature point corresponding to each first transformation feature point, thereby obtaining 4 feature point matching pairs, and construct the initial transformation model through the 4 feature point matching pairs.

[0085] The projection positions of each transformed feature point in the current frame image onto the previous frame image are determined by the initial transformation model. The actual positions of the matching feature points corresponding to each transformed feature point in the current frame image in the previous frame image are also determined. Transformed feature points whose projection positions overlap with their corresponding actual positions in the current frame image are identified as valid feature points, resulting in a set of valid feature points. Transformed feature points whose projection positions do not overlap with their corresponding actual positions in the current frame image are identified as invalid feature points, resulting in a set of invalid feature points.

[0086] A pre-defined evaluation model assesses the value of the initial transformation model, and then adjusts the parameters of the initial transformation model based on the obtained model evaluation value, thereby selecting the initial transformation model with the highest model evaluation value as the target transformation model. The expression for the pre-defined evaluation model can be as follows:

[0087]

[0088]

[0089]

[0090] Where i represents the currently valid region; ε i This represents the percentage of valid feature points in the current valid region; x i Indicates the center location of the currently valid region; denoted by , where represents the average number of effective feature points in each effective region; A represents the image area of ​​the current frame; C represents the covariance matrix of the effective feature point set; and s represents the obtained model evaluation value.

[0091] The probability distribution of the invalid feature point set under the target transformation model is determined using the following formula:

[0092] P=(1-(∑ i p i σ i ) m ) Ks ;

[0093] Where, σ i p represents the percentage of valid points in the region corresponding to the optimal model. i = 1 / N (N is the third number of effective regions), m is the second number of target local region images, and Ks is the number of update iterations of the initial transformation model.

[0094] If the probability distribution of the invalid feature point set is less than the first preset threshold, then the target transformation model is used as the perspective transformation model to express the relative motion relationship between the current frame image and the previous frame image.

[0095] Step S203: Based on the perspective transformation matrix, identify and filter out dynamic transformation feature points in the current frame image to obtain the static transformation feature point set of the current frame image. For details, please refer to [link to relevant documentation]. Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0096] Step S204: Based on the processing result of visual localization of the static transformation feature point set, determine the acquisition position of the current frame image. For details, please refer to [link to relevant documentation]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0097] The positioning method provided in this embodiment determines the perspective transformation matrix between the current frame image and the previous frame image based on the matching relationship between each transformation feature point in the current frame image and each transformation feature point in the previous frame image. This can quickly clarify the relative motion relationship between the current frame image and the previous frame image, and thus effectively identify and filter out dynamic transformation feature points in the current frame image, thereby improving the accuracy of determining the acquisition position of the current frame image.

[0098] This embodiment provides a positioning method that can be used in the aforementioned smart car equipped with a camera, wherein the smart car includes autonomous vehicles, intelligent assisted driving vehicles, etc. Figure 3 This is a flowchart of a positioning method according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps:

[0099] Step S301: Acquire the current frame image.

[0100] Step S302: Determine the perspective transformation matrix between the current frame image and the previous frame image.

[0101] Step S303: Based on the perspective transformation matrix, identify and filter out dynamic transformation feature points in the current frame image to obtain the static transformation feature point set of the current frame image.

[0102] Specifically, step S303 includes:

[0103] Step S3031: Construct the motion estimation image of the previous frame image using the perspective transformation matrix.

[0104] In this embodiment of the invention, in order to determine the motion trajectory of each object in the previous frame image, the coordinates of each transformation feature point in the previous frame image are distorted by a determined perspective transformation matrix to construct a motion estimation image of the previous frame image, so that the coordinate system of each transformation feature point in the obtained motion estimation image can be consistent with the coordinate system of each transformation feature point in the current frame image.

[0105] Step S3032: Determine the optical flow value of each transformed feature point between the current frame image and the motion estimation image.

[0106] In this embodiment of the invention, to describe the motion changes of each transformed feature point in the current frame image, an optical flow algorithm is used to determine the optical flow value of each transformed feature point between the current frame image and the motion estimation image. For example, the optical flow algorithm may include: Lucas-Kanade optical flow (LK optical flow method) and deep learning optical flow method (a method for predicting optical flow values ​​using a deep learning model).

[0107] Step S3033: Select transformation feature points with optical flow values ​​greater than the second preset threshold as dynamic transformation feature points, and filter out dynamic transformation feature points.

[0108] In this embodiment of the invention, the second preset threshold can be understood as a critical value used to distinguish whether the current transformed feature point belongs to a dynamic transformed feature point or a static transformed feature point. If the optical flow value of the transformed feature point is greater than the second preset threshold, it indicates that the transformed feature point has obvious motion changes between two frames of images, and therefore, the transformed feature point can be considered a dynamic transformed feature point. Conversely, if the optical flow value of the transformed feature point is less than or equal to the second preset threshold, it indicates that the transformed feature point does not have obvious motion changes between two frames of images, and therefore, the transformed feature point can be considered a static transformed feature point.

[0109] Therefore, when a transformation feature point with an optical flow value greater than the second preset threshold is detected, the transformation feature point is identified as a dynamic transformation feature point and is filtered out.

[0110] In some optional implementations, step S3033 above includes:

[0111] Step b1: Perform target detection processing on the current frame image to determine at least one object included in the current frame image.

[0112] Step b2: Count the fourth number of dynamically changing feature points in the image of the region where the object is located.

[0113] Step b3: If the fourth quantity is less than the third preset threshold, then filter all dynamic transformation feature points in the image of the area where the object is located.

[0114] Step b4: If the fourth quantity is greater than or equal to the third preset threshold, then filter all transformation feature points in the image of the region where the object is located.

[0115] Specifically, to improve the accuracy of positioning and eliminate the interference of dynamic objects in the current frame image, target detection processing is performed on the current frame image to identify the objects included in the current frame image and the position of each object in the current frame image.

[0116] The fourth number of dynamic transformation feature points in the image of each object's region is counted. A third preset threshold is set, which can be understood as the minimum threshold for determining whether the current object is a dynamic object. If the fourth number of dynamic transformation feature points in the image of the object's region is less than the third preset threshold, the object is considered a static object. To avoid interference from dynamic transformation feature points in the image of the static object's region with the determination of the acquisition location, all dynamic transformation feature points in the image of the object's region are filtered out. If the fourth number of dynamic transformation feature points in the image of the object's region is greater than or equal to the third preset threshold, the object is considered a dynamic object. All transformation feature points in the image of the object's region must be filtered out to eliminate interference from dynamic objects in the positioning of the acquisition location, thereby improving the robustness and accuracy of subsequent visual positioning processing. All transformation feature points include all dynamic transformation feature points and all static transformation feature points in the image of the object's region.

[0117] In some optional implementations, the current frame image can be processed for object detection using a pre-defined semantic segmentation network model. Preferably, the pre-defined semantic segmentation network model can be a pre-trained SegNet network (a semantic segmentation network model based on a Convolutional Neural Network (CNN)). Since the SegNet network is a network model suitable for real-time applications, fast, and with small storage space, semantic segmentation of the current frame image using the SegNet network can achieve pixel-level semantic segmentation. This allows for effective identification of the boundary between the object and the background when determining the local region where the object is located, which facilitates the effectiveness of filtering dynamically changing feature points and greatly improves the robustness and accuracy of subsequent visual localization processing.

[0118] Step S3034: Select transformation feature points whose optical flow values ​​are less than or equal to the second preset threshold as static transformation feature points to obtain a set of static transformation feature points.

[0119] Step S304: Based on the processing result of visual positioning processing of the static transformation feature point set, determine the acquisition position of the current frame image.

[0120] The positioning method provided in this embodiment can effectively identify and filter out dynamic change feature points in the current frame image, thereby effectively reducing the impact of dynamic objects on visual positioning processing, and thus greatly improving the accuracy of determining the acquisition position of the current frame image.

[0121] In some optional implementation scenarios, the specific implementation process of using the camera of a smart car to locate the acquisition position of the current frame image can be as follows: Figure 4 As shown:

[0122] After the current frame image is acquired, target detection processing is performed on the current frame image to determine at least one object included in the current frame image through a semantic segmentation network model.

[0123] Extract the transformation feature points in the current frame image.

[0124] Get the previous frame image and extract the transformation feature points from the previous frame image.

[0125] Each transformed feature point in the current frame image is matched with each transformed feature point in the previous frame image to obtain multiple feature point matching pairs. Each feature point matching pair includes the transformed feature point in the current frame image and its corresponding matching feature point in the previous frame image. The matching feature point is the transformed feature point in the previous frame image that matches that transformed feature point.

[0126] Based on the positional relationship between multiple feature point matching pairs, the perspective transformation matrix between the current frame image and the previous frame image is estimated.

[0127] The motion estimation image of the previous frame is constructed using the perspective transformation matrix.

[0128] The optical flow value of each transformed feature point between the current frame image and the motion estimation image is determined.

[0129] Based on the stream values ​​of each transformation feature point, the dynamic transformation feature points in the current frame image are identified.

[0130] Based on the number of dynamically changing feature points in the image of each object's region, dynamically changing feature points in the current frame image are filtered out to obtain a set of statically changing feature points.

[0131] Start the tracking thread to estimate the initial acquisition position of the current frame image based on the matching results between the static transformation feature point set of the current frame image and the static transformation feature point set of the historical keyframes, and add the current frame image to the historical map to update the historical map.

[0132] A keyframe thread is started to perform keyframe detection processing on the current frame image to identify whether the current frame image is a keyframe image. If the current frame image is a keyframe image, the initial acquisition position is adjusted based on the pose matching relationship between the initial acquisition position in the current frame image and the acquisition position in the historical keyframes to obtain the first adjusted acquisition position.

[0133] A local mapping thread is started to locally optimize the updated historical map based on the current frame image, and the first adjustment acquisition position is adjusted according to the local results to obtain the second adjustment acquisition position.

[0134] A local loopback thread is initiated to detect whether a loopback frame matching the current frame image exists in the historical keyframes. If a loopback frame exists, the second adjustment acquisition position is adjusted based on the pose matching relationship between the second adjustment acquisition position of the current frame image and the acquisition position of the loopback frame to obtain the acquisition position of the current frame image.

[0135] This embodiment also provides a positioning device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0136] This embodiment provides a positioning device, such as Figure 5 As shown, it includes:

[0137] Acquisition module 501 is used to acquire the image of the current frame;

[0138] The determination module 502 is used to determine the perspective transformation matrix between the current frame image and the previous frame image;

[0139] The filtering module 503 is used to identify and filter out dynamic transformation feature points in the current frame image based on the perspective transformation matrix, so as to obtain the static transformation feature point set of the current frame image.

[0140] The positioning module 504 is used to determine the acquisition position of the current frame image based on the processing result of visual positioning processing of the static transformation feature point set.

[0141] In some optional implementations, the determining module 502 includes: a partitioning unit, used to partition the current frame image into regions to obtain a first number of local region images; an extraction unit, used to randomly extract a first transformation feature point from the target local region image to obtain a second number of first transformation feature points, wherein the target local region image is a local region image randomly selected from the first number of local region images, and the second number is less than the first number; a first matching unit, used to determine a second transformation feature point that matches each first transformation feature point from the previous frame image to obtain a second number of feature point matching pairs; a first construction unit, used to construct an initial transformation model based on the second number of feature point matching pairs; and a model estimation unit, used to update the parameters of the initial transformation model based on the projection position of each transformation feature point in the current frame image projected onto the previous frame image through the initial transformation model, and the actual position of each feature matching point in the previous frame image, to obtain a perspective transformation matrix, wherein the feature matching point is the transformation feature point in the previous frame image that matches the transformation feature point.

[0142] In some optional implementations, the model estimation unit includes: a first processing unit, configured to determine transform feature points in the current frame image whose projected positions overlap with their corresponding actual positions as valid feature points, thereby obtaining a set of valid feature points; a second processing unit, configured to determine transform feature points in the current frame image whose projected positions do not overlap with their corresponding actual positions as invalid feature points, thereby obtaining a set of invalid feature points; an evaluation unit, configured to evaluate the model evaluation value of the initial transform model based on the distribution of the valid feature point set in the current frame image; an adjustment unit, configured to adjust the parameters of the initial transform matrix until an optimal model evaluation value is determined, and to use the initial transform matrix corresponding to the optimal model evaluation value as the target transform matrix; a first determining unit, configured to determine the distribution probability of the invalid feature point set under the target transform model; and a second determining unit, configured to determine the target transform model as the perspective transformation matrix if the distribution probability is less than a first preset threshold.

[0143] In some optional implementations, the evaluation unit includes: a third determining unit, configured to determine a third number of effective regions based on the distribution positions of the effective feature point set in the first number of local region images, wherein the effective regions are local region images containing the effective feature points, and the third number is less than or equal to the first number; a covariance matrix determining module, configured to determine the covariance matrix of the effective feature point set based on the proportion of effective feature points in each of the effective regions and the domain center position of the corresponding region; and an evaluation subunit, configured to evaluate the value of the initial transformation model based on the covariance matrix, the image area of ​​the current frame image, and the proportion of effective feature points in each of the effective regions, to obtain a model evaluation value of the initial transformation model.

[0144] In some optional implementations, the filtering module 503 includes: a second construction unit, configured to construct a motion estimation image of the previous frame image using a perspective transformation matrix; a third determination unit, configured to determine the optical flow value of each transformation feature point between the current frame image and the motion estimation image; a third processing unit, configured to use transformation feature points with optical flow values ​​greater than a second preset threshold as dynamic transformation feature points and filter out dynamic transformation feature points; and a fourth processing unit, configured to use transformation feature points with optical flow values ​​less than or equal to the second preset threshold as static transformation feature points to obtain a set of static transformation feature points.

[0145] In some optional implementations, the third processing unit includes: a target detection unit, configured to perform target detection processing on the current frame image to determine at least one object included in the current frame image; a statistics unit, configured to count the fourth number of dynamically changing feature points in the image of the region where the object is located; a first execution unit, configured to filter all dynamically changing feature points in the image of the region where the object is located if the fourth number is less than a third preset threshold; and a second execution unit, configured to filter all changing feature points in the image of the region where the object is located if the fourth number is greater than or equal to the third preset threshold, wherein all changing feature points include all dynamically changing feature points and all static changing feature points in the image of the region where the object is located.

[0146] In some optional implementations, a preset semantic segmentation network model performs object detection processing on the current frame image.

[0147] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0148] In this embodiment, the positioning device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0149] This invention also provides an intelligent vehicle having the above-described features. Figure 5 The positioning device shown.

[0150] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an intelligent vehicle provided by an optional embodiment of the present invention, such as... Figure 6As shown, the intelligent vehicle includes one or more processors 10, memory 20, and cameras 30, as well as interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the intelligent vehicle, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple intelligent vehicles can be connected, each device providing some of the necessary operations (e.g., as a server array, a set of blade servers, or a multiprocessor system). Figure 6 Take a processor 10 as an example.

[0151] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0152] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0153] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the intelligent vehicle. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the intelligent vehicle via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0154] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0155] The intelligent vehicle also includes an input device 40 and an output device 50. The processor 10, memory 20, input device 40, and output device 50 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0156] Input device 40 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the intelligent vehicle, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 50 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0157] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0158] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A positioning method, characterized in that, The method includes: Acquire the current frame image; Determining the perspective transformation matrix between the current frame image and the previous frame image includes: dividing the current frame image into regions to obtain a first number of local region images; randomly extracting a first transformation feature point from the target local region image to obtain a second number of first transformation feature points, wherein the target local region image is a local region image randomly selected from the first number of local region images, and the second number is less than the first number; determining second transformation feature points that match each of the first transformation feature points from the previous frame image to obtain the second number of feature point matching pairs; constructing an initial transformation model based on the second number of feature point matching pairs; and determining the transformation feature points in the current frame image whose projected positions overlap with their corresponding actual positions as valid feature points to obtain a set of valid feature points. In the current frame image, transformation feature points whose projected positions do not overlap with their corresponding actual positions are identified as invalid feature points, resulting in an invalid feature point set. Based on the distribution of the valid feature point set in a first number of local region images, a third number of valid regions are determined, where each valid region is a local region image containing the valid feature points, and the third number is less than or equal to the first number. Based on the proportion of valid feature points within each valid region and the center position of the corresponding region, the covariance matrix of the valid feature point set is determined. Based on the covariance matrix, the image area of ​​the current frame image, and the proportion of valid feature points within each valid region, the initial transformation model is evaluated to obtain a model evaluation value. The parameters of the initial transformation model are adjusted until an optimal model evaluation value is determined, and the initial transformation model corresponding to the optimal model evaluation value is used as the target transformation model. The distribution probability of the invalid feature point set under the target transformation model is determined. If the distribution probability is less than a first preset threshold, the target transformation model is determined as the perspective transformation matrix. Based on the perspective transformation matrix, dynamic transformation feature points in the current frame image are identified and filtered out to obtain the static transformation feature point set of the current frame image; Based on the processing results of visual localization of the static transformation feature point set, the acquisition position of the current frame image is determined.

2. The method according to claim 1, characterized in that, The step of identifying and filtering out dynamic transformation feature points in the current frame image based on the perspective transformation matrix to obtain the static transformation feature point set of the current frame image includes: The motion estimation image of the previous frame is constructed using the perspective transformation matrix. Determine the optical flow value of each transformed feature point between the current frame image and the motion estimation image; Transformation feature points whose optical flow values ​​are greater than a second preset threshold are used as dynamic transformation feature points, and these dynamic transformation feature points are then filtered out. Transformation feature points whose optical flow values ​​are less than or equal to the second preset threshold are used as static transformation feature points to obtain the set of static transformation feature points.

3. The method according to claim 2, characterized in that, The process of filtering out the dynamically changing feature points includes: Perform target detection processing on the current frame image to determine at least one object included in the current frame image; Count the fourth number of dynamically changing feature points in the image of the region where the object is located; If the fourth quantity is less than the third preset threshold, then all dynamic transformation feature points in the image of the region where the object is located are filtered. If the fourth quantity is greater than or equal to the third preset threshold, then all transformation feature points in the image of the region where the object is located are filtered out. All transformation feature points include all dynamic transformation feature points in the image of the region where the object is located and all static transformation feature points in the image of the region where the object is located.

4. The method according to claim 3, characterized in that, The current frame image is processed by a preset semantic segmentation network model for target detection.

5. A positioning device, characterized in that, The device includes: The acquisition module is used to acquire the image of the current frame; The determination module is used to determine the perspective transformation matrix between the current frame image and the previous frame image; The filtering module is used to identify and filter out dynamic transformation feature points in the current frame image based on the perspective transformation matrix, so as to obtain the static transformation feature point set of the current frame image. The positioning module is used to determine the acquisition position of the current frame image based on the processing result of visual positioning processing of the static transformation feature point set; The determining module includes: A partitioning unit is used to partition the current frame image into regions to obtain a first number of local region images; An extraction unit is used to randomly extract a first transformation feature point from a target local region image to obtain a second number of first transformation feature points. The target local region image is a local region image randomly selected from the first number of local region images, and the second number is less than the first number. The first matching unit is used to determine, respectively, second transformation feature points that match each of the first transformation feature points from the previous frame image, to obtain the second number of feature point matching pairs; The first construction unit is used to construct an initial transformation model based on the second number of feature point matching pairs; The model estimation unit is used to update the parameters of the initial transformation model based on the projection position of each transformation feature point in the current frame image projected onto the previous frame image through the initial transformation model, and the actual position of each feature matching point in the previous frame image, to obtain the perspective transformation matrix. The feature matching point is the transformation feature point in the previous frame image that matches the transformation feature point. The model estimation unit includes: The first processing unit is used to determine the transformation feature points in the current frame image whose projected positions overlap with their corresponding actual positions as valid feature points, and obtain a set of valid feature points. The second processing unit is used to determine the transformed feature points in the current frame image whose projected positions do not overlap with their corresponding actual positions as invalid feature points, and to obtain a set of invalid feature points. An evaluation unit is used to evaluate the model evaluation value of the initial transformation model based on the distribution of the effective feature point set on the current frame image. An adjustment unit is used to adjust the parameters of the initial transformation model until the optimal model evaluation value is determined, and the initial transformation model corresponding to the optimal model evaluation value is used as the target transformation model. The first determining unit is used to determine the distribution probability of the invalid feature point set under the target transformation model; The second determining unit is used to determine the target transformation model as the perspective transformation matrix if the distribution probability is less than the first preset threshold. The evaluation unit includes: The third determining unit is used to determine a third number of effective regions based on the distribution positions of the effective feature point set in the first number of local region images. The effective regions are local region images containing the effective feature points, and the third number is less than or equal to the first number. The covariance matrix determination module is used to determine the covariance matrix of the effective feature point set based on the proportion of effective feature points in each effective region and the domain center position of the corresponding region. The evaluation subunit is used to evaluate the value of the initial transformation model based on the covariance matrix, the image area of ​​the current frame image, and the proportion of effective feature points in each effective region, so as to obtain the model evaluation value of the initial transformation model.

6. An intelligent vehicle, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the positioning method of any one of claims 1 to 4; A camera is used to capture the current frame image and / or the previous frame image.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the positioning method according to any one of claims 1 to 4.

Images