Positioning mapping method and device applied to dynamic scene and intelligent device

Abstract

This application relates to the field of intelligent driving technology and provides a method, apparatus, and intelligent device for localization and mapping in dynamic scenes. The method includes: acquiring a target image; inputting the target image into a multi-task neural network model trained based on a preset loss function to obtain a feature point information set and a semantic segmentation result set, wherein the feature point information set includes the orientation information of the extracted feature points; performing a first-layer outlier removal on the feature point information set based on the semantic segmentation result set to obtain a target feature point information set, wherein the first-layer outlier removal is used to remove feature point information of dynamic objects in the target image; matching feature points in the target feature point information set based on the orientation information to obtain a target matching feature pair set; performing camera pose estimation based on the target matching feature pair set, and completing camera localization and map construction based on the pose estimation results. This application can reduce localization drift errors and improve the accuracy of localization and mapping in dynamic scenes.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a localization and mapping method, apparatus and intelligent device for dynamic scenes. Background Technology

[0002] The goal of Simultaneous Localization and Mapping (SLAM) systems is to provide indoor and outdoor robots or autonomous vehicles with a map for automated navigation, along with their own location within the map. With the rapid development of deep learning, numerous deep neural networks have been embedded into traditional purely mathematical systems to improve their robustness and accuracy. However, complex environments present new challenges. For example, in real-world scenarios, the environment changes dynamically, and dynamic objects can interfere with map construction, thus affecting localization performance.

[0003] Existing localization and mapping systems are prone to significant localization drift errors in complex, dynamic scenarios. Reducing these drift errors and improving the accuracy of localization and mapping in dynamic environments is a pressing issue. Summary of the Invention

[0004] This application provides a localization and mapping method, apparatus, and intelligent device for dynamic scenes, which can solve the problem that large depth resolution cameras generate excessive data volume when in use, thereby affecting system performance and resulting in low system performance efficiency.

[0005] In a first aspect, embodiments of this application provide a localization and mapping method applied to dynamic scenes, the localization and mapping method comprising:

[0006] Acquire a target image, wherein the target image is a color image of the target scene;

[0007] The target image is input into a multi-task neural network model trained based on a preset loss function to obtain a feature point information set and a semantic segmentation result set. The feature point information set includes the orientation information of the extracted feature points.

[0008] Based on the semantic segmentation result set, the feature point information set is subjected to a first-layer outlier removal to obtain the target feature point information set. The first-layer outlier removal is used to remove the feature point information of dynamic objects in the target image.

[0009] Based on the directional information, feature points in the target feature point information set are matched to obtain a target matching feature pair set;

[0010] Camera pose estimation is performed based on the target matching feature set, and the camera localization and map construction are completed based on the estimated pose.

[0011] In one possible implementation of the first aspect, before inputting the target image into a multi-task neural network model trained based on a preset loss function, the following steps are included:

[0012] A multi-task neural network model is constructed, comprising a direction estimation network, a feature extraction network, and a semantic segmentation network. The direction estimation network is used to extract direction information in the target image, the feature extraction network is used to extract feature points in the target image, and the semantic segmentation network is used to perform semantic segmentation on the target image.

[0013] Obtain the training sample image set;

[0014] The sample images in the training sample image set are preprocessed to obtain preprocessed sample images;

[0015] The preprocessed sample image is input into the semantic segmentation network to obtain the semantic segmentation result, and the semantic segmentation network is trained based on the first preset loss function;

[0016] The preprocessed sample image is input into the orientation estimation network to extract orientation information from the preprocessed sample image, and the orientation estimation network is trained based on the second preset loss function.

[0017] The preprocessed sample image containing the directional information is input into the feature extraction network to extract feature points from the preprocessed sample image, and the feature extraction network is trained based on the third preset loss function.

[0018] The multi-task neural network model is trained based on a preset loss function, which is determined according to the first loss of the semantic segmentation network, the second loss of the orientation estimation network, and the third loss of the feature extraction network during the training process.

[0019] In one possible implementation of the first aspect, the preprocessing of the sample images in the training sample image set to obtain preprocessed sample images includes:

[0020] The sample image is then scaled in a pyramid shape to obtain a sample image pyramid.

[0021] In one possible implementation of the first aspect, training the multi-task neural network model based on a preset loss function includes:

[0022] The loss (LOSS) of the multi-task neural network model is calculated according to the following formula:

[0023] Loss=βL sum +(1-)·(L kp + ori )

[0024] Among them, L sum L represents the first loss. ori L represents the second loss. kp The third loss is represented by β, which is a preset weight.

[0025] In one possible implementation of the first aspect, training the semantic segmentation network based on a first preset loss function includes:

[0026] The first loss L is calculated according to the following formula. sum :

[0027]

[0028] Where H represents the height of the preprocessed sample image S, W represents the width of S, Q represents the ground truth image corresponding to S, BCELoss is the first preset loss function, and Q ij This represents the true label data for pixel (i,j).

[0029] In one possible implementation of the first aspect, the orientation estimation network consists of a cluster of rotation-invariant convolutional networks, and training the orientation estimation network based on a second preset loss function includes:

[0030] The second loss L is calculated according to the following formula. ori :

[0031]

[0032] Where H represents the height of the preprocessed sample image S, W represents the width of S, G represents the number of clusters in the rotation-invariant convolutional network, k represents the k-th cluster, 1≤k≤G, and O is the optimal orientation vector. Let represent the optimal direction vector corresponding to the coordinates ij pixels generated by the k-th cluster of the a-th image. This represents the optimal direction vector corresponding to the coordinates of the pixel point ij generated by the k-th cluster of the b-th image. The optimal direction vector is the direction vector after passing through the softmax function.

[0033] The feature extraction network L is trained based on a third preset loss function. kp ,include:

[0034] The third loss L is calculated according to the following formula. kp :

[0035]

[0036] Among them, I a Representing image a, I b Representing image b, T gt Let λ represent the affine transformation matrix, l represent the level of the sample image pyramid, L represent the total number of levels in the sample image pyramid, λ be a preset parameter, and N represent the sliding window size. l L represents the size of the l-th sliding window. ip (I a ,I b ,T gt N l ) represents the correspondence between feature points in image a and image b.

[0037] In one possible implementation of the first aspect, the step of matching feature points in the target feature point information set based on the direction information to obtain a target matching feature pair set includes:

[0038] Obtain the descriptor for each feature point in the target matching feature pair set;

[0039] Based on the descriptor, feature points in the target feature point information set are matched to obtain an initial set of matched feature pairs containing matching pairs;

[0040] Based on the direction information, the direction difference between all matching pairs in the initial matching feature pair set is calculated, and the target angle difference is determined. The target angle difference is the three angle differences with the smallest direction difference values ​​in the initial matching feature pair set.

[0041] Based on the target angle difference, a second layer of outlier removal is performed on the matching pairs in the initial matching feature pair set to obtain the target matching feature pair set. The second layer of outlier removal is used to remove all matching pairs that are not within the target angle difference as outliers.

[0042] Secondly, embodiments of this application provide a localization and mapping device for dynamic scenes, the localization and mapping device comprising:

[0043] The target image acquisition unit is used to acquire the target image;

[0044] The feature semantic acquisition unit is used to input the target image into a multi-task neural network model trained based on a preset loss function to obtain a feature point information set and a semantic segmentation result set. The feature point information set includes the orientation information of the extracted feature points.

[0045] The first-layer outlier removal unit is used to remove outliers from the feature point information set based on the semantic segmentation result set to obtain the target feature point information set. The first-layer outlier removal is used to remove feature point information of dynamic objects in the target image.

[0046] The feature point matching unit is used to match feature points in the target feature point information set based on the direction information to obtain a target matching feature pair set;

[0047] The localization and mapping unit is used to estimate the camera pose based on the target matching feature set, and to complete the localization of the camera and map construction based on the estimated pose.

[0048] Thirdly, embodiments of this application provide an intelligent device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the localization and mapping method as described in the first aspect above.

[0049] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the localization and mapping method as described in the first aspect above.

[0050] Fifthly, embodiments of this application provide a computer program product that, when run on a smart device, causes the smart device to execute the positioning and mapping method described in the first aspect above.

[0051] In this embodiment, a target image is acquired and input into a multi-task neural network model trained based on a preset loss function to obtain a feature point information set and a semantic segmentation result set. The feature point information set includes the direction information of the extracted feature points. Based on the semantic segmentation result set, a first-layer outlier removal is performed on the feature point information set to obtain a target feature point information set. The first-layer outlier removal is used to remove feature point information of dynamic objects in the target image. Based on the direction information, feature points in the target feature point information set are matched to obtain a target matching feature pair set. Camera pose estimation is performed based on the target matching feature pair set, and the camera localization and map construction are completed according to the estimated pose. In this application, the multi-task neural network model trained based on the preset loss function performs high-precision semantic segmentation and feature extraction, which is beneficial for removing feature points of dynamic objects and can improve the accuracy and effectiveness of feature extraction, thereby reducing localization drift error and improving the accuracy of localization and mapping in dynamic scenes. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart illustrating the implementation of the localization and mapping method for dynamic scenes provided in this application embodiment;

[0054] Figure 2 This is a flowchart illustrating the specific implementation of the localization and mapping method for dynamic scenes provided in this application, which involves constructing and training a multi-task neural network model.

[0055] Figure 3 This is a flowchart illustrating the specific implementation of step S104 in the localization and mapping method for dynamic scenes provided in this application embodiment;

[0056] Figure 4 This is a structural block diagram of a localization and mapping device for dynamic scenes provided in an embodiment of this application;

[0057] Figure 5 This is a schematic diagram of the smart device provided in the embodiments of this application. Detailed Implementation

[0058] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0059] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0060] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0061] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0062] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0063] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0064] The various method embodiments of this application provide a localization and mapping method applicable to various types of intelligent devices that require localization and mapping in dynamic scenes, specifically intelligent vehicle devices and intelligent robots. This application does not limit the type of intelligent device.

[0065] Figure 1 The implementation flow of the localization and mapping method for dynamic scenes provided in this application embodiment is illustrated. The method flow includes steps S101 to S105. The specific implementation principle of each step is as follows:

[0066] S101: Acquire the target image.

[0067] The target image is a color image of the target scene. Specifically, it can be a single color RGB camera sensor image.

[0068] S102: Input the target image into a multi-task neural network model trained based on a preset loss function to obtain a feature point information set and a semantic segmentation result set. The feature point information set includes the orientation information of the extracted feature points.

[0069] The multi-task neural network model includes a direction estimation network, a feature extraction network, and a semantic segmentation network. The direction estimation network is used to extract direction information in the target image, the feature extraction network is used to extract feature points in the target image, and the semantic segmentation network is used to perform semantic segmentation on the target image.

[0070] In this embodiment of the application, before inputting the target image into the multi-task neural network model trained based on a preset loss function, the target image is preprocessed, and the preprocessed target image is input into the multi-task neural network model.

[0071] In one possible implementation, the target image is pyramidally scaled to obtain a target image pyramid, and the target image pyramid is input into the multi-task neural network model.

[0072] The orientation estimation network described above consists of a cluster of rotation-invariant convolutional networks. In this embodiment, the first layer of the target image pyramid is input to the semantic segmentation network, the target image pyramid is input to the cluster of rotation-invariant convolutional networks to extract potential orientation information, and the target image pyramid after extracting orientation information is input to the feature extraction network.

[0073] The orientation estimation network consists of a cluster of rotation-invariant convolutional networks. After the target image pyramid passes through the rotation-invariant convolutional network cluster, a rotated feature cluster pyramid is obtained. Addressing the differences in coordinate and orientation information structures, this patent incorporates the REKD feature extractor approach, merging the feature cluster pyramid both channel-wise and numerically. Feature point coordinate extraction relies on more data and requires higher dimensions; therefore, the channel-merging result is used for prediction. After interpolating and scaling the feature pyramid after passing through the rotation-invariant network cluster to the same size, pooling is performed on all feature maps, and channel-wise concatenation and merging are performed. The resulting feature maps are then input into a convolutional layer with a kernel size of 1 to obtain a score for each feature point pixel. The feature point coordinates are obtained by filtering this feature map with a prior threshold θ. Let S be the score corresponding to the pixel coordinates in the image. i,j The final feature point map Key i,j The calculation method is as follows:

[0074]

[0075] i and j are pixel indices. Feature point orientation estimation focuses on feature intensity; therefore, weighted summation is used when merging pyramid features. The pyramid feature maps are also multilinearly interpolated and then pooled along the channels to maximize the intensity of the feature point in the direction represented by that channel. Subsequently, all feature maps are scaled to the same resolution, numerically summed, and a softmax operation is performed on the feature vector corresponding to each pixel to obtain the feature point orientation map O. Each value in the orientation feature vector represents the probability of the corresponding direction; a higher value indicates that the pixel's direction is the direction represented by that value. For example, assuming a 360-degree direction is divided into 120-degree segments, it can be divided into 3 channels with corresponding directions of 0, 120, and 240. If the probability values ​​output by the neural network are 0.1, 0.8, and 0.1, then the direction represented by the corresponding pixel is likely 120 degrees.

[0076] As one embodiment of this application, Figure 2 The following illustrates the specific implementation process of constructing and training the multi-task neural network model in the localization and mapping method for dynamic scenes provided in this application embodiment, before inputting the target image into the multi-task neural network model trained based on a preset loss function:

[0077] A1: Construct a multi-task neural network model, which includes a direction estimation network, a feature extraction network, and a semantic segmentation network. The direction estimation network is used to extract direction information in the target image, the feature extraction network is used to extract feature points in the target image, and the semantic segmentation network is used to perform semantic segmentation on the target image.

[0078] A2: Obtain the training sample image set.

[0079] A3: Preprocess the sample images in the training sample image set to obtain preprocessed sample images.

[0080] A4: Input the preprocessed sample image into the semantic segmentation network to obtain the semantic segmentation result, and train the semantic segmentation network based on the first preset loss function.

[0081] In this embodiment, the sample image is scaled in a pyramid shape to obtain a sample image pyramid. The sample image pyramid is then...

[0082] In this embodiment, the semantic segmentation network adopts the mainstream UNet architecture, which includes two modules: an encoder and a decoder, and a connection module (Neck) connecting the two. ResNet is used as the encoder in this invention. Rotated image features are extracted through a four-layer ResNet network to obtain four feature maps of different resolutions, abbreviated as P1, P2, P3, and P4, which are used for the synthesized output of the decoder. Subsequently, an intermediate layer module transforms the P4 features, including two convolutional neural network sequences, each containing a 2D convolutional network, a BatchNorm layer, and a ReLU activation layer. After passing through the intermediate layer module, feature map U4 is obtained, which is then concatenated with feature P4 and input into an upsampling module to obtain feature map U3. The above operation is repeated to finally obtain feature map U1 of the same size as the input features. Finally, U1 is input into a semantic extraction module (a convolutional neural network sequence) to obtain the final semantic segmentation map S.

[0083] As one possible implementation of this application, training the semantic segmentation network based on a first preset loss function includes:

[0084] The first loss L is calculated according to the following formula. sum :

[0085]

[0086] Where H represents the height of the preprocessed sample image S, W represents the width of S, Q represents the ground truth image corresponding to S, BCELoss is the first preset loss function, and Q ij This represents the truth label data for pixel (i,j), where i and j are the pixel indices.

[0087] A5: Input the preprocessed sample image into the orientation estimation network, extract the orientation information in the preprocessed sample image, and train the orientation estimation network based on the second preset loss function.

[0088] As one possible implementation of this application, training the orientation estimation network based on the second preset loss function includes:

[0089] The second loss L is calculated according to the following formula. ori :

[0090]

[0091] Where H represents the height of the preprocessed sample image S, W represents the width of S, G represents the number of clusters in the rotation-invariant convolutional network, k represents the k-th cluster, 1≤k≤G, and O is the optimal direction vector, which refers to the direction vector after the softmax function. Let represent the optimal direction vector corresponding to the coordinates ij pixels generated by the k-th cluster of the a-th image. This represents the optimal direction vector corresponding to the coordinates of pixel ij generated by the k-th cluster of the b-th image.

[0092] In this embodiment, the obtained rotation vectors are first histogram aligned using matrix T, and then the loss L on the orientation vectors of all pixels in each group of the rotation cluster is calculated using cross-entropy. ori .

[0093] As one possible implementation of this application, the feature extraction network L is trained based on a third preset loss function. kp ,include:

[0094] The third loss L is calculated according to the following formula. kp :

[0095]

[0096] I a Representing image a, I b Representing image b, T gt Let λ represent the affine transformation matrix, l represent the level of the sample image pyramid, L represent the total number of levels in the sample image pyramid, λ be a preset parameter, and N represent the sliding window size. l L represents the size of the l-th sliding window. ip (I a ,I b ,T gt N l ) represents the correspondence between feature points in image a and image b.

[0097] A6: Input the preprocessed sample image containing the directional information into the feature extraction network, extract feature points in the preprocessed sample image, and train the feature extraction network based on the third preset loss function.

[0098] A7: The multi-task neural network model is trained based on a preset loss function, which is determined according to the first loss of the semantic segmentation network, the second loss of the orientation estimation network, and the third loss of the feature extraction network during the training process.

[0099] As one possible implementation of this application, the loss (LOSS) of the multi-task neural network model is calculated according to the following formula:

[0100] Loss=βL sum +(1-β)·(L kp +L ori (5)

[0101] Among them, L sum L represents the first loss. ori L represents the second loss. kp The third loss is represented by β, which is a preset weight.

[0102] S103: Based on the semantic segmentation result set, perform a first-layer outlier removal on the feature point information set to obtain a target feature point information set. The first-layer outlier removal is used to remove feature point information of dynamic objects in the target image.

[0103] In this embodiment, based on the semantic segmentation result set, feature points of dynamic objects in the scene are removed. The defined dynamic object regions in the current scene, such as vehicles and pedestrians, can be easily obtained through semantic information. The feature points with corresponding coordinates are removed from the feature point information set to complete the first layer of outlier removal.

[0104] S104: Based on the direction information, match the feature points in the target feature point information set to obtain a target matching feature pair set.

[0105] As one possible implementation of this application Figure 3 The specific implementation flow of step S104 in the localization and mapping method for dynamic scenes provided in the embodiments of this application is illustrated below:

[0106] B1: Obtain the descriptor for each feature point in the target matching feature pair set. A descriptor is a vector that describes the meaning represented by a pixel and is used for subsequent feature matching. The distance between the descriptors of two feature points is compared; the closer the descriptors are, the more similar the feature points they represent.

[0107] B2: Based on the descriptor, match the feature points in the target feature point information set to obtain an initial set of matching feature pairs containing matching pairs.

[0108] In this embodiment, BFMatcher is used to match descriptors of floating-point types.

[0109] B3: Based on the direction information, calculate the direction difference between all matching pairs in the initial matching feature pair set, and determine the target angle difference. The target angle difference is the three angle differences with the smallest direction difference in the initial matching feature pair set.

[0110] Specifically, the directional differences between all matching pairs in the initial matching feature pair set are sorted from largest to smallest, and the three angular differences at the end of the sort are determined as the target angular differences.

[0111] B4: Based on the target angle difference, perform a second layer of outlier removal on the matching pairs in the initial matching feature pair set to obtain a target matching feature pair set. The second layer of outlier removal is used to remove all matching pairs that are not within the target angle difference as outliers.

[0112] Current real-time localization and mapping (RTL) systems in dynamic environments still rely on traditional feature extractors based on manual priors. This application's embodiments extract learning-based dense feature points and their descriptors from images while simultaneously extracting semantic information, resulting in more robust feature point descriptions. Furthermore, the extracted feature points fully consider the instability caused by the rotation of intelligent devices during movement; the feature points extracted by this method contain directional information not found in other learning-based feature points, further improving the stability of the entire RTL system.

[0113] S105: Based on the target matching feature set, perform camera pose estimation, and complete the camera localization and map construction according to the estimated pose.

[0114] In this embodiment, camera pose calculation is performed based on the matching pairs in the target matching feature pair set. The camera pose is optimized by tracking the local map, thereby completing the camera localization and map construction. If a local map already exists, the extracted feature points are matched with the local map.

[0115] In this embodiment, three tracking models exist in the system: a motion model, a keyframe model, and a relocalization model. These three models are progressive, with the next model being used when the previous one fails. For example, the motion model assumes that the automated equipment carrying the camera moves at a constant speed. It estimates the pose of the current frame using the velocity and pose of the previous frame and then performs matching. The specific process is as follows: using the current pose R,t, it tracks the feature points of the previous frame, projects the 3D feature points of the previous frame onto the 2D coordinate system of the current camera using camera intrinsics, and searches for matching points within a given threshold radius. The keyframe model directly uses the nearest keyframe saved in the local map to perform image matching using the FBoW bag-of-words model, and then optimizes the pose by reconstructing the projection error to solve the Perspective-n-Point (PnP: knowing the camera intrinsics, calculating the camera pose using N pairs of matched image coordinates and their world coordinates).

[0116]

[0117] Where u represents the pixel coordinates of the projection, is the camera intrinsic parameter, T is the camera pose, T* represents the newly calculated camera pose, and P is the 3D spatial coordinates of the keypoint in the keyframe. cThis represents the coordinates of the c-th keypoint in the keyframe space, where n is the total number of keypoints and s is the scaling factor.

[0118] The relocalization model performs FBoW bag-of-words matching between the current frame and all keyframes saved in the map, and the camera pose estimation also solves the PnP problem by minimizing the reconstruction error.

[0119] In one possible implementation, loop closure optimization is performed during the mapping process. Loop closure optimization detects closed loops, finds co-view points, and merges poses to reduce map size and improve localization and mapping performance. Loop closure detection relies on the FBoW bag-of-words model. It finds co-view keyframes of the current keyframe and calculates the bag-of-words similarity between them. If the number of keyframes with similarity higher than a manually given prior threshold exceeds a given threshold, a loop closure is detected.

[0120] The specific implementation process of loop closure optimization in this embodiment can refer to existing technologies and will not be elaborated here. However, it should be noted that in this embodiment, both the graph construction and loop closure optimization processes require the use of the bag-of-words model. The reason for using the FBoW bag-of-words model for training and replacing the DPoW2 bag-of-words model in the original system is that the DPoW2 bag-of-words model is not friendly to floating-point data and has low operating efficiency for deep dense descriptors of floating-point data. FBoW natively supports floating-point types and has a good interface that is easy to port. In addition, the DPoW2 bag-of-words model is based on text data, and the system consumes a lot of time when reading the dictionary at startup. The FBoW dictionary is binary stream data, and FBoW also has hardware acceleration capabilities, making it significantly superior to the DPoW2 bag-of-words model in terms of dictionary loading and calculation. For the use of FBoW in this invention, it is first necessary to collect training datasets for different scenarios. For example, the KITTI dataset is used for outdoor autonomous driving scenarios, while the TUM dataset is used for indoor scenarios. Subsequently, the same feature extractor in the front-end module is used to extract features from the images in the dataset. Then, FBoW bag-of-words training is performed on the descriptor subset contained in each image. Finally, the dictionary is saved as a binary file to the disk for use by the localization and mapping system.

[0121] The new bag-of-words model used in this application is adapted to the floating-point descriptor format and supports hardware acceleration, which improves the execution speed of system localization and mapping.

[0122] In this embodiment, a target image is acquired and input into a multi-task neural network model trained based on a preset loss function to obtain a feature point information set and a semantic segmentation result set. The feature point information set includes the direction information of the extracted feature points. Based on the semantic segmentation result set, a first-layer outlier removal is performed on the feature point information set to obtain a target feature point information set. The first-layer outlier removal is used to remove feature point information of dynamic objects in the target image. Based on the direction information, feature points in the target feature point information set are matched to obtain a target matching feature pair set. Camera pose estimation is performed based on the target matching feature pair set, and the camera localization and map construction are completed according to the estimated pose. In this application, the multi-task neural network model trained based on the preset loss function performs high-precision semantic segmentation and feature extraction, which is beneficial for removing feature points of dynamic objects and can improve the accuracy and effectiveness of feature extraction, thereby reducing localization drift error and improving the accuracy of localization and mapping in dynamic scenes.

[0123] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0124] Corresponding to the positioning and mapping method described in the above embodiments, Figure 4 The diagram shows a structural block diagram of a localization and mapping device for dynamic scenes provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0125] Reference Figure 4 The localization and mapping device for dynamic scenes includes: a target image acquisition unit 41, a feature semantic acquisition unit 42, a first-layer culling unit 43, a feature point matching unit 44, and a localization and mapping unit 45, wherein:

[0126] Target image acquisition unit 41 is used to acquire target images;

[0127] The feature semantic acquisition unit 42 is used to input the target image into a multi-task neural network model trained based on a preset loss function to obtain a feature point information set and a semantic segmentation result set. The feature point information set includes the orientation information of the extracted feature points.

[0128] The first-layer removal unit 43 is used to perform a first-layer outlier removal on the feature point information set based on the semantic segmentation result set to obtain a target feature point information set. The first-layer outlier removal is used to remove feature point information of dynamic objects in the target image.

[0129] The feature point matching unit 44 is used to match feature points in the target feature point information set based on the direction information to obtain a target matching feature pair set;

[0130] The localization and mapping unit 45 is used to estimate the camera pose based on the target matching feature set, and to complete the localization of the camera and map construction based on the estimated pose.

[0131] As one possible implementation of this application, the above-mentioned positioning and mapping device further includes:

[0132] The model building unit is used to build a multi-task neural network model, which includes a direction estimation network, a feature extraction network, and a semantic segmentation network. The direction estimation network is used to extract direction information in the target image, the feature extraction network is used to extract feature points in the target image, and the semantic segmentation network is used to perform semantic segmentation on the target image.

[0133] The sample acquisition unit is used to acquire the training sample image set;

[0134] The preprocessing unit is used to preprocess the sample images in the training sample image set to obtain preprocessed sample images;

[0135] The model training unit is configured to: input the preprocessed sample image into the semantic segmentation network to obtain semantic segmentation results, and train the semantic segmentation network based on a first preset loss function; input the preprocessed sample image into the orientation estimation network to extract orientation information from the preprocessed sample image, and train the orientation estimation network based on a second preset loss function; input the preprocessed sample image containing the orientation information into the feature extraction network to extract feature points from the preprocessed sample image, and train the feature extraction network based on a third preset loss function; and train the multi-task neural network model based on a preset loss function, wherein the preset loss function is determined according to the first loss of the semantic segmentation network, the second loss of the orientation estimation network, and the third loss of the feature extraction network during training.

[0136] As one possible implementation of this application, the preprocessing unit is specifically used for:

[0137] The sample image is then scaled in a pyramid shape to obtain a sample image pyramid.

[0138] As one possible implementation of this application, training the multi-task neural network model based on a preset loss function includes:

[0139] The loss (LOSS) of the multi-task neural network model is calculated according to the following formula:

[0140] Loss=βL sum +(1-)·(L kp + ori )

[0141] Among them, L sum L represents the first loss. ori L represents the second loss. kp The third loss is represented by β, which is a preset weight.

[0142] As one possible implementation of this application, training the semantic segmentation network based on a first preset loss function includes:

[0143] The first loss L is calculated according to the following formula. sum :

[0144]

[0145] Where H represents the height of the preprocessed sample image S, W represents the width of S, Q represents the ground truth image corresponding to S, BCELoss is the first preset loss function, and Q ij This represents the true label data for pixel (i,j).

[0146] As one possible implementation of this application, the orientation estimation network is composed of a cluster of rotation-invariant convolutional networks, and training the orientation estimation network based on a second preset loss function includes:

[0147] The second loss L is calculated according to the following formula. ori :

[0148]

[0149] Where H represents the height of the preprocessed sample image S, W represents the width of S, G represents the number of clusters in the rotation-invariant convolutional network, k represents the k-th cluster, 1≤k≤G, and O is the optimal orientation vector. Let represent the optimal direction vector corresponding to the coordinates ij pixels generated by the k-th cluster of the a-th image. This represents the optimal direction vector corresponding to the coordinates of the pixel point ij generated by the k-th cluster of the b-th image. The optimal direction vector is the direction vector after passing through the softmax function.

[0150] The feature extraction network L is trained based on a third preset loss function. kp ,include:

[0151] The third loss L is calculated according to the following formula. kp :

[0152]

[0153] Among them, I a Representing image a, I b Representing image b, T gt Let λ represent the affine transformation matrix, l represent the level of the sample image pyramid, L represent the total number of levels in the sample image pyramid, λ be a preset parameter, and N represent the sliding window size. l L represents the size of the l-th sliding window. ip (I a ,I b ,T gt N l ) represents the correspondence between feature points in image a and image b.

[0154] As one possible implementation of this application, the feature point matching unit 44 includes:

[0155] The descriptor acquisition module is used to acquire the descriptor of each feature point in the target matching feature pair set;

[0156] The matching module is used to match feature points in the target feature point information set based on the descriptor, and obtain an initial set of matching feature pairs containing matching pairs;

[0157] The target angle difference determination module is used to calculate the direction difference between all matching pairs in the initial matching feature pair set based on the direction information, and determine the target angle difference, wherein the target angle difference is the three angle differences with the smallest direction difference values ​​in the initial matching feature pair set;

[0158] The matching pair set determination module is used to perform a second-layer outlier removal on the matching pairs in the initial matching feature pair set based on the target angle difference, to obtain the target matching feature pair set. The second-layer outlier removal is used to remove all matching pairs that are not within the target angle difference as outliers.

[0159] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0160] This application embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements... Figures 1 to 3 This represents the steps of any localization and mapping method applied to dynamic scenes.

[0161] This application embodiment also provides a smart device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements... Figures 1 to 3 This represents the steps of any localization and mapping method applied to dynamic scenes.

[0162] This application also provides a computer program product that, when run on a server, causes the server to execute the implementation as described above. Figures 1 to 3 This represents the steps of any localization and mapping method applied to dynamic scenes.

[0163] Figure 5 This is a schematic diagram of a smart device provided in an embodiment of this application. Figure 5 As shown, the intelligent device 5 in this embodiment includes: a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50. When the processor 50 executes the computer program 52, it implements the steps described in the various embodiments of the localization and mapping method applied to dynamic scenes, for example... Figure 1 The steps S101 to S105 are shown. Alternatively, when the processor 50 executes the computer program 52, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 4 The functions of units 41 to 45 are shown.

[0164] For example, the computer program 52 may be divided into one or more modules / units, which are stored in the memory 51 and executed by the processor 50 to complete this application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the computer program 52 in the smart device 5.

[0165] The intelligent device 5 can be an intelligent robot. The intelligent device 5 may include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art will understand that... Figure 5 This is merely an example of smart device 5 and does not constitute a limitation on smart device 5. It may include more or fewer components than shown, or combine certain components, or different components. For example, smart device 5 may also include input / output devices, network access devices, buses, etc.

[0166] The processor 50 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0167] The memory 51 can be an internal storage unit of the smart device 5, such as a hard drive or memory of the smart device 5. The memory 51 can also be an external storage device of the smart device 5, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the smart device 5. Furthermore, the memory 51 can include both internal and external storage units of the smart device 5. The memory 51 is used to store the computer program and other programs and data required by the smart device. The memory 51 can also be used to temporarily store data that has been output or will be output.

[0168] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0169] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0170] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / terminal equipment, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0171] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0172] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A localization and mapping method for dynamic scenes, characterized in that, The location mapping method includes: Acquire the target image; A multi-task neural network model is constructed, comprising a direction estimation network, a feature extraction network, and a semantic segmentation network. The direction estimation network is used to extract direction information in the target image, the feature extraction network is used to extract feature points in the target image, and the semantic segmentation network is used to perform semantic segmentation on the target image. Obtain the training sample image set; The sample images in the training sample image set are preprocessed to obtain preprocessed sample images; The preprocessed sample image is input into the semantic segmentation network to obtain the semantic segmentation result, and the semantic segmentation network is trained based on the first preset loss function; The preprocessed sample image is input into the orientation estimation network to extract orientation information from the preprocessed sample image, and the orientation estimation network is trained based on the second preset loss function. The preprocessed sample image containing the directional information is input into the feature extraction network to extract feature points from the preprocessed sample image, and the feature extraction network is trained based on the third preset loss function. The multi-task neural network model is trained based on a preset loss function, which is determined according to the first loss of the semantic segmentation network, the second loss of the orientation estimation network, and the third loss of the feature extraction network during the training process. The target image is input into the multi-task neural network model trained based on the preset loss function to obtain a feature point information set and a semantic segmentation result set. The feature point information set includes the orientation information of the extracted feature points. Based on the semantic segmentation result set, the feature point information set is subjected to a first-layer outlier removal to obtain the target feature point information set. The first-layer outlier removal is used to remove the feature point information of dynamic objects in the target image. Based on the directional information, feature points in the target feature point information set are matched to obtain a target matching feature pair set; Camera pose estimation is performed based on the target matching feature set, and the camera localization and map construction are completed based on the estimated pose.

2. The positioning and mapping method according to claim 1, characterized in that, The step of preprocessing the sample images in the training sample image set to obtain preprocessed sample images includes: The sample image is then scaled in a pyramid shape to obtain a sample image pyramid.

3. The positioning and mapping method according to claim 2, characterized in that, The process of training the multi-task neural network model based on a preset loss function includes: The loss (LOSS) of the multi-task neural network model is calculated according to the following formula: Among them, L sum L represents the first loss. ori L represents the second loss. kp The third loss is represented by β, which is a preset weight.

4. The positioning and mapping method according to claim 3, characterized in that, Training the semantic segmentation network based on the first preset loss function includes: The first loss L is calculated according to the following formula. sum : Where H represents the height of the preprocessed sample image S, W represents the width of S, Q represents the ground truth image corresponding to S, BCELoss is the first preset loss function, and Q ij This represents the truth label data for pixel (i,j).

5. The positioning and mapping method according to claim 3, characterized in that, The orientation estimation network is composed of a cluster of rotation-invariant convolutional networks, and training the orientation estimation network based on a second preset loss function includes: The second loss L is calculated according to the following formula. ori : Where H represents the height of the preprocessed sample image S, W represents the width of S, G represents the number of clusters in the rotation-invariant convolutional network, k represents the k-th cluster, 1≤k≤G, and O is the optimal orientation vector. Let represent the optimal direction vector corresponding to the coordinates ij pixels generated by the k-th cluster of the a-th image. Q represents the optimal direction vector corresponding to the coordinates of the ij pixels generated by the k-th cluster of the b-th image. The optimal direction vector is the direction vector after passing through the softmax function. The feature extraction network L is trained based on a third preset loss function. kp ,include: The third loss L is calculated according to the following formula. kp : Among them, I a Representing image a, I b Representing image b, T gt Let λ represent the affine transformation matrix, l represent the level of the sample image pyramid, L represent the total number of levels in the sample image pyramid, λ be a preset parameter, and N represent the sliding window size. l This represents the size of the l-th sliding window. This represents the correspondence between feature points in image a and image b.

6. The positioning and mapping method according to claim 1, characterized in that, The step of matching feature points in the target feature point information set based on the direction information to obtain a target matching feature pair set includes: Obtain the descriptor for each feature point in the target matching feature pair set; Based on the descriptor, feature points in the target feature point information set are matched to obtain an initial set of matched feature pairs containing matching pairs; Based on the direction information, the direction difference between all matching pairs in the initial matching feature pair set is calculated, and the target angle difference is determined. The target angle difference is the three angle differences with the smallest direction difference values ​​in the initial matching feature pair set. Based on the target angle difference, a second layer of outlier removal is performed on the matching pairs in the initial matching feature pair set to obtain the target matching feature pair set. The second layer of outlier removal is used to remove all matching pairs that are not within the target angle difference as outliers.

7. A positioning and mapping device for dynamic scenes, characterized in that, The positioning and mapping device includes: The target image acquisition unit is used to acquire the target image; The model building unit is used to build a multi-task neural network model, which includes a direction estimation network, a feature extraction network, and a semantic segmentation network. The direction estimation network is used to extract direction information in the target image, the feature extraction network is used to extract feature points in the target image, and the semantic segmentation network is used to perform semantic segmentation on the target image. The sample acquisition unit is used to acquire the training sample image set; The preprocessing unit is used to preprocess the sample images in the training sample image set to obtain preprocessed sample images; The model training unit is configured to: input the preprocessed sample image into the semantic segmentation network to obtain semantic segmentation results, and train the semantic segmentation network based on a first preset loss function; input the preprocessed sample image into the orientation estimation network to extract orientation information from the preprocessed sample image, and train the orientation estimation network based on a second preset loss function; input the preprocessed sample image containing the orientation information into the feature extraction network to extract feature points from the preprocessed sample image, and train the feature extraction network based on a third preset loss function; and train the multi-task neural network model based on a preset loss function, wherein the preset loss function is determined according to the first loss of the semantic segmentation network, the second loss of the orientation estimation network, and the third loss of the feature extraction network during training. The feature semantic acquisition unit is used to input the target image into the multi-task neural network model trained based on the preset loss function to obtain a feature point information set and a semantic segmentation result set. The feature point information set includes the orientation information of the extracted feature points. The first-layer outlier removal unit is used to remove outliers from the feature point information set based on the semantic segmentation result set to obtain the target feature point information set. The first-layer outlier removal is used to remove feature point information of dynamic objects in the target image. The feature point matching unit is used to match feature points in the target feature point information set based on the direction information to obtain a target matching feature pair set; The localization and mapping unit is used to estimate the camera pose based on the target matching feature set, and to complete the localization of the camera and map construction based on the estimated pose.

8. A smart device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the localization and mapping method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the localization and mapping method as described in any one of claims 1 to 6.

Images