A method and apparatus for determining a relocation frame, a vehicle, and a storage medium

Abstract

The application discloses a kind of methods for determining reposition frame, device, vehicle and storage medium.The method comprises: obtaining feature map;Determine the image key frame of current image, according to the global feature of the image key frame and the global feature of the map key frame, determine current difference degree matrix;According to the current difference degree matrix, determine reposition mark according to preset dynamic programming algorithm;Based on the reposition mark, determine target reposition frame in the feature map.The technical scheme of the embodiment of the application is smaller in memory resources occupied by the established feature map, solves the problem that great computing resources are consumed in searching reposition frame in the global range of semantic map, is suitable for the scene with high scene similarity and poor communication signal, improves the accuracy and robustness of reposition frame.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle positioning, and in particular to a method and device for determining a repositioning frame, a vehicle and a storage medium. BACKGROUND

[0002] Visual feature mapping is one of the basic and core technologies of indoor autonomous driving and other robot application technologies, and provides effective prior information for feature matching of robot visual repositioning. According to the feature type, image features can be divided into two categories, namely local image features and global image features.

[0003] Currently, there are two main repositioning technologies. One is geometric relationship-based repositioning, that is, ranging or angling of a reference device at a known position, and then calculating the pose of the robot through geometric relationship. The other is feature map-based repositioning, which matches the current observed features with the pre-built feature map to calculate the pose of the robot.

[0004] However, high-precision positioning based on a feature map requires matching the initial position observed at present with a semantic map, and searching for an initial map frame (i.e., a repositioning frame) in the global range of the semantic map usually consumes a large amount of computing resources. For indoor scenes with high scene similarity and poor communication signals, such as underground parking garages, the accuracy and robustness of the initial map frame determined by conventional methods are poor. SUMMARY

[0005] The present application provides a method and device for determining a repositioning frame, a vehicle and a storage medium to solve the problem of poor accuracy and robustness of the determined initial map frame.

[0006] In a first aspect, the present application provides a method for determining a repositioning frame, comprising:

[0007] obtaining a feature map, wherein the feature map comprises a plurality of map key frames, and the map key frames contain global features and local features of map images;

[0008] determining an image key frame of a current image, and determining a current difference degree matrix according to the global features of the image key frame and the global features of the map key frames, wherein the difference degree matrix is used to represent the difference degree between the image key frame and the map key frames, and the image key frame contains global features and local features of the current image;

[0009] determining a repositioning identifier according to a preset dynamic programming algorithm and the current difference degree matrix, wherein the repositioning identifier is determined based on a first preset position in the current difference degree matrix, and the first preset position includes the position of the minimum difference degree in each column or each row of the difference degree in the current difference degree matrix;

[0010] determine a target relocalization frame in the feature map based on the relocalization identifier.

[0011] In a second aspect, an embodiment of the present application provides a relocalization frame determination apparatus, comprising:

[0012] a feature map acquisition module configured to acquire a feature map, wherein the feature map comprises a plurality of map key frames, and the map key frames contain global features and local features of map images;

[0013] a difference degree matrix determination module configured to determine an image key frame of a current image, and determine a current difference degree matrix based on global features of the image key frame and global features of the map key frames, wherein the difference degree matrix is used to represent a difference degree between the image key frame and the map key frames, and the image key frame contains global features and local features of the current image;

[0014] a relocalization identifier determination module configured to determine a relocalization identifier based on a preset dynamic programming algorithm and the current difference degree matrix, wherein the relocalization identifier is determined based on a first preset position in the current difference degree matrix, and the first preset position comprises a position of a minimum difference degree in each column or each row of the current difference degree matrix;

[0015] a relocalization frame determination module configured to determine a target relocalization frame in the feature map based on the relocalization identifier.

[0016] In a third aspect, an embodiment of the present application provides a vehicle, comprising:

[0017] at least one processor;

[0018] and a memory connected in communication with the at least one processor;

[0019] wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the relocalization frame determination method of the first aspect.

[0020] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer instructions for enabling a processor to execute the relocalization frame determination method of the first aspect when executed.

[0021] The embodiment of the present application provides a determination scheme of a repositioning frame, acquires a feature map, wherein the feature map comprises a plurality of map key frames, the map key frame contains global features and local features of a map image, an image key frame of a current image is determined, a current difference matrix is determined according to the global features of the image key frame and the global features of the map key frame, wherein the difference matrix is used to represent the difference degree of the image key frame and the map key frame, the image key frame contains global features and local features of the current image, a repositioning identifier is determined according to a preset dynamic programming algorithm and the current difference matrix, wherein the repositioning identifier is determined based on a first preset position in the current difference matrix, the first preset position comprises a position of the minimum difference degree in each column or each row of the current difference matrix, and a target repositioning frame is determined in the feature map based on the repositioning identifier. By using the above technical scheme, the feature map is acquired, the image key frame of the current image is determined, the current difference matrix is determined according to the global features of the feature map and the image key frame, then the repositioning identifier is determined according to the current difference matrix by using the preset dynamic programming algorithm, and finally the target repositioning frame is determined in the feature map according to the repositioning identifier. In the scheme, the feature map composed of the map key frames occupies smaller memory resources, the problem that great calculation resources are consumed in searching for an initial map frame (namely, a repositioning frame) in the global range of a semantic map is solved, the scheme is suitable for a scene with high scene similarity and poor communication signals, and the accuracy and robustness of the repositioning frame are improved.

[0022] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0024] Figure 1 is a flow chart of a repositioning frame determination method provided by the embodiment one of the present application;

[0025] Figure 2 is a flow chart of a repositioning frame determination method provided by the embodiment two of the present application;

[0026] Figure 3 is a global feature map provided by the embodiment two of the present application;

[0027] Figure 4 is a visualization diagram of a difference matrix according to the second embodiment of the present application;

[0028] Figure 5 is a visualization diagram of an optimal path determined based on a dynamic programming algorithm according to the second embodiment of the present application;

[0029] Figure 6 is a structural diagram of a repositioning frame determination device according to the third embodiment of the present application;

[0030] Figure 7 is a structural diagram of a vehicle according to the fourth embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the technical personnel in the art better understand the present application scheme, the following will be combined with the drawings in the embodiments of the present application, the technical scheme in the embodiments of the present application is described clearly and completely, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In the description of the present application, "a plurality of" means two or more, unless otherwise specified. And / or

[0033] Or", describes the association relationship between the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after it. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, vehicle or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Embodiment one

[0035] Figure 1A flowchart of a method for determining a relocalization frame is provided for the first embodiment of the present application. The method can be used to determine a relocalization frame for a semantic map during relocalization. The method can be performed by a relocalization frame determining device, which can be implemented in hardware and / or software. The relocalization frame determining device can be configured in a vehicle, which can be composed of two or more physical entities or one physical entity.

[0036] As shown in Figure 1 The method for determining a relocalization frame provided by the first embodiment of the present application specifically includes the following steps:

[0037] S101, obtaining a feature map, wherein the feature map includes a plurality of map key frames, and each map key frame contains global features and local features of a map image.

[0038] In this embodiment, the feature map can be established in advance. When the feature map is needed, such as when a relocalization frame is to be determined, the pre-established feature map can be loaded from a set storage area. Before establishing the feature map, a coordinate system usually needs to be defined. The coordinate system can include a world coordinate system and a vehicle body coordinate system. The world coordinate system can be a Cartesian coordinate system, and the origin position of the world coordinate system can be set. The origin position of the vehicle body coordinate system can be a preset position of the vehicle, such as the center of the rear axle of the vehicle. The map key frame can be understood as an image frame obtained by a preset recording device, which meets the first preset requirement. The first preset requirement can be a requirement for the time interval between image frames of a map image or the corresponding position interval between image frames. The local image feature, i.e., the local feature, usually refers to the local expression of the image feature, which is generally the extreme value of the local image. The global image feature, i.e., the global feature, usually refers to the feature on the entire image, which includes texture color and shape features, etc.

[0039] Specifically, positioning technology can be divided into geometric positioning, dead reckoning and feature map positioning. Geometric positioning can provide absolute positioning information. Common geometric positioning technologies include GNSS (Global Navigation Satellite System), Bluetooth communication technology, 5G (5th Generation Mobile Communication Technology), and UWB (Ultra Wide Band) communication technology. GNSS is the most widely used in the autonomous driving industry. It can provide an initial position for high-precision maps such as semantic maps. GNSS and RTK (Real-time kinematic) carrier phase difference technology can also provide observation values for fusion positioning. Dead reckoning can be understood as a technology that uses IMU (Inertial Measurement Unit) and wheel speed sensors to predict the pose at the next time based on the position at the last time. However, as the relative motion progresses, the error of the pose calculated by dead reckoning will continue to increase. Feature map positioning requires matching the observed image features with the pre-built feature map. However, the number, quality and distinguishability of image features directly affect the accuracy and recall rate of feature map positioning. When external factors such as scene affect feature observation, the accuracy and stability of feature map positioning will decrease.

[0040] Specifically, repositioning can be understood as a process of feature matching and pose acquisition in the global range of a feature map without prior pose information. Repositioning is usually part of positioning. Today, there are mainly two repositioning technologies. One is geometric repositioning based on geometric relationships. This technology measures the distance or angle of a reference device with a known position and calculates the pose of the target object through geometric relationships. Common technologies include GPS technology (Global Positioning System) and RTK carrier phase difference technology. The other is feature map-based repositioning. This technology matches the observed features with the pre-built feature map to calculate the pose of the target object. In outdoor environments, geometric repositioning such as GNSS is usually used for map matching initialization. However, GNSS may have signal differences in outdoor environments such as overpasses. GNSS usually has no signal in indoor environments. Therefore, GNSS cannot be used for map matching initialization in poor signal scenarios. In addition, when the semantic map features are few or the related sensors of DR (Dead reckoning) are temporarily dysfunctional, resulting in a large positioning pose covariance, the positioning result is usually not reliable, so repositioning is needed to reinitialize the map matching.

[0041] S102, determine an image key frame of the current image, and determine a current difference degree matrix according to global features of the image key frame and global features of the map key frame, wherein the difference degree matrix is used to represent a difference degree between the image key frame and the map key frame, and the image key frame contains global features and local features of the current image.

[0042] In the embodiment, the image key frame of the current image can be determined by using the determination method of the map key frame, and then the difference degree between the image key frame and the map key frame, i.e. the difference degree, can be determined according to the global features of the image key frame and the global features of the map key frame, and the difference degree can be in the form of a difference degree matrix composed of the difference degree. The determination method of the image key frame and the determination method of the map key frame can be the same or different. The current image can be understood as a current real-time image obtained by using a preset recording device, and the image key frame of the current image can be understood as an image frame that meets the second preset requirement in the current real-time image frame obtained by using the preset recording device, i.e. the image key frame of the current image. The second preset requirement can be a requirement for the time interval between the image frames of the current image or the corresponding position interval between the image frames. The current difference degree matrix can be a multi-dimensional matrix. The first preset requirement of the map key frame and the second preset requirement can be the same or different.

[0043] S103, determine a reposition identifier according to a preset dynamic programming algorithm and the current difference degree matrix, wherein the reposition identifier is determined based on a first preset position in the current difference degree matrix, and the first preset position includes a position of the minimum difference degree in the difference degree of each column or each row in the current difference degree matrix.

[0044] In the embodiment, the position of the minimum difference degree in the difference degree of each column or each row in the current difference degree matrix can be determined by using the preset dynamic programming algorithm, i.e. the map key frame with the minimum difference degree from the image key frame in the feature map can be determined, which can also be understood as the map key frame with the highest similarity to the image key frame in the feature map. The reposition identifier can be used to represent the position of the map key frame that meets the preset position requirement in the map key frame with the highest similarity to the image key frame, wherein the preset position requirement can be the map key frame with the minimum difference degree from the image key frame of the current image.

[0045] S104, determine a target reposition frame in the feature map based on the reposition identifier.

[0046] In the embodiment, according to the repositioning identifier determined according to the above steps, the target repositioning frame can be determined in the current difference matrix, and the initial position of the vehicle can be calculated according to the global pose in the target repositioning frame, and matched with the semantic map, so as to realize the repositioning of the vehicle. The global pose can be understood as the conversion relationship between the global feature in the world coordinate system and the vehicle coordinate system, and the change of the pose can include translation and rotation.

[0047] The method for determining a repositioning frame provided by the embodiment of the application comprises the following steps: acquiring a feature map, wherein the feature map comprises a plurality of map key frames, and each map key frame comprises global features and local features of a map image; determining an image key frame of a current image; determining a current difference matrix according to the global features of the image key frame and the global features of the map key frames, wherein the difference matrix is used to represent the difference degree between the image key frame and the map key frames, and the image key frame comprises global features and local features of the current image; determining a repositioning identifier according to a preset dynamic programming algorithm and the current difference matrix, wherein the repositioning identifier is determined based on a first preset position in the current difference matrix, and the first preset position comprises a position with the minimum difference degree in each column or each row of the current difference matrix; and determining a target repositioning frame in the feature map based on the repositioning identifier. The technical scheme of the embodiment of the application comprises the following steps: acquiring a feature map and determining an image key frame of a current image, determining a current difference matrix according to the global features of the feature map and the image key frame, determining a repositioning identifier according to the current difference matrix by using a preset dynamic programming algorithm, and finally determining a target repositioning frame in the feature map according to the repositioning identifier. The feature map composed of the map key frames has smaller memory resources, solves the problem of consuming a large amount of computing resources in searching for an initial map frame (i.e., a repositioning frame) in the global range of a semantic map, is suitable for a scene with high scene similarity and poor communication signals, and improves the accuracy and robustness of the repositioning frame.

[0048] Embodiment two

[0049] Figure 2 The flowchart of the method for determining a repositioning frame provided by the embodiment two of the application, and the technical scheme of the embodiment of the application is further optimized on the basis of the above optional technical schemes, and a specific way of determining a repositioning frame is given.

[0050] Optionally, the determining the current difference matrix according to the global feature of the image key frame and the global feature of the map key frame comprises: determining a global feature difference according to a difference between the light intensity of a pixel in the global feature of the image key frame and the light intensity of a pixel in the global feature of the map key frame; determining a difference vector by dividing the global feature difference by a preset number of pixels of the global feature, wherein the preset global feature comprises the global feature of the image key frame or the global feature of the map key frame; and determining the updated current difference matrix according to whether a preset parameter of the difference matrix before updating is greater than a preset parameter threshold and the difference vector, wherein the preset parameter comprises a row number and / or a column number of the difference matrix. This arrangement has the advantage that the difference vector determined according to the light intensity of a pixel in the global feature of the image key frame and the light intensity of a pixel in the global feature of the map key frame is time series data, so that the sequence graph matching result determined according to the current difference matrix still has strong robustness even in a real-time changing and high similarity scene.

[0051] Optionally, the determining the reposition identifier according to the preset dynamic programming algorithm and the current difference matrix comprises: determining, by using the preset dynamic programming algorithm, a first preset position of the current difference matrix, a preset matrix corresponding to the dimension of the current difference matrix, and a second preset position in the preset matrix corresponding to the first preset position, wherein the first preset position comprises a position of a minimum difference in a first preset direction in the current difference matrix, the first preset direction comprises each row direction or each column direction, the preset position comprises a row position and a column position, and a value in the preset matrix is determined based on a sum of the differences; and determining, as the reposition identifier, a row position corresponding to a last column in the second preset position or a column position corresponding to a last row, wherein a value in the second preset position corresponding to the reposition identifier is determined based on a sum of the differences in the first preset position. This arrangement has the advantage that, compared with a conventional linear search, the search path of the first preset position can be dynamically planned in the current difference matrix by using the preset dynamic programming algorithm, thereby improving the accuracy of the reposition identifier, i.e. the accuracy of the map key frame with the highest similarity to the image key frame.

[0052] Optionally, the determining the target relocalization frame in the feature map based on the relocalization identifier comprises: determining a confidence of the relocalization identifier, determining a sequence graph matching result according to the confidence, verifying local features of an initial relocalization frame corresponding to the relocalization identifier based on the sequence graph matching result, and determining the target relocalization frame in the feature map according to a verification result. In this way, the accuracy of the target relocalization frame can be improved by verifying the sequence graph matching result corresponding to the relocalization identifier.

[0053] As shown in Figure 2 The embodiment two of the present application provides a method for determining a relocalization frame, which specifically comprises the following steps:

[0054] S201, acquiring a feature map.

[0055] The feature map comprises a plurality of map key frames, and the map key frame contains global features and local features of a map image.

[0056] Specifically, the correlation between local features is small, and in the case of occlusion, the detection and matching of other local features will not be affected by the disappearance of part of the local features. Some excellent local features, such as local features obtained by using SIFT (Scale-invariant feature transform) or SURF algorithm (Speed-up robust features), have the advantages of high detection repetition rate, and the feature descriptor has rotation, illumination, and scale invariance. However, the computing and storage resources of the current mass-produced automatic driving system are limited, and the above-mentioned feature extraction algorithm with rotation, illumination, and scale invariance occupies a large amount of computing and storage resources in the CPU (Central Processing Unit). Although some local features, such as ORB (Oriented FAST and Rotated BRIEF) extracted in the Gaussian space, to some extent, solve the problem of scale invariance, and the RIF (Rotated Invariant Feature) descriptor also solves the problem of rotation invariance to some extent, but the feature map based on ORB features does not work well in the scene where the illumination changes significantly, the viewing angle changes greatly, and the environment is extremely similar, such as underground garage. In addition, the repositioning based on local features usually needs to pre-train a bag-of-words model, which needs to aggregate the local features of the image into global features of the image, and then use the global features to search on the feature map to obtain the repositioning frame, and through the geometric relationship between the image frames in the 3D (3D) and 2D (2D) space of the feature map, the current pose of the robot is calculated. However, this method needs to load the pre-trained bag-of-words model, which occupies a large amount of storage resources, and different application scenarios need to train different bag-of-words models, and the calculation of 3D-2D geometric relationship based on monocular camera, i.e. relative pose calculation, is without scale, so it is difficult to align the feature map obtained by using monocular camera online and reposition.

[0057] Specifically, the global feature is usually a low-level visual feature at the pixel level, and the pixels in the global feature generally have geometric correlation, so the global feature still has good environmental change invariance in the case of environmental change. Global features also have the characteristics of intuitive expression and easy calculation.

[0058] Optionally, the updating method of the feature map comprises:

[0059] 1) determining a current key frame of a map image according to whether an offset distance of a position of a current frame of the map image relative to a position of a previous map key frame is greater than a preset distance threshold, wherein the position is determined based on a world coordinate system in which a vehicle is located.

[0060] Specifically, a preset recording device, such as a front-view fisheye camera, can be used to obtain the map image. The preset recording device and the vehicle body can be regarded as a whole, and the difference between the position of the current frame of the map image in the world coordinate system described above and the position of the previous map key frame in the world coordinate system is the offset distance. If the offset distance is greater than or equal to a preset distance threshold, such as 9 meters, the current frame of the map image can be determined as the current key frame. If the offset distance is less than the preset distance threshold, the current frame is not determined as the current key frame, and the frame image of the map image is continuously obtained. Compared with a monocular camera, the front-view fisheye camera as an input device of the map image increases the spatial geometric information, thereby improving the accuracy and accuracy when calculating the difference between scenes, and the recall rate of the key frame. The map key frame in the feature map is usually stored in a preset cache.

[0061] 2) determining a neighboring global pose estimate of a map key frame from a preset pose cache library based on a first timestamp of the current key frame, and performing interpolation processing on the neighboring global pose estimate by using a preset linear interpolation algorithm to obtain a global pose of the current key frame, wherein the key frame contains a first timestamp representing time information, the neighboring global pose estimate includes at least two global pose estimates closest in time to the first timestamp, the preset pose cache library contains at least two global pose estimates, and the global pose estimate contains a second timestamp representing time information.

[0062] Specifically, the key frame, the global pose estimate, and the global pose can all contain a timestamp representing time information. Generally, the image acquisition frequency is less than the dead reckoning frequency, thereby causing the first timestamp of the current key frame and the second timestamp of the global pose estimate to be misaligned. Therefore, at least two global pose estimates corresponding to times close to the first timestamp of the current key frame can be obtained from the preset pose cache library, such as a global pose estimate corresponding to a time closest to and prior to the first timestamp and a global pose estimate corresponding to a time closest to and later than the first timestamp The two global pose estimations are adjacent global pose estimations, and then a preset linear interpolation algorithm, such as a linear interpolation algorithm based on a quaternion, is used to interpolate the adjacent global pose estimations, so as to calculate the global pose of the current key frame and realize the alignment of the first timestamp and the second timestamp in time. The global pose estimation can be obtained by dead reckoning using a preset sensor such as an IMU and a wheel speed meter. Compared with the global pose obtained by visual dead reckoning alone, the global pose obtained by the IMU and the wheel speed meter has not only scale information but also higher accuracy. Since the global pose estimation is a predicted value, the determination time of the global pose estimation is earlier than the determination time of the current key frame. The IMU coordinate system is usually on the vehicle, and the data measured by the IMU is rigidly moved with the vehicle in the IMU coordinate system. The data measured by the IMU and the result of the dead reckoning are both different from the vehicle coordinate system by an external parameter T BI .

[0063] For example, the determination manner of the translation t c in the global pose of the current key frame can be as follows:

[0064]

[0065] wherein, represents the rotation of the global pose estimation corresponding to the time closest to the first timestamp of the image key frame, represents the rotation of the global pose estimation corresponding to the time closest to the first timestamp of the image key frame, scale0 represents a second coefficient, and scale1 represents a third coefficient. The second coefficient and the third coefficient are determined according to the first timestamp of the image key frame, the second timestamp of and the second timestamp of

[0066] For example, the determination manner of the rotation q c in the global pose of the current key frame can be as follows:

[0067]

[0068] wherein, represents the rotation of the global pose estimation corresponding to the time closest to the first timestamp of the image key frame, represents the rotation of the global pose estimation corresponding to the time closest to the first timestamp of the image key frame, scale0 represents a second coefficient, and scale1 represents a third coefficient. The second coefficient and the third coefficient are determined according to the first timestamp of the image key frame, and The angle between the first coefficient and the third coefficient is determined, and the specific determination manner of the second coefficient is generally different from the specific determination manner of the third coefficient.

[0069] 3) performing a reduction processing on the gray image of the current key frame, and determining the global feature of the current key frame based on the illumination intensity of the pixels in the gray image after the reduction processing.

[0070] Specifically, the current key frame can be subjected to a gray processing to obtain a gray image of the current key frame, and then the gray image is reduced to a small image with a set size, such as 64mm*32mm, by using a preset reduction algorithm, such as a Lanczos algorithm, and the small image can be divided into a plurality of small grids, and then the fast standardization processing is performed on each small grid to obtain the global feature of the current key frame.

[0071] Exemplarily, Figure 3 is a global feature map, and the illumination intensity of each pixel in the small grid after the block standardization processing The calculation method of the illumination intensity of each pixel in the small grid can be as follows:

[0072]

[0073] wherein I represents the illumination intensity value of each pixel in the small grid, represents the average value of the illumination intensity of all the pixels in the small grid, and k represents the sample deviation of the illumination intensity of all the pixels in the small grid, such as Figure 3 As shown in FIG. 1, each small grid can contain a plurality of pixels.

[0074] 4) performing a reduction processing on the gray image of the current key frame, and performing a dimension reduction mapping processing on the gray image after the reduction processing by using a preset model to obtain the local feature of the current key frame.

[0075] Specifically, the gray image after the reduction processing can be obtained by using the method described above, and then a preset number, such as 300, of ORB feature points are extracted from the gray image after the reduction processing, and then the PCA (Principal Component Analysis) dimension reduction mapping processing is performed on the BRIEF descriptor in the ORB feature by using a pre-trained model, so that the redundant information in the gray image is eliminated, the effective information is retained, and the local feature of the current key frame is obtained.

[0076] 5) updating the feature map according to the current key frame, the global feature of the current key frame, the global pose of the current key frame, and the local feature of the current key frame.

[0077] Specifically, a map key frame can be determined using the current key frame, the global feature of the current key frame, the global pose of the current key frame, and the local feature of the current key frame, and the map key frame can be inserted into the feature map to complete the update of the feature map.

[0078] The update method of the feature map has the advantages of improving the determination speed of the feature map, reducing the storage space required by the feature map, and retaining the structural information and rich semantic information of the environment. Compared with the conventional method of determining the feature map based on feature points, the update method of the feature map has stronger robustness when the vehicle faces a scene with real-time changes in the environment.

[0079] Further, the updating of the feature map according to the current key frame, the global feature of the current key frame, the global pose of the current key frame, and the local feature of the current key frame comprises: determining an initial map key frame according to the current key frame, the global feature of the current key frame, the global pose of the current key frame, and the local feature of the current key frame; performing loop detection and loop correction on the initial map key frame to obtain a verified map key frame; and adding the verified map key frame into the feature map according to the first time stamp of the map key frame in the feature map to update the feature map.

[0080] Specifically, the current key frame, the global feature of the current key frame, the global pose of the current key frame, and the local feature of the current key frame obtained above can be integrated to obtain an initial map key frame. Then, loop detection is performed on the initial map key frame in the feature map to determine whether the initial map key frame has appeared in the feature map. If the initial map key frame has appeared in the feature map, loop correction is performed on the feature map, the semantic map, and the trajectory map, and the initial map key frame after loop correction is determined as a map key frame. Finally, the verified map key frame is added to a suitable position in the feature map according to the first time stamp of the map key frame and the first time stamps of the existing map key frames in the feature map, such as the chronological order of the time information displayed, to obtain an updated feature map. If the initial map key frame has not appeared in the feature map, loop correction is not performed, and the initial map key frame is directly determined as an initial map key frame. The trajectory map can be generated according to the driving trajectory of the vehicle.

[0081] S202, determine an image key frame of a current image, and determine a global feature difference degree according to a difference between the illumination intensity of a pixel in the global feature of the image key frame and the illumination intensity of a pixel in the global feature of the map key frame.

[0082] For example, the method for determining map keyframes described above can be used to determine the image keyframes of the current image, and then the global feature difference between the image keyframes and each map keyframe can be calculated. The global feature difference f can be determined as follows:

[0083]

[0084] Where W represents the width of the global feature map, and H represents the height of the global feature map. This represents the illumination intensity value of the small grid corresponding to the position (x, y) in the image coordinate system of the global feature map of the current keyframe. This represents the illumination intensity value of the small grid cell at position (x, y) in the image coordinate system corresponding to the global feature map of a certain map keyframe. This calculation can be understood as counting the number of small grid cells in the global features of a certain map keyframe that differ from the global features of the image keyframe. For example, Figure 3 As shown, the global features can be represented as a global feature map.

[0085] S203. Determine the ratio of the global feature difference degree to the number of pixels of the preset global feature to obtain the difference degree vector.

[0086] The preset global features include the global features of the image keyframes or the global features of the map keyframes.

[0087] For example, the difference degree D can be determined as follows:

[0088]

[0089] Where WH represents the number of pixels in the global feature map of the image keyframe or the number of pixels in the global feature map of the map keyframe, i.e., the number of pixels in the preset global feature map. Usually, the number of pixels in these two maps is the same. Using the above method, the difference between the image keyframe and a set number of map keyframes can be calculated, thus obtaining the difference vector: [D1, D2, D3, ..., D...]. t-1 D t ], D i The difference is represented by i = 1, 2, ..., t, where t represents the preset number. The difference is calculated by dividing the number of non-zero differences between the global features of the image keyframes and the global features of the map keyframes by the preset number of pixels in the global feature map. This standardizes the calculation of the difference and eliminates the need for contrast enhancement operations compared to conventional difference calculation methods.

[0090] S204. Determine the updated current difference matrix based on whether the preset parameters of the difference matrix before the update are greater than the preset parameter threshold and the difference vector.

[0091] wherein the preset parameter comprises a row number and / or a column number of the difference degree matrix.

[0092] Specifically, the preset parameter of the difference degree matrix before updating, such as the row number and / or the column number, can be judged first. When the preset parameter is less than a preset parameter threshold, the difference degree vector obtained above can be horizontally or vertically spliced to the tail of the difference degree matrix before updating in the order of time. When the preset parameter is greater than or equal to the preset parameter threshold, the splicing step above can be performed, and the difference degree vector at the head of the difference degree matrix before updating can be discarded correspondingly, so as to obtain the current difference degree matrix after updating. Wherein the difference degree matrix can be composed of a plurality of difference degree vectors, the tail of the difference degree matrix before updating can be understood as the difference degree vector closest to the image key frame in time in the difference degree matrix before updating, and the head of the difference degree matrix before updating is opposite to the tail. The number of difference degree vectors discarded at the head of the difference degree matrix before updating can be consistent with the number of difference degree vectors to be spliced.

[0093] For example, if the preset parameter is the column number, the preset parameter threshold is 20, the left side of the difference degree matrix before updating is the head and the right side is the tail in the order of time, when the column number of the difference degree matrix before updating is less than 20, the difference degree vector obtained above can be horizontally spliced to the right side of the difference degree matrix before updating, that is, each column vector in the difference degree matrix is a difference degree vector, so as to obtain the current difference degree matrix after updating, Figure 4 For example, as shown in the visualization diagram of the difference degree matrix, Figure 4 The darker the color, the smaller the difference degree, Figure 4 Each column in the difference degree matrix represents a difference degree vector.

[0094] S205, using a preset dynamic programming algorithm, traversing the difference degree of the current difference degree matrix, determining a first preset position of the current difference degree matrix, a preset matrix corresponding to the dimension of the current difference degree matrix, and a second preset position in the preset matrix corresponding to the first preset position.

[0095] Wherein the first preset position comprises the position of the minimum difference degree in the first preset direction in the current difference degree matrix, the first preset direction comprises each row direction or each column direction, the preset position comprises a row position and a column position, and the value in the preset matrix is determined based on the sum value of the difference degree.

[0096] Specifically, a preset dynamic programming algorithm, such as a dynamic programming algorithm, can be used to traverse the difference degrees in the current difference degree matrix, and a preset matrix Total with the same dimension as the current difference degree matrix and a preset step matrix Path are established, the position of the minimum difference degree in each row or each column of the current difference degree matrix, i.e., the first preset position, is searched, and the second preset position in the preset matrix Total is determined correspondingly, such as the first preset position (1, 2) in the current difference degree matrix, i.e., the second value in the first row, and the corresponding second preset position (1, 2) in the preset matrix Total. In the process of traversal and search, a plurality of difference degree search paths and a plurality of steps can be obtained correspondingly. Different difference degree search paths pass through different difference degrees in the current difference degree matrix. The plurality of steps can be written into the preset step matrix Path. The difference degrees and values corresponding to different difference degree search paths can be written into the preset matrix Total. If the current difference degree matrix is horizontally spliced by difference degree vectors, the position of the minimum difference degree in each column of the current difference degree matrix is the first preset position. If the current difference degree matrix is vertically spliced by difference degree vectors, the position of the minimum difference degree in each row of the current difference degree matrix is the first preset position. The difference degree and value at each position in the preset matrix Total can be understood as follows: taking a position A in the preset matrix Total, the corresponding position a in the previous difference degree matrix is obtained by using the preset dynamic programming algorithm to traverse the difference degrees in the current difference degree matrix. There are a plurality of difference degree search paths in the current difference degree matrix, and there is a difference degree search path l passing through the position a. The value at the position A in the preset matrix Total is, and the sum of the difference degrees passed through by the difference degree search path l before passing through the position a.

[0097] In S206, the row position corresponding to the last column in the second preset position or the column position corresponding to the last row is determined as a relocation identifier.

[0098] The value at the second preset position corresponding to the relocation identifier is determined based on the sum of the difference degrees at the first preset position.

[0099] For example, Figure 5 For an optimal path determined based on a dynamic programming algorithm, as shown in Figure 5 the figure is a visualization diagram of the difference degree matrix, Figure 4 and the best path determined by using the dynamic programming algorithm, Figure 5The path composed of the plurality of dots in the current difference matrix is the optimal path, and the position corresponding to each dot is the first preset position in the optimal path. The row position corresponding to the last column in the second preset position of the preset matrix Total is the repositioning identifier. The map key frame corresponding to the repositioning identifier is the map key frame having the highest global feature similarity with the image key frame. If the current difference matrix is horizontally spliced by the difference vectors, the tail of the difference matrix before the update corresponds to the last column in the second preset position, and the repositioning identifier is the row position corresponding to the last column. If the current difference matrix is vertically spliced by the difference vectors, the tail of the difference matrix before the update corresponds to the last row in the second preset position, and the repositioning identifier is the column position corresponding to the last row. In the difference searching path described above, the path having the minimum sum of the difference values is the optimal path, and the sum of the difference values on the optimal path is the value in the second preset position corresponding to the repositioning identifier.

[0100] S207, determining the confidence of the repositioning identifier, and determining the sequence diagram matching result according to the confidence.

[0101] Specifically, the confidence of the sum of the difference values corresponding to the repositioning identifier can be calculated. If the confidence meets the set requirement, it is determined that the sequence diagram matching result corresponding to the repositioning identifier is correct, otherwise, it is determined that the sequence diagram matching result corresponding to the repositioning identifier is incorrect.

[0102] Optionally, the determination of the confidence of the repositioning identifier and the determination of the sequence diagram matching result according to the confidence include: inputting the value in the second preset position corresponding to the repositioning identifier into a preset probability distribution model to obtain a first probability, and inputting the mean value of the values in the row or column of the repositioning identifier in the preset matrix into the preset probability distribution model to obtain a second probability; determining the confidence of the repositioning identifier according to the ratio of the first probability to the second probability; and determining that the sequence diagram matching result is correct when the confidence exceeds a preset confidence threshold, wherein the sequence diagram matching result includes correct and incorrect. The advantage of this setting is that the ratio of the difference probability calculated by the preset probability distribution model, i.e., the ratio of the first probability to the second probability, is used as the confidence, which can filter out incorrect sequence diagram matching results and the corresponding repositioning identifiers.

[0103] Specifically, the difference degree corresponding to the repositioning identifier can be input into a preset probability distribution model, such as a Laplace distribution model, to obtain an output first probability, and the mean value of the values on the row or column where the repositioning identifier is located can be input into the preset probability distribution model to obtain an output second probability, and the ratio of the first probability to the second probability is the confidence of the repositioning identifier. When the confidence is greater than or equal to a preset confidence threshold, such as greater than or equal to 1.08, it can be determined that the sequence graph matching result is correct, otherwise it can be determined that the sequence graph matching result is incorrect. Wherein, if the current difference degree matrix is horizontally spliced by difference degree vectors, the mean value of the values on the column where the repositioning identifier is located is input into the preset probability distribution model, and if the current difference degree matrix is vertically spliced by difference degree vectors, the mean value of the values on the row where the repositioning identifier is located is input into the preset probability distribution model.

[0104] S208, based on the sequence graph matching result, the local feature of the initial repositioning frame corresponding to the repositioning identifier is verified, and the target repositioning frame is determined in the feature map according to the verification result.

[0105] Specifically, when the sequence graph matching result is correct, the local feature of the initial repositioning frame corresponding to the repositioning identifier can be verified, and if the verification result obtained after verification indicates that the sequence graph matching result is correct, the map key frame corresponding to the repositioning identifier can be determined as the target repositioning frame.

[0106] Optionally, the sequence graph matching result is correct, the map key frame corresponding to the repositioning identifier is determined as the initial repositioning frame; the local feature of the image key frame and the local feature of the initial repositioning frame are forcibly matched to obtain a matching point pair; the feature points in the matching point pair are determined, wherein the feature points include inliers and outliers; the verification result is determined according to the proportion of the number of inliers to the number of feature points and the number of inliers, and the target repositioning frame is determined from the feature map according to the verification result, wherein the feature map contains the global pose of the target repositioning frame, the global pose of the target repositioning frame is used to provide an initial position for the semantic map, and the verification result includes correct sequence graph matching result and incorrect sequence graph matching result. The advantage of this setting is that the sequence graph matching result is reverified according to the proportion of the number of inliers to the number of feature points and the number of inliers, which improves the accuracy of the target repositioning frame.

[0107] Specifically, when the sequence graph matching result is correct, the map key frame corresponding to the relocalization identifier can be taken as an initial relocalization frame, then the local features of the image key frame and the local features of the initial relocalization frame are subjected to brute force matching, the feature points with the highest similarity in the two graphs are found, that is, the matching point pair is determined, then a preset sampling algorithm such as the random sample consensus algorithm is used to calculate the fundamental matrix of the matching point, and the inliers and outliers are obtained, if the proportion of the number of inliers to the number of feature points belongs to a preset proportion range, such as greater than or equal to 0.6 and less than 0.65, and the number of inliers satisfies a preset number range, such as less than or equal to 8, it is determined that the verification result of the sequence graph matching result is that the sequence graph matching result is correct, otherwise, it is determined that the verification result of the sequence graph matching result is that the sequence graph matching result is incorrect, if the sequence graph matching result is correct, the initial relocalization frame can be determined as a target relocalization frame, and the global pose in the target relocalization frame can be used to provide an initial position for the semantic map. The outliers can be understood as abnormal value points, and the inliers can be understood as non-abnormal value points.

[0108] The method for determining a relocalization frame provided by the embodiment of the application acquires a map key frame of a feature map, determines a current difference degree matrix according to the light intensity of pixels in the global features of the map key frame and the light intensity of pixels in the global features of an image key frame, then iterates the difference degrees of the current difference degree matrix by using a preset dynamic programming algorithm, thereby determining a relocalization identifier, and then determines the map key frame corresponding to the relocalization identifier as an initial relocalization frame, and verifies the initial relocalization frame, and the target relocalization frame can be determined according to the verification result. The updating mode of the feature map in the method improves the determination speed of the feature map, reduces the storage space required by the feature map, and also retains the structural information and rich semantic information of the environment. When the vehicle faces a scene with real-time changes in the environment, the method has stronger robustness compared to the conventional method for determining a feature map based on feature points. At the same time, the preset dynamic programming algorithm can improve the accuracy of the relocalization identifier and the initial relocalization frame, and avoid local optimal solution. The sequence graph matching result corresponding to the initial relocalization frame is verified twice, which further ensures the accuracy of the target relocalization frame.

[0109] Embodiment three

[0110] Figure 6 A structural schematic diagram of a relocalization frame determination device provided by the embodiment three of the application is shown in FIG. 3. Figure 6 As shown in the figure, the device comprises a feature map acquisition module 301, a difference degree matrix determination module 302, a relocalization identifier determination module 303, and a relocalization frame determination module 304, wherein:

[0111] The feature map acquisition module is configured to acquire a feature map, wherein the feature map comprises a plurality of map key frames, and the map key frames contain global features and local features of map images.

[0112] The difference matrix determination module is configured to determine an image key frame of a current image, determine a current difference matrix according to global features of the image key frame and global features of the map key frames, wherein the difference matrix is used to represent a difference degree between the image key frame and the map key frames, and the image key frame contains global features and local features of the current image.

[0113] The relocalization identification determination module is configured to determine a relocalization identification according to a preset dynamic programming algorithm and the current difference matrix, wherein the relocalization identification is determined based on a first preset position in the current difference matrix, and the first preset position comprises a position of minimum difference degree in each column or each row of the current difference matrix.

[0114] The relocalization frame determination module is configured to determine a target relocalization frame in the feature map based on the relocalization identification.

[0115] The relocalization frame determination device provided by the embodiment of the application acquires a feature map and determines an image key frame of a current image, determines a current difference matrix according to global features of the feature map and the image key frame, determines a relocalization identification according to the current difference matrix by using a preset dynamic programming algorithm, and finally determines a target relocalization frame in the feature map according to the relocalization identification. The feature map composed of the map key frames has smaller occupied memory resources, solves the problem of consuming great calculation resources in searching for an initial map frame (i.e., a relocalization frame) in a global range of a semantic map, is suitable for a scene with high scene similarity and poor communication signals, and improves the accuracy and robustness of the relocalization frame.

[0116] Optionally, the updating manner of the feature map comprises: determining a current key frame of the map image according to whether an offset distance of a position of a current frame of the map image relative to a position of a previous map key frame is greater than a preset distance threshold, wherein the position is determined based on a world coordinate system in which the vehicle is located; determining, based on a first timestamp of the current key frame, a neighboring global pose estimation of the map key frame from a preset pose cache library, and performing interpolation processing on the neighboring global pose estimation by using a preset linear interpolation algorithm to obtain a global pose of the current key frame, wherein the key frame comprises a first timestamp representing time information, the neighboring global pose estimation comprises at least two global pose estimations closest in time to the first timestamp, the preset pose cache library comprises at least two global pose estimations, and the global pose estimation comprises a second timestamp representing time information; performing reduction processing on a gray image of the current key frame, and determining a global feature of the current key frame based on an illumination intensity of a pixel in the gray image after the reduction processing; performing reduction processing on the gray image of the current key frame, and performing dimension reduction mapping processing on the gray image after the reduction processing by using a preset model to obtain a local feature of the current key frame; and updating the feature map according to the current key frame, the global feature of the current key frame, the global pose of the current key frame, and the local feature of the current key frame.

[0117] Further, the updating the feature map according to the current key frame, the global feature of the current key frame, the global pose of the current key frame, and the local feature of the current key frame comprises: determining an initial map key frame according to the current key frame, the global feature of the current key frame, the global pose of the current key frame, and the local feature of the current key frame; performing loop detection and loop correction on the initial map key frame to obtain a verified map key frame; and adding the verified map key frame into the feature map according to the first timestamp of the map key frame in the feature map to update the feature map.

[0118] Optionally, the difference degree matrix determination module comprises:

[0119] The global feature difference degree determination unit is configured to determine a global feature difference degree according to a difference between an illumination intensity of a pixel in the global feature of the image key frame and an illumination intensity of a pixel in the global feature of the map key frame.

[0120] The difference degree vector determination unit is configured to obtain a difference degree vector by dividing the global feature difference degree by a preset number of pixels of a preset global feature, wherein the preset global feature comprises the global feature of the image key frame or the global feature of the map key frame.

[0121] The difference matrix determination unit is configured to determine an updated current difference matrix according to whether a preset parameter of the difference matrix before the update is greater than a preset parameter threshold and the difference vector, wherein the preset parameter comprises a row number and / or a column number of the difference matrix.

[0122] Optionally, the repositioning identifier determination module comprises:

[0123] The position determination unit is configured to determine a first preset position of the current difference matrix, a preset matrix corresponding to a dimension of the current difference matrix, and a second preset position in the preset matrix corresponding to the first preset position by traversing the differences of the current difference matrix using a preset dynamic programming algorithm, wherein the first preset position comprises a position of a minimum difference in a first preset direction in the current difference matrix, the first preset direction comprises a row direction or a column direction, the preset position comprises a row position and a column position, and a value in the preset matrix is determined based on a sum of the differences.

[0124] The repositioning identifier determination unit is configured to determine, as a repositioning identifier, a row position corresponding to a last column in the second preset position or a column position corresponding to a last row, wherein a value in the second preset position corresponding to the repositioning identifier is determined based on a sum of the differences in the first preset position.

[0125] Optionally, the repositioning frame determination module comprises: a matching result determination unit configured to determine a confidence of the repositioning identifier and determine a sequence diagram matching result according to the confidence; and a repositioning frame determination unit configured to verify a local feature of an initial repositioning frame corresponding to the repositioning identifier based on the sequence diagram matching result, and determine a target repositioning frame in the feature map according to a verification result.

[0126] Optionally, the determination of the confidence of the repositioning identifier and the determination of the sequence diagram matching result according to the confidence comprise: inputting a value in the second preset position corresponding to the repositioning identifier into a preset probability distribution model to obtain a first probability, and inputting a mean value of values in a row or a column of the preset matrix in which the repositioning identifier is located into the preset probability distribution model to obtain a second probability; determining the confidence of the repositioning identifier according to a ratio of the first probability to the second probability; and determining that the sequence diagram matching result is correct when the confidence exceeds a preset confidence threshold, wherein the sequence diagram matching result comprises correct and incorrect.

[0127] Optionally, the step of verifying the local features of the initial relocation frame corresponding to the relocation identifier based on the sequence graph matching result, and determining the target relocation frame in the feature map according to the verification result, includes: when the sequence graph matching result is correct, determining the map keyframe corresponding to the relocation identifier as the initial relocation frame; performing brute-force matching on the local features of the image keyframe and the local features of the initial relocation frame to obtain matching point pairs; determining the feature points in the matching point pairs, wherein the feature points include inliers and exliers; determining the verification result based on the ratio of the number of inliers to the number of feature points and the number of inliers, and determining the target relocation frame from the feature map according to the verification result, wherein the feature map contains the global pose of the target relocation frame, the global pose of the target relocation frame is used to provide an initial position for the semantic map, and the verification result includes correct sequence graph matching result and incorrect sequence graph matching result.

[0128] The relocation frame determination device provided in this embodiment of the invention can execute the relocation frame determination method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0129] Example 4

[0130] Figure 7 A schematic diagram of the structure of a vehicle 40 that can be used to implement an embodiment of the present invention is shown. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0131] like Figure 7 As shown, vehicle 40 includes at least one processor 41 and a memory, such as read-only memory (ROM) 42 or random access memory (RAM), communicatively connected to at least one processor 41.

[0132] 43, etc., wherein the memory stores computer programs that can be executed by at least one processor. The processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 into the random access memory (RAM) 43. The RAM 43 can also store various programs and data required for the operation of the vehicle 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. The input / output (I / O) interface 45 is also connected to the bus 44.

[0133] A plurality of components in the vehicle 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a magnetic disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the vehicle 40 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0134] The processor 41 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 performs various methods and processes described above, such as the determination method of the repositioning frame.

[0135] In some embodiments, the determination method of the repositioning frame can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 48. In some embodiments, part or all of the computer program can be loaded and / or installed onto the vehicle 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded onto the RAM 43 and executed by the processor 41, one or more steps of the determination method of the repositioning frame described above can be performed. Alternatively, in other embodiments, the processor 41 can be configured to perform the determination method of the repositioning frame by any other appropriate means, such as by means of firmware.

[0136] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0137] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0138] The computer equipment provided above can be used to execute the relocation frame determination method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0139] Example 5

[0140] In the context of this invention, the computer-readable storage medium may be a tangible medium, and the computer-executable instructions, when executed by a computer processor, are used to perform a method for determining a relocation frame, the method comprising:

[0141] Obtain a feature map, wherein the feature map includes multiple map keyframes, and the map keyframes contain global and local features of the map image;

[0142] Determine the keyframes of the current image, and determine the current difference matrix based on the global features of the keyframes and the map keyframes. The difference matrix is ​​used to characterize the degree of difference between the keyframes and the map keyframes. The keyframes contain the global and local features of the current image.

[0143] Based on the preset dynamic programming algorithm and the current difference matrix, a relocation identifier is determined, wherein the relocation identifier is determined based on a first preset position in the current difference matrix, and the first preset position includes the position of the minimum difference in the difference of each column or each row in the current difference matrix;

[0144] Based on the relocation identifier, the target relocation frame is determined in the feature map.

[0145] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by, or in conjunction with, an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0146] The computer equipment provided above can be used to execute the relocation frame determination method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0147] It is worth noting that in the embodiments of the above-mentioned relocation frame determination device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0148] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for determining a relocation frame, characterized in that, include: Obtain a feature map, wherein the feature map includes multiple map keyframes, and the map keyframes contain global and local features of the map image; Determine the keyframes of the current image, and determine the current difference matrix based on the global features of the keyframes and the map keyframes. The difference matrix is ​​used to characterize the degree of difference between the keyframes and the map keyframes. The keyframes contain the global and local features of the current image. Based on the preset dynamic programming algorithm and the current difference matrix, a relocation identifier is determined, wherein the relocation identifier is determined based on a first preset position in the current difference matrix, and the first preset position includes the position of the minimum difference in the difference of each column or each row in the current difference matrix; Based on the relocation identifier, the target relocation frame is determined in the feature map.

2. The method according to claim 1, characterized in that, The feature map is updated in the following ways: The current keyframe of the map image is determined based on whether the offset distance of the current frame's position relative to the previous keyframe's position is greater than a preset distance threshold, wherein the position is determined based on the world coordinate system in which the vehicle is located. Based on the first timestamp of the current keyframe, the adjacent global pose estimates of the map keyframe are determined from the preset pose cache library, and the adjacent global pose estimates are interpolated using a preset linear interpolation algorithm to obtain the global pose of the current keyframe. The keyframe contains a first timestamp representing time information, the adjacent global pose estimates include at least two global pose estimates that are closest in time to the first timestamp, the preset pose cache library contains at least two of the global pose estimates, and the global pose estimates contain a second timestamp representing time information. The grayscale image of the current keyframe is reduced in size, and the global features of the current keyframe are determined based on the illumination intensity of the pixels in the reduced grayscale image. The grayscale image of the current keyframe is reduced in size, and a preset model is used to perform dimensionality reduction mapping on the reduced grayscale image to obtain the local features of the current keyframe. The feature map is updated based on the current keyframe, the global features of the current keyframe, the global pose of the current keyframe, and the local features of the current keyframe.

3. The method according to claim 2, characterized in that, The step of updating the feature map based on the current keyframe, the global features of the current keyframe, the global pose of the current keyframe, and the local features of the current keyframe includes: The initial map keyframe is determined based on the current keyframe, the global features of the current keyframe, the global pose of the current keyframe, and the local features of the current keyframe. Perform loop closure detection and loop closure correction on the initial map keyframes to obtain the verified map keyframes; Based on the first timestamp of the map keyframe in the feature map, the verified map keyframe is added to the feature map to update the feature map.

4. The method according to claim 2, characterized in that, The step of determining the current dissimilarity matrix based on the global features of the image keyframes and the global features of the map keyframes includes: The global feature difference is determined based on the difference between the illumination intensity of pixels in the global features of the image keyframe and the illumination intensity of pixels in the global features of the map keyframe. Determine the ratio of the global feature difference to the number of pixels of the preset global features to obtain a difference vector, wherein the preset global features include the global features of the image keyframes or the global features of the map keyframes; Based on whether the preset parameters of the difference matrix before the update are greater than the preset parameter threshold, and the difference vector, the updated current difference matrix is ​​determined, wherein the preset parameters include the number of rows and / or columns of the difference matrix.

5. The method according to claim 4, characterized in that, The step of determining the relocation identifier based on the preset dynamic programming algorithm and the current difference matrix includes: Using a preset dynamic programming algorithm, the differences in the current difference matrix are traversed to determine a first preset position of the current difference matrix, a preset matrix corresponding to the dimension of the current difference matrix, and a second preset position in the preset matrix corresponding to the first preset position. The first preset position includes the position of the minimum difference in the current difference matrix in a first preset direction. The first preset direction includes each row direction or each column direction. The preset position includes row position and column position. The values ​​in the preset matrix are determined based on the sum of the differences. The row position corresponding to the last column in the second preset position, or the column position corresponding to the last row, is determined as the relocation identifier, wherein the value at the second preset position corresponding to the relocation identifier is determined based on the sum of the differences at the first preset position.

6. The method according to claim 5, characterized in that, The step of determining the target relocation frame in the feature map based on the relocation identifier includes: Determine the confidence level of the relocation identifier, and determine the sequence graph matching result based on the confidence level; Based on the sequence map matching results, the local features of the initial relocation frame corresponding to the relocation identifier are verified, and the target relocation frame is determined in the feature map according to the verification results.

7. The method according to claim 6, characterized in that, Determining the confidence level of the relocation identifier and determining the sequence graph matching result based on the confidence level includes: The value at the second preset position corresponding to the relocation identifier is input into a preset probability distribution model to obtain a first probability, and the mean value of the values ​​in the row or column where the relocation identifier is located in the preset matrix is ​​input into the preset probability distribution model to obtain a second probability; The confidence level of the relocation identifier is determined based on the ratio of the first probability to the second probability; When the confidence level exceeds a preset confidence threshold, the sequence graph matching result is determined to be correct, wherein the sequence graph matching result includes both correct and incorrect results.

8. The method according to claim 6, characterized in that, The step of verifying the local features of the initial relocation frame corresponding to the relocation identifier based on the sequence map matching result, and determining the target relocation frame in the feature map according to the verification result, includes: When the sequence map matching result is correct, the map keyframe corresponding to the relocation identifier is determined as the initial relocation frame; A brute-force match is performed between the local features of the image keyframe and the local features of the initial relocation frame to obtain matching point pairs; Determine the feature points in the matching point pair, wherein the feature points include interior points and exterior points; The verification result is determined based on the ratio of the number of inliers to the number of feature points and the number of inliers. Based on the verification result, the target relocation frame is determined from the feature map. The feature map contains the global pose of the target relocation frame, which is used to provide an initial position for the semantic map. The verification result includes correct sequence map matching and incorrect sequence map matching.

9. A device for determining a relocation frame, characterized in that, include: A feature map acquisition module is used to acquire a feature map, wherein the feature map includes multiple map keyframes, and the map keyframes contain global features and local features of the map image; The difference matrix determination module is used to determine the image keyframes of the current image and to determine the current difference matrix based on the global features of the image keyframes and the global features of the map keyframes. The difference matrix is ​​used to characterize the degree of difference between the image keyframes and the map keyframes. The image keyframes contain the global features and local features of the current image. The relocation identifier determination module is used to determine a relocation identifier based on a preset dynamic programming algorithm and the current difference matrix, wherein the relocation identifier is determined based on a first preset position in the current difference matrix, and the first preset position includes the position of the minimum difference in the difference of each column or each row in the current difference matrix; The relocation frame determination module is used to determine the target relocation frame in the feature map based on the relocation identifier.

10. A vehicle, characterized in that, The vehicles include: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, which is executed by the at least one processor to enable the at least one processor to perform the method for determining the relocation frame according to any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method for determining a relocation frame as described in any one of claims 1-8.

Images