Map construction method and device, equipment, medium, product and vehicle

Abstract

The invention relates to a map construction method and device, electronic equipment, a computer readable storage medium, a computer program product and a vehicle, at least two frames of images in a driving video of intelligent equipment are processed based on a target model, a position identifier in an environment where the intelligent equipment is located is determined, and the target model is acquired based on a deep learning algorithm; and acquiring map information of driving of the intelligent equipment according to the position identifier. According to the invention, the driving map information of the intelligent equipment can be accurately estimated.

Description

Technical Field

[0001] The present application relates to the technical field of device control, and in particular, to a method, device, equipment, medium, product and vehicle for map construction. Background Art

[0002] The purpose of the Simultaneous Localization and Mapping (SLAM) technology is to solve the two problems of localization and map construction. It is an important implementation approach for assisting intelligent driving today. It is of great significance for both intelligent driving and parking to sense the external environment through the device's own sensors to construct a map and locate the position of the device itself.

[0003] However, traditional SLAM front-end methods are easily affected by the scene, mainly due to the dependence on environmental features and the lack of adaptability to complex conditions. For example, in scenarios such as low light, jitter or weak texture, the map cannot be accurately constructed. Summary of the Invention

[0004] Embodiments of the present application provide a method for map construction, an intelligent device driving assistance device, an electronic device, a storage medium, a program product and a vehicle, aiming to accurately estimate the map information of the intelligent device based on a deep learning algorithm to at least partially solve the above technical problems.

[0005] To achieve the above object, according to the first aspect of the present application, there is provided a method for map construction, including:

[0006] Processing at least two frames of images in the driving video of the intelligent device based on a target model to determine a position identifier in the environment where the intelligent device is located, where the target model is obtained based on a deep learning algorithm;

[0007] Obtaining map information of the intelligent device's travel according to the position identifier.

[0008] Optionally, the target model includes a feature extraction sub-model and a feature matching sub-model. The processing at least two frames of images in the driving video of the intelligent device based on the target model to determine the position identifier in the environment where the intelligent device is located includes:

[0009] Extracting feature points of at least two frames of images in the driving video of the intelligent device through the feature extraction sub-model;

[0010] Determining the position identifier according to the feature points of each image through the feature matching sub-model.

[0011] Optionally, the feature extraction sub-model is trained through the following steps:

[0012] Train the feature extraction sub-model according to the training sample data to obtain the trained feature extraction sub-model.

[0013] Optionally, the step of obtaining the training sample data includes:

[0014] Train the synthetic geometric data set according to the deep learning model to obtain a basic detector;

[0015] Perform a single linear transformation on the driving training image set and the basic detector to obtain training sample data.

[0016] Optionally, the step of obtaining the driving training image set includes:

[0017] Obtain driving video data and perform format conversion on the driving video data;

[0018] Extract images from the converted driving video data to obtain a driving training image set, where the driving training image set includes multiple groups of left-eye images and corresponding right-eye images.

[0019] Optionally, after determining the position identifier according to the feature points of each image through the feature matching sub-model, it further includes:

[0020] Obtain the distance information between the feature points according to the position information of the feature points of each image;

[0021] Filter the feature points according to the distance information and a preset distance threshold to obtain optimized feature points, and determine an optimized position identifier according to the optimized feature points.

[0022] Optionally, determining the position identifier by the feature matching sub-model according to the feature points of each image includes:

[0023] Determine the position identifier according to the feature matching sub-model processing the feature points of each image.

[0024] Optionally, determining the position identifier by processing the feature points of each image according to the feature matching sub-model in the deep learning algorithm includes:

[0025] Match the feature points of each of the images through the feature matching sub-model to determine at least two frames of associated target images;

[0026] Process the target images through the feature matching sub-model to determine the position identifier.

[0027] Optionally, matching the feature points of each of the images through the feature matching sub-model to determine at least two frames of associated target images includes:

[0028] Process the feature points of the first image and the feature points of the second image through the feature matching sub-model to obtain the matching relationship between the first image and the second image; wherein, the first image is any image in the video, and the second image is an associated image of the first image;

[0029] Determine at least two frames of associated target images through the feature matching sub-model according to the matching relationship.

[0030] Optionally, the method further includes:

[0031] When the first image is an initial frame image, extract an image with a different acquisition perspective from the first image in the initial frame image as the second image.

[0032] Optionally, the method further includes:

[0033] When the first image is not a non-initial frame image, extract an image with a different acquisition perspective from the first image in the initial frame image as the second image; and / or,

[0034] When the first image is not a non-initial frame image, use the previous frame adjacent to the first image as the second image.

[0035] Optionally, the process of determining the position identifier by processing the target image through the feature matching sub-model includes:

[0036] Input the feature points of the at least two frames of associated target images within the sliding window into the feature matching sub-model;

[0037] Match the feature points within the sliding window through the feature matching sub-model to obtain the position identifier.

[0038] Optionally, after determining the position identifier in the environment where the intelligent device is located by processing at least two frames of images in the driving video of the intelligent device based on the target model, it includes:

[0039] Obtain the pose information corresponding to the intelligent device according to the spatial position information of the position identifier and the parameter information of the camera that captures the image.

[0040] Optionally, before obtaining the pose information corresponding to the intelligent device according to the spatial position information of the position identifier and the parameter information of the camera that captures the image, it further includes:

[0041] Obtain the operating state information of the intelligent device and perform pre-integration on the operating state information;

[0042] Initialize the pose information corresponding to the intelligent device according to the position identifier and the pre-integrated operating state information.

[0043] Optionally, the obtaining the map information of the intelligent device traveling according to the position identifier further includes:

[0044] Obtain the map information of the intelligent device traveling according to the pose information of the intelligent device and the position identifier.

[0045] Optionally, before obtaining the map information of the intelligent device traveling according to the pose information of the intelligent device and the position identifier, it further includes:

[0046] Optimize the feature points in the at least two frames of images;

[0047] Determine a new position identifier and new pose information according to the optimized feature points.

[0048] Optionally, the optimization process includes non-linear optimization based on an objective function and / or removal of invalid feature points.

[0049] Optionally, the optimizing the feature points in the at least two frames of images includes:

[0050] Optimize the feature points in the at least two frames of images within a sliding window.

[0051] Optionally, after obtaining the map information of the intelligent device traveling according to the position identifier, it further includes:

[0052] Obtain target map information according to the traveling position of the intelligent device and the map information;

[0053] Optimize the target map information according to the position identifier in the target map information.

[0054] Optionally, the optimizing the target map information according to the position identifier in the target map information includes:

[0055] Obtain the current position identifier corresponding to the current frame of image collected by the intelligent device;

[0056] Optimize the target map information according to the position identifier in the target map information and the current position identifier.

[0057] Optionally, optimizing the target map information according to the position identifier in the target map information and the current position identifier includes:

[0058] Determine whether the intelligent device has passed through the same location according to the location identifier in the target map information and the current location identifier;

[0059] If the intelligent device has passed through the same location, optimize the target map information according to the current location identifier.

[0060] Optionally, the method further includes:

[0061] If the intelligent device has passed through the same location, transmit the current location identifier back to optimize the map information according to the transmitted location identifier and the location identifier corresponding to the next frame of image.

[0062] According to a second aspect of the present application, there is provided an intelligent device driving assistance device, including:

[0063] A determination module, configured to process at least two frames of images in the intelligent device driving video based on a target model to determine a location identifier in the environment where the intelligent device is located, and the target model is obtained based on a deep learning algorithm;

[0064] An acquisition module, configured to acquire map information of the intelligent device driving according to the location identifier.

[0065] In a third aspect, the present embodiment further provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to execute the steps of the above method.

[0066] In a fourth aspect, the present embodiment further provides a computer-readable storage medium, which includes a computer program, and when the computer program runs on an electronic device, the computer program is used to cause the electronic device to execute the steps of the above method.

[0067] In a fifth aspect, the present embodiment further provides a computer program product, including a computer program, and the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device executes the steps of the above method.

[0068] In a sixth aspect, the present embodiment further provides a vehicle, and at least one of the above device, electronic device, computer-readable storage medium, and computer program product is provided on the vehicle.

[0069] In summary, through the above technical solutions in the embodiments of the present application, the intelligent device can process at least two frames of images based on the target model, determine the location identifier in the environment where the intelligent device is located, and determine the map information for the intelligent device to travel according to the location identifier. Therefore, the present application can process multiple frames of images using the target model obtained based on the deep learning algorithm, determine the map information of the intelligent device, and is generally applicable to map construction in various types of driving scenarios, avoiding interference from complex scenarios during the map construction process, ensuring the accuracy of the constructed map, and effectively assisting the intelligent device to travel.

[0070] Other features and advantages of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative efforts.

[0072] To more fully understand the present application and its beneficial effects, the following will be described in conjunction with the accompanying drawings, where the same reference numerals represent the same parts in the following description.

[0073] Figure 1 is the first schematic diagram of the intelligent device control process provided in the exemplary embodiment of the present application;

[0074] Figure 2 is the first schematic diagram of feature processing provided in the exemplary embodiment of the present application;

[0075] Figure 3 is the second schematic diagram of feature processing provided in the exemplary embodiment of the present application;

[0076] Figure 4 is the third schematic diagram of feature processing provided in the exemplary embodiment of the present application;

[0077] Figure 5 is the fourth schematic diagram of feature processing provided in the exemplary embodiment of the present application;

[0078] Figure 6 is the schematic diagram of the intelligent device driving assistance device provided in the exemplary embodiment of the present application;

[0079] Figure 7 is the schematic diagram of the architecture of the electronic device provided in the exemplary embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0080] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the protection scope of the present application.

[0081] Combined with the above background technology description, the purpose of the Simultaneous Localization and Mapping (SLAM) technology is to solve the two problems of localization and mapping. It is an important implementation approach for assisting intelligent driving today. It is of great significance for intelligent driving in both driving and parking to construct a map and locate the position of the intelligent device itself through the sensors of the intelligent device to perceive the external environment.

[0082] Traditional SLAM front-end methods mainly perform feature tracking through feature point methods, optical flow methods, and direct methods. Although such technical solutions can achieve the tracking of environmental features, due to the diversity of scenes and the limitation that monocular cameras cannot provide distance information, traditional SLAM front-end methods perform poorly in certain specific scenes, such as low-light, jitter, or weak texture scenes.

[0083] If the lidar solution is adopted, although accurate distance information can be obtained, which helps improve the accuracy of feature tracking in traditional SLAM front-end methods, the high cost limits the types of vehicles that can be equipped with this method.

[0084] To solve the above problems and optimize the existing SLAM front-end methods, the present application proposes a control method, device, electronic device, computer-readable storage medium, computer program product, and intelligent device for an intelligent device. By constructing a SLAM front-end technical solution based on deep learning, it is used for the map construction and positioning processes during driving and parking of intelligent driving.

[0085] Specifically, the map construction method in the present application can be applied to an intelligent device, which can be any device such as an intelligent vehicle or an intelligent robot that can utilize this method.

[0086] Among them, the intelligent vehicle in the present application can be a fuel vehicle, a plug-in hybrid vehicle, or a new energy vehicle, etc. The present application does not make specific limitations on this.

[0087] In an embodiment, the map construction method in the present application, as Figure 1 shown, may include the following steps:

[0088] S10. Process at least two frames of images in the driving video of the intelligent device based on the target model to determine the location identifier in the environment where the intelligent device is located. The target model is obtained based on a deep learning algorithm.

[0089] It should be noted that in this embodiment, a target model can be constructed based on a deep learning algorithm. This model can include a feature extraction sub-model and a feature matching sub-model.

[0090] Furthermore, the intelligent device can process at least two frames of images in the driving video of the intelligent device based on the target model to determine the location identifier in the environment where the intelligent device is located.

[0091] Specifically, for example, the intelligent device can collect at least two frames of images of its driving scene during driving, and then can determine the location identifier based on the at least two frames of images.

[0092] The location identifier in this embodiment can be a road sign. A road sign can be a point or object with significant features in the environment that can be detected and recognized by sensors. Road signs can help the intelligent device for positioning and map construction. This embodiment does not make specific limitations on the type of road signs.

[0093] S20. Obtain the map information of the intelligent device's driving according to the location identifier.

[0094] In this embodiment, after the intelligent device extracts the location identifier in its environment, it can obtain the map information of the intelligent device's driving according to this location identifier.

[0095] For example, the intelligent device can determine the map information of the intelligent device's driving according to the spatial position information of the location identifier and the pose information of the intelligent device.

[0096] Among them, in this embodiment, the spatial position information of the location identifier can include the depth information of the location identifier, and the pose information of the intelligent device can specifically be the pose of the vehicle-mounted camera installed on the intelligent device. This pose information can be calculated based on the spatial position information of the location identifier. It will be described in subsequent embodiments and will not be elaborated here.

[0097] In this way, in this embodiment, the map information of the intelligent device's driving can be determined by combining the spatial position information of the feature points and the pose information of the intelligent device.

[0098] It can be seen that in this embodiment, the intelligent device can process at least two frames of images based on the target model, determine the position identifier in the environment where the intelligent device is located, and determine the map information of the intelligent device's travel according to the spatial position information of the position identifier and the pose information of the intelligent device. Therefore, in this application, the target model obtained based on the deep learning algorithm can be used to process multiple frames of images to determine the map information of the intelligent device, which is generally applicable to map construction of intelligent devices in various types of operation scenarios, ensuring the accuracy of the constructed map.

[0099] In one embodiment, in the above S10, "processing at least two frames of images in the travel video of the intelligent device based on the target model to determine the position identifier in the environment where the intelligent device is located" may include:

[0100] S101, extracting feature points of each image in the travel video of the intelligent device through the feature extraction sub-model;

[0101] S102, processing the feature points of each image through the feature matching sub-model to determine the position identifier.

[0102] It should be noted that in this embodiment, the deep learning algorithm is used to perform feature extraction and matching on the images obtained during the travel to extract the position identifier in the environment where the intelligent device is located.

[0103] Among them, the feature points in this embodiment consist of two parts: key points and descriptors. The key points can represent the position of the feature points in the image, and the descriptor can be a vector used to describe the information of the pixels around the key points.

[0104] Moreover, the feature extraction sub-model and the feature matching sub-model in this embodiment can specifically be models based on the deep learning algorithm, such as CNN (Convolutional Neural Network). This embodiment does not specifically limit the types of the feature extraction sub-model and the feature matching sub-model.

[0105] On this basis, the deep learning feature extraction algorithm based on the feature extraction sub-model can be Local Binary Patterns (LBP, Local Binary Pattern), Scale-Invariant Feauture Transform (SIFT, Scale-Invariant Feature Transform), SuperPoint, etc., and the deep learning feature matching algorithms based on the feature matching sub-model include SuperGlue, LightGlue, etc.

[0106] In one embodiment, in the above S102, "determining the position identifier through the feature matching sub-model according to the feature points of each image" may include:

[0107] S1021. Process the feature points of each image according to the feature matching sub-model to determine the position identifier.

[0108] In this embodiment, the intelligent device can process the feature points of each image according to the feature matching sub-model to determine the position identifier in the environment where the intelligent device is located.

[0109] In a specific embodiment, the above "process the feature points of each image according to the feature matching sub-model to determine the position identifier" may include:

[0110] S1021. Match the feature points of each of the images through the feature matching sub-model to determine at least two frames of associated target images;

[0111] S1022. Process the target images through the feature matching sub-model to determine the position identifier.

[0112] In this embodiment, the intelligent device can match the feature points of each image through the feature matching sub-model to determine at least two frames of associated target images.

[0113] Furthermore, the intelligent device can process the target images through the feature matching sub-model to determine the position identifier. Among them, the position identifiers in the associated target images can be the same spatial point in the environment where the intelligent device is located.

[0114] In a specific embodiment, in the above S1021, "match the feature points of each of the images through the feature matching sub-model to determine at least two frames of associated target images" may include:

[0115] Process the feature points of the first image and the feature points of the second image through the feature matching sub-model to obtain the matching relationship between the first image and the second image; wherein, the first image is any image in the video, and the second image is an associated image of the first image;

[0116] Determine at least two frames of associated target images through the feature matching sub-model according to the matching relationship.

[0117] It should be noted that in this embodiment, the matching relationship between the first image and the second image can represent the matching relationship between the feature points in the first image and the second image, that is, whether the feature points in the first image and the second image are projected from the same position identifier P.

[0118] Furthermore, the intelligent device can determine at least two frames of associated target images through the feature matching sub-model according to the above matching relationship, so that the same position identifier P is included in the determined multiple target images.

[0119] In one embodiment, the map construction method in the present application may further include:

[0120] In a case where the first image is an initial frame image, an image having a different acquisition angle from the first image in the initial frame image is extracted as a second image.

[0121] In this embodiment, after the smart device captures an image, it can pre-determine whether the image is an image in the initial frame.

[0122] If it is determined that the first image is in the initial frame image, an image in the initial frame image with a different acquisition angle from the first image is extracted as the second image. For example, if the first image is the left eye image in the initial frame image, then the second image is the right eye image in the initial frame image.

[0123] In one embodiment, the map construction method in the present application may further include:

[0124] When the first image is not in a non-initial frame image, extracting an image in the initial frame image that has a different acquisition angle from the first image as the second image; and / or,

[0125] When the first image is not a non-initial frame image, a previous frame adjacent to the first image is used as the second image.

[0126] In this embodiment, if it is determined that the first image is not in the initial frame image, an image in the initial frame image that has a different acquisition angle from the first image may be extracted as the second image.

[0127] In addition, the smart device can also use the previous frame adjacent to the first image as the second image. For example, if the first image is a left-eye image, the previous frame adjacent to the left-eye image can be used as the second image; or, if the first image is a right-eye image, the previous frame adjacent to the right-eye image can be used as the second image.

[0128] In one embodiment, if Figure 2For the feature processing flow shown below, the intelligent device can obtain the video recorded by the binocular camera it is equipped with, convert the video into Rosbag data according to the calibration result of the binocular camera, and input the Rosbag data into the feature extraction module. First, it determines whether the current frame image is an initial frame. If it is an initial frame, it performs feature extraction on the left-eye image using the feature extraction sub-model of deep learning. Then, it determines whether there is a right-eye image. If so, it performs feature extraction on the right-eye image to obtain feature points, and then uses the left-eye and right-eye to perform feature matching through the deep learning matching algorithm to obtain the matching relationship and confidence level. If it is not an initial frame, it performs feature extraction on the current frame left-eye image to obtain feature points, and then uses the feature points obtained from the left-eye image of the previous frame and the left-eye image of the current frame respectively to perform feature matching through the deep learning matching algorithm to obtain the matching relationship. Then, it determines whether there is a right-eye image in the current frame. If it exists, it performs feature extraction and feature matching in the same way as above. Finally, it stores the matching result in the feature container and outputs the feature matching image, and visualizes the matching result.

[0129] In a specific implementation, the feature processing flow in the present application is as Figure 3 shown. If it is an initial frame, first perform deep learning feature extraction on the left-eye image in the initial frame to obtain key points and descriptors. If there is an initial frame right-eye image, perform feature point extraction on the right-eye image as well. Then, input the key points and descriptors obtained from the left-eye and right-eye of the initial frame into the deep learning matching algorithm for feature matching, output the matching relationship and confidence level, and input the result into the feature container. If it is not an initial frame, first perform feature point extraction on the current frame left-eye image, use the key points and descriptors to perform feature matching with the feature information of the previous frame left-eye, and then determine whether there is a current frame right-eye image. If it exists, perform feature point extraction on the current frame right-eye once, and then perform feature matching on the key points and descriptors of the current frame left-eye and right-eye to obtain the matching relationship and confidence level and input them into the feature container. Finally, output the feature matching image. It should be noted that after each feature point extraction, the data type of the key points needs to be converted into a vector and then undistorted.

[0130] Through the above method, the embodiments of the present application obtain sufficient features to use these features for the pose initialization of the camera. The subsequent embodiments will be described in detail and will not be elaborated here.

[0131] In one embodiment, in the above S1022, "processing the target image through the feature matching sub-model to determine the position identifier" may include:

[0132] Inputting the feature points of the at least two frames of associated target images in the sliding window into the feature matching sub-model;

[0133] Through the feature matching sub-model, the feature points within the sliding window are matched to obtain the position identifier.

[0134] In this embodiment, in order to obtain the best matching relationship between images, a sliding window mechanism is adopted. Suppose there are two target images named image1 and image2 respectively. Among them, image1 has N1 feature points, image2 has N2 feature points, and the descriptor dimension corresponding to each feature point is 256. First, the descriptors of image1 are multiplied by the transposed descriptors of image2 to obtain a matrix M. Then, some constraint conditions are set - for example, first filter out the descriptors whose confidence levels belong to a certain threshold range, limit the rows and columns within a certain fixed range when traversing the matrix M, etc., allowing each match to be performed only within a small window and only calculating the elements on the diagonal. The window slides step by step, and the final matching result is obtained through sequential calculation to obtain the position identifier.

[0135] In one embodiment, the feature extraction sub-model in this embodiment can be trained through the following steps:

[0136] S30. According to the training sample data, the feature extraction sub-model is trained to obtain the trained feature extraction sub-model.

[0137] In this embodiment, the feature extraction sub-model can be pre-trained using the training sample data to obtain the trained feature extraction sub-model.

[0138] In a specific embodiment, the steps for obtaining the training sample data include:

[0139] Train the synthetic geometric data set according to the deep learning model to obtain a basic detector;

[0140] Perform a single linear transformation on the driving training image set and the basic detector to obtain the training sample data.

[0141] As Figure 4 shown, the synthetic geometric data set in this embodiment can be randomly generated, including geometric bodies of various shapes such as cuboids, polygons, stars, and lines, for training the feature extraction sub-model.

[0142] The driving training image set FeauMatch can be the unlabeled pictures obtained by recording with the binocular camera mounted on the intelligent device during driving on the road and then through the ROS-based data extraction method.

[0143] The ROS-based data extraction steps may include: obtaining the video recorded by the binocular camera mounted on the intelligent device, converting the video into Rosbag data according to the calibration result of the binocular camera, then accessing the Rosbag to obtain topics, and obtaining each frame of image message according to the topics to extract the images and save them as a data set (i.e., the driving training image set FeauMatch in this embodiment). The above topics may include the IMU topic, the left-eye image topic, and the right-eye image topic. Specifically, each topic contains a timestamp and corresponding IMU data or image data, etc. This data extraction method can be widely applied to various formats of data and multi-degree-of-freedom data output.

[0144] In this way, the intelligent device can train the synthetic geometric data set according to the deep learning model to obtain a basic detector.

[0145] Specifically, in this embodiment, a deep learning model suitable for geometric data is selected, such as PointNet (for point cloud data), CNN (for image data), or Transformer (for multi-modal data), etc. The process of training the synthetic geometric data set to obtain a basic detector in this embodiment can refer to the construction process of the existing basic detector, which will not be elaborated here.

[0146] Furthermore, the intelligent device can perform a single linear transformation on the above basic detector and the driving training image set to obtain training sample data.

[0147] It should be noted that the basic detector in this embodiment can be a network model of the SuperPoint architecture. The two are combined to perform a random homography transformation, specifically including: first, randomly performing image processing operations such as scaling, translation, rotation, and partial cropping on the images in the driving training image set, then applying the basic detector to the images to generate a set of pseudo-ground truth feature point positions, and finally merging each image with pseudo-ground truth feature points to obtain the result after the homography transformation (i.e., the training sample data in this embodiment).

[0148] Furthermore, the intelligent device can input the above training sample data into the feature extraction sub-model, train the feature extraction sub-model, and obtain the trained feature extraction sub-model.

[0149] In one embodiment, the steps for obtaining the above driving training image set include:

[0150] Obtaining driving video data and performing format conversion on the driving video data;

[0151] Performing image extraction on the converted driving video data to obtain a driving training image set, where the driving training image set contains multiple groups of left-eye images and corresponding right-eye images.

[0152] In this embodiment, in combination with the above embodiments, this embodiment can provide a data extraction method based on ROS (Robot Operating System), including:

[0153] Obtain the video recorded by the binocular camera mounted on the intelligent device, convert the video into Rosbag data (i.e., the converted driving video data in this embodiment) according to the calibration result of the binocular camera, then access the Rosbag to obtain topics, and extract and save the pictures as a driving training image set according to the picture messages of each frame.

[0154] In one embodiment, after S102, "determine the position identifier according to the feature points of each image through the feature matching submodel", the following may also be included:

[0155] S103, obtain the distance information between the feature points according to the position information of the feature points of each image;

[0156] S104, screen the feature points according to the distance information and a preset distance threshold to obtain optimized feature points, and determine an optimized position identifier according to the optimized feature points.

[0157] In combination with the above embodiments, after extracting the features of two target images to obtain feature points including key points and descriptors, and performing feature matching on the two images through a deep learning matching algorithm to obtain the matching feature points between the two images, the Euclidean distance d can be calculated for the position coordinates of the feature points in the two images (such as the left-eye image and the right-eye image, the left-eye image and the left-eye image, or the right-eye image and the right-eye image). If the Euclidean distance d is less than the preset distance threshold, it is considered that the two feature points are actually projections of the same position identifier (i.e., the two feature points correspond to the same position identifier in the same environment), so as to screen out a better matching effect.

[0158] In a specific embodiment, in this embodiment, the internal parameter distortion coefficient and external parameters of the binocular camera are calibrated, and the obtained results can be used in image processing processes such as distortion correction. In this way, the collected images can be processed for distortion correction, and then the deep learning feature extraction submodel SuperPoint can be used to extract features. The feature information can include key points, descriptors, confidence levels, etc. Then, the trained deep learning feature matching LightGlue model is used for feature matching to obtain the matching relationship and confidence level, determine the position identifier in the image, and calculate the Euclidean distance d between the pixel coordinates of the position identifier to obtain the similarity between two feature vectors. The distance d calculation formula is as follows:

[0159]

[0160] Among them, p1 and p2 are the same feature points on two pictures respectively (that is, p1 and p2 are projected by the same position identifier), and u and v are the coordinates of the two points on the image respectively. These parameters can be obtained from the IMU and the wheel speedometer and calculated.

[0161] In one embodiment, after step S10, "processing at least two frames of images in the driving video of the intelligent device based on the target model to determine the position identifier in the environment where the intelligent device is located", the following steps may further be included:

[0162] S40, obtaining the pose information corresponding to the intelligent device according to the spatial position information of the position identifier and the parameter information of the camera that captures the image.

[0163] In this embodiment, the depth of the spatial point in the map can be estimated by the method of triangulation. Assume that the optical centers of the cameras are O1 and O2. Theoretically, the straight line O1p1 and O2p2 will intersect at a point P in the scene, and this point is the position of the spatial point corresponding to the two feature points in the three-dimensional scene. According to the definition in the epipolar geometry, let x1 and x2 be the normalized coordinates of the two feature points, then they satisfy:

[0164] s2x2 = s1Rx1 + t

[0165] Among them, s1 and s2 are the depth information in the spatial position information of the feature points, and R and t are the rotation matrix and translation vector corresponding to the intelligent device.

[0166] In this way, in this embodiment, the operation of the in-vehicle device can be estimated by PnP (Perspective-n-Point), and the pose information of the in-vehicle camera can be calculated:

[0167]

[0168] Among them, T * is the Lie group representation of the camera pose, K is the internal parameter matrix, s i is the depth information of the feature point, P i is the coordinate of the position identifier corresponding to the feature point, and u i is the pixel coordinate of the projection of the position identifier.

[0169] In one embodiment, before step S40, "obtaining the pose information corresponding to the intelligent device according to the spatial position information of the position identifier and the parameter information of the camera that captures the image", the following steps may further be included:

[0170] S50, obtaining the operation state information of the intelligent device and pre-integrating the operation state information;

[0171] S60. Obtain the initial pose information corresponding to the intelligent device according to the position identifier and the pre-integrated operation state information.

[0172] In this embodiment, as Figure 5 shown, while determining the position identifier based on at least two frames of images in the driving video of the intelligent device, the intelligent device can also obtain the operation state information through the IMU and pre-integrate the operation state information. The purpose of pre-integration is to compress these high-frequency operation state information into a more compact form.

[0173] Furthermore, the initial pose information corresponding to the intelligent device can be determined according to the position identifier and the pre-integrated operation state information to initialize the camera pose.

[0174] Specifically, for example, after the intelligent device obtains the matching result and the pre-integration result, visual and IMU alignment is performed to determine the initial pose. Specifically, it can include aligning the visual estimated rotation with the IMU pre-integrated rotation, aligning the visual estimated translation with the IMU pre-integrated position increment to establish constraints between all frames, optimizing the gravity direction to the vertical gravity direction in the world coordinate system, etc., to achieve pose initialization, which will not be elaborated here.

[0175] In one embodiment, in the above S20, "obtain the map information of the intelligent device's travel according to the position identifier" may include:

[0176] S201. Obtain the map information of the intelligent device's travel according to the pose information of the intelligent device and the position identifier.

[0177] Combined with the above description, in this embodiment, the intelligent device can generate the map information of the intelligent device's travel according to the pose information and the position identifier of the intelligent device.

[0178] In one embodiment, after the above S40, "obtain the pose information corresponding to the intelligent device according to the spatial position information of the position identifier and the parameter information of the camera that captures the image", it may further include:

[0179] S70. Optimize the feature points in the at least two frames of images;

[0180] S80. Determine a new position identifier and new pose information according to the optimized feature points.

[0181] In this embodiment, as Figure 5 shown, in the non-linear optimization stage, Ceres can be used for curve fitting. Ceres is a widely used least squares problem solving library, and the most general form of the least squares problem it solves is as follows:

[0182]

[0183] s.t.l j ≤x j ≤u j

[0184] wherein, x1, ..., x n are optimization variables, also known as parameter blocks, and f i is called the cost function, also known as the residual block, and l j and u j are the upper and lower limits of the j-th optimization variable.

[0185] It can be understood that the measurement data of sensors (such as cameras, lidars, etc.) in the SLAM system usually contains noise, and the model also has uncertainties. Nonlinear optimization adjusts the camera pose and the positions of map points by minimizing the error between the measured values and the estimated values, thereby improving the accuracy and robustness of the system.

[0186] After optimization, some feature points at the edge positions or those that cannot be used (i.e., the invalid feature points in this embodiment) are removed to obtain valid feature points, and the sliding window is updated.

[0187] Updating the sliding window can be understood as follows: a sliding window can contain multiple key-frame images and is updated as the intelligent device moves. Assuming the size of the sliding window is 3, when the intelligent device moves to the 4th key image frame, the 1st key frame is deleted, and the 4th key frame is added to the sliding window.

[0188] Through the above operations, this embodiment can determine a new position identifier and new pose information based on the optimized feature points and generate an initial global map.

[0189] In one embodiment, after the above S20, "obtaining the map information of the intelligent device according to the position identifier", the following may further be included:

[0190] S90, obtaining target map information according to the driving position of the intelligent device and the map information;

[0191] S100, optimizing the target map information according to the position identifier in the target map information.

[0192] In this embodiment, intelligent device relocalization can also be performed based on deep learning.

[0193] For example, when the intelligent device travels to a certain place again, the GPS of the intelligent device sends a signal. The intelligent device locates based on the GPS signal, and filters out a matching global map from the map library according to the positioning result, that is, the initial global map generated above. Then the intelligent device initializes the scene, further locates the position of the intelligent device in the global map, and then activates the global map. After the global map is loaded, as the intelligent device travels, this embodiment can execute a relocalization method based on deep learning, which is the feature extraction method and feature matching method based on deep learning in the above embodiments.

[0194] Specifically, for example, the intelligent device can obtain the current position identifier in the current frame image collected by the intelligent device according to the feature extraction method and feature matching method based on deep learning in the above embodiments.

[0195] Then the intelligent device can optimize the target map information according to the position identifier in the target map information and the current position identifier.

[0196] In a specific embodiment, in the above S100, "optimize the target map information according to the position identifier in the target map information" may include:

[0197] Determine whether the intelligent device has passed through the same position according to the position identifier in the target map information and the current position identifier;

[0198] If the intelligent device has passed through the same position, optimize the target map information according to the current position identifier.

[0199] In this embodiment, if the position identifier in the target map information is the same as the current position identifier, it can be determined that the intelligent device has passed through the same position. At this time, the target map information can be optimized according to the current position identifier. For example, update the spatial point corresponding to the position identifier in the target map information.

[0200] In another embodiment, the intelligent device can also use the feature extraction sub-model to extract feature points from the historical frame and the current frame respectively, and then use the feature matching sub-model to find the similarity of the current frame data from the historical frame data. If most of the feature points can be matched, it can be considered that the intelligent device has passed through the same location. For example, assume that 5 historical frames are selected, then the current frame needs to be feature-matched with the previous 5 frames. Assume that 3 consecutive frames out of 5 frames are used as the standard for the final successful match. When 3 consecutive frames are matched, it is determined that the intelligent device has passed through the same location.

[0201] It can be understood that the relocalization method in this application can reposition the intelligent device by matching the current observation with the feature points of the known global map, correct the cumulative error, and improve the positioning accuracy. Secondly, relocalization can also be used for loop closure detection, that is, to identify whether the intelligent device has passed through the current position. If it is confirmed that it has passed, global optimization is triggered to optimize the map and trajectory. In addition, the system robustness can also be improved. Assuming that the initial positioning is lost, the relocalization technology can quickly restore the positioning.

[0202] In an embodiment, if the intelligent device passes through the same position, the current position identifier is transmitted back to optimize the map information according to the transmitted position identifier and the position identifier corresponding to the next frame of image.

[0203] In this embodiment, combined with the above description, if it is detected that the intelligent device passes through the same position, the current position identifier can be transmitted back, providing global constraints, thereby reducing the cumulative error, optimizing the map and trajectory, and enhancing the robustness and consistency of the system. At the same time, the corresponding map information can also be determined according to the transmitted position identifier and the position identifier in the next frame of image in the future.

[0204] Generally speaking, in the embodiment of this application, as Figure 5 shown, the input is the image obtained by the camera. The feature information extracted by the trained feature extraction sub-model includes key points, descriptors, and confidence levels. Then, the trained feature matching is used for feature matching to obtain the matching relationship and confidence level. At the same time, the data obtained by the IMU is pre-integrated. Then, the pose is initialized according to the results of feature matching and pre-integration, and then the PnP process is solved, that is, the operation of the camera is estimated. Then, optimization is performed in the non-linear optimization stage. After optimization, some points at the edge positions or those that cannot be used are removed, the sliding window is updated to obtain the pose and road signs, and an initial global map is generated. When the intelligent device travels to a certain place again, the GPS signal can be used to obtain the positioning of the intelligent device. According to the GPS positioning result, the matching global map is screened out from the map library, that is, the above-generated initial global map. Then, the scene is initialized to further locate where the intelligent device is in the global map, and the global map is activated and loaded according to the positioning. As the intelligent device travels, the relocalization method based on deep learning is executed to determine whether the intelligent device has passed through the same place. This application has the advantages of good feature extraction effect and more robust matching, thereby reducing the cumulative error, improving the accuracy of the map, and can be applied to fields such as domestic robots, valet parking of intelligent vehicles, and memory parking.

[0205] Correspondingly, the embodiment of this application also provides an intelligent device driving assistance device, as Figure 6 shown, this device may include:

[0206] A determination module 1001, configured to process at least two frames of images in the driving video of the intelligent device based on a target model, and determine a location identifier in the environment where the intelligent device is located, where the target model is obtained based on a deep learning algorithm;

[0207] An acquisition module 1002, configured to acquire map information of the driving of the intelligent device according to the location identifier.

[0208] Optionally, the determination module 1001 in this application is further configured to:

[0209] Extract feature points of at least two frames of images in the driving video of the intelligent device through the feature extraction sub-model;

[0210] Determine the location identifier through the feature matching sub-model according to the feature points of each image.

[0211] Optionally, the intelligent device driving assistance device in this application further includes:

[0212] A first training module, configured to train the feature extraction sub-model according to training sample data to obtain a trained feature extraction sub-model.

[0213] Optionally, the intelligent device driving assistance device in this application further includes:

[0214] A second training module, configured to train a synthetic geometric data set according to a deep learning model to obtain a basic detector;

[0215] A single linear transformation module, configured to perform a single linear transformation on a driving training image set and the basic detector to obtain training sample data.

[0216] Optionally, the acquisition method of the above-mentioned driving training image set includes:

[0217] Acquire driving video data and perform format conversion on the driving video data;

[0218] Extract images from the converted driving video data to obtain a driving training image set, where the driving training image set includes multiple groups of left-eye images and corresponding right-eye images.

[0219] Optionally, the determination module 1001 in this application is further configured to:

[0220] Obtain distance information between the feature points according to the position information of the feature points of each image;

[0221] Screen the feature points according to the distance information and a preset distance threshold to obtain optimized feature points, and determine an optimized location identifier according to the optimized feature points.

[0222] Optionally, the determination module 1001 in the present application is further configured to:

[0223] Process the feature points of each image according to the feature matching sub-model, and determine the position identifier.

[0224] Optionally, the determination module 1001 in the present application is further configured to:

[0225] Match the feature points of each of the images through the feature matching sub-model to determine at least two frames of associated target images;

[0226] Process the target images through the feature matching sub-model to determine the position identifier.

[0227] Optionally, the determination module 1001 in the present application is further configured to:

[0228] Process the feature points of the first image and the feature points of the second image through the feature matching sub-model to obtain the matching relationship between the first image and the second image; wherein, the first image is any image in the video, and the second image is an associated image of the first image;

[0229] Determine at least two frames of associated target images through the feature matching sub-model according to the matching relationship.

[0230] Optionally, the determination module 1001 in the present application is further configured to:

[0231] When the first image is an initial frame image, extract an image with a different acquisition perspective from the first image in the initial frame image as the second image.

[0232] Optionally, the determination module 1001 in the present application is further configured to:

[0233] When the first image is not a non-initial frame image, extract an image with a different acquisition perspective from the first image in the initial frame image as the second image; and / or,

[0234] When the first image is not a non-initial frame image, use the previous frame adjacent to the first image as the second image.

[0235] Optionally, the determination module 1001 in the present application is further configured to:

[0236] Input the feature points of the at least two frames of associated target images within the sliding window into the feature matching sub-model;

[0237] Match the feature points within the sliding window through the feature matching sub-model to obtain the position identifier.

[0238] Optionally, the intelligent device driving assistance device in the present application further includes:

[0239] A pose information acquisition module, configured to acquire the pose information corresponding to the intelligent device according to the spatial position information of the position identifier and the parameter information of the camera that acquires the image.

[0240] Optionally, the intelligent device driving assistance device in the present application further includes:

[0241] A pre-integration module, configured to acquire the operating state information of the intelligent device and perform pre-integration on the operating state information;

[0242] An initialization module, configured to perform pose initialization on the pose information corresponding to the intelligent device according to the position identifier and the pre-integrated operating state information.

[0243] Optionally, the acquisition module 1002 in the present application is further configured to:

[0244] Acquire the map information of the intelligent device according to the pose information of the intelligent device and the position identifier.

[0245] Optionally, the acquisition module 1002 in the present application is further configured to:

[0246] A first optimization module, configured to perform optimization processing on the feature points in the at least two frames of images;

[0247] An update module, configured to determine a new position identifier and new pose information according to the optimized feature points.

[0248] Optionally, the optimization processing includes non-linear optimization processing based on an objective function and / or removal processing of invalid feature points.

[0249] Optionally, the acquisition module 1002 in the present application is further configured to:

[0250] Perform optimization processing on the feature points of the at least two frames of images within a sliding window.

[0251] Optionally, the intelligent device driving assistance device in the present application further includes:

[0252] A map information acquisition module, configured to acquire target map information according to the driving position of the intelligent device and the map information;

[0253] A second optimization module, configured to optimize the target map information according to the position identifier in the target map information.

[0254] Optionally, the above-mentioned second optimization module is further configured to:

[0255] Obtain the current position identifier corresponding to the current frame image collected by the intelligent device;

[0256] Optimize the target map information according to the position identifier in the target map information and the current position identifier.

[0257] Optionally, the above second optimization module is further configured to:

[0258] Determine whether the intelligent device has passed through the same position according to the position identifier in the target map information and the current position identifier;

[0259] If the intelligent device has passed through the same position, optimize the target map information according to the current position identifier.

[0260] Optionally, the intelligent device driving assistance device in this application further includes:

[0261] A feedback module, configured to, if the intelligent device has passed through the same position, feedback the current position identifier, so as to optimize the map information according to the feedback position identifier and the position identifier corresponding to the next frame image.

[0262] For the specific implementation of each of the above operations, reference may be made to the previous embodiments, which will not be elaborated here.

[0263] Correspondingly, an embodiment of this application further provides an electronic device, as Figure 7 shown, Figure 7 is a schematic structural diagram of the electronic device provided by the embodiment of this application. The electronic device 1100 includes a processor 1101 with one or more processing cores, a memory 1102 with one or more computer-readable storage media, and a computer program stored on the memory 1102 and executable on the processor. Among them, the processor 1101 is electrically connected to the memory 1102. Those skilled in the art can understand that the structure of the electronic device shown in the figure does not constitute a limitation on it, and it may include more or fewer components than shown in the figure, or combine certain components, or arrange different components.

[0264] The processor 1101 is the control center of the electronic device 1100, connecting various parts of the entire electronic device 1100 through various interfaces and circuits. By running or loading software programs and / or units stored in the memory 1102, and invoking the data stored in the memory 1102, it executes various functions of the electronic device 1100 and processes data, thereby monitoring the electronic device 1100 as a whole. The processor 1101 can be a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc., and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application.

[0265] In the embodiments of the present application, the processor 1101 in the electronic device 1100 will load the instructions corresponding to the processes of one or more application programs into the memory 1102 according to the following steps, and the processor 1101 will run the application programs stored in the memory 1102 to implement various functions, such as:

[0266] Processing at least two frames of images in the driving video of the intelligent device based on a target model to determine a position identifier in the environment where the intelligent device is located, and the target model is obtained based on a deep learning algorithm;

[0267] Obtaining map information of the driving of the intelligent device according to the position identifier.

[0268] For the specific implementation of the above operations, reference can be made to the previous embodiments, which will not be elaborated here.

[0269] Optionally, as Figure 7 shown, the electronic device 1100 further includes: a touch display screen 1103, a radio frequency circuit 1104, an audio circuit 1105, an input unit 1106, and a power supply 1107. Among them, the processor 1101 is electrically connected to the touch display screen 1103, the radio frequency circuit 1104, the audio circuit 1105, the input unit 1106, and the power supply 1107 respectively. Those skilled in the art can understand that Figure 7 the structure of the electronic device shown in

[0270] The touch display screen 1103 can be used to display a graphical user interface and receive operation instructions generated by a user's interaction with the graphical user interface. The touch display screen 1103 may include a display panel and a touch panel. Among them, the display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the electronic device. These graphical user interfaces can be composed of graphics, text, icons, videos, and any combination thereof. Optionally, the display panel can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. The touch panel can be used to collect touch operations of the user on or near it (such as operations of the user using a finger, a stylus, or any suitable object or accessory on or near the touch panel), and generate corresponding operation instructions, and the operation instructions execute the corresponding program. Optionally, the touch panel can include two parts: a touch display system and a touch controller. Among them, the touch display system detects the touch orientation of the user and detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch display system, converts it into contact coordinates, and then sends it to the processor 1101, and can receive and execute the commands sent by the processor 1101. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it transmits it to the processor 1101 to determine the type of touch event. Subsequently, the processor 1101 provides a corresponding visual output on the display panel according to the type of touch event. In the embodiments of the present application, the touch panel and the display panel can be integrated into the touch display screen 1103 to implement input and output functions. However, in some embodiments, the touch panel and the touch panel can be implemented as two independent components to implement input and output functions. That is, the touch display screen 1103 can also be used as part of the input unit 1106 to implement the input function.

[0271] The radio frequency circuit 1104 can be used to transmit and receive radio frequency signals to establish wireless communication with a network device or other electronic devices through wireless communication, and transmit and receive signals with the network device or other electronic devices.

[0272] The audio circuit 1105 can be used to provide an audio interface between the user and the electronic device through a speaker and a microphone. The audio circuit 1105 can convert the received audio data into an electrical signal and transmit it to the speaker, which converts it into a sound signal for output. On the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 1105, converted into audio data, and then the audio data is output to the processor 1101 for processing. After that, it is sent to another electronic device, for example, through the radio frequency circuit 1104, or the audio data is output to the memory 1102 for further processing. The audio circuit 1105 may also include an earphone jack to provide communication between the peripheral earphone and the electronic device.

[0273] The input unit 1106 can be used to receive input digital, character information or user feature information (such as fingerprint, iris, face information, etc.), and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0274] The power supply 1107 is used to supply power to each component of the electronic device 1100. Optionally, the power supply 1107 can be logically connected to the processor 1101 through a power management device, so as to realize functions such as management of charging, discharging, and power consumption management through the power management device. The power supply 1107 may also include any components such as one or more DC or AC power supplies, recharge devices, power failure detection circuits, power converters or inverters, and power status indicators.

[0275] Although Figure 7 not shown in the figure, the electronic device 1100 may also include a camera, a sensor, a Wi-Fi module, a Bluetooth module, etc., which will not be elaborated here.

[0276] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0277] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling relevant hardware through instructions. The instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0278] Therefore, an embodiment of the present application provides a computer-readable storage medium, in which multiple computer programs are stored. The computer programs can be loaded by a processor to execute any method provided by the embodiments of the present application. The computer programs can execute the steps of the following methods:

[0279] Processing at least two frames of images in the driving video of the intelligent device based on a target model to determine a location identifier in the environment where the intelligent device is located, and the target model is obtained based on a deep learning algorithm;

[0280] Obtaining map information of the driving of the intelligent device according to the location identifier.

[0281] For the specific implementation of each of the above operations, reference may be made to the previous embodiments, which will not be elaborated here.

[0282] Among them, the computer-readable storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc.

[0283] Since the computer program stored in the computer-readable storage medium can execute any method provided by the embodiments of the present application, the beneficial effects achievable by any method provided by the embodiments of the present application can be realized. For details, refer to the previous embodiments, which will not be elaborated here.

[0284] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0285] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowcharts and / or block diagrams can be implemented by computer program instructions, and the combination of processes and / or blocks in the flowcharts and / or block diagrams can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in one process Figure 1 ]one process or multiple processes and / or blocks Figure 1 a device for implementing the functions specified in one block or multiple blocks.

[0286] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device implements the process Figure 1one or more processes and / or blocks Figure 1 the functions specified in one or more blocks.

[0287] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one or more processes and / or blocks Figure 1 one or more blocks.

[0288] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0289] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of computer-readable media.

[0290] Computer-readable media includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated communication signals and carrier waves.

[0291] In the description of this application, the terms "first" and "second" are used only for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more features. In the description of this application, "a plurality of" means two or more unless otherwise specifically defined.

[0292] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0293] Among the embodiments, implementation manners and related technical features of the present application, they can be combined and replaced with each other without conflict.

[0294] The above are only the preferred embodiments of the present application, and do not impose any form of limitation on the present application. However, any simple modifications, equivalent changes and decorations made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application still fall within the scope of the technical solution of the present application.

Claims

1. A method for map construction, characterized in that, Applied to an intelligent device, the method includes: Processing at least two frames of images in the driving video of the intelligent device based on a target model to determine a location identifier in the environment where the intelligent device is located, where the target model is obtained based on a deep learning algorithm; Obtaining map information of the intelligent device's driving according to the location identifier.

2. The method according to claim 1, wherein The target model includes a feature extraction sub-model and a feature matching sub-model. Processing at least two frames of images in the driving video of the intelligent device based on the target model to determine a location identifier in the environment where the intelligent device is located includes: Extracting feature points of at least two frames of images in the driving video of the intelligent device through the feature extraction sub-model; Determining the location identifier through the feature matching sub-model according to the feature points of each image.

3. The method according to claim 2, characterized in that The feature extraction sub-model is trained through the following steps: Training the feature extraction sub-model according to training sample data to obtain a trained feature extraction sub-model.

4. The method according to claim 3, wherein The steps for obtaining the training sample data include: Training a synthetic geometric data set according to a deep learning model to obtain a basic detector; Performing a single linear transformation on the driving training image set and the basic detector to obtain training sample data.

5. The method according to claim 4, wherein The steps for obtaining the driving training image set include: Obtaining driving video data and performing format conversion on the driving video data; Performing image extraction on the converted driving video data to obtain a driving training image set, where the driving training image set includes multiple groups of left-eye images and corresponding right-eye images.

6. The method according to claim 2, characterized in that, After determining the location identifier through the feature matching sub-model according to the feature points of each image, it further includes: Obtaining distance information between the feature points according to the position information of the feature points of each image; Filtering the feature points according to the distance information and a preset distance threshold to obtain optimized feature points, and determining an optimized location identifier according to the optimized feature points.

7. The method according to claim 2, wherein Determining the location identifier through the feature matching sub-model according to the feature points of each image includes: Determining the location identifier according to the feature points of each image processed by the feature matching sub-model.

8. The method according to claim 7, characterized in that, Determining the location identifier according to the feature points of each image processed by the feature matching sub-model in the deep learning algorithm includes: Matching the feature points of each of the images through the feature matching sub-model to determine at least two frames of associated target images; Processing the target images through the feature matching sub-model to determine the location identifier.

9. The method according to claim 8, wherein Matching the feature points of each of the images through the feature matching sub-model to determine at least two frames of associated target images includes: Processing the feature points of the first image and the feature points of the second image through the feature matching sub-model to obtain a matching relationship between the first image and the second image; where the first image is any image in the video, and the second image is an associated image of the first image; Determining at least two frames of associated target images through the feature matching sub-model according to the matching relationship.

10. The method according to claim 9, characterized in that, The method further includes: When the first image is an initial frame image, extract an image in the initial frame image that has a different acquisition perspective from the first image as the second image.

11. The method according to claim 9, characterized in that, The method further includes: When the first image is not a non-initial frame image, extract an image in the initial frame image that has a different acquisition perspective from the first image as the second image; and / or, When the first image is not a non-initial frame image, use the previous frame adjacent to the first image as the second image.

12. The method according to claim 8, wherein The process of processing the target image through the feature matching sub-model to determine the position identifier includes: Input the feature points of the at least two associated target images within the sliding window into the feature matching sub-model; Through the feature matching sub-model, match the feature points within the sliding window to obtain the position identifier.

13. The method according to claim 1, wherein After processing at least two frames of images in the intelligent device driving video based on the target model to determine the position identifier in the environment where the intelligent device is located, it includes: According to the spatial position information of the position identifier and the parameter information of the camera that captures the image, obtain the pose information corresponding to the intelligent device.

14. The method according to claim 13, characterized in that Before obtaining the pose information corresponding to the intelligent device according to the spatial position information of the position identifier and the parameter information of the camera that captures the image, it further includes: Obtain the operating state information of the intelligent device and perform pre-integration on the operating state information; According to the position identifier and the pre-integrated operating state information, perform pose initialization on the pose information corresponding to the intelligent device.

15. The method according to claim 13, wherein When obtaining the map information of the intelligent device's travel according to the position identifier, it further includes: According to the pose information of the intelligent device and the position identifier, obtain the map information of the intelligent device's travel.

16. The method according to claim 15, wherein Before obtaining the map information of the intelligent device's travel according to the pose information of the intelligent device and the position identifier, it further includes: Perform optimization processing on the feature points in the at least two frames of images; Determine a new position identifier and new pose information according to the optimized feature points.

17. The method according to claim 16, wherein The optimization processing includes non-linear optimization processing based on an objective function and / or removal processing of invalid feature points.

18. The method according to claim 16, characterized in that, The process of performing optimization processing on the feature points in the at least two frames of images includes: Perform optimization processing on the feature points of the at least two frames of images within the sliding window.

19. The method according to claim 1, characterized in that, After obtaining the map information of the intelligent device's travel according to the position identifier, it further includes: According to the travel position of the intelligent device and the map information, obtain target map information; Optimize the target map information according to the position identifier in the target map information.

20. The method according to claim 19, wherein The process of optimizing the target map information according to the position identifier in the target map information includes: Obtain the current position identifier corresponding to the current frame image captured by the intelligent device; Optimize the target map information according to the position identifier in the target map information and the current position identifier.

21. The method according to claim 20, wherein The process of optimizing the target map information according to the position identifier in the target map information and the current position identifier includes: Determine whether the intelligent device passes through the same location according to the location identifier in the target map information and the current location identifier; If the intelligent device passes through the same location, optimize the target map information according to the current location identifier.

22. The method according to claim 21, wherein The method further includes: If the intelligent device passes through the same location, transmit the current location identifier back to optimize the map information according to the transmitted location identifier and the location identifier corresponding to the next frame of image.

23. An intelligent device driving assistance device, characterized in that, The intelligent device driving assistance device includes: A determination module, configured to process at least two frames of images in the intelligent device driving video based on a target model, and determine a location identifier in the environment where the intelligent device is located, where the target model is obtained based on a deep learning algorithm; An acquisition module, configured to acquire the map information of the intelligent device driving according to the location identifier.

24. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes any one of the methods described in claims 1 to 22.

25. A computer-readable storage medium, characterized in that, It includes a computer program, and when the computer program runs on an electronic device, the computer program is used to cause the electronic device to execute any one of the methods described in claims 1 to 22.

26. A computer program product, characterized in that, Includes a computer program, and the computer program is stored in a computer-readable storage medium; when the processor of the electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device executes any one of the methods described in claims 1 to 22.

27. A vehicle, characterized in that, At least one of the device described in claim 23, the electronic device described in claim 24, the computer-readable storage medium described in claim 25, and the computer program product described in claim

Images