Map generation method, device, electronic device and computer-readable storage medium

Abstract

The present application provides a map generation method, apparatus, electronic device and computer-readable storage medium, which relate to the field of computer technology. The method includes: obtaining multi-channel fisheye camera data, wheel odometer data and inertial sensing unit (IMU) data, then determining the trajectory information of the target object based on the wheel odometer data and the IMU data, and extracting the respective corresponding visual information for each channel based on the multi-channel fisheye camera data, and then creating the respective corresponding map information for each channel based on the respective corresponding visual information for each channel and the trajectory information of the target object, and then generating a composite map based on the respective corresponding map information for each channel. The map generation method, apparatus, electronic device and computer-readable storage medium provided by the present application can use multiple low-cost sensors to perform automatic and high-precision map generation, so as to reduce the dependence on high-cost lidar for mapping.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to a map generation method, apparatus, electronic device, and computer-readable storage medium. Background Art

[0002] Currently, with the development of information technology, intelligent robots, intelligent target objects, etc. have also achieved rapid development, and the demand for high-precision positioning technology is becoming stronger and stronger. High-precision positioning technology is the core technology to solve the problem of "where", and it is of great significance in both global path planning and local path planning. High-precision maps are the core and key to achieving high-precision positioning, and they are the core components of autonomous unmanned machine systems such as logistics vehicles and passenger cars.

[0003] In related technologies, high-precision maps are usually constructed based on lidar, integrating vision and GNSS, etc. The map construction cost is high, the process is cumbersome, and manual participation may be required. However, the reasons for the high cost and cumbersome process of high-precision map construction mainly lie in the dependence on high-precision and high-cost lidar. Therefore, how to solve the dependence of high-precision map construction on high-precision and high-cost lidar has become a key problem. Summary of the Invention

[0004] The purpose of this application is to provide a map generation method, apparatus, electronic device, and computer-readable storage medium to solve at least one of the above technical problems.

[0005] The above application purpose of this application is achieved through the following technical solutions:

[0006] In a first aspect, a map generation method is provided, including:

[0007] Obtain multi-channel fisheye camera data, wheel odometer data, and inertial sensing unit IMU data;

[0008] Based on the wheel odometer data and IMU data, determine the trajectory information of the target object;

[0009] Extract the visual information corresponding to each channel respectively based on the multi-channel fisheye camera data;

[0010] Based on the visual information corresponding to each channel respectively, and the trajectory information of the target object, create the map information corresponding to each channel respectively;

[0011] Generate a composite map based on the map information corresponding to each channel respectively.

[0012] In a possible implementation, obtain wheel odometer data and inertial sensing unit IMU data; based on the wheel odometer data and IMU data, determine the trajectory information of the target object, including:

[0013] Obtain the wheel odometer file and the IMU file;

[0014] Based on the data in the wheel odometer file and the data in the IMU file, perform pose calculation to obtain a result sequence of poses;

[0015] Based on the result sequence of poses, determine the trajectory information of the target object.

[0016] In another possible implementation, the method further includes:

[0017] Based on the current IMU bias value, and calculate the poses at different times through the data in the wheel speed odometer file and the data in the IMU file;

[0018] Obtain the pose at the start point time and the pose at the end point time from the calculated poses at different times;

[0019] Construct an error function with the difference between the poses at the start and end points of the loop data acquisition, and calculate the error value;

[0020] Update the IMU bias value based on the error value;

[0021] Loop to execute: take the updated IMU bias value as the current IMU bias value; based on the current IMU bias value, and calculate the poses at different times through the data in the wheel speed odometer file and the data in the IMU file; obtain the pose at the start point time and the pose at the end point time from the calculated poses at different times; construct an error function with the difference between the poses at the start and end points of the loop data acquisition, and calculate the error value; update the IMU bias value based on the error value; until the error value is less than a preset threshold;

[0022] Determine the optimized IMU bias value as the IMU bias value when the error value is less than the preset threshold.

[0023] In another possible implementation, the updating the IMU bias value based on the error value includes:

[0024] If the error value is greater than the error value calculated last time, use a negative bias change to reduce the IMU bias value;

[0025] If the error value is less than the error value calculated last time, use a positive bias change to increase the IMU bias value.

[0026] In another possible implementation, the performing pose calculation based on the data in the wheel odometer file and the data in the IMU file includes:

[0027] Perform pose calculation based on the data in the wheel odometer file, the data in the IMU file, and the IMU bias value;

[0028] Among them, the IMU bias value includes at least one of the following:

[0029] Default bias value;

[0030] Bias value set in the static calibration of the IMU;

[0031] Updated IMU bias value;

[0032] Optimized IMU bias value.

[0033] In another possible implementation, the extraction of the respective corresponding visual information based on the data of each fisheye camera includes:

[0034] Through the embedded platform, and extract the respective corresponding visual information based on the data of each fisheye camera;

[0035] Among them, the extraction of the corresponding visual information based on the data of each fisheye camera includes:

[0036] Generate a grayscale image corresponding to the road based on the data of each fisheye camera;

[0037] Construct an image pyramid corresponding to the road based on the grayscale image corresponding to each road;

[0038] Calculate the expected number of feature points required for each layer in the image pyramid based on the total number of features, the number of pyramid layers, and the scale factor;

[0039] Based on the expected number of feature points required for each layer, divide the image of the corresponding layer into at least one image block, and select at least one valid feature point from each image block;

[0040] Calculate the direction and descriptor corresponding to each valid feature point, and determine the direction and descriptor corresponding to at least one valid feature point as the visual information corresponding to the road.

[0041] In another possible implementation, the creation of the respective corresponding map information based on the respective corresponding visual information and the trajectory information of the target object includes:

[0042] Based on the trajectory information of the target object and the trajectory information of each map stored in the map database, determine the current mapping method, or determine the current mapping method through the preset mapping method. The current mapping method includes: new map construction and incremental mapping;

[0043] If the current mapping method is the new map construction, map information corresponding to each road is created based on the visual information corresponding to each road and the trajectory information of the target object, and through a preset method;

[0044] If the current mapping method is the incremental mapping, map information matching the trajectory information of the target object is obtained from the map database, and incremental mapping is performed based on the visual information corresponding to each road and the trajectory information of the target object on the basis of the matching map information.

[0045] In another possible implementation, incremental mapping is performed based on the visual information corresponding to any one road and the trajectory information of the target object on the basis of the matching map information, including:

[0046] Initialization and key frame sequence construction are performed based on the visual information corresponding to any one road and the trajectory information of the target object, where the key frame sequence contains multiple key frames, and the first key frame with successful initialization is the I key frame; for each key frame, repositioning is performed on the matching map information, and the key frame with successful repositioning is determined as the S key frame, and the key frame in the matching map information that is closest to the S key frame is set as the S_Y key frame;

[0047] Key frame construction step: Determine the visual frame to be processed and the pose to be processed for mapping tracking processing, construct a new key frame, and perform local map optimization through BA;

[0048] Repositioning determination step: Determine whether the newly constructed key frame is successfully repositioned on the matching map information;

[0049] Key frame determination step: If the repositioning is successful, determine the key frame with successful positioning as the E key frame, and the key frame in the matching map information that is closest to the E key frame is set as the E_Y key frame;

[0050] Incremental map construction step: Based on the points between the S_Y key frame and the E_Y key frame, all the key frames and points between the S key frame and the E key frame, and all the key frames and points between the I key frame and the S key frame, and perform joint optimization processing through BA to obtain the incrementally updated map information;

[0051] The next visual frame is determined as the visual frame to be processed, the next pose is determined as the pose to be processed, and the key frame construction step, repositioning determination step, key frame determination step, and incremental map construction step are executed in a loop until a preset condition is met to achieve incremental mapping.

[0052] In another possible implementation, generating a composite map based on the map information corresponding to each path includes:

[0053] Generating multi-submap package files corresponding to each path based on the map information corresponding to each path and preset memory constraint information; generating pose trajectory information corresponding to each path based on the multi-submap package files corresponding to each path and the visual information corresponding to each path;

[0054] Generating a composite map based on the pose trajectory information corresponding to each path.

[0055] In another possible implementation, before generating the multi-submap package files corresponding to each path based on the map information corresponding to each path and preset memory constraint information, it further includes:

[0056] Obtaining the key-frame pose trajectory information corresponding to each path from the map information corresponding to each path, and obtaining the trajectory information corresponding to each key frame from the trajectory information of the target object;

[0057] Determining the relative pose error (RPE) corresponding to the map information of each path based on the key-frame pose trajectory information corresponding to each path and the trajectory information corresponding to each key frame;

[0058] Determining the absolute trajectory error (ATE) corresponding to the map information of each path based on the key-frame pose trajectory information corresponding to each path and the trajectory information corresponding to each key frame;

[0059] Determining the cross ATE corresponding to the map information of each path based on the key-frame pose trajectory information corresponding to each path and the trajectory information corresponding to each key frame;

[0060] Determining the evaluation result corresponding to the map information of each path based on the RPE corresponding to the map information of each path, the ATE corresponding to the map information of each path, and the cross ATE corresponding to the map information of each path;

[0061] Among them, generating the multi-submap package files corresponding to each path based on the map information corresponding to each path and preset memory constraint information includes:

[0062] If the evaluation results corresponding to the map information of each path are all not greater than the preset threshold, then generating the multi-submap package files corresponding to each path based on the map information corresponding to each path and preset memory constraint information.

[0063] In another possible implementation, generating the multi-submap package file corresponding to any one path based on the map information corresponding to the any one path and preset memory constraint information includes:

[0064] Based on the map information corresponding to any one of the roads and the preset memory constraint information, determine the number of sub - graphs corresponding to any one of the roads and the effective space occupied by each sub - graph;

[0065] Based on the effective space occupied by each sub - graph and the map information corresponding to the map of any one of the roads, determine the key - frame information corresponding to each sub - graph, where the key - frame information includes: starting key - frame serial number, effective starting key - frame serial number, effective ending key - frame serial number, and ending key - frame serial number;

[0066] Based on the starting key - frame serial number and the ending key - frame serial number, segment the map information corresponding to any one of the roads to obtain each sub - graph file;

[0067] Based on the number of sub - graphs corresponding to any one of the roads, the key - frame information corresponding to each sub - graph, and each sub - graph file, generate a multi - sub - graph packet file corresponding to any one of the roads.

[0068] In another possible implementation, before generating the multi - sub - graph packet file corresponding to any one of the roads based on the number of sub - graphs corresponding to any one of the roads, the key - frame information corresponding to each map, and each sub - graph file, it further includes: determining the statistical histogram of each key - frame corresponding to any one of the roads based on the map information corresponding to any one of the roads and through a tape model;

[0069] Determine a relocalization scenario recognition file based on the histogram;

[0070] Among them, generating the multi - sub - graph packet file corresponding to any one of the roads based on the number of sub - graphs corresponding to any one of the roads, the key - frame information corresponding to each sub - graph, and each sub - graph file includes:

[0071] Based on the number of sub - graphs corresponding to any one of the roads and the key - frame information corresponding to each sub - graph, construct an index file for the multi - sub - graphs;

[0072] Based on the index file of the multi - sub - graphs, the number of sub - graphs corresponding to any one of the roads, the key - frame information corresponding to each sub - graph, and each sub - graph file, generate the multi - sub - graph packet file corresponding to any one of the roads.

[0073] In another possible implementation, generating the pose trajectory information corresponding to each road based on the multi - sub - graph packet files corresponding to each road respectively and the visual information corresponding to each road respectively includes:

[0074] Decode the multi - sub - graph packet files corresponding to each road respectively to obtain the map files corresponding to each road respectively;

[0075] Perform visual relocalization and tracking localization on the visual information corresponding to each path in their respective corresponding map files to obtain the pose trajectory information corresponding to each path.

[0076] In another possible implementation, generating a composite map based on the pose trajectory information corresponding to each path includes:

[0077] Construct a primary navigation map based on preset pose trajectory information;

[0078] Determine the mapping relationship between the pose trajectory information of other paths and the preset pose trajectory information, where the preset pose trajectory information is any one of the pose trajectory information corresponding to each path;

[0079] Generate a composite map based on the primary navigation map and the mapping relationship.

[0080] In another possible implementation, constructing the primary navigation map based on preset pose trajectory information includes:

[0081] Determine the type information and location information corresponding to each POI to be added;

[0082] Construct the primary navigation map based on the preset pose trajectory information and the type information and location information corresponding to each POI.

[0083] In another possible implementation, the method further includes:

[0084] Generate trace following trajectory information based on the preset pose trajectory information;

[0085] Obtain the generated top view and generate a virtual boundary line in the generated top view;

[0086] Generate a navigation map based on the primary navigation map, the trace following trajectory, and the virtual boundary line.

[0087] In another possible implementation, generating the trace following trajectory information based on the preset pose trajectory information includes at least one of the following:

[0088] Determine the preset pose trajectory information as the trace following trajectory information;

[0089] Generate the trace following trajectory information based on the mapped pose trajectory information of each path, where the mapped pose trajectory information of each path is the pose trajectory information after mapping with the preset pose trajectory information as the reference.

[0090] In another possible implementation, before obtaining the generated top view, it further includes:

[0091] Generate the top view;

[0092] Among them, the method for generating the top view includes:

[0093] Obtain the external parameter matrix of the multi-channel fisheye camera installation, and based on the external parameter matrix of the multi-channel fisheye camera installation and through the projection imaging model of the multi-channel fisheye camera, determine the inverse perspective transformation (IPM) of the multi-channel fisheye camera;

[0094] Generate the top view according to the multi-channel fisheye camera data and the IPM of the multi-channel fisheye camera.

[0095] In another possible implementation manner, before obtaining the multi-channel fisheye camera data, wheel odometer data, and inertial sensing unit (IMU) data, it further includes:

[0096] Obtain the encoded compressed image data, IMU data, and wheel odometer data from the database;

[0097] Calibrate the timestamps corresponding to the obtained encoded compressed image data, IMU data, and wheel odometer data respectively;

[0098] Decode the calibrated compressed image data, and perform segmentation processing on the decoded image data to obtain the image data corresponding to each fisheye camera;

[0099] Encode the image data corresponding to each fisheye camera into files corresponding to each path, and the files corresponding to each path contain the image data corresponding to each path respectively;

[0100] Generate an IMU file based on the calibrated IMU data, and generate a wheel odometer file based on the calibrated wheel odometer data. The IMU file contains the calibrated IMU data, and the wheel odometer file contains the calibrated wheel odometer data;

[0101] Among them, obtaining the multi-channel fisheye camera data, wheel odometer data, and inertial sensing unit (IMU) data includes: obtaining the image data in the files corresponding to each path; and,

[0102] Obtaining the wheel odometer data in the wheel odometer file; and,

[0103] Obtaining the IMU data in the IMU file.

[0104] In another possible implementation manner, calibrating the timestamps corresponding to the obtained encoded compressed image data, IMU data, and wheel odometer data respectively includes:

[0105] Obtain the preset data corresponding to each sensor. The preset data corresponding to each sensor includes: the sampling frequency corresponding to each sensor, the time stamps corresponding to each sensor at the data start time, the time stamps corresponding to each sensor at the data end time, the sampling start data sequence numbers corresponding to each sensor, and the sampling end data sequence numbers. The sensors include: multiple fisheye cameras, a wheel odometer, and an IMU;

[0106] Based on the time stamp of the preset sensor at the data start time, determine the first difference information between the time stamps of other sensors at the data start time and the time stamp of the preset sensor at the data start time based on a specific relationship;

[0107] Based on the time stamp of the preset sensor at the data end time, determine the second difference information between the time stamps of other sensors at the data end time and the time stamp of the preset sensor at the data end time based on a specific relationship;

[0108] Based on the sampling start data sequence number of the preset sensor, determine the third difference information between the sampling start data sequence numbers of other sensors and the sampling start data sequence number of the preset sensor based on a specific relationship;

[0109] Based on the sampling end data sequence number of the preset sensor, determine the fourth difference information between the sampling end data sequence numbers of other sensors and the sampling end data sequence number of the preset sensor based on a specific relationship;

[0110] Based on the first difference information, the second difference information, the third difference information, and the fourth difference information corresponding to other sensors respectively, determine the time stamp differences between the data of other sensors and the data of the preset sensor based on a specific relationship;

[0111] Based on the time stamp differences between the data of other sensors and the data of the preset sensor respectively, determine to calibrate the corresponding sensor data;

[0112] Wherein, the specific relationship is the relationship between the sampling frequencies corresponding to other sensors and the sampling frequency of the preset sensor.

[0113] In a second aspect, a map generation device is provided, including:

[0114] A first acquisition module for acquiring fisheye camera data, wheel odometer data, and inertial sensing unit (IMU) data; a first determination module for determining the trajectory information of a target object based on the wheel odometer data and the IMU data;

[0115] An extraction module, configured to extract visual information corresponding to each path respectively based on the data of each fisheye camera;

[0116] A creation module, configured to create map information corresponding to each path respectively based on the visual information corresponding to each path respectively and the trajectory information of the target object;

[0117] A first generation module, configured to generate a composite map based on the pose trajectory information corresponding to each path respectively.

[0118] In a possible implementation manner, when the first acquisition module acquires wheel odometer data and inertial sensing unit IMU data, it is specifically configured to: acquire a wheel odometer file and an IMU file;

[0119] When the first determination module determines the trajectory information of the target object based on the wheel odometer data and the IMU data, it is specifically configured to:

[0120] Perform pose calculation based on the data in the wheel odometer file and the data in the IMU file to obtain a result sequence of poses;

[0121] Determine the trajectory information of the target object based on the result sequence of the poses.

[0122] In another possible implementation manner, the device further includes: a first calculation module, a second acquisition module, a second calculation module, an update module, a loop module, and a second determination module, where,

[0123] The first calculation module is configured to calculate poses at different moments based on the current IMU bias value and the data in the wheel speed odometer file and the IMU file;

[0124] The second acquisition module is configured to acquire the pose at the start point moment and the pose at the end point moment from the poses at different moments calculated;

[0125] The second calculation module is configured to construct an error function with the difference between the poses at the start and end points of the loop data acquisition and calculate an error value;

[0126] The update module is configured to update the IMU bias value based on the error value;

[0127] The loop module is used to repeatedly execute the following steps: use the updated IMU bias value as the current IMU bias value; based on the current IMU bias value, calculate the poses at different times through the data in the wheel odometer file and the data in the IMU file; obtain the pose at the starting point and the pose at the ending point from the calculated poses at different times; construct an error function with the difference between the poses at the starting and ending points of the loop data acquisition and calculate the error value; update the IMU bias value based on the error value; until the error value is less than a preset threshold.

[0128] The second determination module is used to determine the IMU bias value corresponding to when the error value is less than the preset threshold as the optimized IMU bias value.

[0129] In another possible implementation, when the update module updates the IMU bias value based on the error value, it specifically is used for:

[0130] When the error value is greater than the error value calculated in the previous time, use a negative bias change to reduce the IMU bias value;

[0131] When the error value is less than the error value calculated in the previous time, use a positive bias change to increase the IMU bias value.

[0132] In another possible implementation, when the first determination module calculates the pose based on the data in the wheel odometer file and the data in the IMU file, it specifically is used for:

[0133] Calculate the pose based on the data in the wheel odometer file, the data in the IMU file, and the IMU bias value;

[0134] Wherein, the IMU bias value includes at least one of the following:

[0135] Default bias value;

[0136] Bias value set in the static calibration of the IMU;

[0137] Updated IMU bias value;

[0138] Optimized IMU bias value.

[0139] In another possible implementation, when the extraction module extracts the respective corresponding visual information based on the data of each fisheye camera, it specifically is used for:

[0140] Through an embedded platform, and based on the data of each fisheye camera, extract the respective corresponding visual information;

[0141] Among them, when the extraction module extracts the corresponding visual information for each path based on the data of each fisheye camera, it is specifically used for:

[0142] Generating a grayscale image corresponding to each path based on the data of each fisheye camera;

[0143] Constructing an image pyramid corresponding to each path based on the grayscale image corresponding to each path;

[0144] Calculating the expected number of feature points required for each layer in the image pyramid based on the total number of features, the number of pyramid layers, and the scale factor;

[0145] Based on the expected number of feature points required for each layer, dividing the image of the corresponding layer into at least one image block, and selecting at least one valid feature point from each image block;

[0146] Calculating the direction and descriptor corresponding to each valid feature point respectively, and determining the direction and descriptor corresponding to at least one valid feature point as the visual information corresponding to each path.

[0147] In another possible implementation manner, when the creation module creates the map information corresponding to each path based on the visual information corresponding to each path and the trajectory information of the target object, it is specifically used for:

[0148] Based on the trajectory information of the target object and the trajectory information of each map stored in the map database, determining the current mapping method, or determining the current mapping method through a preset mapping method. The current mapping method includes: new map construction and incremental mapping;

[0149] When the current mapping method is the new map construction, based on the visual information corresponding to each path and the trajectory information of the target object, and creating the map information corresponding to each path through a preset method;

[0150] When the current mapping method is the incremental mapping, obtaining the map information matching the trajectory information of the target object from the map database, and performing incremental mapping on the basis of the matching map information based on the visual information corresponding to each path and the trajectory information of the target object.

[0151] In another possible implementation manner, when the creation module performs incremental mapping on the basis of the matching map information based on the visual information corresponding to any path and the trajectory information of the target object, it is specifically used for:

[0152] Initialize and construct a keyframe sequence based on the visual information corresponding to any one of the paths and the target object trajectory information, where the keyframe sequence contains multiple keyframes, and the first keyframe with successful initialization is the I keyframe; for each keyframe, perform relocalization on the matched map information, determine the keyframe with successful relocalization as the S keyframe, and determine the keyframe in the matched map information that is closest to the S keyframe, denoted as the S_Y keyframe;

[0153] Keyframe construction steps: Determine the visual frame to be processed and the pose to be processed for mapping tracking, construct a new keyframe, and perform local map optimization through BA;

[0154] Relocalization determination steps: Determine whether the newly constructed keyframe is successfully relocalized on the matched map information;

[0155] Keyframe determination steps: If the relocalization is successful, determine the keyframe with successful localization as the E keyframe, and determine the keyframe in the matched map information that is closest to the E keyframe, denoted as the E_Y keyframe;

[0156] Incremental map construction steps: Based on the points between the S_Y keyframe and the E_Y keyframe, all the keyframes and points between the S keyframe and the E keyframe, and all the keyframes and points between the I keyframe and the S keyframe, and perform joint optimization processing through BA to obtain the incrementally updated map information;

[0157] Loop to execute by determining the next visual frame as the visual frame to be processed, the next pose as the pose to be processed, the keyframe construction steps, the relocalization determination steps, the keyframe determination steps, and the incremental map construction steps until a preset condition is met to achieve incremental mapping.

[0158] In another possible implementation, when the first generation module generates a composite map based on the map information corresponding to each path respectively, it is specifically used for:

[0159] Generate multi-submap package files corresponding to each path based on the map information corresponding to each path respectively and the preset memory constraint information; generate pose trajectory information corresponding to each path based on the multi-submap package files corresponding to each path respectively and the visual information corresponding to each path;

[0160] Generate a composite map based on the pose trajectory information corresponding to each path respectively.

[0161] In another possible implementation, the device further includes: a third acquisition module, a third determination module, a fourth determination module, a fifth determination module, and a sixth determination module, where,

[0162] The third acquisition module is configured to acquire the key-frame pose trajectory information corresponding to each path from the map information corresponding to each path respectively, and acquire the trajectory information corresponding to each key frame from the trajectory information of the target object;

[0163] The third determination module is configured to determine the relative pose error (RPE) corresponding to each path of map information based on the key-frame pose trajectory information corresponding to each path respectively and the trajectory information corresponding to each key frame respectively;

[0164] The fourth determination module is configured to determine the absolute trajectory error (ATE) corresponding to each path of map information based on the key-frame pose trajectory information corresponding to each path respectively and the trajectory information corresponding to each key frame respectively;

[0165] The fifth determination module is configured to determine the cross ATE corresponding to each path of map information based on the key-frame pose trajectory information corresponding to each path respectively and the trajectory information corresponding to each key frame respectively;

[0166] The sixth determination module is configured to determine the evaluation result corresponding to each path of map information based on the RPE corresponding to each path of map information, the ATE corresponding to each path of map information, and the cross ATE corresponding to each path of map information;

[0167] Wherein, when the first generation module generates the multi-submap package files corresponding to each path based on the map information corresponding to each path respectively and the preset memory constraint information, it is specifically configured to:

[0168] When the evaluation results corresponding to each path of map information are all not greater than the preset threshold, generate the multi-submap package files corresponding to each path based on the map information corresponding to each path respectively and the preset memory constraint information.

[0169] In another possible implementation manner, when the first generation module generates the multi-submap package file corresponding to any one path based on the map information corresponding to any one path and the preset memory constraint information, it is specifically configured to:

[0170] Determine the number of submaps corresponding to any one path and the effective space occupied by each submap based on the map information corresponding to any one path and the preset memory constraint information;

[0171] Determine the key-frame information corresponding to each submap based on the effective space occupied by each submap and the map information corresponding to the map of any one path, where the key-frame information includes: the starting key-frame serial number, the effective starting key-frame serial number, the effective ending key-frame serial number, and the ending key-frame serial number;

[0172] Based on the starting key frame number and the ending key frame number, split the map information corresponding to any one path to obtain each sub-map file;

[0173] Based on the number of sub-maps corresponding to any one path, the key frame information corresponding to each sub-map, and each sub-map file, generate a multi-sub-map package file corresponding to any one path.

[0174] In another possible implementation, the apparatus further includes: a seventh determination module and an eighth determination module, where,

[0175] The seventh determination module is configured to determine the statistical histogram of each key frame corresponding to any one path based on the map information corresponding to any one path and through a tape model;

[0176] The eighth determination module is configured to determine a relocalization scenario recognition file based on the histogram;

[0177] Wherein, when the first generation module generates a multi-sub-map package file corresponding to any one path based on the number of sub-maps corresponding to any one path, the key frame information corresponding to each sub-map, and each sub-map file, it is specifically configured to:

[0178] Based on the number of sub-maps corresponding to any one path and the key frame information corresponding to each sub-map, construct an index file for the multi-sub-maps;

[0179] Based on the index file of the multi-sub-maps, the number of sub-maps corresponding to any one path, the key frame information corresponding to each sub-map, and each sub-map file, generate a multi-sub-map package file corresponding to any one path.

[0180] In another possible implementation, when the first generation module generates pose trajectory information corresponding to each path based on the multi-sub-map package files corresponding to each path and the visual information corresponding to each path, it is specifically configured to:

[0181] Decode the multi-sub-map package files corresponding to each path to obtain the map files corresponding to each path;

[0182] Perform visual relocalization and tracking localization on the visual information corresponding to each path in the respective corresponding map files to obtain the pose trajectory information corresponding to each path.

[0183] In another possible implementation, when the first generation module generates a composite map based on the pose trajectory information corresponding to each path, it is specifically configured to:

[0184] Construct a primary navigation map based on the preset pose trajectory information;

[0185] Determine the mapping relationship between the pose trajectory information of other paths and the preset pose trajectory information, where the preset pose trajectory information is any one of the pose trajectory information corresponding to each path;

[0186] Generate a composite map based on the primary navigation map and the mapping relationship.

[0187] In another possible implementation manner, when constructing the primary navigation map based on the preset pose trajectory information, the first generation module is specifically configured to:

[0188] Determine the type information and position information corresponding to each POI to be added;

[0189] Construct the primary navigation map based on the preset pose trajectory information and the type information and position information corresponding to each POI.

[0190] In another possible implementation manner, the device further includes: a second generation module, a fourth acquisition module, a third generation module, and a fourth generation module, where

[0191] The second generation module is configured to generate tracking trajectory information based on the preset pose trajectory information;

[0192] The fourth acquisition module is configured to acquire the generated top view;

[0193] The third generation module is configured to generate virtual boundary lines in the generated top view;

[0194] The fourth generation module is configured to generate a navigation map based on the primary navigation map, the tracking trajectory, and the virtual boundary lines.

[0195] In another possible implementation manner, when generating the tracking trajectory information based on the preset pose trajectory information, the second generation module is specifically configured to perform at least one of the following:

[0196] Determine the preset pose trajectory information as the tracking trajectory information;

[0197] Generate the tracking trajectory information based on the mapped pose trajectory information of each path, where the mapped pose trajectory information of each path is the pose trajectory information after being mapped with the preset pose trajectory information as the reference.

[0198] In another possible implementation manner, the device further includes: a fifth generation module, where

[0199] The fifth generation module is configured to generate the top view;

[0200] Wherein, when generating the top view, the fifth generation module is specifically configured to:

[0201] Obtain the external parameter matrix of the installation of the multi-channel fisheye cameras, and determine the inverse perspective transformation (IPM) of the multi-channel fisheye cameras based on the external parameter matrix of the installation of the multi-channel fisheye cameras and through the projection imaging model of the multi-channel fisheye cameras;

[0202] Generate the top view according to the multi-channel fisheye camera data and the IPM of the multi-channel fisheye cameras.

[0203] In another possible implementation manner, the device further includes: a fifth acquisition module, a calibration module, a processing module, an encoding module, and a sixth generation module, wherein,

[0204] The fifth acquisition module is configured to acquire the encoded compressed image data, IMU data, and wheel odometer data from a database;

[0205] The calibration module is configured to calibrate the timestamps corresponding to the acquired encoded compressed image data, IMU data, and wheel odometer data respectively;

[0206] The processing module is configured to perform decoding processing on the calibrated compressed image data, and perform segmentation processing on the decoded image data to obtain the image data corresponding to each of the multi-channel fisheye cameras;

[0207] The encoding module is configured to encode the image data corresponding to each of the multi-channel fisheye cameras into files corresponding to each channel, and each file corresponding to each channel contains the image data corresponding to it;

[0208] The sixth generation module is configured to generate an IMU file based on the calibrated IMU data, and generate a wheel odometer file based on the calibrated wheel odometer data, wherein the IMU file contains the calibrated IMU data, and the wheel odometer file contains the calibrated wheel odometer data;

[0209] Wherein, when acquiring the multi-channel fisheye camera data, wheel odometer data, and inertial sensing unit (IMU) data, the first acquisition module is specifically configured to:

[0210] Acquire the image data in the files corresponding to each of the channels; and,

[0211] Acquire the wheel odometer data in the wheel odometer file; and,

[0212] Acquire the IMU data in the IMU file.

[0213] In another possible implementation, when the calibration module calibrates the timestamps corresponding to the encoded compressed image data, IMU data, and wheel odometer data obtained respectively, it is specifically configured to:

[0214] Obtain the preset data corresponding to each sensor. The preset data corresponding to each sensor includes: the sampling frequency corresponding to each sensor, the timestamps corresponding to each sensor at the start time of data, the timestamps corresponding to each sensor at the end time of data, the sampling start data sequence numbers corresponding to each sensor, and the sampling end data sequence numbers corresponding to each sensor. The each sensor includes: multiple fish-eye cameras, a wheel odometer, and an IMU;

[0215] Taking the timestamp of the preset sensor at the start time of data as a reference, determine the first difference information between the timestamps of other sensors at the start time of data and the timestamp of the preset sensor at the start time of data based on a specific relationship;

[0216] Taking the timestamp of the preset sensor at the end time of data as a reference, determine the second difference information between the timestamps of other sensors at the end time of data and the timestamp of the preset sensor at the end time of data based on a specific relationship;

[0217] Taking the sampling start data sequence number of the preset sensor as a reference, determine the third difference information between the sampling start data sequence numbers of other sensors and the sampling start data sequence number of the preset sensor based on a specific relationship;

[0218] Taking the sampling end data sequence number of the preset sensor as a reference, determine the fourth difference information between the sampling end data sequence numbers of other sensors and the sampling end data sequence number of the preset sensor based on a specific relationship;

[0219] Based on the first difference information, second difference information, third difference information, and fourth difference information corresponding to other sensors respectively, determine the timestamp differences between the other sensor data and the data of the preset sensor respectively based on a specific relationship;

[0220] Based on the timestamp differences between the other sensor data and the data of the preset sensor respectively, determine to calibrate the corresponding sensor data of each;

[0221] Wherein, the specific relationship is the relationship between the sampling frequencies corresponding to other sensors and the sampling frequency of the preset sensor.

[0222] In a third aspect, an electronic device is provided, and the electronic device includes:

[0223] One or more processors;

[0224] Memory;

[0225] One or more applications, where the one or more applications are stored in the memory and are configured to be executed by one or more processors, and the one or more programs are configured to: execute operations corresponding to map generation as shown in any possible implementation of the first aspect.

[0226] A fourth aspect provides a computer-readable storage medium storing at least one instruction, at least one program segment, a code set, or an instruction set, and the at least one instruction, at least one program segment, the code set, or the instruction set is loaded and executed by a processor to implement the map generation method as shown in any possible implementation of the first aspect.

[0227] In summary, the present application has the following beneficial effects:

[0228] The present application provides a map generation method, apparatus, electronic device, and computer-readable storage medium. Compared with the related art, by acquiring multi-channel fisheye camera data, wheel odometer data, and inertial sensing unit (IMU) data, and based on the wheel odometer data and IMU data, the trajectory information of the target object is determined, and visual information corresponding to each channel is extracted respectively based on the multi-channel fisheye camera data. Then, based on the visual information corresponding to each channel and the trajectory information of the target object, map information corresponding to each channel is created, and then a composite map is generated based on the map information corresponding to each channel. That is, in the present application, it is not necessary to rely on high-precision and high-cost lidar. Only by using the data collected by low-cost multi-channel fisheye cameras, wheel odometer data, and IMU data, composite map information can be generated, thereby avoiding the dependence on high-precision and high-cost lidar for high-precision map construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0229] Figure 1 It is a schematic flowchart of a map generation method provided by an embodiment of the present application;

[0230] Figure 2 It is a schematic flowchart of new map construction and incremental mapping provided by an embodiment of the present application;

[0231] Figure 3 It is an example diagram of generating a composite map provided by an embodiment of the present application;

[0232] Figure 4 It is an example diagram of a generated navigation map provided by an embodiment of the present application;

[0233] Figure 5 It is a schematic flowchart of an example process of offline map generation in an embodiment of the present application;

[0234] Figure 6 Schematic structural diagram of the device for generating a map provided by an embodiment of the present application;

[0235] Figure 7 Schematic structural diagram of the device of the electronic device provided by an embodiment of the present application. Detailed implementation manners

[0236] The present application will be further described in detail below with reference to the accompanying drawings.

[0237] This specific embodiment is only an interpretation of the present application, and it does not limit the present application. Those skilled in the art can make modifications without creative contributions to this embodiment after reading this specification, but as long as they are within the scope of the claims of the present application, they are protected by the patent law.

[0238] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0239] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after, unless otherwise specified.

[0240] The embodiment of the present application provides a map generation method, which can generate map information corresponding to a target object. In the embodiment of the present application, the target object can be a vehicle, a robot, or other devices that need to build a map, which is not limited in the embodiment of the present application. Among them, at least a multi-channel fisheye camera, a wheel odometer, and an inertial sensing unit (Inertial Measurement Unit, IMU) are installed in the target object.

[0241] The following will further describe the embodiment of the present application in detail with reference to the drawings in the specification.

[0242] The embodiment of the present application provides a map generation method, as Figure 1As shown, the method of this map can be executed by an electronic device, which can be a server or a terminal device. Among them, the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc., but is not limited thereto. The terminal device and the server can be directly or indirectly connected through wired or wireless communication methods. This application embodiment does not make any restrictions here. The method includes:

[0243] Step S101, obtain multi-channel fisheye camera data, wheel odometer data, and IMU data.

[0244] For the embodiments of this application, the multi-channel fisheye camera data refers to the data collected by the multi-channel fisheye camera; the wheel odometer data refers to the data collected by the wheel odometer, and the IMU data refers to the data collected by the IMU. In the embodiments of this application, the multi-channel fisheye camera refers to a fisheye camera with multiple cameras, such as a dual-channel fisheye camera, a four-channel fisheye camera, and an eight-channel fisheye camera, etc. Among them, the fisheye camera refers to a camera with a fisheye lens, which is a lens with an extremely short focal length and a viewing angle close to or equal to 180°.

[0245] Step S102, determine the trajectory information of the target object based on the wheel odometer data and the IMU data.

[0246] For the embodiments of this application, after obtaining the wheel odometer data and the IMU data, perform pose calculation based on the wheel odometer data and the IMU data to obtain the pose result sequence of the target object. Among them, the pose result sequence of the target object can form the trajectory information of the target object.

[0247] Step S103, extract the visual information corresponding to each path based on the multi-channel fisheye camera data respectively.

[0248] For the embodiments of this application, after obtaining the data collected by each multi-channel fisheye camera, extract the visual features and corresponding timestamps of each frame of image based on the data collected by each fisheye camera to obtain the visual information corresponding to this path. In the embodiments of this application, the visual information corresponding to any path can include: the visual information corresponding to each frame of image and their respective corresponding timestamps.

[0249] Further, in the embodiments of this application, step S102 can be executed before step S103, can be executed after step S103, or can be executed simultaneously with step S103. This application embodiment does not make any limitations.

[0250] Step S104: Create map information corresponding to each path based on the visual information corresponding to each path and the trajectory information of the target object.

[0251] Specifically, based on the visual information corresponding to each path obtained in the above step S103 and the trajectory information of the target object obtained in step S102, the map information corresponding to each path can be created by means of incremental mapping or conventional mapping. In the embodiments of the present application, the incremental mapping method refers to performing incremental mapping on the basis of the existing map, and the conventional mapping refers to reconstructing a new map.

[0252] Step S105: Generate a composite map based on the map information corresponding to each path.

[0253] For the embodiments of the present application, after creating the map information corresponding to each path in the above manner, a composite map is generated based on the created map information corresponding to each path.

[0254] The embodiments of the present application provide a map generation method. Compared with the related art, in the embodiments of the present application, by acquiring multi-path fisheye camera data, wheel odometer data, and inertial sensing unit (IMU) data, and based on the wheel odometer data and IMU data, the trajectory information of the target object is determined, and based on the fisheye camera data of each path, the visual information corresponding to each path is extracted respectively. Then, based on the visual information corresponding to each path and the trajectory information of the target object, the map information corresponding to each path is created, and then a composite map is generated based on the map information corresponding to each path. That is, in the present application, there is no need to rely on high-precision and high-cost lidar, and only the data collected by low-cost multi-path fisheye cameras, wheel odometer data, and IMU data can be used to generate composite map information, thereby avoiding the dependence on high-precision lidar for high-precision map construction.

[0255] Further, in the embodiments of the present application, before step S101 of acquiring multi-path fisheye camera data, wheel odometer data, and IMU data, it further includes: the data acquisition front end collects the corresponding data from each sensor. In the embodiments of the present application, the data acquisition front end can be an embedded controller. For example, it can be a TDA4 domain controller installed on a vehicle.

[0256] Specifically, in the embodiments of the present application, after the data acquisition front end obtains multi-channel fisheye camera data (taking four-channel fisheye camera data as an example for introduction in the embodiments of the present application), for the four-channel fisheye camera data, first perform time-domain alignment of the image data, that is, the time stamp difference between adjacent images is less than 20 milliseconds to be considered synchronizable; for the synchronizable 4-channel images, splice them into one image, and the resolution of the spliced image is 2 times the resolution of the original single-channel image; considering the encoding resource limitation of the embedded platform, it is necessary to downsample the spliced image. For example, the resolution of the original input image for each channel is 1280*720, after splicing it is 2560*1440, and after downsampling it is 1920*1280; secondly, compress and encode the downsampled image based on a video encoder, specifically, an H.264 encoder or an H.265 encoder can be used, and the encoded sequence can be an I, P frame sequence or an I, B, P frame sequence. The collected images will be used for mapping processing later, and the images are required to be as clear as possible after encoding. When the encoder hardware of the embedded platform supports it, it is recommended to give priority to using a configuration and sequence output with a high compression ratio, that is, give priority to using an H.265 encoder + I, B, P frame sequence; if not supported, use an H.264 encoder + I, P frame sequence.

[0257] Furthermore, for the wheel odometer and IMU, data acquisition is directly performed through the driving and data parsing of relevant sensors, and then publishing can be carried out using ROS1 or ROS2. Similarly, if there are GNSS / RTK sensors, data acquisition and publishing are performed.

[0258] Furthermore, after the data acquisition front end obtains the compressed and encoded multi-channel fisheye camera data, wheel odometer data, and IMU data, and may also obtain GNSS data and / or RTK data, the data acquisition backend (i.e., the electronic device in the embodiments of the present application) can also perform corresponding processing on the data obtained by the data acquisition front end and store it in the database. Specifically, it can include: listening to and receiving the published data based on ROS1 or ROS2. Decode the received image data; perform 1-to-4 processing, that is, upsample the decoded image with a resolution of 1920*1280 to an image with a resolution of 2560*1440, and divide it into four sub-images, each with a resolution of 1280*720, and each sub-image corresponds to one of the four channels of images; generate a top view for the four-channel images and the internal and external parameters of the four cameras. Specifically, the top view generation is completed using inverse perspective mapping (IPM); check the effect of the top view. If the lines on both sides are straight and continuous, the internal and external parameters of the camera and the camera image are okay; secondly, if it is judged to be okay, store the received 4-in-1 compressed image data, IMU and wheel odometer data, GNSS / RTK data, and camera internal and external parameter data.

[0259] Further, in the embodiments of the present application, the acquisition of multi-channel fisheye camera data in step S101 can be directly obtained from the above data acquisition front end or obtained from a database; similarly, the wheel odometer data can be directly obtained from the data acquisition front end or obtained from a database; similarly, the IMU data can be directly obtained from the data acquisition front end or obtained from a database. In the embodiments of the present application, the multi-channel fisheye camera data, wheel odometer data, and IMU data stored in the database are respectively obtained from the multi-channel fisheye camera, wheel odometer, and IMU and stored.

[0260] Further, before obtaining the multi-channel fisheye camera data, wheel odometer data, and IMU data from the database, it may further include: step S10 (not shown in the figure), step S11 (not shown in the figure), step S12 (not shown in the figure), step S13 (not shown in the figure), and step S14 (not shown in the figure), where

[0261] Step S10: Obtain the encoded compressed image data, IMU data, and wheel odometer data from the database.

[0262] Step S11: Calibrate the timestamps corresponding to the obtained encoded compressed image data, IMU data, and wheel odometer data respectively.

[0263] For the embodiments of the present application, the timestamps corresponding to the obtained encoded compressed image data, IMU data, and wheel odometer data are calibrated based on the termination time alignment method or the statistical alignment method.

[0264] Specifically, the assumption of termination time alignment is that the data acquisition of each sensor at the termination acquisition moment is physically at the same time (the timestamps marked on the data may be different). Based on this, according to the timestamp differences, respective sampling frequencies, and respective sampling data serial numbers of the last moments of the obtained data of each sensor (encoded compressed image data, IMU data, and wheel odometer data), the timestamps of the sensor data at each moment are aligned from back to front. In the embodiments of the present application, each sensor refers to the multi-channel fisheye camera, wheel odometer, and IMU.

[0265] Specifically, the calibration of the timestamps corresponding to the encoded compressed image data, IMU data, and wheel odometer data obtained in step S11 may specifically include: step S11a (not shown in the figure), step S11b (not shown in the figure), step S11c (not shown in the figure), step S11d (not shown in the figure), step S11e (not shown in the figure), step S11f (not shown in the figure), and step S11g (not shown in the figure), where

[0266] Step S11a: Obtain the preset data corresponding to each sensor respectively.

[0267] Among them, the preset data corresponding to each sensor respectively includes: the sampling frequency corresponding to each sensor respectively, the timestamps corresponding to each sensor respectively at the data start time, the timestamps corresponding to each sensor respectively at the data end time, the sampling start data sequence number corresponding to each sensor respectively, and the sampling end data sequence number. Each sensor includes: a multi-channel fisheye camera, a wheel odometer, and an IMU.

[0268] Step S11b: Based on the timestamp of the preset sensor at the data start time, determine the first difference information between the timestamps of other sensors at the data start time and the timestamp of the preset sensor at the data start time based on a specific relationship.

[0269] Among them, the specific relationship is the relationship between the sampling frequencies corresponding to other sensors respectively and the sampling frequency of the preset sensor.

[0270] Step S11c: Based on the timestamp of the preset sensor at the data end time, determine the second difference information between the timestamps of other sensors at the data end time and the timestamp of the preset sensor at the data end time based on a specific relationship.

[0271] Step S11d: Based on the sampling start data sequence number of the preset sensor, determine the third difference information between the sampling start data sequence numbers of other sensors respectively and the sampling start data sequence number of the preset sensor based on a specific relationship.

[0272] Step S11e: Based on the sampling end data sequence number of the preset sensor, determine the fourth difference information between the sampling end data sequence numbers of other sensors respectively and the sampling end data sequence number of the preset sensor based on a specific relationship.

[0273] Step S11f: Based on the first difference information, second difference information, third difference information, and fourth difference information corresponding to other sensors respectively, determine the timestamp differences between other sensor data and the data of the preset sensor based on a specific relationship.

[0274] Step S11g: Based on the timestamp differences between other sensor data and the preset sensor data respectively, determine the corresponding sensor data for calibration.

[0275] Specifically, the assumption of the statistical alignment method is that the data acquisitions at the start and end of the acquisition are physically simultaneous (the timestamps marked on the data may be different). The following introduces the method of calibrating sensor data through specific examples, as follows:

[0276] (a) Obtain the sampling frequencies of each sensor, which are determined values: Camera_Sampling_Freq = 25hz, Wheel_Sampling_Freq = 50hz, IMU_Sampling_Freq = 100hz. The timestamps of the different sensor data at the start of the data are respectively represented by Camera_Start_Stamp, Wheel_Start_Stamp, and IMU_Start_Stamp; the timestamps of the different sensor data at the end are respectively represented by Camera_End_Stamp, Wheel_End_Stamp, and IMU_End_Stamp; the starting sampling data numbers are respectively represented by Camera_Start_Numb, Wheel_Start_Numb, and IMU_Start_Numb; the ending sampling numbers are respectively represented by Camera_End_Numb, Wheel_End_Numb, and IMU_End_Numb. Among them, Camera_Sampling_Freq represents the acquisition frequency of the multi-channel fisheye camera, Wheel_Sampling_Freq represents the acquisition frequency of the wheel odometer, and IMU_Sampling_Freq represents the acquisition frequency of the IMU.

[0277] (b) Taking the IMU data with the highest frequency as the reference, calculate the timestamp differences and sequence number differences between the image data and the wheel odometer data and it respectively, to obtain the start timestamp difference, end timestamp difference, start sequence number difference, and end sequence number difference;

[0278] Among them, the timestamp difference between the image data and the IMU data at the start moment is characterized by delta_Camera_Start_Stamp, and the timestamp difference between the wheel odometer data and the IMU data at the start moment is characterized by delta_Wheel_Start_Stamp; the timestamp difference between the image data and the IMU data at the end moment is characterized by delta_Camera_End_Stamp, and the timestamp difference between the wheel odometer data and the IMU data at the end moment is characterized by delta_Wheel_End_Stamp; the sequence number difference between the image data and the IMU data at the start moment is characterized by delta_Camera_Start_Numb, and the sequence number difference between the wheel odometer data and the IMU data at the start moment is characterized by delta_Wheel_Start_Numb, the sequence number difference between the image data and the IMU data at the end moment is characterized by delta_Camera_End_Numb, and the sequence number difference between the wheel odometer and IMU data at the end moment is characterized by delta_Wheel_End_Numb.

[0279] Specifically, delta_Camera_Start_Stamp = Camera_Start_Stamp - IMU_Start_Stamp, delta_Wheel_Start_Stamp = Wheel_Start_Stamp - IMU_Start_Stamp;

[0280] Specifically, delta_Camera_End_Stamp = Camera_End_Stamp - IMU_End_Stamp, delta_Wheel_End_Stamp = Wheel_End_Stamp - IMU_End_Stamp;

[0281] Specifically, delta_Camera_Start_Numb = Camera_Start_Numb * 4 - IMU_Start_Numb, delta_Wheel_Start_Numb = Wheel_Start_Numb * 2 - IMU_Start_Numb;

[0282] Specifically, delta_Camera_End_Numb = Camera_End_Numb * 4 - IMU_End_Numb, delta_Wheel_End_Numb = Wheel_End_Numb * 2 - IMU_End_Numb;

[0283] It should be noted that in the embodiments of the present application, although the IMU data is used as a reference to calculate the timestamp difference and sequence number difference between the image data and the wheel odometer data respectively, obtaining the start timestamp difference, end timestamp difference, start sequence number difference, and end sequence number difference, it is not limited to using the IMU data as a reference. It is also possible to use the image data as a reference to calculate the timestamp difference and sequence number difference between the IMU data and the odometer data respectively, obtaining the start timestamp difference, end timestamp difference, start sequence number difference, and end sequence number difference; or use the wheel odometer data as a reference to calculate the timestamp difference and sequence number difference between the IMU data and the image data respectively, obtaining the start timestamp difference, end timestamp difference, start sequence number difference, and end sequence number difference.

[0284] (c) Statistically calculate the timestamp difference between the image data and the wheel odometer data and the IMU data. The specific calculation is as follows: The average data timestamp difference of the image data, Delta_camera = (delta_Camera_Start_Stamp + delta_Camera_End_Stamp + (delta_Wheel_Start_Numb + delta_Camera_End_Numb) * 10) / 4, with the unit of milliseconds; the average timestamp difference of the wheel odometer data, Delta_Wheel = (delta_Wheel_Start_Stamp + delta_Wheel_End_Stamp + (delta_Wheel_Start_Numb + delta_Wheel_End_Numb) * 10) / 4.

[0285] (d) Correct the data based on the statistically obtained timestamp difference.

[0286] Specifically, for each data timestamp of the image data, subtract Delta_camera, and for each data timestamp of the wheel odometer data, subtract Delta_Wheel to achieve data correction.

[0287] Furthermore, if a GNSS sensor and / or an RTK sensor are provided on the target object, that is, GNSS data and / or RTK data can be obtained from the database. Considering the sampling frequency of GNSS is 1hz and that of RTK is 50hz, time statistics and synchronization processing of GNSS / RTK are added in the above steps (a), (b), (c), and (d) to obtain the synchronized GNSS / RTK data and its timestamp.

[0288] Step S12: Decode the calibrated compressed image data and perform segmentation processing on the decoded image data to obtain the image data corresponding to each fisheye camera respectively.

[0289] Specifically, decode the compressed image calibrated by the above method, and then perform segmentation processing on the decoded image to obtain the image data corresponding to each path respectively.

[0290] For example, decode the compressed H264 or H265 bitstream, and upsample the decoded data with a resolution of 1920*1280 to restore it to twice the resolution of a single path resolution of 1280*720; segment the upsampled image data of 2560*1440, that is, divide the image data into four parts on average, upper left, upper right, lower left, and lower right, with respective resolutions of 1280*720, to obtain the image data of 4 fisheye cameras. The corresponding relationship is that the upper left corresponds to the front fisheye, the upper right corresponds to the right fisheye, the lower left corresponds to the left fisheye, and the lower right corresponds to the rear fisheye, that is, obtain the image data corresponding to the front fisheye, the image data corresponding to the right fisheye, the image data corresponding to the left fisheye, and the image data corresponding to the rear fisheye respectively.

[0291] Furthermore, since the data volume of each path of image after decoding is large, image compression is performed on each frame of the decoded image to obtain files corresponding to each path respectively. For details, see step S13. Among them, step S13: Encode the image data corresponding to each fisheye camera into files corresponding to each path respectively.

[0292] Among them, the files corresponding to each path contain the image data corresponding to each of them respectively.

[0293] For the embodiments of the present application, encode the image data corresponding to each fisheye camera into JPEG files corresponding to each path respectively, and it can also be png files or other types of files. In the embodiments of the present application, the files corresponding to each path contain the image data corresponding to each of them respectively, and the files of different paths are stored in different directories.

[0294] Furthermore, a unified image timestamp text file can also be generated. The content of each line of text in this file is: the frame number of the image and the timestamp of this frame of image. In the embodiments of the present application, the method of generating a unified image timestamp text file can be executed after step S13, or can be executed simultaneously with step S13.

[0295] Step S14: Generate an IMU file based on the calibrated IMU data, and generate a wheel odometer file based on the calibrated wheel odometer data.

[0296] Among them, the IMU file contains calibrated IMU data, and the wheel odometer file contains calibrated wheel odometer data.

[0297] Specifically, for the wheel odometer data, the parsed and published data is saved as a text file. The specific format of each line is: timestamp, number of teeth of the left rear wheel, number of teeth of the right rear wheel; for the IMU data, the parsed and published data is saved as a text file. The specific format of each line is: timestamp, acceleration values of three axes, angular velocity values of three axes.

[0298] Specifically, if there is GNSS data, the parsed and published data is saved as text data, and the specific format is: timestamp, longitude, latitude, altitude; if there is RTK data, the parsed and published data is saved as text data, and the specific format is: timestamp, longitude, latitude, altitude, and quaternion (representation of direction in 3D space).

[0299] Furthermore, in the embodiment of the present application, step S14 can be executed after step S13, or can be executed before step S12. Of course, it can also be executed simultaneously with step S12 and step S13, and there is no limitation in the embodiment of the present application.

[0300] Specifically, obtaining multi-channel fisheye camera data, wheel odometer data, and IMU data in step S101 can specifically include: obtaining image data in the files corresponding to each channel; and obtaining wheel odometer data in the wheel odometer file; and obtaining IMU data in the IMU file.

[0301] Specifically, obtain the image data corresponding to each channel from the file generated in step S13, obtain the calibrated wheel odometer data from the wheel odometer file generated in step S14, and obtain the calibrated IMU data from the IMU file generated in step S14.

[0302] Furthermore, if the multi-channel fisheye camera data, wheel odometer data, and IMU data are directly obtained from the sensors, it is also necessary to perform timestamp calibration on the obtained data (multi-channel fisheye camera data, wheel odometer data, and IMU data). The specific calibration method is as detailed in the above embodiments and will not be elaborated here.

[0303] Specifically, wheel odometer data and inertial sensing unit (IMU) data are acquired; based on the wheel odometer data and the IMU data, the trajectory information of the target object is determined, which may specifically include: acquiring a wheel odometer file and an IMU file; based on the data in the wheel odometer file and the data in the IMU file, pose calculation is performed to obtain a result sequence of poses; based on the result sequence of poses, the trajectory information of the target object is determined. In the embodiments of the present application, based on the data in the wheel odometer file and the data in the IMU file, pose calculation may specifically include: performing pose calculation based on the data in the wheel odometer file, the data in the IMU file, and the IMU bias value. Wherein, the IMU bias value includes at least one of a default bias value and a bias value set in the static calibration of the IMU.

[0304] Specifically, in a certain time change segment Delta_T, the default value can be set to 100 milliseconds, and its range is from 40 milliseconds to 1 second. Based on the changes of the left and right wheel encoders Wheel(nl,nr), the mileage changes Sr and Sl within the Delta_T time are calculated, and the average of the two can be taken to obtain the vehicle mileage change S=(Sr + Sl) / 2 within the Delta_T time. Considering that the noise influence of the IMU is relatively large in the low-speed state, only the IMU angular velocity data IMU(v_ang_x,v_ang_y,v_ang_z) is used to integrate within the Delta_T time to obtain the angle change amounts IMU(Roll,Pitch,Yaw) in three directions. The target object (for example, a vehicle) moves in the ground 2D space, and the position and attitude change of the vehicle can be determined as Vehicle(S*sin(Yaw),S*cos(Yaw),Yaw). The pose of the vehicle at any moment t, Vehicle_t = Vehicle(S*sin(Yaw),S*cos(Yaw),Yaw)+Vehicle_t-1, that is, the pose at the previous moment plus the pose change within the Delta_T time.

[0305] To further improve the accuracy of the output pose, the wheel speed odometer and the IMU can be more optimally fused to improve the accuracy of the output pose. Specifically, within a certain time change segment Delta_T, the wheel speed odometer calculates the vehicle pose change odometry(x,y,yaw) and its covariance odometry_cov during the Delta_T time based on Wheel(nl,nr,wheel_ang), that is, the left and right wheel encoders and the steering wheel angle change; within the Delta_T time, the IMU obtains the angle change amounts IMU(Roll,Pitch,Yaw) in three directions and its covariance imu_cov by integrating the angular velocity data; based on the extended Kalman filter, the loose coupling method is used to fuse the calculation outputs of the wheel odometer and the IMU to obtain the changing 3D position and 3D attitude.

[0306] Furthermore, the calculation and selection of the IMU bias are performed based on an iterative attempt method. Usually, when the data collection is kilometer-level data collection over a long distance and the starting and ending positions of the data collection are the same, the optimization calculation of the bias is carried out; in other scenarios, the default bias value in the IMU hardware manual or the bias value given in the static calibration of the IMU can be directly used. Specifically, the method for optimizing the calculation of the IMU bias can also include: step S201 (not shown in the figure), step S202 (not shown in the figure), step S203 (not shown in the figure), step S204 (not shown in the figure), and step S205 (not shown in the figure). Among them, in step S201, based on the current IMU bias value, the poses at different times are calculated using the data in the wheel speed odometer file and the data in the IMU file.

[0307] In step S202, the pose at the start point time and the pose at the end point time are obtained from the poses at different times calculated.

[0308] In step S203, an error function is constructed based on the difference between the poses of the loop data collection start and end points, and the error value is calculated.

[0309] In step S204, the IMU bias value is updated based on the error value.

[0310] Specifically, updating the IMU bias value based on the error value may specifically include: the first update is to add a positive bias change to the original IMU bias value, and subsequent updates determine whether to use a positive or negative bias change based on whether the error change direction is increasing or decreasing. If the error value is greater than the error value obtained in the previous calculation, a negative bias change is used; if the error value is less than the error value obtained in the previous calculation, a positive bias change is used. In the embodiments of the present application, the bias change can be characterized by delta_bias, and the default value of delta_bias is 0.001, which can also be other values and is not limited in the embodiments of the present application. Step S205: Loop and execute the step of using the updated IMU bias value as the current IMU bias value, step S201, step S202, step S203, and step S204 until the error value is less than a preset threshold. In the embodiments of the present application, when the error is less than the preset threshold, the latest IMU bias value is obtained, and the threshold is default to 0.5 meters.

[0311] Step S206: Determine the IMU bias value corresponding to when the error value is less than the preset threshold as the optimized IMU bias value.

[0312] Further, after obtaining the optimized IMU bias value, pose calculation is performed based on the data in the wheel odometer file, the data in the IMU file, and the optimized IMU bias value to obtain a pose sequence, and the optimized trajectory information is determined based on the obtained pose sequence and the time stamp.

[0313] Specifically, in step S103, extracting the respective corresponding visual information based on the data of each fisheye camera may specifically include: through the embedded platform, and extracting the respective corresponding visual information based on the data of each fisheye camera.

[0314] Specifically, during the operation of the logistics vehicle, the high-precision map can be constructed offline in the cloud, but visual positioning must be run in real time on the embedded platform. Due to the computing power and resource limitations of the embedded platform, high-consumption tasks such as image scale change, feature point detection, and descriptor calculation need to be implemented by the hard core of the system on chip (SoC) or related acceleration libraries, and most of them use fixed-point operations. There are certain differences between the image scale change, feature points, and descriptor calculation on the embedded platform and the floating-point implementation on the cloud. To ensure the effective operation of map building on the cloud platform and positioning on the embedded platform, the operation results on the embedded platform are used for scale transformation, feature point detection, and descriptor calculation related to map building.

[0315] Among them, visual information corresponding to each path is extracted based on the data of each fisheye camera, which may specifically include: step Sa (not shown in the figure), step Sb (not shown in the figure), step Sc (not shown in the figure), and step Sd (not shown in the figure). Among them, in step Sa, a grayscale image corresponding to each path is generated based on the data of each fisheye camera; an image pyramid corresponding to each path is constructed based on the grayscale image corresponding to each path.

[0316] Specifically, a JPEG or PNG file corresponding to each fisheye camera is obtained, decoded, and a grayscale image is generated; an image pyramid is constructed based on the grayscale image.

[0317] In step Sb, the expected number of feature points required for each layer in the image pyramid is calculated based on the total number of features, the number of pyramid layers, and the scale factor. Usually, the number of feature points can be set to 2000 - 3000, the number of layers is 6 - 8, and the scale factor is 1.2.

[0318] In step Sc, the image of the corresponding layer is segmented into at least one image block based on the expected number of feature points required for each layer, and at least one valid feature point is selected from each image block.

[0319] Specifically, according to the number of target feature points of each layer, the image of this layer is pre - segmented into small image blocks according to a quadtree, feature points are detected by FAST or Harris corner detection within the blocks, and valid feature points are evenly selected within each image block.

[0320] In step Sd, the direction and descriptor corresponding to each valid feature point are calculated, and the direction and descriptor corresponding to at least one valid feature point are determined as the visual information corresponding to this path.

[0321] Specifically, for valid feature points, the direction and Brief descriptor of the feature points are calculated. In the embodiment of the present application, the feature point direction is determined by the gray centroid method; specifically, 360 degrees are divided into 30 quantization angle intervals from 0 degrees to 360 degrees according to the quantization interval delta_Angle = 12 degrees, and the quantization ordinal number m is recorded. The feature point angle is quantized according to the quantization interval delta_ang, and the middle value of each quantization angle interval is taken as the quantization result of the interval. For example, for the quantization interval 0 to 12°, its quantization result is taken as 6; then, according to the quantization result, the patternQ lookup table is searched, the point pair coordinates in the patternQ table are sequentially obtained, and then the image gray values are obtained according to the point pair coordinates, and the brief descriptor is calculated. The following formulas (Formula 1, Formula 2, and Formula 3) are used to determine the coordinates of the quantization point pair corresponding to the descriptor point pair of the target feature point. In the embodiment of the present application, the generation process of the patternQ lookup table may specifically include: storing the quantization result and the 2 quantization coordinates corresponding to the descriptor point pair of the target feature point corresponding to the quantization result in the order of the quantization ordinal number to generate the patternQ lookup table.

[0322] X m = X p * cos(A m ) - Y p * sin(A m ) Formula 1

[0323] Y m = X p * sin(A m ) - Y p * cos(A m ) Formula 2

[0324] Q pmn = X m + Y m * W n Formula 3

[0325] Among them, (X p , Y p ) is the descriptor point pair coordinate of the target feature point, A m is the quantization result, m is the quantization ordinal number, n is the image pyramid ordinal number, W n is the width of the nth layer image, and Q pmn is the quantization coordinate of the point pair coordinate.

[0326] Further, in the embodiments of the present application, features and timestamps of each frame of image in the data of each fisheye camera are calculated based on an embedded platform for subsequent mapping. In the embodiments of the present application, before mapping, it is possible to first determine whether to incrementally map through GNSS and / or RTK data, image features, or user configuration. If incremental mapping is not required, a new map is constructed. If incremental mapping is required, initialization is performed and relocalization, frame tracking, and local optimization are carried out on the map to be updated, optimization of the locked points of the key frames of the map to be updated, optimization of the newly created key frames and points, and the loop of performing initialization and relocalization, frame tracking, and local optimization on the map to be updated, optimization of the locked points of the key frames of the map to be updated, and optimization of the newly created key frames and points is executed until the image feature sequence obtained in step S102 and the pose sequence obtained in step S103 are processed; then the newly constructed map or the map incrementally constructed is adjusted in map size and stored, specifically as Figure 2 shown. Among them, the specific methods for new map construction or incremental mapping can be specifically referred to the following embodiments.

[0327] Further, in the embodiments of the present application, in step S104, map information corresponding to each path is created based on the visual information corresponding to each path and the trajectory information of the target object, which may specifically include: step S1041 (not shown in the figure), step S1042 (not shown in the figure), and step S1043 (not shown in the figure). Among them, in step S1041, based on the trajectory information of the target object and the trajectory information of each map stored in the map database, the current mapping method is determined, or the current mapping method is determined through a preset mapping method.

[0328] Among them, the current mapping method includes: new map construction and incremental mapping. In the embodiments of the present application, new map construction is to construct a new map, and incremental mapping is to create a map incrementally based on the map stored in the VSLAM map library.

[0329] Specifically, in the embodiments of the present application, it is determined whether to perform incremental mapping through the obtained trajectory information of the target object or user configuration.

[0330] Specifically, in the embodiments of the present application, based on the matching of image features with the key frames of each map in the map library, relocalization is performed to obtain the located pose, and it is determined whether the pose coincides with the trajectory of each map. If there is a coincidence and the coincidence length is greater than the threshold, it is determined as incremental mapping; otherwise, a new map is constructed. The determination criterion is the same as the coincidence determination of RTK below.

[0331] Further, since it consumes a lot of resources and takes a long time to make a determination based on image features, it is also possible to determine whether to incrementally build a map according to the user's configuration. If the configuration is clearly to build a new map, directly build a new map; if the configuration is to incrementally build a map, incrementally build the map in the map library that needs to be incrementally built according to the configuration.

[0332] Further, if Global Navigation Satellite System (GNSS) data and / or Real Time Kinematic (RTK) data can be obtained, it is possible to determine whether to incrementally build a map through the GNSS data and / or RTK data. In the embodiments of the present application, the poses of each map in the Vslam map library are retrieved based on the GNSS / RTK pose data. If the GNSS / RTK pose data coincides with the trajectory of a certain map in the map library, and the coincidence length is greater than a given threshold, it is determined to incrementally build a map. The determination criterion for coincidence is the Euclidean distance between the pose and the trajectory. The distance threshold can be adjusted, and the default value is 1 meter; if it is RTK, it can be reduced to 0.5 meter, and if it is GPS, it can be relaxed to 3 to 5 meters. If the Euclidean distance is less than the distance threshold, it is considered to coincide. When two trajectories coincide, the coincidence length is defined as the length of the coincident trajectory part in the map. Among them, if the coincidence length is greater than the length threshold, it is determined to incrementally build a map; if the coincidence length is less than or equal to the length threshold, it is determined to build a new map. Among them, the length threshold can be adjusted, and the default value is set to 5 meters.

[0333] Among them, for the map in the map library that needs to be incrementally built, it is set as the map Y to be updated.

[0334] Step S1042: If the current map building method is to build a new map, based on the visual information corresponding to each path and the trajectory information of the target object, create the map information corresponding to each path through a preset method.

[0335] Specifically, for building a new map, the feature method is used to build the map. The inputs are the image feature sequence output in step S103 and the pose sequence output in step S102. The specific map building method can use ORB2, ORB3 or VINS to build the map. When using ORB2, select the monocular-based map building method to build a new map; when using ORB3, directly build a new map using the image feature sequence and the pose sequence; when using VINS, based on the image feature sequence and the pose sequence, and at the same time modify the sparse optical flow part to feature matching to complete the new map building.

[0336] Further, the new map obtained by the method of new map construction is denoted as map N. If ORB2 is not used, the pose sequence output in step S102 is optimized by PoseGraph with the poses of the key frames of the new map N as constraints to obtain an updated pose sequence.

[0337] Step S1043: If the current mapping method is incremental mapping, map information matching the trajectory information of the target object is obtained from the map database, and incremental mapping is performed based on the visual information corresponding to each path and the trajectory information of the target object on the basis of the matching map information.

[0338] Specifically, incremental mapping is performed based on the visual information corresponding to any path and the trajectory information of the target object on the basis of the matching map information, which may specifically include: step S1043a (not shown in the figure), step S1043b (not shown in the figure), and step S1043c (not shown in the figure), where

[0339] Step S1043a: Initialize and construct a key frame sequence based on the visual information corresponding to any path and the trajectory information of the target object. For each key frame, perform relocalization on the matching map information, and determine the key frame with successful relocalization as the S key frame, and determine the key frame in the matching map information that is closest to the S key frame, denoted as the S_Y key frame. In the embodiments of the present application, the key frame sequence includes multiple key frames, and the first key frame with successful initialization is the I key frame.

[0340] Specifically, perform conventional initialization and construction of the key frame sequence on the image feature sequence output in step S103 and the pose sequence output in step S102. The first key frame with successful initialization is denoted as the I key frame. For the constructed key frame sequence, for each key frame, perform relocalization on the map Y to be updated. If the relocalization is successful, set this key frame as the S key frame, and record the key frame on the map Y to be updated that is closest to the S key frame, denoted as the S_Y key frame.

[0341] Step S1043b, determine the visual frame to be processed and the pose to be processed for tracking processing of mapping, and construct a new key frame, and optimize the local map through BA. In the embodiment of the present application, the next frame image feature in the image feature sequence output by step S103 and the next pose in the pose sequence output by step S102 are processed in chronological order to track the mapping; at the same time, a new key frame is constructed, and the local map is optimized through BA. In the embodiment of the present application, the BA optimization in SLAM first calculates the normalized spatial point coordinates corresponding to the pixel coordinates on the A image according to the pixel coordinates matched by the camera model and the A and B image features, and then calculates the pixel coordinates reprojected to the B image according to the coordinates of the spatial point. The reprojected pixel coordinates (estimated values) and the matched pixel coordinates (measured values) on the B image will not completely overlap. The purpose of BA is to establish equations for each matched feature point, and then combine them to form an overdetermined equation to solve the optimal pose matrix or spatial point coordinates (both can be optimized at the same time).

[0342] Step S1043c, determine whether the newly constructed key frame is successfully relocated on the matching map information; if the relocation is successful, determine that the successfully located key frame is the E key frame, and determine the key frame in the matching map information that is closest to the E key frame, and set it as the E_Y key frame; based on the points between the S_Y key frame and the E_Y key frame, all key frames and points between the S key frame and the E key frame, and all key frames and points between the I key frame and the S key frame, and perform joint optimization processing through BA to obtain the incrementally updated map information.

[0343] Specifically, in the embodiment of the present application, it is determined whether the key frame in the map constructed in step S1043b is successfully relocated on the map to be updated Y. If the relocation is successful, the key frame is set as the E key frame, and the key frame on the map to be updated Y that is closest to the E key frame is recorded as E_Y. The position and posture of the key frame of the map to be updated are locked, and the points between the S_Y key frame and the E_Y key frame on the update map Y, all the key frames and points from the newly constructed S key frame to the E key frame, and all the key frames and points from the newly constructed I key frame to the S key frame are jointly optimized through BA to obtain the update map Y after incremental update.

[0344] Step S1043c, loop execution to determine the next visual frame as the visual frame to be processed, determine the next pose as the pose to be processed, step S1043b, until the preset conditions are met to achieve incremental mapping.

[0345] Specifically, in the embodiment of the present application, steps S1043b and S1043c are repeated until the image feature sequence output by step S103 and the pose sequence output by step S102 are processed, and then the incremental mapping of the map Y to be updated is completed.

[0346] Further, during incremental mapping, if ORB2 is not used, the pose of the key frame of the map Y to be updated is used as a constraint to optimize the pose sequence output in step S102 through PoseGraph to obtain an updated pose sequence.

[0347] Further, if both the map Y to be updated and the newly created map N are pure visual monocular maps based on ORB2, their mapping has scale uncertainty. Therefore, both the newly created map N and the map Y to be updated need to perform scale update. Specifically, it includes: calculating the scale factor based on the wheel speed and the pose trajectory of the IMU and the pose trajectory of the newly created map N or the map Y to be updated, adjusting the poses and points of all key frames in the monocular map based on the calculated scale factor to obtain a new map N or an updated map Y with restored scale, using the pose of the key frame of the new map N or the updated map Y with restored scale as a constraint to optimize the pose sequence output in step S102 through PoseGraph to obtain an updated pose sequence, and then storing the new map N or the updated map Y.

[0348] Further, after creating the map information corresponding to each path through the method of constructing a new map or incremental mapping, generating a composite map based on the map information corresponding to each path in step S105 may specifically include: step S1051 (not shown in the figure), step S1052 (not shown in the figure), and step S1053 (not shown in the figure), where step S1051 generates multi-submap package files corresponding to each path based on the map information corresponding to each path and the preset memory constraint information.

[0349] Specifically, based on the size of the map information and the memory constraint of the target platform for the Simultaneous Localization And Mapping (SLAM) map, calculate the size of the submap and determine the number of submaps. For example, the memory constraint for the SLAM map is 200MB, and the map size is 800MB. To ensure smooth transition of positioning between different submaps, two submaps need to be loaded into memory simultaneously, and there should be an overlapping area between different submaps, with a 10% redundancy between each submap. That is, the size of each submap can be determined to be 100M, but the effective map size is 80MB. Thus, the number of submaps to be generated is determined to be 800 / 80 = 10 submaps, and each submap is 100MB. In the embodiment of the present application, multi-submap package files corresponding to each path are generated based on the size of the submap, the number of submaps, and the map information corresponding to each path.

[0350] Step S1052: Generate the pose trajectory information corresponding to each path based on the multi-submap package files corresponding to each path and the visual information corresponding to each path.

[0351] For the embodiments of the present application, extract the map information corresponding to each path from the multi-submap package files corresponding to each path, and then generate the pose trajectory information corresponding to each path based on the map information corresponding to each path and the visual information corresponding to each path.

[0352] Step S1053: Generate a composite map based on the pose trajectory information corresponding to each path.

[0353] Specifically, select the pose trajectory information corresponding to one path from the pose trajectory information corresponding to each path as the reference pose trajectory information, establish the mapping relationship between the pose trajectory information corresponding to other branches and the reference pose information, and then generate a composite map through the mapping relationship.

[0354] Furthermore, after creating the maps corresponding to each path by the newly constructed method or the incremental mapping method, the self-evaluation and cross-evaluation of the mapping quality of the constructed maps can be performed to determine whether the constructed maps meet the requirements for continuing to generate a composite map. In the embodiments of the present application, the Relative Pose Error (RPE) index, the Absolute Trajectory Error (ATE) index, and the cross-evaluation method are used to determine whether the constructed maps meet the requirements for continuing to generate a composite map. In the embodiments of the present application, the RPE index is used to evaluate the stability of the visual maps established by each path of cameras. The RPE includes rotational error and translational error, which mainly reflect the accuracy and stability of the algorithm. The global consistency between multiple visual pose trajectories is evaluated based on the ATE. The specific evaluation method is described in detail in the following embodiments.

[0355] Furthermore, in the embodiments of the present application, before step S1051 (generate the multi-submap package files corresponding to each path based on the map information corresponding to each path and the preset memory constraint information), it may further include: step Se, step Sf, step Sg, and step Sh, where

[0356] Step Se: Obtain the key frame pose trajectory information corresponding to each path from the map information corresponding to each path.

[0357] Specifically, obtain the key frame pose trajectories of the maps constructed in the above embodiments, denoted as: P1,..., P n ∈SE(3).

[0358] Further, the trajectory information of the target object corresponding to the key frame pose trajectory information (i.e., the trajectory information generated from wheel odometer and IMU data) is set as the true pose, and through Q1, ..., Q n ∈ SE(3), where the subscript n represents the time t. Step Se: Determine the relative pose error RPE corresponding to each visual pose trajectory information respectively.

[0359] For the embodiments of the present application, the RPE relative pose error mainly describes the accuracy of the pose difference between two frames separated by a fixed time difference Δ (compared with the true pose). Therefore, the RPE of the i-th frame is defined as follows:

[0360]

[0361] Given the total number n and the interval Δ, m = n - Δ RPEs can be obtained, and then the root mean square error RMSE is used to statistically analyze this error to obtain an overall value:

[0362]

[0363] where trans(E i ) represents taking the translation part translation in the relative pose error. RPE includes two parts of errors, namely rotation error and translation error. Usually, it is sufficient to evaluate using the translation error, but if necessary, the error of the rotation angle can also be statistically analyzed using the same method.

[0364] Specifically, the pose trajectory of the wheel speed IMU is relatively smooth and stable, and it is used as the true value; calculate the RPE of the 4 visual pose trajectories to obtain rpe i , i = 1, 2, 3, 4.

[0365] Step Sf: Determine the absolute trajectory error ATE corresponding to each visual pose trajectory information respectively.

[0366] Specifically, the ATE of the i-th frame is defined as follows:

[0367]

[0368] Similar to RPE, RMSE can be used to statistically analyze ATE:

[0369]

[0370] Specifically, taking the pose trajectory of the wheel speed IMU as the true value; calculate the ATE of the 4 visual pose trajectories to obtain ate i , i = 1, 2, 3, 4.

[0371] Step Sg: Determine the cross ATE corresponding to each visual pose trajectory information respectively.

[0372] For the embodiments of this application, for each visual pose trajectory, calculate its cross-ATE with other visual pose trajectories. Specifically, the cross-ATE between the visual pose trajectory of the i-th path and the pose trajectory of the j-th path is denoted as cross_ate ij , where i ∈ (1, 2, 3, 4), j ≠ i ∩ j ∈ (1, 2, 3, 4). Sum them up and take the average to obtain the absolute trajectory error of the i-th visual trajectory:

[0373]

[0374] Step Sh: Based on the RPE corresponding to each visual pose trajectory information, the ATE corresponding to each visual pose trajectory information, and the cross-ATE corresponding to each visual pose trajectory information, determine the evaluation results corresponding to each map information.

[0375] Specifically, in the embodiments of this application, the evaluation results corresponding to each map information are determined through formula (6).

[0376] q i = α * rpe i + β * ate i + χ * cross_ate i (6)

[0377] where q i represents the evaluation result corresponding to the i-th map information, and α, β, χ are weighting coefficients, which can be determined according to experience, and the default values are 0.3, 0.3, 0.4. When q i is greater than a given threshold, it is determined that there is a problem with the map. The threshold is determined according to experience, and the default value is 2.0; if q i is not greater than the given threshold, it is determined that there is no problem with the map, and then the subsequent steps are continued.

[0378] Furthermore, in the above embodiments, a strategy of single-layer self-evaluation and multi-layer cross-evaluation is adopted for quality evaluation. In self-evaluation, RPE is used to evaluate stability and ATE is used to evaluate global consistency. In cross-evaluation, ATE is used to evaluate the global consistency between multiple visual pose trajectories. The self-evaluation and cross-evaluation are weighted and fused, so as to solve the problem of automatic evaluation of the mapping quality of multiple different types of sensors.

[0379] Furthermore, step S1051 may specifically include: if the evaluation results corresponding to each map information are not greater than the preset threshold, then based on the map information corresponding to each path and the preset memory constraint information, generate multi-submap package files corresponding to each path. That is, if there is no problem with each map information through the above evaluation, the subsequent steps can be continued.

[0380] Specifically, based on the map information corresponding to any road and the preset memory constraint information, a multi-submap package file corresponding to any road is generated, which may specifically include: step S1051a (not shown in the figure), step S1051b (not shown in the figure), step S1051c (not shown in the figure), and step S1051d (not shown in the figure), where

[0381] Step S1051a: Based on the map information corresponding to any road and the preset memory constraint information, determine the number of submaps corresponding to any road and the effective space occupied by each submap. In the embodiments of the present application, the map information corresponding to any road includes: the storage space occupied by any road map.

[0382] Specifically, after obtaining the storage space occupied by the road map, based on the occupied storage space and the memory constraint available for the SLAM map of the target platform, calculate the size of the submap and determine the number of submaps. For example, the memory constraint for the SLAM map is 200MB, and the map size is 800MB; to ensure smooth transition of positioning between different submaps, two submaps need to be loaded into memory simultaneously, and there needs to be an overlapping area between different submaps, with a 10% redundancy between each submap; it can be determined that the size of each submap is 100M, but the effective map size is 80MB. Thus, it is determined that the number of submaps to be generated is 800 / 80 = 10 submaps, and each submap is 100MB.

[0383] Step S1051b: Based on the effective space occupied by each submap and the map information corresponding to any road map, determine the key frame information corresponding to each submap.

[0384] Among them, the key frame information includes: the starting key frame number, the effective starting key frame number, the effective ending key frame number, and the ending key frame number.

[0385] Step S1051c: Based on the starting key frame number and the ending key frame number, split the map information corresponding to any road to obtain each submap file.

[0386] Further, after obtaining each submap file, store the split submap files.

[0387] Step S1051d: Based on the number of submaps corresponding to any road, the key frame information corresponding to each submap, and each submap file, generate a multi-submap package file corresponding to any road.

[0388] Specifically, based on the number of subgraphs corresponding to any path, the key frame information corresponding to each subgraph, and each subgraph file, a multi-subgraph packet file corresponding to any path is generated, which may specifically include: constructing an index file for the multi-subgraph based on the number of subgraphs corresponding to any path and the key frame information corresponding to each subgraph; generating a multi-subgraph packet file corresponding to any path based on the index file of the multi-subgraph, the number of subgraphs corresponding to any path, the key frame information corresponding to each subgraph, and each subgraph file.

[0389] Further, before step S1051d, it may further include: determining the statistical histogram of each key frame corresponding to any path based on the map information corresponding to any path and through a tape model; determining a relocalization scene recognition file based on the histogram. Further, in the embodiment of the present application, after obtaining the relocalization scene recognition file, the relocalization scene recognition file is stored, where the relocalization scene file is used for searching for similar key frames during global relocalization.

[0390] Further, on this basis, a multi-subgraph packet file corresponding to any path is generated based on the index file of the multi-subgraph, the number of subgraphs corresponding to any path, the key frame information corresponding to each subgraph, and each subgraph file, which may specifically include: packing the index file, the relocalization scene recognition file, and each subgraph file to generate a multi-subgraph packet file for the vision of this path.

[0391] Further, after obtaining the multi-subgraph packet file based on the above method, in step S1052, based on the multi-subgraph packet files respectively corresponding to each path and the vision information respectively corresponding to each path, pose trajectory information respectively corresponding to each path is generated, which may specifically include: decoding the multi-subgraph packet files respectively corresponding to each path to obtain the map files respectively corresponding to each path; performing visual relocalization and tracking localization on the vision information respectively corresponding to each path in their respective corresponding map files to obtain the pose trajectory information respectively corresponding to each path. In the embodiment of the present application, taking four fisheye cameras as an example, the pose trajectory information respectively corresponding to each path includes: the positioning pose trajectory of the front fisheye, the positioning pose trajectory of the rear fisheye, the positioning pose trajectory of the left fisheye, and the positioning pose trajectory of the right fisheye.

[0392] Further, the multi-subgraph packets and positioning pose trajectories of different paths constitute a layer of map; the maps of each layer are independently constructed and stored independently.

[0393] Further, after obtaining the pose trajectory information respectively corresponding to each path, in step S1053, based on the pose trajectory information respectively corresponding to each path, a composite map is generated, which may specifically include: step S10531 (not shown in the figure), step S10532 (not shown in the figure), and step S10533 (not shown in the figure), where

[0394] Step S10531: Construct a primary navigation map based on the preset pose trajectory information.

[0395] Specifically, in the embodiments of the present application, the preset pose trajectory information may be any one of the positioning pose trajectories of the front-view fisheye, the rear-view fisheye, the left-view fisheye, and the right-view fisheye.

[0396] Specifically, the steps of constructing a primary navigation map based on the preset pose trajectory information may specifically include: determining the type information and position information corresponding to each point of interest (POI) to be added; constructing a primary navigation map based on the preset pose trajectory information and the type information and position information corresponding to each POI. In the embodiments of the present application, the case where the preset pose trajectory information is the positioning pose trajectory of the front-view fisheye is taken as an example for introduction.

[0397] Specifically, based on the positioning pose trajectory of the front-view fisheye, add the type and position of the detected POI or the type and position of the manually set POI through the HMI, and store the positioning pose trajectory of the front-view fisheye and the type and position of each POI, then the primary navigation map can be obtained.

[0398] Step S10532: Determine the mapping relationship between the pose trajectory information of other channels and the preset pose trajectory information.

[0399] For the embodiments of the present application, taking the positioning pose trajectory of the front-view fisheye as the reference trajectory, establish the mapping relationship between the pose trajectories of each layer map and the reference trajectory. Specifically, the mapping from a certain layer map trajectory to the front-view fisheye map trajectory may be an overall mapping relationship, and the SIM(3) mapping of the two trajectories is calculated based on the evo tool, that is, the overall translation, rotation, and scale mapping of the two trajectories. Further, the mapping from a certain map trajectory to the positioning pose trajectory of the front-view fisheye may also be a continuous local dynamic mapping. The sliding window mechanism is used to continuously calculate the SIM(3) mapping of the local trajectories of the two trajectories, and this mapping is associated with the position value of the sliding window, thereby obtaining a SIM(3) mapping sequence, which is used to accurately map a certain map trajectory and map points to the trajectory and points in the front-view fisheye map coordinate system.

[0400] Step S10533: Generate a composite map based on the primary navigation map and the mapping relationship.

[0401] For the embodiments of the present application, graph transformation is performed based on the mapping relationship to complete the construction of the composite map. Since the different sensors (different road cameras, RTK) start the mapping initialization at different times, there are differences in the starting poses of the maps established by the different sensors. At the same time, the scales of mapping by different sensors are also inconsistent and unstable. Therefore, in the embodiments of the present application, based on the positioning pose trajectory of the front fisheye camera, a mapping from other pose trajectories to the front fisheye map is established.

[0402] For the embodiments of the present application, the map of the non-front vision camera is transformed based on the mapping relationship. Specifically, the coordinates of all points and all poses in the map need to be transformed. The transformation matrix used for the transformation is the SIM(3) mapping calculated in the above steps, or a sequence of SIM(3) mappings. Through this mapping or sequence of mappings, the trajectory and points of the non-front vision camera map are accurately mapped into the coordinate system of the front fisheye map, completing the fusion and transformation of the coordinate systems. Through the above steps, the transformation of the maps of other road cameras or other sensors is obtained. The transformed maps of all road cameras and other sensors, together with the preliminary navigation map, constitute the composite map. Among them, an example diagram of the composite map is as Figure 3 shown. The front fisheye point cloud map data and its positioning pose trajectory are the core layer of the multi-layer composite map. It is the benchmark for the pose trajectory mapping of each layer of maps such as the left vision camera, right vision camera, rear vision camera, and RTK, and is also the benchmark for the construction of the navigation map. The four fisheye camera layers are essential in the composite map, and the RTK layer is optional, specifically depending on whether the corresponding sensor is equipped.

[0403] Another possible implementation manner of the embodiments of the present application is that the method may further include: step Si (not shown in the figure), step Sj (not shown in the figure), and step Sk (not shown in the figure), where

[0404] Step Si: Generate the trace following trajectory information based on the preset pose trajectory information.

[0405] Specifically, step Si may specifically include at least one of step Si1 (not shown in the figure) and step Si2 (not shown in the figure), where

[0406] Step Si1: Determine the preset pose trajectory information as the trace following trajectory information.

[0407] Step Si2: Generate the trace following trajectory information based on the pose trajectory information of each road after mapping processing.

[0408] Among them, the pose trajectory information of each road after mapping processing is the pose trajectory information obtained by mapping processing based on the preset pose trajectory information.

[0409] That is to say, in the embodiments of the present application, the positioning pose trajectory of the front fisheye can be determined as the tracking trajectory information, or the positioning pose trajectories of the four-way fisheye can be mapped based on the positioning pose trajectory of the front fisheye as a reference, and then the average is taken to generate the tracking trajectory information.

[0410] Step Sj: Obtain the generated top view and generate a virtual boundary line in the generated top view.

[0411] Further, before the step of obtaining the generated top view, it may further include: generating a top view. In the embodiments of the present application, the method for generating a top view may specifically include: obtaining the external parameter matrix of the installation of the multi-way fisheye camera, and based on the external parameter matrix of the installation of the multi-way fisheye camera and through the projection imaging model of the multi-way fisheye camera, determining the inverse perspective transformation IPM of the multi-way fisheye camera; generating a top view according to the multi-way fisheye camera data and the IPM of the multi-way fisheye camera.

[0412] For the embodiments of the present application, according to the projection imaging model of the fisheye camera and the external parameter matrix of the installation of the fisheye camera, the inverse perspective transformation IPM (Inverse Perspective Mapping) of the camera can be determined, and the specific formula (7) is as follows.

[0413]

[0414] Among them, the pixels in the fisheye camera are mapped to the ground with the vehicle center as the coordinate origin, and z is taken as zero. Specifically, π c is the imaging projection model of the fisheye camera, and the Mei model or the Davide Scaramuzzax model can be adopted. Its inverse projection model, [R c t c is the external parameter matrix from the fisheye camera to the vehicle center, [u v] is the pixel position in the image coordinate system of the fisheye camera, and [x y] is the position in the top view coordinate system with the vehicle center as the origin. λ is a scalar, and the default value is 1; col:i refers to taking the i-th column of the matrix. After the inverse projection transformation, the images of the 4-way fisheye camera generate a top view.

[0415] Further, on the top view, road boundary points are obtained based on semantic segmentation; the road boundary points are sampled, and the sampling strategy is to discretely and uniformly collect the road boundary points closest to the vehicle center; for the areas without road boundary points, a sequence of points at a specified distance from the vehicle is sampled to replace them, and the default distance is set to be half of the vehicle width plus 1 meter from the vehicle center.

[0416] Further, in the above embodiments, a tracking trajectory is automatically generated based on the positioning pose trajectory of the front fisheye, and a virtual road boundary is automatically generated based on the positioning pose trajectory of the front fisheye, the road boundary, and the vehicle geometric features, which can solve the problem of automatically generating the navigation trajectory and road constraints for global planning.

[0417] Step Sk: Generate a navigation map based on the primary navigation map, the tracking trajectory, and the virtual boundary line.

[0418] For the embodiments of the present application, a navigation map is generated based on the preliminary navigation layer, the tracking trajectory, and the virtual lane line. It can be used as the basis for HMI display and can also be used for global path planning. Among them, the tracking trajectory for global planning has a definite direction, and its direction is determined by the time sequence of the vehicle's travel during map building, with irreversibility.

[0419] Specifically, an example of the constructed navigation map is Figure 4 as shown, Figure 4 where the white line in it is the virtual boundary line. The color blocks with different shades represent different POIs, such as the North Gate, the East Gate, the main building, and the artificial lake, etc.

[0420] Further, on the basis of the above embodiments, the embodiments of the present application introduce a specific example for generating a high-precision offline map, as Figure 5 shown. The method specifically includes: Step S110, data synchronization and road segment processing. The data source is the data in database 110, mainly completing data synchronization, compressed image decoding, and segmentation; Step S120, trajectory generation based on the wheel odometer and IMU and IMU bias calculation; Step S130, generation based on image scale processing, feature point position detection, and feature point descriptor extraction on the embedded platform; Step S140, automatically perform new map creation, map update, and fusion determination, supporting automatic addition and update of maps for changes in lighting and scenes; Step S150, complete automatic evaluation of the mapping quality of multiple different types of sensors based on single-layer self-evaluation and multi-layer cross-evaluation; Step S160, automatically split the map based on the memory constraint of the target positioning platform and the map size, supporting real-time operation of the map on the target platform with low memory consumption; Step S170, complete map fusion based on the global / local sliding window collaborative mapping mechanism with the visual trajectory of the front fisheye camera as the basis, realizing fast fusion and expansion of maps constructed by multiple different types of sensors; Step S180, automatically generate a tracking trajectory based on the front vision pose trajectory, and automatically generate a virtual road boundary based on the front vision pose trajectory, the road boundary, and the vehicle geometric features, completing the automatic generation of the navigation trajectory and road constraints.

[0421] The above embodiments introduce a map generation method from the perspective of the method flow. The following embodiments introduce a map generation device from the perspective of modules. For details, see the following embodiments.

[0422] An embodiment of the present application provides a map generation device. As Figure 6 shown, the map generation device 60 may specifically include: a first acquisition module 61, a first determination module 62, an extraction module 63, a creation module 64, and a first generation module 65. Among them,

[0423] The first acquisition module 61 is configured to acquire multi-channel fisheye camera data, wheel odometer data, and inertial sensing unit (IMU) data;

[0424] The first determination module 62 is configured to determine the trajectory information of the target object based on the wheel odometer data and the IMU data; the extraction module 63 is configured to extract the visual information corresponding to each path respectively based on the fisheye camera data of each path;

[0425] The creation module 64 is configured to create the map information corresponding to each path respectively based on the visual information corresponding to each path and the trajectory information of the target object;

[0426] The first generation module 65 is configured to generate a composite map based on the map information corresponding to each path respectively.

[0427] In another possible implementation manner of the embodiment of the present application, when the first acquisition module 61 acquires the wheel odometer data and the inertial sensing unit (IMU) data, it is specifically configured to: acquire a wheel odometer file and an IMU file;

[0428] When the first determination module 62 determines the trajectory information of the target object based on the wheel odometer data and the IMU data, it is specifically configured to:

[0429] Perform pose calculation based on the data in the wheel odometer file and the data in the IMU file to obtain a result sequence of poses;

[0430] Determine the trajectory information of the target object based on the result sequence of poses.

[0431] In another possible implementation manner of the embodiment of the present application, the device 60 further includes: a first calculation module, a second acquisition module, a second calculation module, an update module, a loop module, and a second determination module. Among them,

[0432] The first calculation module is configured to calculate the poses at different times based on the current IMU bias value and the data in the wheel speed odometer file and the IMU file;

[0433] The second acquisition module is configured to acquire the pose at the start point time and the pose at the end point time from the poses at different times calculated;

[0434] A second calculation module, configured to construct an error function with the difference in poses between the loop data acquisition start point and the end point, and calculate an error value;

[0435] An update module, configured to update the IMU bias value based on the error value;

[0436] A loop module, configured to loop and execute the following steps: use the updated IMU bias value as the current IMU bias value; based on the current IMU bias value, calculate the poses at different times through the data in the wheel odometer file and the data in the IMU file; obtain the pose at the start point time and the pose at the end point time from the calculated poses at different times; construct an error function with the difference in poses between the loop data acquisition start point and the end point, and calculate an error value; update the IMU bias value based on the error value; until the error value is less than a preset threshold;

[0437] A second determination module, configured to determine the IMU bias value corresponding to when the error value is less than the preset threshold as the optimized IMU bias value.

[0438] Another possible implementation manner of the embodiment of the present application. When the update module updates the IMU bias value based on the error value, it is specifically configured to:

[0439] When the error value is greater than the error value calculated last time, use a negative bias change to reduce the IMU bias value;

[0440] When the error value is less than the error value calculated last time, use a positive bias change to increase the IMU bias value.

[0441] Another possible implementation manner of the embodiment of the present application. When the first determination module 62 calculates the pose based on the data in the wheel odometer file and the data in the IMU file, it is specifically configured to: calculate the pose based on the data in the wheel odometer file, the data in the IMU file, and the IMU bias value;

[0442] Wherein, the IMU bias value includes at least one of the following:

[0443] The default bias value;

[0444] The bias value set in the static calibration of the IMU;

[0445] The updated IMU bias value;

[0446] The optimized IMU bias value.

[0447] Another possible implementation manner of the embodiment of the present application. When the extraction module 63 extracts the respective corresponding visual information based on the data of each fisheye camera, it is specifically configured to: through an embedded platform, and extract the respective corresponding visual information based on the data of each fisheye camera;

[0448] Among them, when the extraction module 63 extracts the corresponding visual information for each path based on the data of each fisheye camera, it is specifically used for:

[0449] Generating a grayscale image corresponding to each path based on the data of each fisheye camera;

[0450] Constructing an image pyramid corresponding to each path based on the grayscale image corresponding to each path;

[0451] Based on the total number of features, the number of pyramid layers, and the scale factor, calculating the expected number of feature points required for each layer in the image pyramid; based on the expected number of feature points required for each layer, dividing the image of the corresponding layer into at least one image block, and selecting at least one valid feature point from each image block;

[0452] Calculating the direction and descriptor corresponding to each valid feature point respectively, and determining the direction and descriptor corresponding to at least one valid feature point as the visual information corresponding to this path.

[0453] In another possible implementation manner of the embodiment of the present application, when the creation module 64 creates the map information corresponding to each path based on the visual information corresponding to each path and the trajectory information of the target object, it is specifically used for:

[0454] Based on the trajectory information of the target object and the trajectory information of each map stored in the map database, determining the current map building method, or determining the current map building method through a preset map building method. The current map building method includes: new map construction and incremental map building;

[0455] When the current map building method is new map construction, based on the visual information corresponding to each path and the trajectory information of the target object, and creating the map information corresponding to each path through a preset method;

[0456] When the current map building method is incremental map building, obtaining the map information matching the trajectory information of the target object from the map database, and performing incremental map building on the basis of the matching map information based on the visual information corresponding to each path and the trajectory information of the target object.

[0457] In another possible implementation manner of the embodiment of the present application, when the creation module 64 performs incremental map building on the basis of the matching map information based on the visual information corresponding to any path and the trajectory information of the target object, it is specifically used for:

[0458] Performing initialization and key frame sequence construction based on the visual information corresponding to any path and the trajectory information of the target object, where the key frame sequence contains multiple key frames, and the first key frame with successful initialization is the I key frame;

[0459] For each key frame, perform relocalization on the matching map information, determine the key frame with successful relocalization as the S key frame, and determine the key frame in the matching map information that is closest to the S key frame, denoted as the S_Y key frame;

[0460] Key frame construction steps: Determine the visual frame to be processed and the pose to be processed for mapping tracking, construct a new key frame, and perform local map optimization through BA;

[0461] Relocalization determination steps: Determine whether the newly constructed key frame is successfully relocalized on the matching map information;

[0462] Key frame determination steps: If the relocalization is successful, determine the key frame with successful localization as the E key frame, and determine the key frame in the matching map information that is closest to the E key frame, denoted as the E_Y key frame;

[0463] Incremental map construction steps: Based on the points between the S_Y key frame and the E_Y key frame, all the key frames and points between the S key frame and the E key frame, and all the key frames and points between the I key frame and the S key frame, and perform joint optimization processing through BA to obtain the incrementally updated map information;

[0464] Loop to execute by determining the next visual frame as the visual frame to be processed, the next pose as the pose to be processed, the key frame construction steps, the relocalization determination steps, the key frame determination steps, and the incremental map construction steps until the preset conditions are met to achieve incremental mapping.

[0465] Another possible implementation manner of the embodiment of the present application is that the first generation module 105 generates a composite map based on the map information respectively corresponding to each path, specifically for: generating multi-submap package files respectively corresponding to each path based on the map information respectively corresponding to each path and the preset memory constraint information; generating pose trajectory information respectively corresponding to each path based on the multi-submap package files respectively corresponding to each path and the visual information respectively corresponding to each path; generating a composite map based on the pose trajectory information respectively corresponding to each path.

[0466] Another possible implementation manner of the embodiment of the present application is that the device 60 further includes: a third acquisition module, a third determination module, a fourth determination module, a fifth determination module, and a sixth determination module, where,

[0467] The third acquisition module is used to acquire the key frame pose trajectory information respectively corresponding to each path from the map information respectively corresponding to each path, and acquire the trajectory information respectively corresponding to each key frame from the trajectory information of the target object;

[0468] A third determination module, configured to determine the relative pose error (RPE) corresponding to each road map information based on the key frame pose trajectory information corresponding to each of them and the trajectory information corresponding to each key frame respectively;

[0469] A fourth determination module, configured to determine the absolute trajectory error (ATE) corresponding to each road map information based on the key frame pose trajectory information corresponding to each of them and the trajectory information corresponding to each key frame respectively;

[0470] A fifth determination module, configured to determine the cross ATE corresponding to each road map information based on the key frame pose trajectory information corresponding to each of them and the trajectory information corresponding to each key frame respectively;

[0471] A sixth determination module, configured to determine the evaluation result corresponding to each road map information based on the RPE corresponding to each road map information, the ATE corresponding to each road map information, and the cross ATE corresponding to each road map information;

[0472] Wherein, when the first generation module 65 generates the multi-submap package file corresponding to each road based on the map information corresponding to each road and the preset memory constraint information, it is specifically configured to: when the evaluation results corresponding to the map information corresponding to each road are not greater than the preset threshold, generate the multi-submap package file corresponding to each road based on the map information corresponding to each road and the preset memory constraint information.

[0473] Another possible implementation manner of the embodiment of the present application is that when the first generation module 65 generates the multi-submap package file corresponding to any one road based on the map information corresponding to any one road and the preset memory constraint information, it is specifically configured to:

[0474] Determine the number of submaps corresponding to any one road and the effective space occupied by each submap based on the map information corresponding to any one road and the preset memory constraint information;

[0475] Determine the key frame information corresponding to each submap based on the effective space occupied by each submap and the map information corresponding to any one road map, where the key frame information includes: the starting key frame number, the effective starting key frame number, the effective ending key frame number, and the ending key frame number;

[0476] Slice the map information corresponding to any one road based on the starting key frame number and the ending key frame number to obtain each submap file;

[0477] Generate the multi-submap package file corresponding to any one road based on the number of submaps corresponding to any one road, the key frame information corresponding to each submap, and each submap file.

[0478] Another possible implementation manner of the embodiment of the present application is that the device 60 further includes: a seventh determination module and an eighth determination module, wherein,

[0479] The seventh determination module is configured to determine the statistical histograms of the key frames corresponding to any path based on the map information corresponding to any path and through a tape model;

[0480] The eighth determination module is configured to determine a relocalization scenario recognition file based on the histogram;

[0481] Among them, when the first generation module 65 generates a multi-submap packet file corresponding to any path based on the number of submaps corresponding to any path, the key frame information corresponding to each submap, and each submap file, it is specifically configured to:

[0482] Construct an index file for multi-submaps based on the number of submaps corresponding to any path and the key frame information corresponding to each submap;

[0483] Generate a multi-submap packet file corresponding to any path based on the index file of multi-submaps, the number of submaps corresponding to any path, the key frame information corresponding to each submap, and each submap file.

[0484] In another possible implementation manner of the embodiment of the present application, when the first generation module 65 generates pose trajectory information corresponding to each path based on the multi-submap packet files corresponding to each path and the visual information corresponding to each path, it is specifically configured to:

[0485] Decode the multi-submap packet files corresponding to each path to obtain the map files corresponding to each path;

[0486] Perform visual relocalization and tracking localization on the visual information corresponding to each path in the map file corresponding to it to obtain the pose trajectory information corresponding to each path.

[0487] In another possible implementation manner of the embodiment of the present application, when the first generation module 65 generates a composite map based on the pose trajectory information corresponding to each path, it is specifically configured to:

[0488] Construct a primary navigation map based on the preset pose trajectory information;

[0489] Determine the mapping relationship between the pose trajectory information of other paths and the preset pose trajectory information, where the preset pose trajectory information is any one of the pose trajectory information corresponding to each path;

[0490] Generate a composite map based on the primary navigation map and the mapping relationship.

[0491] In another possible implementation manner of the embodiment of the present application, when the first generation module 65 constructs a primary navigation map based on the preset pose trajectory information, it is specifically configured to:

[0492] Determine the type information and location information corresponding to each POI to be added;

[0493] Construct a primary navigation map based on the preset pose trajectory information and the type information and location information corresponding to each POI respectively.

[0494] In another possible implementation manner of the embodiment of the present application, the apparatus 60 further includes: a second generation module, a fourth acquisition module, a third generation module, and a fourth generation module, wherein,

[0495] The second generation module is configured to generate a trace following trajectory information based on the preset pose trajectory information;

[0496] The fourth acquisition module is configured to acquire the generated top view;

[0497] The third generation module is configured to generate a virtual boundary line in the generated top view;

[0498] The fourth generation module is configured to generate a navigation map based on the primary navigation map, the trace following trajectory, and the virtual boundary line.

[0499] In another possible implementation manner of the embodiment of the present application, when the second generation module generates the trace following trajectory information based on the preset pose trajectory information, it is specifically configured to at least one of the following:

[0500] Determine the preset pose trajectory information as the trace following trajectory information;

[0501] Generate the trace following trajectory information based on the mapped various pose trajectory information, and the mapped various trajectory information is the pose trajectory information mapped based on the preset pose trajectory information as a reference.

[0502] In another possible implementation manner of the embodiment of the present application, the apparatus 60 further includes: a fifth generation module, wherein,

[0503] The fifth generation module is configured to generate a top view;

[0504] Wherein, when the fifth generation module generates the top view, it is specifically configured to:

[0505] Acquire the external parameter matrix of the installation of multiple fisheye cameras, and determine the inverse perspective transformation (IPM) of the multiple fisheye cameras based on the external parameter matrix of the installation of the multiple fisheye cameras and through the projection imaging model of the multiple fisheye cameras;

[0506] Generate a top view according to the multiple fisheye camera data and the IPM of the multiple fisheye cameras.

[0507] In another possible implementation manner of the embodiment of the present application, the apparatus 60 further includes: a fifth acquisition module, a calibration module, a processing module, an encoding module, and a sixth generation module, wherein,

[0508] A fifth acquisition module, configured to acquire encoded compressed image data, IMU data, and wheel odometer data from a database;

[0509] A calibration module, configured to calibrate the timestamps respectively corresponding to the acquired encoded compressed image data, IMU data, and wheel odometer data;

[0510] A processing module, configured to perform decoding processing on the calibrated compressed image data, and perform segmentation processing on the decoded image data to obtain the image data respectively corresponding to each fisheye camera;

[0511] An encoding module, configured to encode the image data respectively corresponding to each fisheye camera into files respectively corresponding to each path, and the files respectively corresponding to each path contain the image data respectively corresponding to each path;

[0512] A sixth generation module, configured to generate an IMU file based on the calibrated IMU data, and generate a wheel odometer file based on the calibrated wheel odometer data, where the IMU file contains the calibrated IMU data, and the wheel odometer file contains the calibrated wheel odometer data;

[0513] Among them, when the first acquisition module 61 acquires multi-path fisheye camera data, wheel odometer data, and inertial sensing unit IMU data, it is specifically configured to:

[0514] Acquire the image data in the files respectively corresponding to each path; and,

[0515] Acquire the wheel odometer data in the wheel odometer file; and,

[0516] Acquire the IMU data in the IMU file.

[0517] In another possible implementation manner of the embodiment of the present application, when the calibration module calibrates the timestamps respectively corresponding to the acquired encoded compressed image data, IMU data, and wheel odometer data, it is specifically configured to:

[0518] Acquire the preset data respectively corresponding to each sensor, where the preset data respectively corresponding to each sensor includes: the sampling frequency respectively corresponding to each sensor, the timestamps respectively corresponding to each sensor at the data start time, the timestamps respectively corresponding to each sensor at the data end time, the sampling start data sequence number respectively corresponding to each sensor, and the sampling end data sequence number, and each sensor includes: multi-path fisheye cameras, wheel odometers, and IMUs;

[0519] Based on the timestamp of the preset sensor at the data start time as a reference, determine the first difference information between the timestamps of other sensors at the data start time and the timestamp of the preset sensor at the data start time based on a specific relationship;

[0520] Based on the timestamp of the preset sensor at the data termination moment, determine the second difference information between the timestamps corresponding to other sensors at the data end moment and the timestamp corresponding to the preset sensor at the data end moment based on a specific relationship;

[0521] Based on the sampling start data sequence number corresponding to the preset sensor, determine the third difference information between the sampling start data sequence numbers corresponding to other sensors and the sampling start data sequence number corresponding to the preset sensor based on a specific relationship;

[0522] Based on the sampling end data sequence number corresponding to the preset sensor, determine the fourth difference information between the sampling end data sequence numbers corresponding to other sensors and the sampling end data sequence number corresponding to the preset sensor based on a specific relationship;

[0523] Based on the first difference information, the second difference information, the third difference information, and the fourth difference information corresponding to other sensors respectively, determine the timestamp differences between the data of other sensors and the data of the preset sensor based on a specific relationship;

[0524] Based on the timestamp differences between the data of other sensors and the data of the preset sensor respectively, determine the calibration of the corresponding sensor data for each;

[0525] Wherein, the specific relationship is the relationship between the sampling frequencies corresponding to other sensors and the sampling frequency corresponding to the preset sensor.

[0526] Further, in the embodiments of the present application, the first acquisition module 61, the second acquisition module, the third acquisition module, the fourth acquisition module, and the fifth acquisition module may be the same acquisition module, or may be different acquisition modules, or may be partially the same acquisition module, which is not limited in the embodiments of the present application.

[0527] Further, in the embodiments of the present application, the first determination module 62, the second determination module, the third determination module, the fourth determination module, the fifth determination module, the sixth determination module, the seventh determination module, and the eighth determination module may be the same determination module, or may all be different determination modules, or may be partially the same determination module, which is not limited in the embodiments of the present application.

[0528] Further, in the embodiments of the present application, the first generation module 65, the second generation module, the third generation module, the fourth generation module, the fifth generation module, and the sixth generation module may all be the same generation module, or may all be different generation modules, or may be partially the same generation module, which is not limited in the embodiments of the present application.

[0529] Further, in the embodiments of the present application, the first calculation module, the second calculation module, and the third calculation module may be the same calculation module, or may all be different calculation modules, or some may be the same calculation module, which is not limited in the embodiments of the present application.

[0530] The embodiments of the present application provide a map generation device. Compared with the related art, in the embodiments of the present application, by acquiring multi-channel fisheye camera data, wheel odometer data, and inertial sensing unit (IMU) data, and based on the wheel odometer data and IMU data, the trajectory information of the target object is determined, and based on the fisheye camera data of each channel, the visual information corresponding to each channel is extracted respectively. Then, based on the visual information corresponding to each channel and the trajectory information of the target object, the map information corresponding to each channel is created, and then the composite map is generated based on the map information corresponding to each channel. That is, in the present application, it is not necessary to rely on high-precision and high-cost lidar, and the composite map information can be generated only by the data collected by low-cost multi-channel fisheye cameras, wheel odometer data, and IMU data, thereby avoiding the dependence on high-precision lidar for high-precision map construction.

[0531] The embodiments of the present application provide a map generation device applicable to the above method embodiments, which will not be elaborated here.

[0532] In the embodiments of the present application, an electronic device is provided, such as Figure 7 shown. Figure 7 As shown, the electronic device 700 includes: a processor 701 and a memory 703. Among them, the processor 701 and the memory 703 are connected, such as through a bus 702. Optionally, the electronic device 700 may further include a transceiver 704. It should be noted that in practical applications, the transceiver 704 is not limited to one, and the structure of the electronic device 700 does not constitute a limitation to the embodiments of the present application.

[0533] The processor 701 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logical blocks, modules, and circuits described in combination with the disclosure of the present application. The processor 701 may also be a combination that realizes computing functions, such as a combination including one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0534] The bus 702 may include a path for transmitting information between the above components. The bus 702 can be a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus 702 can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity of representation, Figure 7 it is only represented by a thick line in the figure, but it does not mean that there is only one bus or one type of bus.

[0535] The memory 703 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or it can also be an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0536] The memory 703 is used to store the application program code for implementing the solution of this application and is controlled by the processor 701 for execution. The processor 701 is used to execute the application program code stored in the memory 703 to implement the content shown in the foregoing method embodiments.

[0537] Among them, the electronic device includes but is not limited to: mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. It can also be a server, etc. In the embodiments of this application, the server can be a cloud server. Figure 7 The electronic device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of this application.

[0538] Embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored. When it runs on a computer, the computer can execute the corresponding content in the foregoing method embodiments. Compared with the related art, by acquiring multi-channel fisheye camera data, wheel odometer data, and inertial sensing unit (IMU) data, and based on the wheel odometer data and IMU data, the trajectory information of the target object is determined, and visual information corresponding to each channel is extracted respectively based on the multi-channel fisheye camera data. Then, based on the visual information corresponding to each channel and the trajectory information of the target object, map information corresponding to each channel is created, and then a composite map is generated based on the map information corresponding to each channel. That is, in the present application, there is no need to rely on high-precision and high-cost lidar. Only by using the data collected by low-cost multi-channel fisheye cameras, wheel odometer data, and IMU data can composite map information be generated, thereby avoiding the dependence on high-precision lidar for high-precision map construction.

[0539] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0540] In the embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the devices or units can be in an electrical, mechanical, or other form.

[0541] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0542] In addition, in each embodiment of the present application, the functional units may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0543] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories, random access memories, magnetic disks, or optical discs.

[0544] As mentioned above, the above embodiments are only used to introduce the technical solution of the present application in detail. However, the description of the above embodiments is only used to help understand the method and its core idea of the present application, and should not be construed as a limitation to the present application. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application should be covered within the protection scope of the present application.

Claims

1. A method for generating a map, characterized in that, Including: Obtaining data from multiple fisheye cameras, wheel odometer data, and inertial measurement unit (IMU) data; Determining the trajectory information of the target object based on the wheel odometer data and the IMU data; Extracting the respective corresponding visual information from each path of fisheye camera data; Creating the respective corresponding map information for each path based on the respective corresponding visual information for each path and the trajectory information of the target object; Generating a composite map based on the respective corresponding map information for each path; Wherein, generating the composite map based on the respective corresponding map information for each path includes: Generating the respective corresponding multi-submap packet files for each path based on the respective corresponding map information for each path and the preset memory constraint information; Generating the respective corresponding pose trajectory information for each path based on the respective corresponding multi-submap packet files for each path and the respective corresponding visual information for each path; Generating a composite map based on the respective corresponding pose trajectory information for each path; Wherein, before generating the respective corresponding multi-submap packet files for each path based on the respective corresponding map information for each path and the preset memory constraint information, it further includes: Obtaining the respective corresponding key frame pose trajectory information from the respective corresponding map information for each path, and obtaining the trajectory information corresponding to each key frame from the trajectory information of the target object; Determining the relative pose error (RPE) corresponding to each path of map information based on the respective corresponding key frame pose trajectory information and the trajectory information corresponding to each key frame; Determining the absolute trajectory error (ATE) corresponding to each path of map information based on the respective corresponding key frame pose trajectory information and the trajectory information corresponding to each key frame; Determining the cross ATE corresponding to each path of map information based on the respective corresponding key frame pose trajectory information and the trajectory information corresponding to each key frame; Determining the respective corresponding evaluation results for each path of map information based on the RPE corresponding to each path of map information, the ATE corresponding to each path of map information, and the cross ATE corresponding to each path of map information; specifically, determining the respective corresponding evaluation results for each path of map information through the following formula; qi = α * rpe i + β * ate i + χ * cross_ate i ; where q i represents the evaluation result corresponding to the i-th road map information, α, β, χ are weighting coefficients, and rpe i is the relative pose error RPE corresponding to the i-th road map information, and ate i is the absolute trajectory error ATE corresponding to the i-th road map information, and cross_ate i is the cross ATE of the i-th visual pose trajectory with the pose trajectories of other roads; Wherein, generating the respective corresponding multi-submap packet files for each path based on the respective corresponding map information for each path and the preset memory constraint information includes: If the respective corresponding evaluation results for each path of map information are not greater than the preset threshold, then generating the respective corresponding multi-submap packet files for each path based on the respective corresponding map information for each path and the preset memory constraint information.

2. The method according to claim 1, wherein Obtaining wheel odometer data and inertial measurement unit (IMU) data; determining the trajectory information of the target object based on the wheel odometer data and the IMU data includes: Obtaining the wheel odometer file and the IMU file; Performing pose calculation based on the data in the wheel odometer file and the data in the IMU file to obtain a result sequence of poses; determining the trajectory information of the target object based on the result sequence of poses.

3. The method according to claim 2, wherein The method further includes: Based on the current IMU bias value, calculate the poses at different times through the data in the wheel odometer file and the data in the IMU file; obtain the pose at the starting point time and the pose at the ending point time from the calculated poses at different times; Construct an error function with the difference between the poses at the starting and ending points of the loop closure data acquisition, and calculate the error value; Update the IMU bias value based on the error value; Loop and execute: use the updated IMU bias value as the current IMU bias value; based on the current IMU bias value, calculate the poses at different times through the data in the wheel odometer file and the data in the IMU file; obtain the pose at the starting point time and the pose at the ending point time from the calculated poses at different times; construct an error function with the difference between the poses at the starting and ending points of the loop closure data acquisition, and calculate the error value; update the IMU bias value based on the error value; until the error value is less than the preset threshold; Determine the IMU bias value corresponding to when the error value is less than the preset threshold as the optimized IMU bias value.

4. The method according to claim 3, wherein The updating the IMU bias value based on the error value includes: If the error value is greater than the error value obtained in the previous calculation, use a negative bias change to reduce the IMU bias value; If the error value is less than the error value obtained in the previous calculation, use a positive bias change to increase the IMU bias value.

5. The method according to claim 3 or 4, characterized in that, The performing pose calculation based on the data in the wheel odometer file and the data in the IMU file includes: Perform pose calculation based on the data in the wheel odometer file, the data in the IMU file, and the IMU bias value; Wherein, the IMU bias value includes at least one of the following: Default bias value; Bias value set in the static calibration of the IMU; Updated IMU bias value; Optimized IMU bias value.

6. The method according to claim 1, characterized in that The extracting the respective corresponding visual information based on the data of each fisheye camera includes: Through an embedded platform, extract the respective corresponding visual information based on the data of each fisheye camera; Wherein, extracting the visual information corresponding to each path based on the data of each fisheye camera includes: Generate the grayscale image corresponding to each path based on the data of each fisheye camera; Construct the image pyramid corresponding to each path based on the grayscale image corresponding to each path; Calculate the expected number of feature points required for each layer in the image pyramid based on the total number of features, the number of pyramid layers, and the scale factor; Based on the expected number of feature points required for each layer, divide the image of the corresponding layer into at least one image block, and select at least one valid feature point from each image block; Calculate the direction and descriptor corresponding to each valid feature point respectively, and determine the direction and descriptor corresponding to at least one valid feature point as the visual information corresponding to the path.

7. The method according to claim 1, wherein The creating the respective corresponding map information based on the respective corresponding visual information and the trajectory information of the target object includes: Determine the current mapping method based on the trajectory information of the target object and the trajectory information of each map stored in the map database, or determine the current mapping method through the preset mapping method. The current mapping method includes: new map construction and incremental mapping; If the current mapping method is the new map construction, based on the visual information corresponding to each path and the trajectory information of the target object, create the map information corresponding to each path through a preset method; If the current mapping method is the incremental mapping, obtain the map information matching the trajectory information of the target object from the map database, and perform incremental mapping on the basis of the matching map information based on the visual information corresponding to each path and the trajectory information of the target object.

8. The method according to claim 7, characterized in that, Performing incremental mapping on the basis of the matching map information based on the visual information corresponding to any path and the trajectory information of the target object includes: Initialization and key frame sequence construction are performed based on the visual information corresponding to any path and the trajectory information of the target object. Among them, the key frame sequence contains multiple key frames, and the first key frame with successful initialization is the I key frame; For each key frame, perform re-localization on the matching map information, determine the key frame with successful re-localization as the S key frame, and determine the key frame in the matching map information that is closest to the S key frame, denoted as the S_Y key frame; Key frame construction step: Determine the visual frame to be processed and the pose to be processed for mapping tracking processing, construct a new key frame, and perform local map optimization through BA; Re-localization determination step: Determine whether the newly constructed key frame is successfully re-localized on the matching map information; Key frame determination step: If the re-localization is successful, determine the key frame with successful localization as the E key frame, and determine the key frame in the matching map information that is closest to the E key frame, denoted as the E_Y key frame; Incremental map construction step: Based on the points between the S_Y key frame and the E_Y key frame, all the key frames and points between the S key frame and the E key frame, and all the key frames and points between the I key frame and the S key frame, and perform joint optimization processing through BA to obtain the incrementally updated map information; Loop to execute the steps of determining the next visual frame as the visual frame to be processed, determining the next pose as the pose to be processed, the key frame construction step, the re-localization determination step, the key frame determination step, and the incremental map construction step until the preset conditions are met to achieve incremental mapping.

9. The method according to claim 1, characterized in that Generate the multi-submap package file corresponding to any path based on the map information corresponding to any path and the preset memory constraint information, including: Based on the map information corresponding to any path and the preset memory constraint information, determine the number of submaps corresponding to any path and the effective space occupied by each submap; Determine the key frame information corresponding to each sub - map based on the effective space occupied by each sub - map and the map information corresponding to any one of the road maps. The key frame information includes: starting key frame serial number, effective starting key frame serial number, effective ending key frame serial number, and ending key frame serial number; Based on the starting key frame serial number and the ending key frame serial number, split the map information corresponding to any one of the roads to obtain each sub - map file; Generate a multi - sub - map package file corresponding to any one of the roads based on the number of sub - maps corresponding to any one of the roads, the key frame information corresponding to each sub - map, and each sub - map file; 10. The method according to claim 9, wherein Before the step of generating a multi - sub - map package file corresponding to any one of the roads based on the number of sub - maps corresponding to any one of the roads, the key frame information corresponding to each map, and each sub - map file, further includes: Based on the map information corresponding to any one of the roads and through a tape model, determine the statistical histogram of each key frame corresponding to any one of the roads; Determine a relocalization scenario recognition file based on the histogram; Among them, the step of generating a multi - sub - map package file corresponding to any one of the roads based on the number of sub - maps corresponding to any one of the roads, the key frame information corresponding to each sub - map, and each sub - map file includes: Construct an index file of multi - sub - maps based on the number of sub - maps corresponding to any one of the roads and the key frame information corresponding to each sub - map; Generate a multi - sub - map package file corresponding to any one of the roads based on the index file of multi - sub - maps, the number of sub - maps corresponding to any one of the roads, the key frame information corresponding to each sub - map, and each sub - map file; 11. The method according to claim 1, wherein The step of generating pose trajectory information corresponding to each road based on the multi - sub - map package files corresponding to each road respectively and the visual information corresponding to each road respectively includes: Decode the multi - sub - map package files corresponding to each road respectively to obtain the map files corresponding to each road respectively; Perform visual relocalization and tracking localization on the visual information corresponding to each road respectively in the respective corresponding map files to obtain the pose trajectory information corresponding to each road respectively; 12. The method according to claim 1, characterized in that, The step of generating a composite map based on the pose trajectory information corresponding to each road respectively includes: Construct a primary navigation map based on the preset pose trajectory information; Determine the mapping relationship between the pose trajectory information of other roads and the preset pose trajectory information. The preset pose trajectory information is any one of the pose trajectory information corresponding to each road respectively; Generate a composite map based on the primary navigation map and the mapping relationship; 13. The method according to claim 12, wherein The step of constructing a primary navigation map based on the preset pose trajectory information includes: Determine the type information and location information corresponding to each POI to be added respectively; Construct the primary navigation map based on the preset pose trajectory information and the type information and location information corresponding to each POI respectively; 14. The method according to claim 13, characterized in that The method further includes: Generate a tracing trajectory information based on the preset pose trajectory information; Obtain the generated top - view and generate virtual boundary lines in the generated top - view; Generate a navigation map based on the primary navigation map, the tracing trajectory, and the virtual boundary lines; 15. The method according to claim 14, wherein Generating the trace following trajectory information based on the preset pose trajectory information includes at least one of the following: Determining the preset pose trajectory information as the trace following trajectory information; Generating the trace following trajectory information based on the mapped pose trajectory information of each path, where the mapped pose trajectory information of each path is the pose trajectory information obtained by mapping with the preset pose trajectory information as the reference.

16. The method according to claim 14 or 15, characterized in that, Before obtaining the generated top view, it further includes: generating the top view; Among them, the method for generating the top view includes: Obtaining the external parameter matrices of the installation of multiple fisheye cameras, and determining the inverse perspective transformation (IPM) of the multiple fisheye cameras based on the external parameter matrices of the installation of the multiple fisheye cameras and through the projection imaging model of the multiple fisheye cameras; Generating the top view according to the data of the multiple fisheye cameras and the IPM of the multiple fisheye cameras.

17. The method according to claim 1, characterized in that, Before obtaining the data of the multiple fisheye cameras, the wheel odometer data, and the inertial sensing unit (IMU) data, it further includes: Obtaining the encoded compressed image data, IMU data, and wheel odometer data from the database; Calibrating the timestamps corresponding to the obtained encoded compressed image data, IMU data, and wheel odometer data respectively; Performing decoding processing on the calibrated compressed image data, and performing segmentation processing on the decoded image data to obtain the image data corresponding to each fisheye camera respectively; Encoding the image data corresponding to each fisheye camera into files corresponding to each path, where the files corresponding to each path contain the image data corresponding to each of them respectively; Generating an IMU file based on the calibrated IMU data, and generating a wheel odometer file based on the calibrated wheel odometer data, where the IMU file contains the calibrated IMU data, and the wheel odometer file contains the calibrated wheel odometer data; Among them, obtaining the data of the multiple fisheye cameras, the wheel odometer data, and the inertial sensing unit (IMU) data includes: Obtaining the image data in the files corresponding to each path; and, Obtaining the wheel odometer data in the wheel odometer file; and, Obtaining the IMU data in the IMU file.

18. The method according to claim 17, characterized in that, Calibrating the timestamps corresponding to the obtained encoded compressed image data, IMU data, and wheel odometer data respectively includes: Obtaining the preset data corresponding to each sensor, where the preset data corresponding to each sensor includes: the sampling frequency corresponding to each sensor, the timestamps corresponding to each sensor at the data start time, the timestamps corresponding to each sensor at the data end time, the sampling start data sequence number corresponding to each sensor, and the sampling end data sequence number, and each sensor includes: multiple fisheye cameras, wheel odometer, and IMU; Taking the timestamp corresponding to the preset sensor at the data start time as the reference, and determining the first difference information between the timestamps corresponding to other sensors at the data start time and the timestamp corresponding to the preset sensor at the data start time based on a specific relationship; Based on the timestamp of the preset sensor at the data termination moment, determine the second difference information between the timestamps corresponding to other sensors at the data end moment and the timestamp corresponding to the preset sensor at the data end moment based on a specific relationship; Based on the sampling start data sequence number corresponding to the preset sensor, determine the third difference information between the sampling start data sequence numbers corresponding to other sensors and the sampling start data sequence number corresponding to the preset sensor based on a specific relationship; Based on the sampling end data sequence number corresponding to the preset sensor, determine the fourth difference information between the sampling end data sequence numbers corresponding to other sensors and the sampling end data sequence number corresponding to the preset sensor based on a specific relationship; Based on the first difference information, second difference information, third difference information, and fourth difference information corresponding to other sensors respectively, determine the timestamp differences between the data of other sensors and the data of the preset sensor respectively based on a specific relationship; Based on the timestamp differences between the data of other sensors and the data of the preset sensor respectively, determine the calibration of the sensor data corresponding to each; Wherein, the specific relationship is the relationship between the sampling frequencies corresponding to other sensors and the sampling frequency corresponding to the preset sensor.

19. A map generation device, characterized in that, It includes: A first acquisition module, configured to acquire multi-channel fisheye camera data, wheel odometer data, and inertial sensing unit (IMU) data; A first determination module, configured to determine the trajectory information of the target object based on the wheel odometer data and the IMU data; An extraction module, configured to extract the visual information corresponding to each channel respectively based on the multi-channel fisheye camera data; A creation module, configured to create the map information corresponding to each channel respectively based on the visual information corresponding to each channel respectively and the trajectory information of the target object; A first generation module, configured to generate a composite map based on the map information corresponding to each channel respectively; wherein, generating a composite map based on the map information corresponding to each channel respectively includes: generating multi-submap packet files corresponding to each channel respectively based on the map information corresponding to each channel respectively and the preset memory constraint information; generating the pose trajectory information corresponding to each channel respectively based on the multi-submap packet files corresponding to each channel respectively and the visual information corresponding to each channel respectively; generating a composite map based on the pose trajectory information corresponding to each channel respectively; Wherein, before generating the multi-submap packet files corresponding to each channel respectively based on the map information corresponding to each channel respectively and the preset memory constraint information, it further includes: Obtain the key frame pose trajectory information corresponding to each from the map information corresponding to each channel respectively, and obtain the trajectory information corresponding to each key frame from the trajectory information of the target object; Determine the relative pose error (RPE) corresponding to each channel of map information based on the key frame pose trajectory information corresponding to each and the trajectory information corresponding to each key frame respectively; Determine the absolute trajectory error (ATE) corresponding to each channel of map information based on the key frame pose trajectory information corresponding to each and the trajectory information corresponding to each key frame respectively; Determine the cross-ATE corresponding to each road map information based on the respective corresponding key frame pose trajectory information and the trajectory information corresponding to each key frame. Based on the RPE corresponding to each road map information, the ATE corresponding to each road map information, and the cross-ATE corresponding to each road map information, determine the evaluation result corresponding to each road map information; specifically, determine the evaluation result corresponding to each road map information through the following formula. q i = α * rpe i + β * ate i + χ * cross_ate i ; where q i represents the evaluation result corresponding to the i-th road map information, α, β, χ are weighting coefficients, and rpe i is the relative pose error RPE corresponding to the i-th road map information, and ate i is the absolute trajectory error ATE corresponding to the i-th road map information, and cross_ate i is the cross ATE between the i-th visual pose trajectory and the pose trajectories of other roads; Among them, generating multi-submap packet files corresponding to each road based on the map information corresponding to each road and the preset memory constraint information includes: If the evaluation results corresponding to each road map information are not greater than the preset threshold, generate the multi-submap packet files corresponding to each road based on the map information corresponding to each road and the preset memory constraint information.

20. An electronic device, characterized in that, It includes: One or more processors; A memory; One or more applications, where the one or more applications are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs are configured to: execute the map generation method according to any one of claims 1 to 18.

21. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one segment of program, a code set or an instruction set, and the at least one instruction, the at least one segment of program, the code set or the instruction set are loaded and executed by the processor to implement the map generation method according to any one of claims 1 to 18.

Images