Global relocation method and apparatus, electronic device, and computer storage medium

Abstract

Embodiments of the present application provide a global repositioning method and device, electronic equipment and computer storage medium, wherein the global repositioning method comprises: acquiring three-dimensional point cloud data and two-dimensional image data for describing an environment in which a to-be-positioned object is located; determining a global descriptor and feature point cloud data corresponding to the to-be-positioned object according to the three-dimensional point cloud data and the two-dimensional image data, wherein the global descriptor carries overall structure information, local semantic information and image detail descriptor information of the environment; retrieving a first pose corresponding to the global descriptor from a preset global dictionary, wherein the global dictionary stores a correspondence relationship between a plurality of global descriptors and a plurality of poses; and obtaining a second pose of the to-be-positioned object from a preset positioning point cloud map in combination with the feature point cloud data, taking the first pose as an initial pose. Through the embodiments of the present application, accurate global repositioning is achieved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of automatic driving, and in particular to a global repositioning method and device, electronic equipment and computer storage medium. BACKGROUND

[0002] With the development of automation technology, more and more devices can realize automatic driving, such as automatic driving vehicles or automatic driving robots, etc. Because these devices have great advantages in automation degree, driving safety, traffic efficiency, etc., they have become a research hotspot in the industry.

[0003] In the automatic driving technology, high-precision positioning of these devices (such as vehicles or robots, etc.) is crucial for normal automatic driving. And this high-precision positioning relies on global repositioning of the device, that is, the process of determining the initial pose of the device within the global map range in the start-up stage of the device. Then, based on this pose, continuous high-precision positioning of the device can be performed.

[0004] At present, a commonly used global repositioning method is to reposition with an external positioning source, for example, to use GNSS (Global Navigation Satellite System) as a positioning source for global repositioning. GNSS can provide global pose in the Earth coordinate system, with an accuracy of up to meters. However, GNSS may have unstable or no signal in scenarios such as buildings, viaducts, tunnels, etc., resulting in inaccurate or even failed global repositioning. SUMMARY

[0005] Therefore, embodiments of the present application provide a global repositioning scheme to at least partially solve the above problems.

[0006] According to a first aspect of embodiments of the present application, a global repositioning method is provided, comprising: acquiring three-dimensional point cloud data and two-dimensional image data for describing an environment in which a to-be-positioned object is located; determining a global descriptor and feature point cloud data corresponding to the to-be-positioned object according to the three-dimensional point cloud data and the two-dimensional image data, wherein the global descriptor carries overall structure information, local semantic information and image detail descriptor information of the environment; retrieving a first pose corresponding to the global descriptor from a preset global dictionary, wherein the global dictionary stores a correspondence relationship between a plurality of global descriptors and a plurality of poses; and obtaining a second pose of the to-be-positioned object from a preset positioning point cloud map in combination with the feature point cloud data, taking the first pose as an initial pose.

[0007] According to a second aspect of the embodiments of the present application, a global repositioning device is provided, comprising: an acquisition module configured to acquire three-dimensional point cloud data and two-dimensional image data describing an environment in which a to-be-positioned object is located; a determination module configured to determine a global descriptor and feature point cloud data corresponding to the to-be-positioned object according to the three-dimensional point cloud data and the two-dimensional image data, wherein the global descriptor carries overall structure information, local semantic information and image detail descriptor information of the environment; a retrieval module configured to retrieve a first pose corresponding to the global descriptor from a preset global dictionary, wherein the global dictionary stores a correspondence relationship between a plurality of global descriptors and a plurality of poses; and a positioning module configured to obtain a second pose of the to-be-positioned object from a preset positioning point cloud map by taking the first pose as an initial pose and combining the feature point cloud data.

[0008] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus; the memory is configured to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the global repositioning method according to the first aspect.

[0009] According to a fourth aspect of the embodiments of the present application, a computer storage medium is provided, and the computer storage medium stores a computer program, and the program is executed by a processor to implement the global repositioning method according to the first aspect.

[0010] According to the global repositioning scheme provided in the embodiments of the present application, when repositioning the object to be positioned, the three-dimensional point cloud data and the two-dimensional image data of the environment where the object to be positioned is located are considered simultaneously, wherein the three-dimensional point cloud data is not sensitive to light, seasonal change and the like, but lacks local detailed information, and the two-dimensional image data carries rich image detailed information, but is susceptible to light, seasonal change and the like. Combining the two can effectively make up for the respective shortcomings of the two. Further, based on this, the embodiments of the present application are based on the three-dimensional point cloud data and the two-dimensional image data to extract the overall structure information, the local semantic information and the image detailed descriptor information of the environment where the object to be positioned is located, and realize multi-level information acquisition from the overall to the local and then to the details based on multi-source data. The global descriptors in the global dictionary also carry the overall structure information, the local semantic information and the image detailed descriptor information, and have corresponding poses. Then, according to the global descriptor corresponding to the object to be positioned and the global dictionary, the coarse positioning of the global repositioning of the object to be positioned can be realized. Further, taking the coarse positioning result, i.e., the first pose, as the initial pose, and combining the feature point cloud data of the object to be positioned, the second pose of the object to be positioned can be obtained from the preset positioning point cloud map. Since the positioning point cloud map is used for high-precision positioning, the second pose determined based on the same is also a high-precision pose. In this way, the global repositioning from coarse positioning to fine positioning is realized. Moreover, the global repositioning does not depend on GNSS, and will not cause the global repositioning to be inaccurate or even fail due to unstable or no signal, and realizes accurate global repositioning. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments described in the embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0012] Figure 1A A step flow chart of a global repositioning method according to the embodiment one of the present application;

[0013] Figure 1B A schematic diagram of an equal division partitioning method in the embodiment shown in Figure 1A

[0014] Figure 1C A schematic diagram of a global descriptor in the embodiment shown in Figure 1A

[0015] Figure 1D A schematic diagram of a global repositioning process in the embodiment shown in Figure 1A ​​​

[0016] Figure 2 FIG. 1 is a structural block diagram of a global repositioning device according to an embodiment of the present application;

[0017] Figure 3 FIG. 2 is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the personnel in the art better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art should belong to the scope of protection of the embodiments of the present application.

[0019] The specific implementation of the embodiments of the present application will be further described below in conjunction with the drawings of the embodiments of the present application.

[0020] Referring to Figure 1A FIG. 3 is a step flowchart of a global repositioning method according to an embodiment of the present application.

[0021] The global repositioning method of the present embodiment includes the following steps:

[0022] Step S101: Obtain three-dimensional point cloud data and two-dimensional image data for describing the environment in which the object to be positioned is located.

[0023] In the present embodiment, the object to be positioned can be any object with automatic driving function, including but not limited to an autonomous vehicle, an automatic driving robot, etc. It should be noted that in the present application, an autonomous vehicle is taken as an example, but those skilled in the art should understand that other objects with automatic driving function can also be implemented according to the description in the embodiments of the present application to achieve corresponding global repositioning.

[0024] When the object to be positioned is performing global repositioning, it will first collect data of the environment in which it is located, such as a road environment or a parking environment, etc. In a conventional manner, three-dimensional point cloud data is collected, while in the embodiments of the present application, not only three-dimensional point cloud data is collected, but also two-dimensional image data is collected.

[0025] For example, an autonomous vehicle collects three-dimensional point cloud data through a three-dimensional laser radar LIDAR carried by the autonomous vehicle, and collects two-dimensional image data through a camera carried by the autonomous vehicle. The 3D point cloud data collected by the LIDAR is robust to light and seasonal changes; the 2D image data collected by the camera carries rich image detail information.

[0026] Step S102: determining a global descriptor and feature point cloud data corresponding to the object to be positioned according to the three-dimensional point cloud data and the two-dimensional image data.

[0027] The global descriptor carries the overall structure information, local semantic information and image detail descriptor information of the environment.

[0028] In an available manner, the determination of the global descriptor corresponding to the object to be positioned can be implemented as follows: converting the three-dimensional point cloud data and the two-dimensional image data to a reference coordinate system to generate corresponding reference point cloud data and reference image data, the reference coordinate system having a Z-axis direction determined based on the ground on which the object to be positioned is located, a heading angle direction determined around the Z-axis direction, and a target orientation of the object to be positioned on the ground; in the reference coordinate system, setting Nr equal divisions along the target orientation and Ns equal divisions along the heading angle direction, and dividing the reference point cloud data into Nr×Ns equal divisions; and generating overall structure information, local semantic information and image detail descriptor information of the environment in which the object to be positioned is located according to the reference point cloud data and the reference image data. The X-axis direction can be a front direction of the object to be positioned, and the Z-axis can be an axis perpendicular to the ground and pointing to the sky direction.

[0029] Since the three-dimensional point cloud data and the two-dimensional image data are usually in different coordinate systems, when the information of both needs to be comprehensively considered, they need to be converted to the same coordinate system, i.e., the reference coordinate system. Alternatively, the reference coordinate system can be the coordinate system of the three-dimensional point cloud data, so that only the two-dimensional image data needs to be converted, thereby reducing the cost of coordinate conversion. However, the reference coordinate system can also be other coordinate systems, such as the IMU (Inertial Measurement Unit) coordinate system used by an autonomous vehicle, and the present application is not limited in this regard.

[0030] In the reference coordinate system, the reference point cloud data converted from the three-dimensional point cloud data is partitioned, and the subsequent global descriptor generation process is based on the partitioning, thereby reducing the complexity of data processing and effectively ensuring the accuracy of the data of each partition and the accuracy of the overall global descriptor.

[0031] An exemplary equal division partitioning manner is as follows: Figure 1BAs shown, the LIDAR collects 3D point cloud data, and the Camera collects 2D image data; the 2D image data collected by the Camera is converted to the LIDAR coordinate system, and the 3D point cloud data based on the LIDAR coordinate system is the reference point cloud data, and the 2D image data is the reference image data; then, in the LIDAR coordinate system, the reference point cloud data is divided into Nr equal parts along the X axis, and the reference point cloud data is divided into Ns equal parts along the yaw angle (heading angle). Thus, the reference point cloud data is divided into Nr x Ns equal parts, and each equal part is a partition. Figure 1B In the figure, only two equal parts are simply shown, i.e., the equal part represented by the diagonal line and the equal part represented by the horizontal line. However, those skilled in the art should understand that, according to the above manner, Nr x Ns equal parts corresponding to the reference point cloud data can be obtained.

[0032] wherein the specific settings of Nr and Ns can be appropriately set by those skilled in the art according to actual needs. Alternatively, Nr can be set to be divided every 5M, Ns can be set to be divided into 60 equal parts, etc., and the embodiments of the present application do not limit this.

[0033] Further, based on the divided equal partitions, as well as the reference point cloud data and the reference image data, a global descriptor can be generated, including the overall structure information of the environment where the object to be positioned is located, the local semantic information, and the image detail descriptor information.

[0034] In the following, the processes of generating the overall structure information, the local semantic information, and the image detail descriptor information described above are described respectively, but those skilled in the art should understand that, in actual applications, those skilled in the art can generate corresponding information by using part of the manners, and other information can be generated by using other manners; of course, all of them can be generated by using the manners described in the embodiments of the present application. However, whether part of the manners or all of the manners are used, they should fall within the protection scope of the embodiments of the present application.

[0035] (I) Generating the overall structure information of the environment where the object to be positioned is located

[0036] including: for each of the Nr x Ns equal parts, taking the Z-axis maximum value of the reference point cloud data in the equal part as the value of the equal part, wherein the Z-axis is the axis perpendicular to the ground and pointing to the sky direction; according to the Z-axis maximum value of each equal part, forming an Nr x Ns-dimensional two-dimensional matrix, and taking the two-dimensional matrix to represent the overall structure information of the environment where the object to be positioned is located.

[0037] For example, in each of the Nr x Ns equal parts, the maximum value (denoted as Z value) of all the reference point cloud data on the Z-axis is taken as the value of the equal part, thereby composing a two-dimensional matrix, and the matrix records the overall structure information of the environment where the object to be positioned is located.

[0038] The two-dimensional matrix formed by the Z value as the data element based on the Nr×Ns, retains the equal division information and the Z value information, so that the information of the point cloud data in the three-dimensional coordinate system is completely retained.

[0039] (ii) generating local semantic information of an environment in which the object to be positioned is located

[0040] comprises: for each equal division in the Nr×Ns equal divisions, based on the reference point cloud data in the equal division, extracting semantic information in the equal division according to a preset target object category; and generating local semantic information of an environment in which the object to be positioned is located according to the semantic information of each equal division. Extracting semantic information according to Nr×Ns equal divisions makes the obtained semantic information more accurate, and also has a good corresponding relationship with Nr×Ns equal divisions, which facilitates subsequent data processing.

[0041] Wherein, the target object category can be appropriately set by those skilled in the art according to actual needs, for example, pedestrians, other vehicles, road facilities, rod-shaped facilities (such as power poles, traffic light poles, etc.), etc., and the embodiments of the present application do not limit this. The specific extraction process of semantic information can also be realized by those skilled in the art according to actual needs in an appropriate manner, for example, neural network model, etc., and the embodiments of the present application do not limit this.

[0042] Further optionally, after the semantic information in the equal division corresponding to each target object category is encoded in a 0-1 coding manner to generate a 0-1 coding sequence corresponding to the equal division with a length of the number S of target object categories, in this case, the generating local semantic information of an environment in which the object to be positioned is located according to the semantic information of each equal division comprises: generating a three-dimensional 0-1 array of Nr×Ns×S corresponding to the reference point cloud data according to the 0-1 coding sequence corresponding to each equal division, and using the array to represent the local semantic information of the environment in which the object to be positioned is located. Through the sequence coding manner, different semantic information can be effectively represented, and the processing of semantic information is simple and efficient.

[0043] For example, the semantic information of the reference point cloud data in each of the Nr x Ns partitions can be extracted in a manner such as PointNet, PointNet++, PointCNN, RSNet, or FuseSeg, that is, the extracted semantic information is also divided in the Nr x Ns partition manner. The semantic information is counted in the partition in the Nr x Ns partition, and the semantic information is encoded in a 0-1 encoding manner. For example, in the case where s target object categories are set, there are a total of 1, 2... i... s, a total of S semantic information, and in a certain partition, if the i-th semantic information exists, it is recorded as 1, otherwise it is recorded as 0. Taking an example in which the target object includes three categories, pedestrians, other vehicles, and power poles, there are correspondingly 3 kinds of semantic information. If the semantic information corresponding to the first partition indicates that there are pedestrians in the partition but no other target objects, the semantic information corresponding to the first partition can be 【1, 0, 0】, where the first element in the brackets 【】 is used to indicate whether there are pedestrians, the second element is used to indicate whether there are other vehicles, and the third element is used to indicate whether there are power poles. As can be seen from the above semantic information 【1, 0, 0】, there are only pedestrians in the first partition, and there are no vehicles and power poles.

[0044] The S semantic information is finally aggregated into a 0-1 encoding sequence with a length of S in each partition. The semantic information corresponding to the entire reference point cloud data will be aggregated into a three-dimensional 0-1 array with dimensions of Nr x Ns x S, and this array records the local semantic information of the environment in which the object to be positioned is located.

[0045] (Three) Generating image detail descriptor information of the environment in which the object to be positioned is located

[0046] It includes: extracting feature points in the reference image data and detail descriptors corresponding to the feature points; projecting the feature points into the reference point cloud data to determine the partition to which the feature points belong in the Nr x Ns partition; and generating image detail descriptor information of the environment in which the object to be positioned is located according to the detail descriptors corresponding to the feature points in each partition in the Nr x Ns partition. The detail descriptors obtained based on the reference image data effectively record the detail information of the image, and can be an effective supplement to the information recorded by the point cloud data. The specific implementation of extracting feature points and feature point detail descriptors from the reference image data can be implemented by a person skilled in the art according to actual needs in any appropriate manner, including but not limited to neural network, feature point extraction algorithm, etc.

[0047] For example, the feature points and their corresponding detail descriptors can be extracted for the reference image data corresponding to the 2D image data collected by the Camera, and the feature point positions can be projected into the 3D point cloud using the Camera-LIDAR extrinsic parameters. By using the Nr x Ns equal division method, the detail descriptors can be stored in the corresponding equal division, and the extraction and recording of the detail descriptors are completed.

[0048] An overall description is shown in Figure 1C As shown in Figure 1C As can be seen from the above, the Camera collects 2D image data of the environment where the autonomous vehicle is located, and the LIDAR collects 3D point cloud data of the environment where the autonomous vehicle is located; then, the 2D image data is converted to the coordinate system where the 3D point cloud data is located; further, the 3D point cloud data and the 2D image data in the coordinate system are processed as described above to obtain the overall structure information, the local semantic information and the image detail descriptor information of the environment where the autonomous vehicle is located; the overall structure information, the local semantic information and the image detail descriptor information are all based on the Nr x Ns equal division, and the data in each part of the overall description generated thereby has both rich and comprehensive information and good coupling.

[0049] In the above manner, the data collected by the multi-source sensors LIDAR and Camera is information-extracted to obtain the overall structure information, the local semantic information and the image detail descriptor information of the environment where the object to be positioned is located. Through the LIDAR-Camera extrinsic parameters, these information is put into a unified overall description in a tightly coupled manner. Since the information from the overall to the local and then to the details in the environment where the object to be positioned is recorded, the overall description describes the environment more comprehensively, and thus robust global repositioning can be achieved.

[0050] In addition, in this step, the feature point cloud data of the object to be positioned is also extracted, such as by a neural network model or by a feature extraction algorithm to extract the feature point cloud data. The feature point cloud data can effectively represent the object to be positioned to provide a more accurate basis for subsequent point cloud matching.

[0051] Step S103: retrieving a first pose corresponding to the overall description from a preset global dictionary.

[0052] The global dictionary stores the correspondence between a plurality of overall descriptions and a plurality of poses.

[0053] In this embodiment, the plurality of global descriptors in the global dictionary can be generated in advance by extracting overall structure information, local semantic information and image detail descriptor information from different environments. The extraction of overall structure information, the extraction of local semantic information and the extraction of image detail descriptor information can be implemented in the manner described above in step S102, and will not be described again here.

[0054] The pose in the global dictionary can be generated based on the pose of the three-dimensional point cloud data in the GLOBAL coordinate system. That is, three-dimensional point cloud data and two-dimensional image data are collected for an object in an environment at a time. After the three-dimensional point cloud data and the two-dimensional image data are processed, the corresponding global descriptor can be generated, and the pose of the object at the time of data collection is known. Based on this, a corresponding relationship between the pose of the object and the global descriptor of the environment in which the object is located can be established. By performing the above processing on a plurality of objects and various environments in which the objects are located, a plurality of corresponding relationships are obtained, and a corresponding global dictionary is generated.

[0055] Based on this, after obtaining the global descriptor of the object to be positioned, it can be matched with the global descriptor in the global dictionary, and the coarse pose of the object to be positioned, i.e., the first pose, can be determined according to the pose corresponding to the global descriptor in the global dictionary matched.

[0056] Step S104: Obtain the second pose of the object to be positioned from the pre-set positioning point cloud map in combination with the feature point cloud data, taking the first pose as the initial pose.

[0057] The positioning point cloud map provides a comprehensive scanning data of the environment in which the object to be positioned is located, and the data sampling rate of the positioning point cloud map is generally high (such as 0.1 m), which can be used for high-precision positioning and provide high-precision pose (cm level).

[0058] In one possible way, the positioning point cloud map for high-precision positioning can be established by those skilled in the art in a conventional way, and the process is briefly described as follows:

[0059] (1) Use a collection vehicle / robot (hereinafter referred to as an autonomous driving application scenario, and a vehicle as an application carrier, but other application scenarios can also be equally applicable) equipped with high-precision positioning equipment and sensors (LIDAR / Camera, etc.) to scan the target environment area to obtain mapping data, i.e., 3D point cloud data (in this example, the body coordinate system is the LIDAR coordinate system) collected by LIDAR, 2D image data collected by Camera, etc., and the corresponding pose.

[0060] (2) Using the pose to convert the 3D point cloud data into the GLOBAL coordinate system, and splicing to obtain a dense point cloud.

[0061] (3) After bundle adjustment, dynamic object filtering, downsampling, etc., a final positioning point cloud map is obtained.

[0062] Based on the positioning point cloud map, a corresponding second pose (fine pose) can be obtained by taking the feature point cloud data as the registration point, and taking the first pose (coarse pose) obtained by the global dictionary as the initial pose. That is, taking the feature point cloud data as the registration point, determining the pose in the positioning point cloud map that can be registered with the initial pose, and determining the registered pose as the second pose of the object to be positioned.

[0063] Thus, the coarse-to-fine global repositioning of the object to be positioned is realized.

[0064] A process of global repositioning is shown in Figure 1D As can be seen from Figure 1D It can be seen that:

[0065] First, a global dictionary for global coarse positioning and a positioning point cloud map for global fine positioning are obtained.

[0066] The global dictionary is a set of global descriptor-pose correspondences, where the pose is the pose of the LIDAR sensor in the GLOBAL coordinate system, and the global descriptor is a global descriptor generated in advance by the method described in step S102.

[0067] Then, the autonomous vehicle uses the LIDAR sensor and the Camera sensor to collect 3D point cloud data and 2D image data of the environment it is in at the current pose, and extracts the corresponding current global descriptor, and extracts the current feature point cloud data of the autonomous vehicle.

[0068] Next, the extracted current global descriptor is used to search in the global dictionary, and the global descriptor with the closest Euclidean distance to the current global descriptor is retrieved, and its corresponding pose is the global coarse pose (first pose).

[0069] Finally, using the global coarse pose as the initial pose, the current feature point cloud data is matched with the positioning point cloud map to obtain the fine pose (second pose) of the autonomous vehicle.

[0070] Thus, a robust global repositioning based on the global dictionary and the positioning point cloud map is realized.

[0071] Through the embodiment, when globally repositioning the object to be positioned, the three-dimensional point cloud data and the two-dimensional image data of the environment where the object to be positioned is located are considered at the same time, the three-dimensional point cloud data is not sensitive to light, seasonal change and the like, but lacks local detailed information, and the two-dimensional image data carries rich image detailed information, but is susceptible to light, seasonal change and the like. Combining the two can effectively make up for the respective shortcomings of the two. Further, based on this, based on the respective characteristics of the three-dimensional point cloud data and the two-dimensional image data, the overall structure information, the local semantic information and the image detailed descriptor information of the environment where the object to be positioned is located are extracted based on the two, and multi-level information acquisition from the overall to the local and then to the details based on multi-source data is realized. The global descriptor in the global dictionary also carries a plurality of overall structure information, local semantic information and image detailed descriptor information, and has a corresponding pose. Then, according to the global descriptor corresponding to the object to be positioned and the global dictionary, coarse positioning of the global repositioning of the object to be positioned can be realized. Further, taking the coarse positioning result, i.e., the first pose, as an initial pose, and combining the feature point cloud data of the object to be positioned, the second pose of the object to be positioned can be obtained from the preset positioning point cloud map. Because the positioning point cloud map is used for high-precision positioning, the second pose determined based on the positioning point cloud map is also a high-precision pose. Thus, the global repositioning from coarse positioning to fine positioning is realized. Moreover, the global repositioning does not depend on GNSS, and will not cause the global repositioning to be inaccurate or even fail due to unstable or no signal, and realizes accurate global repositioning.

[0072] The global repositioning method of the embodiment can be executed by any appropriate electronic device with data processing capability, including but not limited to: an electronic device that can be carried in an automatic driving device, or a server, a mobile terminal (such as a mobile phone, a PAD, etc.), a PC, etc.

[0073] Reference Figure 2 , a structural block diagram of a global repositioning device according to Embodiment Two of the present application is shown.

[0074] The global repositioning apparatus of the embodiment comprises: an acquisition module 201, configured to acquire three-dimensional point cloud data and two-dimensional image data for describing an environment in which an object to be positioned is located; a determination module 202, configured to determine a global descriptor and feature point cloud data corresponding to the object to be positioned according to the three-dimensional point cloud data and the two-dimensional image data, wherein the global descriptor carries overall structure information, local semantic information and image detail descriptor information of the environment; a retrieval module 203, configured to retrieve a first pose corresponding to the global descriptor from a preset global dictionary, wherein the global dictionary stores a corresponding relationship between a plurality of global descriptors and a plurality of poses; and a positioning module 204, configured to take the first pose as an initial pose, and obtain a second pose of the object to be positioned from a preset positioning point cloud map in combination with the feature point cloud data.

[0075] Optionally, the determination module 202 is configured to convert the three-dimensional point cloud data and the two-dimensional image data to a reference coordinate system to generate corresponding reference point cloud data and reference image data; divide the reference point cloud data into Nr×Ns divisions in the reference coordinate system, along an X-axis direction and an azimuth direction, wherein the X-axis direction is a forward direction of the object to be positioned and a direction in which a right hand points; and generate overall structure information, local semantic information and image detail descriptor information of the environment in which the object to be positioned is located according to the reference point cloud data, the reference image data and the Nr×Ns divisions.

[0076] Optionally, when the determination module 202 generates the overall structure information of the environment in which the object to be positioned is located according to the reference point cloud data, the reference image data and the Nr×Ns divisions: for each division in the Nr×Ns divisions, takes a Z-axis maximum value of the reference point cloud data in the division as a value of the division, wherein the Z-axis is an axis direction perpendicular to the ground and pointing to the sky; and forms an Nr×Ns two-dimensional matrix according to the Z-axis maximum value of each division, so as to represent the overall structure information of the environment in which the object to be positioned is located by the two-dimensional matrix.

[0077] Optionally, when the determination module 202 generates the local semantic information of the environment in which the object to be positioned is located according to the reference point cloud data, the reference image data and the Nr×Ns divisions: for each division in the Nr×Ns divisions, extracts semantic information in the division according to a preset target object category based on the reference point cloud data in the division; and generates the local semantic information of the environment in which the object to be positioned is located according to the semantic information of each division.

[0078] Optionally, the determining module 202 encodes the semantic information corresponding to each target object category in each of the divisions by using a 0-1 encoding method to generate a 0-1 encoding sequence with a length of S corresponding to the number of target object categories in each of the divisions after determining the semantic information in each of the divisions based on the reference point cloud data in the division and the preset target object category; when generating the local semantic information of the environment where the object to be positioned is located according to the semantic information of each division, the determining module 202 generates a three-dimensional 0-1 array of Nr×Ns×S corresponding to the reference point cloud data according to the 0-1 encoding sequence corresponding to each division, and uses the array to represent the local semantic information of the environment where the object to be positioned is located.

[0079] Optionally, when generating the image detail descriptor information of the environment where the object to be positioned is located according to the reference point cloud data, the reference image data, and the Nr×Ns divisions, the determining module 202 extracts feature points in the reference image data and detail descriptors corresponding to the feature points; projects the feature points into the reference point cloud data to determine the division to which the feature points belong in the Nr×Ns divisions; and generates the image detail descriptor information of the environment where the object to be positioned is located according to the detail descriptors corresponding to the feature points in each of the Nr×Ns divisions.

[0080] Optionally, the positioning module 204 is configured to determine a pose that can be registered with the initial pose in the positioning point cloud map by taking the feature point cloud data as a registration point, and determine the registered pose as the second pose of the object to be positioned.

[0081] The global repositioning apparatus of the embodiment is configured to implement the corresponding global repositioning method in the foregoing method embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be described herein again. In addition, the functions of each module in the global repositioning apparatus of the embodiment can be implemented by referring to the description of the corresponding part in the foregoing method embodiment, which will not be described herein again.

[0082] Referring to Figure 3 , a structural schematic diagram of an electronic device according to Embodiment Three of the present application is shown, and the specific implementation of the electronic device is not limited in the specific embodiments of the present application.

[0083] As shown in Figure 3 , the electronic device can include a processor 302, a communications interface 304, a memory 306, and a communications bus 308.

[0084] Among them:

[0085] The processor 302, the communication interface 304, and the memory 306 communicate with each other through a communication bus 308.

[0086] The communication interface 304 is configured to communicate with other electronic devices or servers.

[0087] The processor 302 is configured to execute the program 310, and specifically, can execute the related steps in the above global repositioning method embodiments.

[0088] Specifically, the program 310 can include program code including computer operation instructions.

[0089] The processor 302 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device can be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0090] The memory 306 is configured to store the program 310. The memory 306 can include a high-speed RAM memory, and can also include a non-volatile memory such as at least one disk memory.

[0091] The program 310 can be specifically used to cause the processor 302 to perform the following operations: obtaining three-dimensional point cloud data and two-dimensional image data for describing an environment in which an object to be positioned is located; determining a global descriptor and feature point cloud data corresponding to the object to be positioned according to the three-dimensional point cloud data and the two-dimensional image data, wherein the global descriptor carries overall structure information, local semantic information, and image detail descriptor information of the environment; retrieving a first pose corresponding to the global descriptor from a preset global dictionary, wherein the global dictionary stores a correspondence between a plurality of global descriptors and a plurality of poses; and obtaining a second pose of the object to be positioned from a preset positioning point cloud map in combination with the feature point cloud data, taking the first pose as an initial pose.

[0092] In an optional implementation, the program 310 is further configured to cause the processor 302 to, when determining the global descriptor corresponding to the object to be positioned according to the three-dimensional point cloud data and the two-dimensional image data: convert the three-dimensional point cloud data and the two-dimensional image data into a reference coordinate system to generate corresponding reference point cloud data and reference image data; divide the reference point cloud data into Nr×Ns divisions along an X-axis direction and a heading angle direction in the reference coordinate system, where the X-axis direction is a forward direction of the object to be positioned and a direction in which a right hand points; and generate overall structure information, local semantic information, and image detail descriptor information of an environment in which the object to be positioned is located according to the reference point cloud data, the reference image data, and the Nr×Ns divisions.

[0093] In an optional implementation, the program 310 is further configured to cause the processor 302 to, when generating the overall structure information of the environment in which the object to be positioned is located according to the reference point cloud data, the reference image data, and the Nr×Ns divisions: for each division of the Nr×Ns divisions, take a maximum value of a Z-axis of the reference point cloud data in the division as a value of the division, where the Z-axis is an axis perpendicular to the ground and pointing to the sky; and form a two-dimensional matrix of Nr×Ns dimensions according to the maximum value of the Z-axis of each division, so as to represent the overall structure information of the environment in which the object to be positioned is located by the two-dimensional matrix.

[0094] In an optional implementation, the program 310 is further configured to cause the processor 302 to, when generating the local semantic information of the environment in which the object to be positioned is located according to the reference point cloud data, the reference image data, and the Nr×Ns divisions: for each division of the Nr×Ns divisions, extract semantic information in the division according to a preset target object category based on the reference point cloud data in the division; and generate the local semantic information of the environment in which the object to be positioned is located according to the semantic information of each division.

[0095] In an optional implementation, the program 310 is further configured to cause the processor 302 to, after the semantic information in each division is extracted based on the reference point cloud data in the division according to the preset target object category, encode the semantic information corresponding to each target object category in the division by using a 0-1 encoding mode to generate a 0-1 encoding sequence corresponding to the division and having a length of a number S of the target object categories; and the program 310 is further configured to cause the processor 302 to, when generating the local semantic information of the environment in which the object to be positioned is located according to the semantic information of each division: generate a three-dimensional 0-1 array of Nr×Ns×S corresponding to the reference point cloud data according to the 0-1 encoding sequence corresponding to each division, and use the array to represent the local semantic information of the environment in which the object to be positioned is located.

[0096] In an optional implementation, the program 310 is further configured to cause the processor 302 to, when generating the image detail descriptor information of the environment where the object to be positioned is located according to the reference point cloud data, the reference image data and the Nr x Ns partitions, extract feature points in the reference image data and detail descriptors corresponding to the feature points; project the feature points into the reference point cloud data, and determine the partitions to which the feature points belong in the Nr x Ns partitions; and generate the image detail descriptor information of the environment where the object to be positioned is located according to the detail descriptors corresponding to the feature points in each partition in the Nr x Ns partitions.

[0097] In an optional implementation, the program 310 is further configured to cause the processor 302 to, when obtaining the second pose of the object to be positioned from the preset positioning point cloud map in combination with the feature point cloud data, taking the first pose as an initial pose, determine, taking the feature point cloud data as a registration point, a pose in the positioning point cloud map that can be registered with the initial pose, and determine the registered pose as the second pose of the object to be positioned.

[0098] The specific implementation of each step in the program 310 can refer to the corresponding description in the corresponding steps and units of the foregoing global repositioning method embodiments, and will not be described herein. It can be clearly understood by those skilled in the art that, for the convenience and brevity of description, the specific working process of the foregoing device and module can refer to the corresponding process description in the foregoing method embodiments, and will not be described herein.

[0099] Through the electronic device of the embodiment, when globally repositioning the object to be positioned, the three-dimensional point cloud data and the two-dimensional image data of the environment in which the object to be positioned is located are considered at the same time, the three-dimensional point cloud data is not sensitive to light, seasonal change and the like, but lacks local detailed information, and the two-dimensional image data carries rich image detailed information, but is susceptible to light, seasonal change and the like. Combining the two can effectively make up for the respective shortcomings of the two. Further, based on this, the present application scheme further extracts the overall structure information, local semantic information and image detailed descriptor information of the environment in which the object to be positioned is located based on the three-dimensional point cloud data and the two-dimensional image data, and realizes multi-level information acquisition from the overall to the local and then to the details based on multi-source data. The global descriptors in the global dictionary also carry a plurality of overall structure information, local semantic information and image detailed descriptor information, and have corresponding poses. Then, according to the global descriptor corresponding to the object to be positioned and the global dictionary, the coarse positioning of the global repositioning of the object to be positioned can be realized. Further, taking the coarse positioning result, i.e., the first pose, as the initial pose, and combining the feature point cloud data of the object to be positioned, the second pose of the object to be positioned can be obtained from the preset positioning point cloud map. Since the positioning point cloud map is used for high-precision positioning, the second pose determined based on the same is also a high-precision pose. Thus, the global repositioning from coarse positioning to fine positioning is realized. Moreover, the global repositioning is independent of GNSS, and will not cause the global repositioning to be inaccurate or even fail due to unstable or no signal, and thus precise global repositioning is realized.

[0100] Of course, these algorithm modules will also be different according to the types of autonomous vehicles. For example, different algorithm modules will be involved for logistics vehicles, public service vehicles, medical service vehicles and terminal service vehicles. The algorithm modules will be exemplified below for the four kinds of autonomous vehicles respectively:

[0101] Among them, the logistics vehicle refers to a vehicle used in a logistics scene, which can be a logistics vehicle with automatic sorting function, a logistics vehicle with cold storage function, or a logistics vehicle with measurement function. These logistics vehicles will involve different algorithm modules.

[0102] For example, for a logistics vehicle, an automatic sorting device can be provided, which can automatically take out, accurately carry, sort and store goods after the logistics vehicle arrives at the destination. This involves global repositioning related to high-precision positioning before the vehicle departs, i.e., the process of determining the initial pose of the device within the global map range during the start-up stage of the device. Then, based on this pose, continuous high-precision positioning of the device can be performed.

[0103] For example, for the cold chain logistics scene, the logistics vehicle can also be equipped with a refrigeration device, which can refrigerate or keep the fruits, vegetables, aquatic products, frozen foods and other perishable foods transported at a suitable temperature environment, solving the problem of long-distance transportation of perishable foods. In order to ensure the transportation efficiency and accurate driving of the logistics vehicle, it involves global repositioning related to high-precision positioning, that is, the process of determining the initial pose of the device within the global map range in the starting stage of the device. Then, based on this pose, continuous high-precision positioning of the device can be performed.

[0104] Among them, the public service vehicle refers to a vehicle that provides a certain public service, which can be a fire truck, a deicing vehicle, a water truck, a snowplow, a garbage disposal vehicle, a traffic command vehicle, etc. These public service vehicles involve different algorithm modules.

[0105] For example, for an automatic driving fire truck, its main task is to perform reasonable fire extinguishing tasks on the fire scene, which involves an algorithm module for accurate positioning, that is, a module for global repositioning related to high-precision positioning, that is, the process of determining the initial pose of the device within the global map range in the starting stage of the device. Then, based on this pose, continuous high-precision positioning of the device can be performed.

[0106] For example, for a deicing vehicle, its main task is to remove ice and snow on the road surface, which involves a deicing algorithm module for accurate positioning, which is related to global repositioning of high-precision positioning, that is, the process of determining the initial pose of the device within the global map range in the starting stage of the device. Then, based on this pose, continuous high-precision positioning of the device can be performed.

[0107] Among them, the medical service vehicle refers to an automatic driving vehicle that can provide one or more medical services, which can provide disinfection, temperature measurement, drug dispensing, isolation, etc. Medical services, in order to provide better medical services, more accurate vehicle positioning is needed, which involves global repositioning related to high-precision positioning, that is, the process of determining the initial pose of the device within the global map range in the starting stage of the device. Then, based on this pose, continuous high-precision positioning of the device can be performed.

[0108] Among them, the terminal service vehicle refers to a self-service automatic driving vehicle that can replace some terminal devices to provide certain convenient services to users, for example, these vehicles can provide printing, attendance, scanning, unlocking, payment, retail, etc. In order to provide better services, more accurate vehicle positioning is needed, which involves global repositioning related to high-precision positioning, that is, the process of determining the initial pose of the device within the global map range in the starting stage of the device. Then, based on this pose, continuous high-precision positioning of the device can be performed.

[0109] For example, in some application scenarios, users often need to go to a specific location to print or scan documents, which is time-consuming and laborious. Therefore, a terminal service vehicle that can provide printing / scanning services for users appears. These service vehicles can interconnect with user terminal devices, and users can issue a printing instruction through the terminal device. The service vehicle responds to the printing instruction, automatically prints the document required by the user, and can automatically deliver the printed document to the user's location. The user does not need to queue at the printer, which can greatly improve the printing efficiency. Alternatively, the service vehicle can move to the user's location in response to the scanning instruction issued by the user through the terminal device. The user places the document to be scanned on the scanning tool of the service vehicle to complete the scanning, without the need to queue at the printer / scanner, saving time and effort. This involves an algorithm module for providing printing / scanning services, which at least needs to identify the interconnection with the user terminal device, the response to the printing / scanning instruction, the positioning of the user's location, and the travel control, etc.

[0110] For another example, with the development of new retail business services, more and more e-commerce companies use self-service vending machines to deliver goods to major office buildings and public areas. However, these self-service vending machines are placed in fixed locations and cannot be moved. Users need to be close to the self-service vending machine to purchase the required goods, which is still not very convenient. Therefore, self-driving vehicles that can provide retail services appear. These service vehicles can carry goods and automatically move, and can provide corresponding self-service shopping APPs or shopping portals. Users can use mobile phones and other terminal devices to place orders through the APPs or shopping portals. The order includes the name and quantity of the goods to be purchased and the user's location. After receiving the order request, the vehicle can determine whether it has the goods to be purchased and whether the quantity is sufficient. If it has the goods to be purchased and the quantity is sufficient, it can automatically move to the user's location and provide the goods to the user, further improving the convenience of user shopping, saving user time, and allowing users to use their time for more important things. This involves algorithm modules for providing retail services, which mainly implement the logic of responding to user order requests, order processing, goods information maintenance, user location positioning, and payment management. In order to provide better new retail business services, more accurate vehicle positioning is needed, which involves global repositioning related to high-precision positioning, i.e., the process of determining the initial pose of the device within the global map range during the startup phase of the device. Then, based on this pose, continuous high-precision positioning of the device can be performed.

[0111] It should be noted that, according to the needs of implementation, each component / step described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or part of the operations of the components / steps can be combined into new components / steps, to achieve the purpose of the embodiments of the present application.

[0112] The methods according to embodiments of the present application described above can be implemented in hardware, firmware, or software, or a combination of hardware, firmware or software, and can be stored in a recording medium such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk, or be downloaded by a network from a remote recording medium or non-transitory machine-readable medium originally stored in a local recording medium and stored in a local recording medium, so that the methods described herein can be processed by such software stored on a recording medium using a general purpose computer, a special purpose processor, or programmable or dedicated hardware such as an ASIC or FPGA. It can be understood that the computer, processor, microprocessor controller, or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the global relocation methods described herein. Furthermore, when a general purpose computer accesses code for implementing the global relocation methods shown herein, the execution of the code transforms the general purpose computer into a special purpose computer for executing the global relocation methods shown herein.

[0113] Those skilled in the art can understand that the units and method steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of the present application.

[0114] The above embodiments are only used to illustrate but not limit the embodiments of the present application, and ordinary skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application, therefore all equivalent technical solutions belong to the scope of the embodiments of the present application, and the patent protection scope of the embodiments of the present application should be defined by the claims.

Claims

1. A global repositioning method, comprising: obtaining three-dimensional point cloud data and two-dimensional image data describing an environment in which an object to be positioned is located; determining a global descriptor and feature point cloud data corresponding to the object to be positioned according to the three-dimensional point cloud data and the two-dimensional image data, wherein the global descriptor carries overall structure information, local semantic information and image detail descriptor information of the environment; retrieving a first pose corresponding to the global descriptor from a preset global dictionary, wherein the global dictionary stores a correspondence between a plurality of global descriptors and a plurality of poses; obtaining a second pose of the object to be positioned from a preset positioning point cloud map, taking the first pose as an initial pose and combining the feature point cloud data, wherein the global descriptor corresponding to the object to be positioned is determined according to the three-dimensional point cloud data and the two-dimensional image data, comprising: converting the three-dimensional point cloud data and the two-dimensional image data to a reference coordinate system to generate corresponding reference point cloud data and reference image data, the reference coordinate system having a Z-axis direction determined based on a ground surface on which the object to be positioned is located, a heading angle direction determined around the Z-axis direction, and a target orientation of the object to be positioned on the ground surface; dividing the reference point cloud data into Nr×Ns sections along the target orientation and the heading angle direction; generating overall structure information, local semantic information and image detail descriptor information of the environment in which the object to be positioned is located according to the reference point cloud data, the reference image data and the Nr×Ns sections.

2. The method of claim 1, wherein, generating overall structure information of the environment in which the object to be positioned is located according to the reference point cloud data, the reference image data and the Nr×Ns sections, comprising: for each section in the Nr×Ns sections, taking the maximum value of the Z-axis of the reference point cloud data in the section as the value of the section; forming a two-dimensional matrix of Nr×Ns dimensions according to the maximum value of the Z-axis of each section, and taking the two-dimensional matrix to represent the overall structure information of the environment in which the object to be positioned is located.

3. The method of claim 1, wherein, generating local semantic information of the environment in which the object to be positioned is located according to the reference point cloud data, the reference image data and the Nr×Ns sections, comprising: for each section in the Nr×Ns sections, extracting semantic information in the section according to a preset target object category based on the reference point cloud data in the section; generating local semantic information of the environment in which the object to be positioned is located according to the semantic information of each section.

4. The method of claim 3, wherein, after the semantic information in the section is extracted according to the preset target object category based on the reference point cloud data in the section, the method further comprises: encoding the semantic information corresponding to each target object category in the section using a 0-1 encoding method to generate a 0-1 encoding sequence corresponding to the section with a length of S, the number of target object categories. The generating of the local semantic information of the environment where the object to be positioned is located according to the semantic information of each partition comprises: generating a three-dimensional 0-1 array of Nr×Ns×S corresponding to the reference point cloud data according to the 0-1 encoding sequence corresponding to each partition, and using the array to represent the local semantic information of the environment where the object to be positioned is located.

5. The method of claim 1, wherein, The generating of the image detail descriptor information of the environment where the object to be positioned is located according to the reference point cloud data, the reference image data and the Nr×Ns partition comprises: extracting feature points in the reference image data and detail descriptors corresponding to the feature points; projecting the feature points into the reference point cloud data to determine the partition to which the feature points belong in the Nr×Ns partition; generating the image detail descriptor information of the environment where the object to be positioned is located according to the detail descriptors corresponding to the feature points in each partition in the Nr×Ns partition.

6. The method according to any one of claims 1 to 5, wherein, Taking the first pose as an initial pose, the second pose of the object to be positioned is obtained from a preset positioning point cloud map in combination with the feature point cloud data, comprising: taking the feature point cloud data as a registration point, determining a pose in the positioning point cloud map that can be registered with the initial pose, and determining the registered pose as the second pose of the object to be positioned.

7. A global repositioning device, comprising: an acquisition module configured to acquire three-dimensional point cloud data and two-dimensional image data for describing an environment where an object to be positioned is located; a determination module configured to determine a global descriptor and feature point cloud data corresponding to the object to be positioned according to the three-dimensional point cloud data and the two-dimensional image data, wherein the global descriptor carries overall structure information, local semantic information and image detail descriptor information of the environment, and wherein the determination of the global descriptor corresponding to the object to be positioned according to the three-dimensional point cloud data and the two-dimensional image data comprises: converting the three-dimensional point cloud data and the two-dimensional image data to reference coordinate system to generate corresponding reference point cloud data and reference image data, the reference coordinate system having a Z-axis direction determined based on a ground surface where the object to be positioned is located, a heading direction determined around the Z-axis direction, and a target orientation of the object to be positioned on the ground surface; setting Nr partitions along the target orientation and Ns partitions along the heading direction, and dividing the reference point cloud data into Nr×Ns partitions; and generating overall structure information, local semantic information and image detail descriptor information of the environment where the object to be positioned is located according to the reference point cloud data, the reference image data and the Nr×Ns partitions, respectively; a retrieval module configured to retrieve a first pose corresponding to the global descriptor from a preset global dictionary, wherein the global dictionary stores a correspondence relationship between a plurality of global descriptors and a plurality of poses; a positioning module configured to take the first pose as an initial pose, and obtain a second pose of the object to be positioned from a preset positioning point cloud map in combination with the feature point cloud data.

8. An electronic device comprising: a processor, a memory, a communication interface, and a communication bus, which enable communication among the processor, the memory, and the communication interface through the communication bus; the memory is configured to store at least one executable instruction, which causes the processor to perform operations corresponding to the global relocation method according to any one of claims 1-6.

9. A computer storage medium, which stores a computer program, and the computer program, when executed by a processor, implements the global relocation method according to any one of claims 1-6.

Images