Point cloud labeling model training method and device, storage medium and computer equipment

Abstract

The invention discloses a point cloud labeling model training method and device, a storage medium and computer equipment. The method comprises the steps that pre-labeled sample point cloud data and a digital map of a target area are acquired, and point clouds in the sample point cloud data are located in the area range of the target area; map information is extracted from the digital map, and the map information is used for describing a static environment in the target area; the original point cloud labeling model is trained according to the map information and the sample point cloud data, a target point cloud labeling model is generated, and the target point cloud labeling model is used for labeling the point cloud data. According to the invention, the technical problem of inaccurate point cloud recognition of an obstacle at an ultra-long distance by a model is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a point cloud annotation model training method, device, storage medium and computer equipment. Background Art

[0002] In recent years, with the rapid development of autonomous driving, intelligent transportation and other fields, point cloud annotation technology has also received extensive attention and research. The point cloud annotation method provided in the relevant technology requires a large amount of annotation data to train the point cloud annotation model. For some specific scenes or objects, if the annotation data used for training is insufficient, the annotation effect of the trained model is not good. For example, the ability to handle noise and incompleteness in point cloud data is weak, which is prone to mis-annotation or omission, or requires interaction with human annotators, and the annotation efficiency is low.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] Embodiments of the present invention provide a point cloud annotation model training method, apparatus, storage medium and computer equipment to at least solve the technical problem of inaccurate point cloud recognition of obstacles at ultra-long distances by the model.

[0005] According to one aspect of an embodiment of the present invention, a point cloud annotation model training method is provided, comprising: obtaining pre-annotated sample point cloud data and a digital map of a target area, wherein the point cloud in the sample point cloud data is located within the area range of the target area; extracting map information from the digital map, wherein the map information is used to describe the static environment in the target area; and training an original point cloud annotation model according to the map information and the sample point cloud data to generate a target point cloud annotation model, wherein the target point cloud annotation model is used to annotate the point cloud data.

[0006] Optionally, before obtaining a digital map of the target area, the method further includes: determining first positioning information, wherein the first positioning information represents the spatial position of the sample point cloud data; and determining an area range of the target area based on the first positioning information, wherein the area range of the target area includes the spatial position represented by the first positioning information.

[0007] Optionally, determining the first positioning information includes: acquiring second positioning information, wherein the second positioning information represents the spatial position of the point cloud acquisition device when acquiring the sample point cloud data; determining, based on the sample point cloud data, a relative position relationship between the spatial position of the sample point cloud data and the spatial position of the point cloud acquisition device; and determining the first positioning information based on the second positioning information and the relative position relationship.

[0008] Optionally, determining the area range of the target area based on the first positioning information includes: acquiring second positioning information, wherein the second positioning information represents the spatial position at which a point cloud acquisition device acquires the sample point cloud data; generating the target area based on the first positioning information and the second positioning information, wherein the area range of the target area is centered on the spatial position represented by the second positioning information.

[0009] Optionally, extracting map information from the digital map includes: acquiring multiple map layers from the digital map, wherein the multiple map layers respectively represent different aspects of the digital map; generating multiple layer images corresponding to the multiple map layers one by one according to the multiple map layers, wherein the map information includes the multiple layer images.

[0010] Optionally, the training of the original point cloud annotation model according to the map information and the sample point cloud data to generate a target point cloud annotation model includes: inputting the map information into a first feature extraction network of the original point cloud annotation model to output a first feature map; inputting the sample point cloud data into a second feature extraction network of the original point cloud annotation model to output a second feature map; and training the original point cloud annotation model according to the first feature map and the second feature map to generate the target point cloud annotation model.

[0011] Optionally, the training of the original point cloud annotation model according to the first feature map and the second feature map to generate the target point cloud annotation model includes: inputting the first feature map and the second feature map into a fusion network layer to generate fusion features, wherein the feature dimensions of the first feature map and the second feature map are the same; inputting the fusion features into the neck network of the original point cloud annotation model, training the neck network and the head network of the original point cloud annotation model and updating the parameters of the neck network and the head network to obtain the target point cloud annotation model.

[0012] Optionally, the pre-labeled sample point cloud data is point cloud data selected based on the following conditions: the distance between the sample point cloud data and the point cloud acquisition device is greater than a distance threshold; and / or the point cloud density of the sample point cloud data is less than a point cloud density threshold.

[0013] According to another aspect of an embodiment of the present invention, a point cloud annotation model training device is also provided, including: an acquisition module, used to acquire pre-annotated sample point cloud data and a digital map of a target area, wherein the point cloud in the sample point cloud data is located within the area range of the target area; an extraction module, used to extract map information from the digital map, wherein the map information is used to describe the static environment in the target area; a training module, used to train an original point cloud annotation model according to the map information and the sample point cloud data to generate a target point cloud annotation model, wherein the target point cloud annotation model is used to annotate the point cloud data.

[0014] According to another aspect of an embodiment of the present invention, a non-volatile storage medium is further provided, wherein the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute any one of the above-mentioned point cloud annotation model training methods.

[0015] According to another aspect of an embodiment of the present invention, a computer device is further provided, comprising a memory and a processor, wherein the memory is used to store programs, and the processor is used to run the programs stored in the memory, wherein when the program is run, any one of the above-mentioned point cloud annotation model training methods is executed.

[0016] In an embodiment of the present invention, pre-annotated sample point cloud data and a digital map of a target area are obtained, wherein the point cloud in the sample point cloud data is located within the area range of the target area; map information is extracted from the digital map, wherein the map information is used to describe the static environment in the target area; an original point cloud annotation model is trained according to the map information and the sample point cloud data to generate a target point cloud annotation model for annotating the point cloud data, thereby achieving the purpose of using the static environment information in the map to assist in training the point cloud annotation model, thereby achieving the technical effect of optimizing the recognition accuracy of the point cloud annotation model for specific point cloud data, and further solving the technical problem of the model's inaccurate point cloud recognition of obstacles at ultra-long distances. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 A hardware structure block diagram of a computer terminal for implementing a point cloud annotation model training method is shown;

[0019] Figure 2is a flow chart of a point cloud annotation model training method provided according to an embodiment of the present invention;

[0020] Figure 3 is a schematic diagram of sample point cloud data provided according to an optional embodiment of the present invention;

[0021] Figure 4 is a schematic diagram of providing static environment superimposed sample point cloud data according to an optional embodiment of the present invention;

[0022] Figure 5 is a schematic diagram of a training method for a point cloud annotation model provided according to an optional embodiment of the present invention;

[0023] Figure 6 It is a structural block diagram of a point cloud annotation model training device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] According to an embodiment of the present invention, a method embodiment of a point cloud annotation model training is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a point cloud annotation model training method. Figure 1 As shown, the computer terminal 10 may include one or more processors (processors 102a, 102b, ..., 102n are used as examples in the figure) (the processors may include but are not limited to processing devices such as microprocessors MCU or programmable logic devices FPGA), and a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply, and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0028] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0029] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the point cloud annotation model training method in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the point cloud annotation model training method of the above-mentioned application is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0030] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0031] In the related technologies, there are some defects in the annotation of point clouds, such as heavy reliance on annotation data, high cost, high requirements for sensor accuracy and stability, low annotation efficiency, etc. These defects limit the application scope and effect of point cloud annotation technology, and further research and improvement are needed.

[0032] The present invention relates to a technology that provides prior information to assist in offline pre-annotation of ultra-long-distance obstacle point clouds, especially for the annotation of LiDAR point clouds. Manual annotation of ultra-long-distance obstacles, especially the annotation of LiDAR point clouds, is extremely difficult due to the sparse laser points at a distance (>100 meters). The previous method is to use the human eye combined with images to determine the type of point cloud for annotation. It is efficient to annotate places with rich point clouds at short distances, but it is difficult to accurately annotate point clouds at long distances or even ultra-long distances. The annotation results are unreliable and inefficient.

[0033] The present invention can provide a technical solution for offline pre-annotation of ultra-long-distance obstacle point clouds based on map prior information. The solution uses high-precision maps to provide information on the static environment around long-distance obstacles, thereby improving the ability of offline point cloud models to identify long-distance obstacles and providing richer pre-annotation results for online point cloud annotation models.

[0034] Figure 2 is a flow chart of a point cloud annotation model training method provided in an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0035] Step S202 , obtaining pre-labeled sample point cloud data and a digital map of the target area, wherein the point cloud in the sample point cloud data is located within the area range of the target area.

[0036] Optionally, the digital map can be a high-precision map commonly used in the field of intelligent driving. A three-dimensional scene of the target area can be constructed based on the high-precision map. The high-precision map records accurate positioning coordinates, and also includes lane boundaries, traffic signs, guardrails, street lamp poles, gantries and other road attribute elements, as well as the material, shape, width and other information of various objects. The pre-labeled sample point cloud data includes the point cloud data itself and the classification true value of the point cloud data, which can be used as a sample to train a point cloud classification model or a point cloud labeling model.

[0037] Figure 3 is a schematic diagram of sample point cloud data provided according to an optional embodiment of the present invention, such as Figure 3As shown, the white dots in the figure can be the points obtained by laser radar scanning, and the point cloud in the circle indicated by the arrow in the figure is the sample point cloud data. The sample point cloud data may be very far away from the vehicle, for example, more than 100 meters, and the point cloud is very sparse, so it is not easy to get the correct result when using the conventional point cloud annotation model for object recognition. The present application can obtain static environment data near the sample point cloud data as an aid to improve the training effect of the point cloud annotation model. Figure 4 is a schematic diagram of providing a static environment superimposed with sample point cloud data according to an optional embodiment of the present invention, such as Figure 4 As shown in Figure 1, superimposing the static environment information obtained from the digital map with the sample point cloud data can enrich the relevant information of the sample point cloud data, improve the model's feature learning of the sample point cloud data, and eliminate some misidentified objects, such as Figure 4 As shown, the sample point cloud data in the white box is located within the lane, so it is impossible to be an object outside the lane, such as a telephone pole. Therefore, based on the static environment information provided by the digital map, the model's mislearning can be avoided and the recognition accuracy of the point cloud annotation model can be improved.

[0038] As an optional embodiment, the pre-labeled sample point cloud data is point cloud data selected based on the following conditions: the distance between the sample point cloud data and the point cloud acquisition device is greater than the distance threshold; and / or the point cloud density of the sample point cloud data is less than the point cloud density threshold. The sample point cloud data may be the point cloud of a long-distance obstacle. Such an obstacle is difficult to accurately identify. Its point cloud data usually has two characteristics: one is that the distance from the point cloud acquisition device is particularly far, for example, more than 100 meters, and the other is that the point cloud data of the obstacle is particularly sparse, so it is also difficult to accurately identify the obstacle.

[0039] As an optional embodiment, before obtaining a digital map of the target area, the method may further include the following steps: determining first positioning information, wherein the first positioning information represents the spatial position of the sample point cloud data; determining an area range of the target area based on the first positioning information, wherein the area range of the target area includes the spatial position represented by the first positioning information.

[0040] In order to improve the training effect of the sample point cloud data on the model, the static environment information around the sample point cloud data can be obtained so that the model can use the static environment information to learn how to classify the point cloud data, so the range of the target area can be determined based on the location of the sample point cloud data. In this embodiment, the sample point cloud data can be a point cloud gathered in one place, rather than a point cloud scattered in a wide area.

[0041] As an optional embodiment, when determining the first positioning information, the second positioning information can be obtained first, wherein the second positioning information represents the spatial position of the point cloud acquisition device when collecting sample point cloud data; then, based on the sample point cloud data, the relative position relationship between the spatial position of the sample point cloud data and the spatial position of the point cloud acquisition device is determined; based on the second positioning information and the relative position relationship, the first positioning information is determined.

[0042] In this optional embodiment, the point cloud acquisition device may be a laser radar or a depth camera on the vehicle. Therefore, the spatial position of the vehicle can be determined based on the high-precision positioning function of the vehicle (such as the vehicle's own GPS) as the spatial position of the point cloud acquisition device, the second positioning information can be determined, and then the spatial position of the point cloud data can be determined based on the relative position relationship between the point cloud data and the point cloud acquisition device.

[0043] As an optional embodiment, the area range of the target area can be determined based on the first positioning information, and the following method can also be used: obtain second positioning information, wherein the second positioning information represents the spatial position at which the point cloud acquisition device collects sample point cloud data; generate a target area based on the first positioning information and the second positioning information, wherein the area range of the target area is centered on the spatial position represented by the second positioning information.

[0044] Based on this optional embodiment, a relatively convenient local digital map acquisition method can be adopted, that is, a circular area is delineated with the location of the point cloud acquisition device as the center and a radius not less than the distance from the sample point cloud data to the point cloud acquisition device, and the area is determined as the target area, and a digital map in the area is used to provide static environmental information. The target area delineation method is simple to operate and does not require the coordinate system to be transformed to delineate the target area with the sample point cloud data as the center.

[0045] Step S204: extracting map information from the digital map, wherein the map information is used to describe the static environment in the target area.

[0046] The static environment information of the map obtained in this step can be used to assist in the identification and annotation of point cloud data, and can be input into the model as a priori information when training the point cloud annotation model, so that the model can improve the annotation accuracy of the point cloud data.

[0047] As an optional embodiment, extracting map information from a digital map may include the following steps: obtaining multiple map layers from the digital map, wherein the multiple map layers respectively represent different aspects of the digital map; generating multiple layer images corresponding to the multiple map layers based on the multiple map layers, wherein the map information includes the multiple layer images.

[0048] High-precision maps usually consist of multiple different layers, each of which is responsible for describing different aspects or features of the map. Optionally, common layer divisions of high-precision maps can include the following categories: 1. Road network layer: The road network layer contains the topological structure information of the road, including the connection relationship of the road, road grade, number of lanes, lane width, etc. This layer describes the structure and attributes of the road network. 2. Lane marking and traffic sign layer: This layer contains the geometry and location information of lane markings and traffic signs. It describes traffic signs such as lane divisions, turn arrows, stop signs, and traffic lights on the road. 3. Building layer: The building layer describes the location, shape, and attribute information of buildings in a city or region. It can include different types of buildings such as houses, commercial buildings, and public facilities. 4. Vegetation and landform layer: This layer describes the location and morphological information of vegetation, trees, and other landforms. It can be used for applications such as landscape analysis and greening planning. 5. Water body layer: The water body layer describes the location and shape information of water bodies such as rivers, lakes, and reservoirs. This layer is very important for applications such as water resource management and flood prediction. 6. Terrain and elevation layer: This layer describes the terrain and elevation information of the ground, including mountains, valleys, altitude, etc. It can be used for applications such as terrain analysis, navigation, and visual effects. 7. POI layer: The POI (Point of Interest) layer contains the location and attribute information of places of interest or landmarks, such as restaurants, hotels, gas stations, hospitals, etc. It is very important for applications such as navigation, location services, and business analysis. The combination of these layers can provide rich geographic information and support various applications such as navigation systems, autonomous driving, urban planning, traffic management, etc. Different high-precision map providers may have slightly different layer divisions and naming, but usually include some of the main layers mentioned above.

[0049] In this step, the generated multiple layer images corresponding one-to-one to multiple map layers can be two-dimensional images. The data of some layers in the high-precision map contain height information, so it is equivalent to 2.5-dimensional data. In this step, such map data is also converted into two-dimensional layer images, which is convenient for subsequent unified input into the point cloud annotation model for model training.

[0050] Step S206, training the original point cloud annotation model according to the map information and the sample point cloud data to generate a target point cloud annotation model, wherein the target point cloud annotation model is used to annotate the point cloud data.

[0051] Among them, the original point cloud annotation model can be a model used for point cloud annotation after pre-training, that is, a pre-trained model. The pre-trained model is a model that has been trained before. The model is only trained with point cloud data, and no static environmental information is used to assist in the training process. Therefore, the model parameters in the pre-trained model are only sensitive to point cloud data. Based on the pre-trained model, this application adds map information and combines it with point cloud data for further training, so as to obtain a point cloud annotation model with higher recognition accuracy.

[0052] As an optional embodiment, the original point cloud annotation model is trained according to the map information and the sample point cloud data to generate the target point cloud annotation model, which may include the following steps: inputting the map information into the first feature extraction network of the original point cloud annotation model to output the first feature map; inputting the sample point cloud data into the second feature extraction network of the original point cloud annotation model to output the second feature map; training the original point cloud annotation model according to the first feature map and the second feature map to generate the target point cloud annotation model.

[0053] Optionally, the first feature extraction network can be a 2D image feature extraction network including five convolutional layers, which is used to obtain beyond-horizon information based on the static environment information provided by the static layer of the digital map (such as the distance to the roadside of the map, the reflectivity of the objects on the map, etc.), and output 128 feature maps after inputting these map information into the first feature extraction network, upsampling to the same dimension as the 384 feature maps output by the second feature extraction network, and then putting the first feature map and the second feature map into the neck network for fusion training. This method can improve the recognition of long-distance obstacles and filter out misidentified objects that should not appear, thereby providing richer pre-labeling results and improving pre-labeling accuracy.

[0054] The first feature map is equivalent to the result of abstracting the static 2D image output, such as some features in the static layer, such as the matrix result after abstracting the outline, background, brightness, etc. of the static layer. If only point cloud data is used for model training, it will lead to misrecognition due to the sparsity of the point cloud of distant obstacles, distant objects or environmental limitations. For example, some distant obstacles are recognized as cars, or reflective points appear in some specific environments (such as glass / reflectors), causing errors in the point cloud data. Point clouds appear in the range where objects should not be scanned (such as outside the road / wall), which will cause noise and lead to mislearning of the model. After adding the features of the static map image, it is equivalent to making a priori to filter out these misrecognized objects, thereby improving the recognition effect.

[0055] As an optional embodiment, the original point cloud annotation model is trained according to the first feature map and the second feature map to generate a target point cloud annotation model, including: inputting the first feature map and the second feature map into the fusion network layer to generate fusion features, wherein the feature dimensions of the first feature map and the second feature map are the same; inputting the fusion features into the neck network of the original point cloud annotation model, training the neck network and the head network of the original point cloud annotation model and updating the parameters of the neck network and the head network to obtain the target point cloud annotation model.

[0056] In deep learning, the head network, neck network and backbone network in the model refer to different parts or components in the network model. The backbone network refers to the main part of the model, which is usually composed of a series of convolutional layers. It is responsible for extracting the feature representation of the input data. The backbone network is usually a network with high depth and complexity, such as ResNet, VGG, etc. The role of the backbone network is to gradually abstract and extract the high-level features of the input data through layers of convolution and pooling operations. The first feature extraction network and the second feature extraction network in this application are the backbone networks. The neck network is located between the backbone network and the head network, and usually contains some dimensionality reduction or feature fusion operations for further processing the features extracted by the backbone network. The purpose of the neck network is to reduce the dimension of the features or fuse the features through specific operations to provide a more compact and useful feature representation. The neck network can have different designs, such as convolutional layers that reduce the feature dimension, global average pooling, attention mechanisms, etc. The head network is the last part of the model, responsible for performing specific tasks such as classification, target detection, semantic segmentation, etc. It is generally composed of fully connected layers, convolutional layers and other layers related to specific tasks. The head network further processes and interprets the features output by the neck network to produce the final prediction results. The hierarchical structure of the model allows flexible adjustment and transfer learning on different tasks and data sets. The backbone network can be shared, while the head network can be customized and adjusted according to specific tasks, and the neck network provides flexibility in intermediate feature representation. In this embodiment, a fusion network layer is added between the neck networks to fuse the first feature map and the second feature map to obtain the input of the neck network.

[0057] In the above steps, pre-annotated sample point cloud data and a digital map of the target area are obtained, wherein the point cloud in the sample point cloud data is located within the area range of the target area; map information is extracted from the digital map, wherein the map information is used to describe the static environment in the target area; the original point cloud annotation model is trained according to the map information and the sample point cloud data to generate a target point cloud annotation model for annotating the point cloud data, thereby achieving the purpose of using the static environment information in the map to assist in the training of the point cloud annotation model, thereby achieving the technical effect of optimizing the recognition accuracy of the point cloud annotation model for specific point cloud data, and further solving the technical problem of the model's inaccurate point cloud recognition of obstacles at ultra-long distances.

[0058] Figure 5 is a schematic diagram of a training method for a point cloud annotation model according to an optional embodiment of the present invention. Figure 5 As shown, the method may include the following steps:

[0059] Step (1): Obtain a high-precision map of the three-dimensional scene in advance. Obtain a high-precision map around the point cloud data in advance. The high-precision map has multiple layers such as height and reflectivity, and includes semantic information such as roads, road signs, and traffic lights.

[0060] Step (2): High-precision positioning. The high-precision positioning of the vehicle position is achieved by using multi-sensor fusion technologies such as laser radar, GPS, and inertial measurement unit. Positioning can be turned on when annotating data is collected, and the positioning results can be directly reused during annotation. The annotated data in this step is the sample point cloud data, and the vehicle is the vehicle where the point cloud acquisition device is located when the sample point cloud data is collected.

[0061] Step (3): Extract static environment information. Based on the vehicle positioning information, extract the static environment information around the long-distance obstacle (i.e., the obstacle corresponding to the point cloud data) from the pre-established high-precision map, mainly including information such as roads, lanes, and traffic signs.

[0062] Step (4): Offline point cloud pre-annotation. Use the provided static environmental information (such as the distance inside and outside the map road, the reflectivity of objects on the map, etc.) to obtain the beyond-line-of-sight environmental information. By inputting this environmental information into the image feature extraction network and integrating it with the offline point cloud model for training, the recognition of long-distance obstacles is improved and the misidentified objects that should not appear are filtered out, thereby providing richer pre-annotation results and improving the pre-annotation accuracy.

[0063] The difference between this solution and the existing technology is that this solution solves the problem of difficulty in labeling long-distance obstacles by providing prior information to assist offline point cloud pre-labeling, and improves the accuracy and reliability of labeling. At the same time, this solution uses pre-established three-dimensional scenes and high-precision positioning technology to provide richer static environment information, filter out some misidentified objects, and thus provide more accurate prediction results for the online model.

[0064] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0065] Through the description of the above implementation methods, those skilled in the art can clearly understand that the point cloud annotation model training method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of various embodiments of the present invention.

[0066] According to an embodiment of the present invention, a point cloud annotation model training device for implementing the above-mentioned point cloud annotation model training method is also provided. Figure 6 is a structural block diagram of a point cloud annotation model training device provided according to an embodiment of the present invention. Figure 6 As shown, the point cloud annotation model training device includes: an acquisition module 62, an extraction module 64 and a training module 66. The point cloud annotation model training device is described below.

[0067] An acquisition module 62 is used to acquire pre-labeled sample point cloud data and a digital map of the target area, wherein the point cloud in the sample point cloud data is located within the area range of the target area;

[0068] An extraction module 64, connected to the acquisition module 62, is used to extract map information from the digital map, wherein the map information is used to describe the static environment in the target area;

[0069] The training module 66 is connected to the extraction module 64 and is used to train the original point cloud annotation model according to the map information and the sample point cloud data to generate a target point cloud annotation model, wherein the target point cloud annotation model is used to annotate the point cloud data.

[0070] It should be noted that the acquisition module 62, the extraction module 64 and the training module 66 correspond to steps S202 to S206 in the embodiment, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in the embodiment.

[0071] An embodiment of the present invention may provide a computer device. Optionally, in this embodiment, the computer device may be located in at least one network device among multiple network devices of a computer network. The computer device includes a memory and a processor.

[0072] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the point cloud annotation model training method and device in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned point cloud annotation model training method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0073] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain pre-annotated sample point cloud data and a digital map of the target area, wherein the point cloud in the sample point cloud data is located within the area range of the target area; extract map information from the digital map, wherein the map information is used to describe the static environment in the target area; train the original point cloud annotation model according to the map information and the sample point cloud data to generate a target point cloud annotation model, wherein the target point cloud annotation model is used to annotate the point cloud data.

[0074] Optionally, the processor may also execute program code of the following steps: before obtaining a digital map of the target area, the method may further include: determining first positioning information, wherein the first positioning information represents the spatial position of the sample point cloud data; determining an area range of the target area based on the first positioning information, wherein the area range of the target area includes the spatial position represented by the first positioning information.

[0075] Optionally, the processor may also execute the program code of the following steps: determining the first positioning information, including: obtaining the second positioning information, wherein the second positioning information represents the spatial position of the point cloud acquisition device when collecting the sample point cloud data; determining the relative position relationship between the spatial position of the sample point cloud data and the spatial position of the point cloud acquisition device based on the sample point cloud data; determining the first positioning information based on the second positioning information and the relative position relationship.

[0076] Optionally, the processor may also execute program code of the following steps: determining the area range of the target area based on the first positioning information, including: obtaining second positioning information, wherein the second positioning information represents the spatial position at which the point cloud acquisition device collects sample point cloud data; generating a target area based on the first positioning information and the second positioning information, wherein the area range of the target area is centered on the spatial position represented by the second positioning information.

[0077] Optionally, the processor may also execute program code of the following steps: extracting map information from a digital map, including: obtaining multiple map layers from the digital map, wherein the multiple map layers respectively represent different aspects of the digital map; generating multiple layer images corresponding to the multiple map layers one by one based on the multiple map layers, wherein the map information includes the multiple layer images.

[0078] Optionally, the processor may also execute program code for the following steps: training the original point cloud annotation model according to map information and sample point cloud data to generate a target point cloud annotation model, including: inputting map information into a first feature extraction network of the original point cloud annotation model to output a first feature map; inputting sample point cloud data into a second feature extraction network of the original point cloud annotation model to output a second feature map; training the original point cloud annotation model according to the first feature map and the second feature map to generate a target point cloud annotation model.

[0079] Optionally, the processor may also execute the program code of the following steps: training the original point cloud annotation model according to the first feature map and the second feature map to generate a target point cloud annotation model, including: inputting the first feature map and the second feature map into the fusion network layer to generate fusion features, wherein the feature dimensions of the first feature map and the second feature map are the same; inputting the fusion features into the neck network of the original point cloud annotation model, training the neck network and the head network of the original point cloud annotation model and updating the parameters of the neck network and the head network to obtain the target point cloud annotation model.

[0080] Optionally, the processor may also execute the program code of the following steps: the pre-labeled sample point cloud data is point cloud data selected based on the following conditions: the distance between the sample point cloud data and the point cloud acquisition device is greater than a distance threshold; and / or the point cloud density of the sample point cloud data is less than a point cloud density threshold.

[0081] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a non-volatile storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0082] The embodiment of the present invention further provides a non-volatile storage medium. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the point cloud annotation model training method provided in the above embodiment.

[0083] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0084] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: obtaining pre-annotated sample point cloud data and a digital map of the target area, wherein the point cloud in the sample point cloud data is located within the area range of the target area; extracting map information from the digital map, wherein the map information is used to describe the static environment in the target area; training an original point cloud annotation model based on the map information and the sample point cloud data to generate a target point cloud annotation model, wherein the target point cloud annotation model is used to annotate the point cloud data.

[0085] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: before obtaining a digital map of the target area, the method further includes: determining first positioning information, wherein the first positioning information represents the spatial position of the sample point cloud data; determining an area range of the target area based on the first positioning information, wherein the area range of the target area includes the spatial position represented by the first positioning information.

[0086] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: determining first positioning information, including: obtaining second positioning information, wherein the second positioning information represents the spatial position of the point cloud acquisition device when acquiring sample point cloud data; determining, based on the sample point cloud data, a relative position relationship between the spatial position of the sample point cloud data and the spatial position of the point cloud acquisition device; determining the first positioning information based on the second positioning information and the relative position relationship.

[0087] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: determining the area range of the target area based on the first positioning information, including: obtaining second positioning information, wherein the second positioning information represents the spatial position at which the point cloud acquisition device collects sample point cloud data; generating a target area based on the first positioning information and the second positioning information, wherein the area range of the target area is centered on the spatial position represented by the second positioning information.

[0088] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: extracting map information from a digital map, including: obtaining multiple map layers from the digital map, wherein the multiple map layers respectively represent different aspects of the digital map; generating multiple layer images corresponding to the multiple map layers one by one based on the multiple map layers, wherein the map information includes the multiple layer images.

[0089] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: training the original point cloud annotation model according to map information and sample point cloud data to generate a target point cloud annotation model, including: inputting map information into a first feature extraction network of the original point cloud annotation model to output a first feature map; inputting sample point cloud data into a second feature extraction network of the original point cloud annotation model to output a second feature map; training the original point cloud annotation model according to the first feature map and the second feature map to generate a target point cloud annotation model.

[0090] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: training the original point cloud annotation model according to the first feature map and the second feature map to generate a target point cloud annotation model, including: inputting the first feature map and the second feature map into the fusion network layer to generate fusion features, wherein the feature dimensions of the first feature map and the second feature map are the same; inputting the fusion features into the neck network of the original point cloud annotation model, training the neck network and the head network of the original point cloud annotation model and updating the parameters of the neck network and the head network to obtain the target point cloud annotation model.

[0091] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: the pre-labeled sample point cloud data is point cloud data selected based on the following conditions: the distance between the sample point cloud data and the point cloud acquisition device is greater than a distance threshold; and / or the point cloud density of the sample point cloud data is less than a point cloud density threshold.

[0092] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0093] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0095] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0096] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0097] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0098] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A point cloud annotation model training method, characterized in that: include: Acquire pre-labeled sample point cloud data and a digital map of a target area, wherein the point cloud in the sample point cloud data is located within the area range of the target area; Extracting map information from the digital map, wherein the map information is used to describe a static environment in the target area; The original point cloud annotation model is trained according to the map information and the sample point cloud data to generate a target point cloud annotation model, wherein the target point cloud annotation model is used to annotate the point cloud data.

2. The method according to claim 1, characterized in that: Before obtaining the digital map of the target area, the method further includes: Determine first positioning information, wherein the first positioning information represents the spatial position of the sample point cloud data; The area range of the target area is determined according to the first positioning information, wherein the area range of the target area includes the spatial position represented by the first positioning information.

3. The method according to claim 2, characterized in that The determining the first positioning information includes: Acquire second positioning information, wherein the second positioning information represents a spatial position at which the point cloud acquisition device acquires the sample point cloud data; Determine, based on the sample point cloud data, a relative positional relationship between a spatial position of the sample point cloud data and a spatial position of the point cloud acquisition device; The first positioning information is determined according to the second positioning information and the relative position relationship.

4. The method according to claim 2, characterized in that: The determining the area range of the target area according to the first positioning information includes: Acquire second positioning information, wherein the second positioning information represents a spatial position at which the point cloud acquisition device acquires the sample point cloud data; The target area is generated according to the first positioning information and the second positioning information, wherein the area range of the target area is centered on the spatial position represented by the second positioning information.

5. The method according to claim 1, characterized in that The extracting map information from the digital map comprises: Acquire a plurality of map layers from the digital map, wherein the plurality of map layers respectively represent different aspects of the features of the digital map; According to the plurality of map layers, a plurality of layer images corresponding to the plurality of map layers are generated, wherein the map information includes the plurality of layer images.

6. The method according to any one of claims 1 to 5, characterized in that The training of the original point cloud annotation model according to the map information and the sample point cloud data to generate a target point cloud annotation model includes: Inputting the map information into a first feature extraction network of the original point cloud annotation model, and outputting a first feature map; Inputting the sample point cloud data into a second feature extraction network of the original point cloud annotation model, and outputting a second feature map; The original point cloud annotation model is trained according to the first feature map and the second feature map to generate the target point cloud annotation model.

7. The method according to claim 6, characterized in that The step of training the original point cloud annotation model according to the first feature map and the second feature map to generate the target point cloud annotation model includes: Inputting the first feature map and the second feature map into a fusion network layer to generate fusion features, wherein the first feature map and the second feature map have the same feature dimension; The fusion feature is input into the neck network of the original point cloud annotation model, the neck network and the head network of the original point cloud annotation model are trained and the parameters of the neck network and the head network are updated to obtain the target point cloud annotation model.

8. A point cloud annotation model training device, characterized in that: include: An acquisition module, used for acquiring pre-labeled sample point cloud data and a digital map of a target area, wherein the point cloud in the sample point cloud data is located within the area range of the target area; An extraction module, used to extract map information from the digital map, wherein the map information is used to describe the static environment in the target area; The training module is used to train the original point cloud annotation model according to the map information and the sample point cloud data to generate a target point cloud annotation model, wherein the target point cloud annotation model is used to annotate the point cloud data.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the point cloud annotation model training method described in any one of claims 1 to 7.

10. A computer device, characterized in that: The computer device includes a memory and a processor, the memory is used to store programs, and the processor is used to run the programs stored in the memory, wherein the point cloud annotation model training method described in any one of claims 1 to 7 is executed when the program is run.

Images