Method, device, equipment and medium for determining label

Abstract

The present application discloses a method, device, equipment and medium for label determination, and relates to the field of computer technology. The method comprises: obtaining point cloud semantic information of multiple frames of point clouds to be processed in a specified area; classifying and processing the point cloud semantic information of each frame to obtain static semantic point cloud and dynamic semantic point cloud of each frame; splicing the static semantic point cloud and dynamic semantic point cloud of each frame respectively to obtain dense static semantic point cloud and dense dynamic semantic point cloud; based on the dense static semantic point cloud and dense dynamic semantic point cloud, using a merging strategy, determining the dense semantic point cloud of each frame; based on the voxel grid network of the specified area, voxelizing each point cloud in the dense semantic point cloud of each frame to obtain the point cloud occupancy of each voxel grid; based on the point cloud occupancy of each voxel grid, selecting and processing the semantics of the point cloud in each voxel grid to obtain the occupied grid semantic label.

Description

Technical Field

[0001] The present application relates to the field of computer technology, specifically to technical fields such as positioning technology and autonomous driving, and more particularly to a method, device, equipment and medium for determining a label. Background Art

[0002] The general obstacle occupancy network model based on deep learning can directly learn to generate the mapping relationship of occupancy grids from raw sensor data, can handle complex environments and dynamic changes, and provide high-precision and real-time environmental perception capabilities. When training the obstacle occupancy network model, the true value label of the obstacle occupancy grid is required.

[0003] At present, the process of generating the true value label of the obstacle occupancy grid can be to collect point cloud data, segment the point cloud data, convert the segmented point cloud semantics into occupancy grid labels, and store the generated occupancy grid labels. Segmenting point cloud data usually requires manual or semi-automatic annotation tools. The annotator needs to classify each point and segment the point cloud into different categories, such as roads, pedestrians, vehicles, buildings, etc. Converting the point cloud into an occupancy grid is to assign each point cloud point to a fixed-size grid cell, and the state and category of each grid cell is determined by the label of the point it contains. Summary of the invention

[0004] The present application provides a method, device, equipment and medium for determining a label, which can solve the problem that the determined obstacle occupancy grid label data is not accurate. The technical solution is as follows:

[0005] In a first aspect, a method for determining a label is provided, the method comprising:

[0006] Obtaining point cloud semantic information of at least one frame of point cloud to be processed in a specified area;

[0007] Classify the point cloud semantic information of each frame to obtain a static semantic point cloud and a dynamic semantic point cloud of each frame;

[0008] The static semantic point cloud and the dynamic semantic point cloud of each frame are spliced ​​to obtain a dense static semantic point cloud and a dense dynamic semantic point cloud respectively;

[0009] Based on the dense static semantic point cloud and the dense dynamic semantic point cloud, a dense semantic point cloud of each frame is determined using a preset merging strategy;

[0010] Based on the preset voxel grid network of the designated area, voxelize each point cloud in the dense semantic point cloud of each frame to obtain the point cloud occupancy of each voxel grid in the voxel grid network;

[0011] Based on the point cloud occupancy of each voxel grid, the semantics of the point cloud in each voxel grid is selected and processed to determine the occupied grid semantic label of each voxel grid.

[0012] In a possible implementation, the point cloud semantic information of each frame is classified to obtain a static semantic point cloud and a dynamic semantic point cloud of each frame, including:

[0013] Using the preset target detection model, each frame of point cloud is detected and processed to obtain the dynamic obstacle detection results of each frame;

[0014] Based on the dynamic obstacle detection result of each frame, extracting and processing the dynamic semantic point cloud in the point cloud semantic information of each frame to obtain the dynamic semantic point cloud of each frame;

[0015] Based on the dynamic semantic point cloud of each frame, the dynamic semantic point cloud in the point cloud semantic information of each frame is filtered to obtain the static semantic point cloud of each frame.

[0016] In a possible implementation, the splicing processing of the dynamic semantic point cloud of each frame to obtain a dense dynamic semantic point cloud includes:

[0017] Get the dynamic obstacle detection results corresponding to the dynamic semantic point cloud of each frame;

[0018] Based on the dynamic obstacle detection result corresponding to the dynamic semantic point cloud of each frame, splicing the dynamic semantic point cloud of each frame;

[0019] Based on the results of the splicing process, a dense dynamic semantic point cloud is obtained.

[0020] In a possible implementation, the static semantic point cloud of each frame is spliced ​​to obtain a dense static semantic point cloud, including:

[0021] Based on a preset order, stitching the static semantic point cloud of each frame;

[0022] Based on the results of the stitching process, a dense static semantic point cloud is obtained.

[0023] In a possible implementation, determining a dense semantic point cloud of each frame based on the dense static semantic point cloud and the dense dynamic semantic point cloud using a preset merging strategy includes:

[0024] The following operations are performed for each frame of point cloud: obtain the position data of the point cloud of the current frame;

[0025] Based on the position data of the point cloud of the current frame, converting the dense static semantic point cloud to obtain a dense static semantic point cloud of the current frame;

[0026] Get the dynamic obstacle detection results of the point cloud of the current frame;

[0027] Performing matching processing on the dynamic obstacle detection result and the dense dynamic semantic point cloud to obtain a dense dynamic semantic point cloud of the current frame;

[0028] Merging the dense static semantic point cloud of the current frame and the dense dynamic semantic point cloud of the current frame to obtain a dense semantic point cloud of the current frame;

[0029] Based on the dense semantic point cloud of each current frame, a dense semantic point cloud of each frame is determined.

[0030] In a possible implementation, the voxel grid network preset based on the designated area performs voxelization processing on each point cloud in the dense semantic point cloud of each frame to obtain the point cloud occupancy of each voxel grid in the voxel grid network, including:

[0031] Based on the preset voxel grid network of the designated area, voxelize each point cloud in the dense semantic point cloud of each frame to obtain a position index of each point cloud;

[0032] Determining the number of point clouds within each voxel grid based on the position index of each point cloud and the preset position index of the voxel grid;

[0033] Based on the number of point clouds in each voxel grid, the point cloud occupancy of each voxel grid is obtained.

[0034] In a possible implementation, the semantics of the point cloud in each voxel grid is selected and processed based on the point cloud occupancy of each voxel grid to determine the occupied grid semantic label of each voxel grid, including:

[0035] In response to the point cloud occupancy of the voxel grid being point cloud occupied, obtaining the number of point clouds in the voxel grid and the semantics of the point clouds;

[0036] Based on the number of point clouds in the voxel grid and the semantics of the point clouds, a voting mechanism is used to select the semantics of the point cloud with the highest vote rate in the voxel grid;

[0037] The semantics of the point cloud with the highest vote rate is determined as the occupied grid semantic label of the voxel grid.

[0038] In a second aspect, a method for training an obstacle occupancy network model is provided, the method comprising:

[0039] Acquire sample data and an occupied grid semantic label corresponding to the sample data, wherein the occupied grid semantic label is determined by processing the sample data according to the method described in the first aspect and any possible implementation manner thereof;

[0040] Based on the sample data and the occupied grid semantic labels corresponding to the sample data, performing iterative training processing on the obstacle occupancy network model to be trained;

[0041] In response to the training reaching a preset termination condition, a trained obstacle occupancy network model is obtained.

[0042] In a third aspect, a device for determining a label is provided, the device comprising:

[0043] An acquisition unit, used for acquiring point cloud semantic information of at least one frame of point cloud to be processed in a specified area;

[0044] A classification unit, used for classifying the point cloud semantic information of each frame to obtain a static semantic point cloud and a dynamic semantic point cloud of each frame;

[0045] A splicing unit, used for splicing the static semantic point cloud and the dynamic semantic point cloud of each frame respectively, so as to obtain a dense static semantic point cloud and a dense dynamic semantic point cloud respectively;

[0046] A merging unit, configured to determine a dense semantic point cloud of each frame based on the dense static semantic point cloud and the dense dynamic semantic point cloud using a preset merging strategy;

[0047] An obtaining unit, configured to voxelize each point cloud in the dense semantic point cloud of each frame based on a preset voxel grid network of the designated area, so as to obtain the point cloud occupancy of each voxel grid in the voxel grid network;

[0048] The determination unit is used to select and process the semantics of the point cloud in each voxel grid based on the point cloud occupancy of each voxel grid, so as to determine the occupied grid semantic label of each voxel grid.

[0049] In a fourth aspect, a training device for an obstacle occupancy network model is provided, the device comprising:

[0050] an acquiring unit, configured to acquire sample data and an occupied grid semantic label corresponding to the sample data, wherein the occupied grid semantic label is determined by processing the sample data according to the method described in the first aspect and any possible implementation manner thereof;

[0051] A training unit, configured to perform iterative training processing on an obstacle occupancy network model to be trained based on the sample data and the occupancy grid semantic labels corresponding to the sample data;

[0052] The obtaining unit is used to obtain the obstacle occupancy network model after training is completed in response to the training reaching a preset termination condition.

[0053] In a fifth aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the first aspect and any possible implementation thereof, and the method of the second aspect as described above.

[0054] In a sixth aspect, an electronic device is provided, including:

[0055] at least one processor; and

[0056] a memory communicatively connected to the at least one processor; wherein,

[0057] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the first aspect and any possible implementation thereof, and the method of the second aspect as described above.

[0058] In a seventh aspect, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the first aspect and any possible implementation manner thereof, and the method of the second aspect as described above.

[0059] In an eighth aspect, an autonomous driving vehicle is provided, comprising the electronic device as described above.

[0060] The beneficial effects of the technical solution provided by this application include at least:

[0061] It can be seen from the above technical scheme that, on the one hand, the embodiment of the present application can obtain the point cloud semantic information of at least one frame of point cloud to be processed in the specified area, and then classify the point cloud semantic information of each frame to obtain a static semantic point cloud and a dynamic semantic point cloud for each frame, respectively splice the static semantic point cloud and the dynamic semantic point cloud of each frame to obtain a dense static semantic point cloud and a dense dynamic semantic point cloud, respectively, determine the dense semantic point cloud of each frame based on the dense static semantic point cloud and the dense dynamic semantic point cloud using a preset merging strategy, and voxelize each point cloud in the dense semantic point cloud of each frame based on a preset voxel grid network of the specified area to obtain the point cloud occupancy of each voxel grid, so that the occupancy of each voxel grid can be obtained, so that the occupancy of each voxel grid can be obtained. The point cloud occupancy of the voxel grid is determined, and the semantics of the point cloud in each voxel grid are selected and processed to determine the occupied grid semantic label of each voxel grid. Since the static semantic point cloud and dynamic semantic point cloud of each frame can be determined based on the point cloud semantic information of each frame, the static semantic point cloud and dynamic semantic point cloud of each frame are spliced ​​respectively to obtain dense static semantic point cloud and dense dynamic semantic point cloud, and the dense static semantic point cloud and dense dynamic semantic point cloud of each frame are merged, and then the dense semantic point cloud obtained after the merger can be used to determine the occupied grid semantic label corresponding to the voxel grid, which can realize the automatic determination of the occupied grid semantic label, accurately and effectively convert the semantics of the point cloud to the occupied grid semantic label of the corresponding grid, thereby improving the accuracy and reliability of the occupied grid semantic label.

[0062] On the other hand, the embodiment of the present application can classify the point cloud semantic information of each frame corresponding to the sample data to obtain a static semantic point cloud and a dynamic semantic point cloud of each frame, respectively splice the static semantic point cloud and the dynamic semantic point cloud of each frame to respectively obtain a dense static semantic point cloud and a dense dynamic semantic point cloud, based on the dense static semantic point cloud and the dense dynamic semantic point cloud, using a preset merging strategy, determine the dense semantic point cloud of each frame, based on the preset voxel grid network of the specified area, voxelize each point cloud in the dense semantic point cloud of each frame to obtain a point of each voxel grid. The cloud occupancy situation makes it possible to select and process the semantics of the point cloud in each voxel grid based on the point cloud occupancy situation of each voxel grid, and obtain the occupied grid semantic label corresponding to the sample data, and then based on the sample data and the occupied grid semantic label corresponding to the sample data, the obstacle occupancy network model to be trained can be iteratively trained, and in response to the training reaching the preset termination condition, a trained obstacle occupancy network model is obtained. Since the obstacle occupancy network model can be trained by utilizing the occupied grid semantic label of the sample data more effectively and accurately, the robustness and performance of the trained obstacle occupancy network model can be improved.

[0063] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0065] Figure 1 It is a flowchart of a method for determining a label provided by an embodiment of the present application;

[0066] Figure 2 is a flowchart of a method for training an obstacle occupancy network model provided by another embodiment of the present application;

[0067] Figure 3 is a flowchart of a method for determining a label provided by another embodiment of the present application;

[0068] Figure 4 is a structural block diagram of a device for determining a label provided in yet another embodiment of the present application;

[0069] Figure 5 is a structural block diagram of a training device for an obstacle occupancy network model provided in yet another embodiment of the present application;

[0070] Figure 6 It is a block diagram of an electronic device used to implement the tag determination method of an embodiment of the present application. DETAILED DESCRIPTION

[0071] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0072] Obviously, the described embodiments are only part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0073] It should be noted that the terminal devices involved in the embodiments of the present application may include but are not limited to mobile phones, personal digital assistants (PDAs), wireless handheld devices, tablet computers and other smart devices; display devices may include but are not limited to personal computers, televisions and other devices with display functions.

[0074] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0075] Universal obstacle occupancy grid technology is a key technology in autonomous driving, intelligent robots, environmental perception and navigation systems. It achieves accurate representation and understanding of three-dimensional space by dividing the environment into grids and recording the occupancy status of each grid cell.

[0076] At present, the obstacle occupancy network model based on deep learning can directly learn to generate the mapping relationship of the occupancy grid from the original sensor data. It can handle complex environments and dynamic changes, provide high-precision and real-time environmental perception capabilities, and provide safe and reliable navigation and decision-making support for autonomous driving vehicles and robots.

[0077] Training the obstacle occupancy network model requires the obstacle occupancy grid true value label, that is, the occupancy grid semantic label. Generating the occupancy grid true value label generally involves several key steps: collecting data, segmenting the scene point cloud, converting the point cloud into an occupancy grid, and storing the generated occupancy grid semantic label. Point cloud semantic segmentation usually requires manual or semi-automatic annotation tools. The annotator needs to classify each point and divide the point cloud into different categories, such as roads, pedestrians, vehicles, buildings, etc. Converting the point cloud into an occupancy grid is to assign each point to a fixed-size grid cell. The state and category of each grid cell is determined by the labels of the points it contains.

[0078] However, there are still some problems with the above-mentioned related technologies. The manual annotation process of point cloud segmentation is very time-consuming, labor-intensive and costly. Due to sensor noise, complex real environment and subjective differences in the annotation process, the accuracy of the generated occupancy grid labels is limited, and large amounts of occupancy grid label data will take up a lot of storage space.

[0079] Therefore, there is an urgent need for a label determination method to automatically and effectively semantically annotate occupied grids, thereby ensuring the reliability and accuracy of occupied grid semantic labels.

[0080] Please refer to Figure 1 , which shows a flow chart of a method for determining a label provided by an embodiment of the present application. The method for determining a label may specifically include:

[0081] Step 101: Obtain point cloud semantic information of at least one frame of point cloud to be processed in a specified area.

[0082] Step 102: Classify the point cloud semantic information of each frame to obtain a static semantic point cloud and a dynamic semantic point cloud of each frame.

[0083] Step 103 : splicing the static semantic point cloud and the dynamic semantic point cloud of each frame respectively to obtain a dense static semantic point cloud and a dense dynamic semantic point cloud respectively.

[0084] Step 104: Based on the dense static semantic point cloud and the dense dynamic semantic point cloud, a dense semantic point cloud of each frame is determined using a preset merging strategy.

[0085] Step 105 : Based on the preset voxel grid network of the designated area, voxelize each point cloud in the dense semantic point cloud of each frame to obtain the point cloud occupancy of each voxel grid in the voxel grid network.

[0086] Step 106 : Based on the point cloud occupancy of each voxel grid, the semantics of the point cloud in each voxel grid is selected and processed to determine the occupied grid semantic label of each voxel grid.

[0087] It should be noted that the designated area may be an area where point clouds are collected. The at least one frame of point clouds may include point clouds of each frame of consecutive frames.

[0088] It should be noted that the dense static semantic point cloud may be a dense point cloud representing a static object or a static scene, that is, it may be a dense point cloud with a static semantic category.

[0089] It should be noted that the dense dynamic semantic point cloud may be a dense point cloud representing a dynamic object, that is, it may be a dense point cloud with dynamic semantic categories.

[0090] It should be noted that the dense semantic point cloud may be a dense point cloud with semantic categories. Each point cloud in the dense semantic point cloud may have a corresponding semantic category, that is, a semantic label.

[0091] It should be noted that part or all of the execution subject of steps 101 to 106 may be an application located in the local terminal, or may also be a functional unit such as a plug-in or software development kit (SDK) set in the application located in the local terminal, or may also be a processing engine located in a network-side server, or may also be a distributed system located on the network side, for example, a processing engine or distributed system in a data processing platform on the network side, etc. This embodiment does not specifically limit this.

[0092] It is understandable that the application may be a local program (nativeApp) installed on the local terminal, or may also be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0093] In this way, the static semantic point cloud and dynamic semantic point cloud of each frame can be determined based on the point cloud semantic information of each frame, and then the static semantic point cloud and dynamic semantic point cloud of each frame can be spliced ​​respectively to obtain dense static semantic point cloud and dense dynamic semantic point cloud, and the dense static semantic point cloud and dense dynamic semantic point cloud of each frame can be merged, and then the dense semantic point cloud obtained after the merger can be used to determine the occupied grid semantic label corresponding to the voxel grid, which can realize the automatic determination of the occupied grid semantic label, and accurately and effectively convert the semantics of the point cloud to the occupied grid semantic label of the corresponding grid, thereby improving the accuracy and reliability of the occupied grid semantic label.

[0094] Optionally, in a possible implementation of this embodiment, in step 101, specifically, first, at least one frame of point cloud and an image corresponding to the point cloud may be obtained. Secondly, the image corresponding to the point cloud is semantically segmented using an image semantic segmentation model to obtain an image semantic segmentation result. Thirdly, the point cloud is mapped to the image semantic segmentation result to use the semantic segmentation labels of the corresponding pixels on the image as the semantic segmentation result of the point cloud. Thirdly, the semantic segmentation result of the point cloud is used as the semantic information of the point cloud.

[0095] In this way, the image corresponding to the point cloud can be semantically segmented by utilizing the image semantic segmentation model, thereby realizing the automation of point cloud semantic annotation and improving data processing efficiency.

[0096] Optionally, in a possible implementation of this embodiment, in step 101, at least one frame of point cloud and an image corresponding to the point cloud may be first acquired. Secondly, the point cloud may be subjected to ground point cloud recognition processing using a first recognition algorithm to obtain the first ground point cloud. Thirdly, the point cloud may be subjected to obstacle recognition processing using a second recognition algorithm to obtain the first obstacle point cloud. Thirdly, based on the point cloud and the image, the first point cloud semantic segmentation corresponding to the point cloud is determined using a preset image semantic segmentation strategy. Thirdly, based on the first ground point cloud, the point cloud of ground semantics in the first point cloud semantic segmentation is filtered to obtain second point cloud semantic segmentation data and second ground point cloud data. Thirdly, based on the first obstacle point cloud data, the point cloud data of obstacle semantics in the second point cloud semantic segmentation data is filtered to obtain third point cloud semantic segmentation data and second obstacle point cloud data. Thirdly, based on the second ground point cloud data, the second obstacle point cloud data, and the third point cloud semantic segmentation data, a point cloud semantic segmentation result of the point cloud is obtained. Finally, the semantic segmentation result of the point cloud can be used as the point cloud semantic information.

[0097] In this implementation, the first recognition algorithm may include a ground point cloud recognition algorithm. The second recognition algorithm may include a target obstacle detection algorithm based on a 3D target detection model.

[0098] In this way, the first point cloud semantic segmentation data determined by using the image semantic segmentation strategy can be filtered and processed respectively according to the determined obstacle point cloud data and ground point cloud data, so as to obtain the third point cloud semantic segmentation data, the second ground point cloud data, and the second obstacle point cloud data. Then, the second ground point cloud data, the second obstacle point cloud data, and the third point cloud semantic segmentation data are used as the point cloud semantic segmentation results of the point cloud data. The multi-dimensional point cloud semantic recognition results can be combined to obtain a more accurate point cloud semantic segmentation result, thereby improving the reliability of the point cloud semantic segmentation results and ensuring the reliability of the point cloud semantic information. At the same time, the automation of point cloud semantic annotation is realized, thereby improving data processing efficiency.

[0099] It should be noted that the specific implementation process provided in this implementation can be combined with the various specific implementation processes provided in the aforementioned implementation to implement the label determination method of this embodiment. For detailed description, please refer to the relevant content in the aforementioned implementation, which will not be repeated here.

[0100] Optionally, in a possible implementation of the present embodiment, in step 102, a preset target detection model can be used to perform detection processing on each frame of point cloud to obtain a dynamic obstacle detection result for each frame, and then based on the dynamic obstacle detection result for each frame, the dynamic semantic point cloud in the point cloud semantic information of each frame can be extracted to obtain the dynamic semantic point cloud for each frame, and based on the dynamic semantic point cloud of each frame, the dynamic semantic point cloud in the point cloud semantic information of each frame can be filtered to obtain a static semantic point cloud for each frame.

[0101] In this implementation, the dynamic obstacle detection result may include, but is not limited to, a dynamic obstacle detection frame and a dynamic obstacle category identifier (ID). The dynamic obstacle category identifier may be a dynamic obstacle detection frame category identifier.

[0102] In this implementation, the preset target detection model may include a 3D target detection model.

[0103] It can be understood that here, for each single-frame point cloud, a 3D target detection model is used to detect and process each single-frame point cloud, so that dynamic obstacles, such as 3D detection frames of vehicles and pedestrians, can be obtained, and the point cloud within the detection frame can be extracted to obtain a dynamic semantic point cloud. Then, using the dynamic semantic point cloud, the point cloud semantic information of the point cloud is filtered to obtain a static semantic point cloud, thereby, the single-frame point cloud can be divided into a dynamic semantic point cloud and a static semantic point cloud. The dynamic semantic point cloud can be a dynamic object point cloud. The static semantic point cloud can be a static scene point cloud. Moreover, the dynamic semantic point cloud and the static semantic point cloud can be saved separately to achieve dynamic and static separation of the single-frame point cloud.

[0104] In this way, the target detection model can be used to detect dynamic obstacles in each frame of point cloud, and then the dynamic semantic point cloud and static semantic point cloud in the point cloud semantic information can be separated based on the detected dynamic obstacles, so as to obtain the static semantic point cloud and dynamic semantic point cloud of each frame respectively. This can improve the reliability of the static semantic point cloud and the dynamic semantic point cloud, and improve the reliability of the semantics of the point cloud, thereby further ensuring the accuracy of the subsequently determined occupied grid semantic labels.

[0105] It should be noted that the specific implementation process provided in this implementation can be combined with the various specific implementation processes provided in the aforementioned implementation to implement the label determination method of this embodiment. For detailed description, please refer to the relevant content in the aforementioned implementation, which will not be repeated here.

[0106] Optionally, in a possible implementation of this embodiment, in step 103, specifically, first, a dynamic obstacle detection result corresponding to the dynamic semantic point cloud of each frame may be obtained. Secondly, based on the dynamic obstacle detection result corresponding to the dynamic semantic point cloud of each frame, the dynamic semantic point cloud of each frame may be spliced. Thirdly, based on the result of the splicing process, a dense dynamic semantic point cloud may be obtained.

[0107] In this implementation, the dynamic obstacle detection result includes a dynamic obstacle category ID. The dynamic semantic point cloud of each frame may be arranged in a preset time sequence, that is, the dynamic semantic point cloud of consecutive frames.

[0108] In a specific implementation process of this implementation method, the dynamic semantic point cloud of each frame can be spliced ​​based on the dynamic obstacle category ID of the dynamic obstacle detection result corresponding to the dynamic semantic point cloud of each frame. Furthermore, a dense dynamic semantic point cloud can be obtained based on the result of the splicing process.

[0109] In this implementation, the dense dynamic semantic point cloud may be a dynamic semantic point cloud of consecutive frames in a preset time sequence. The dense dynamic semantic point cloud may be a dense point cloud representing a dynamic object.

[0110] Optionally, in a possible implementation of the present embodiment, in step 103, the static semantic point cloud of each frame may be stitched together based on a preset order, and a dense static semantic point cloud may be obtained based on the result of the stitching process.

[0111] In this implementation, the preset order may be the time order of each frame of static semantic point cloud.

[0112] In a specific implementation process of this implementation method, the static semantic point cloud of each frame is spliced ​​based on a preset time sequence, and then a dense static semantic point cloud can be obtained based on the result of the splicing process.

[0113] In this implementation, the dense static semantic point cloud may be a static scene point cloud obtained by splicing consecutive frames. The dense static semantic point cloud may be a dense point cloud representing a static scene.

[0114] Optionally, in a possible implementation of the present embodiment, in step 104, specifically, first, the following operations may be performed for each frame of the point cloud: obtain the position data of the point cloud of the current frame. Secondly, based on the position data of the point cloud of the current frame, the dense static semantic point cloud may be converted to obtain a dense static semantic point cloud of the current frame. Thirdly, the dynamic obstacle detection result of the point cloud of the current frame is obtained. Thirdly, the dynamic obstacle detection result and the dense dynamic semantic point cloud are matched to obtain a dense dynamic semantic point cloud of the current frame. Thirdly, the dense static semantic point cloud of the current frame and the dense dynamic semantic point cloud of the current frame are merged to obtain a dense semantic point cloud of the current frame. Thirdly, based on the dense semantic point cloud of each current frame, the dense semantic point cloud of each frame is determined.

[0115] Optionally, in a possible implementation of the present embodiment, in step 105, specifically, first, based on the preset voxel grid network of the designated area, voxelize each point cloud in the dense semantic point cloud of each frame, and obtain the position index of each point cloud. Next, based on the position index of each point cloud and the preset position index of the voxel grid, determine the number of point clouds in each voxel grid. Next, based on the number of point clouds in each voxel grid, obtain the point cloud occupancy of each voxel grid.

[0116] In this implementation, the preset voxel grid network of the designated area may include multiple voxel grids. The preset position index of the voxel grid may be determined according to the position information of each voxel grid in the preset voxel grid network.

[0117] In a specific implementation process of the implementation method, for a point cloud in the dense semantic point cloud of each frame, the position coordinates of the point cloud can be (x, y, z), and the point cloud can be voxelized based on the preset voxel grid network of the specified area to obtain the position index of the point cloud. ,i [x, y, z] can be expressed as follows: (1)

[0118] Among them, i_ub is the lower limit of the preset valid point cloud range in each coordinate axis direction, and i_size can be the size of the unit voxel grid in each coordinate axis direction in the coordinate system defined by it, or the length.

[0119] For example, for a three-dimensional Euclidean geometric space with meters as units, the unit voxel grid, that is, a voxel grid, can be a cube with a length, width and height of 1m. In this case, i_size can be 1, i [x,y,z].

[0120] Here, the preset effective point cloud range can be determined based on the effective range of point cloud collection corresponding to one scan of the laser radar.

[0121] In another specific implementation process of the implementation method, the position index of each point cloud is matched with the preset position index of each voxel grid, and the number of point clouds in each voxel grid is determined based on the result of the matching process. Thirdly, based on the number of point clouds in each voxel grid, the point cloud occupancy of each voxel grid is obtained.

[0122] Exemplarily, each position of the voxelized grid is traversed, and if the number of point clouds in the voxel grid is greater than 0, it can be determined that the point cloud occupancy status of the voxel grid is point cloud occupied, otherwise, its point cloud occupancy status is point cloud unoccupied.

[0123] Optionally, in a possible implementation of this embodiment, in step 105, specifically, first, in response to the point cloud occupancy of the voxel grid being point cloud occupied, the number of point clouds in the voxel grid and the semantics of the point clouds are obtained. Secondly, based on the number of point clouds in the voxel grid and the semantics of the point clouds, a voting mechanism is used to select the semantics of the point cloud with the highest vote rate in the voxel grid. Thirdly, the semantics of the point cloud with the highest vote rate is determined as the occupied grid semantic label of the voxel grid.

[0124] In this implementation, the number of semantics of the point cloud may be multiple.

[0125] Optionally, in a possible implementation of the present embodiment, after step 106, first, the position information of each voxel grid can be obtained. Secondly, based on the position information of each voxel grid, the position index of each voxel grid is determined. Thirdly, based on the occupied grid semantic label of each voxel grid, the occupied grid semantic label index of each voxel grid is determined. Thirdly, based on the acquisition range corresponding to the preset voxel grid network, the position index of each voxel grid and the occupied grid semantic label index, each voxel grid is encoded to obtain each encoded voxel grid, and each encoded voxel grid can be stored.

[0126] In a specific implementation process of this implementation method, the collection range corresponding to the preset voxel grid network can be determined based on the collection coverage area of ​​the laser radar.

[0127] Here, the acquisition coverage area of ​​the laser radar may be the area that the laser radar can acquire. The position index of each voxel grid may be a preset position index of each voxel grid. The preset position index of each voxel grid may be determined based on the position information of each voxel grid.

[0128] In another specific implementation process of the implementation method, for any voxel grid, each voxel grid can be encoded based on the position index of the voxel grid and the occupied grid semantic label to obtain the encoded voxel grid index, which can be shown in the following formula (2): index=x_index*y_size*z_size*c_size+y_index*z_size*c_size+z_index*c_size+c_index (2)

[0129] Among them, x_index, y_index, and z_index may represent the position index of the voxel grid, c_index may represent the occupied grid semantic label of the voxel grid, that is, the category index, and y_size, z_size, and c_size may be the maximum values ​​of y_index, z_index, and c_index, respectively.

[0130] Exemplarily, here, the encoded voxel grid may be a 32-bit integer value.

[0131] In this way, each voxel grid with an occupied grid semantic label can be encoded and stored in an encoded form, which can greatly reduce the storage space of a large amount of occupied grid semantic label data.

[0132] It should be noted that the specific implementation process provided in this implementation can be combined with the various specific implementation processes provided in the aforementioned implementation to implement the label determination method of this embodiment. For detailed description, please refer to the relevant content in the aforementioned implementation, which will not be repeated here.

[0133] Figure 2 is a flowchart of a training method for an obstacle occupancy network model provided by another embodiment of the present application, such as Figure 2 shown.

[0134] Step 201: Obtain sample data and an occupied grid semantic label corresponding to the sample data, wherein the occupied grid semantic label is determined by processing the sample data according to the method described in the above embodiment.

[0135] Step 202: Based on the sample data and the occupied grid semantic labels corresponding to the sample data, an iterative training process is performed on the obstacle occupancy network model to be trained.

[0136] Step 203: In response to the training reaching a preset termination condition, a trained obstacle occupancy network model is obtained.

[0137] It should be noted that the sample data may include a point cloud of a specified area.

[0138] It should be noted that the preset termination conditions may include the number of iterations reaching a preset threshold, the training results satisfying a preset convergence condition, etc.

[0139] It should be noted that before step 201, the point cloud semantic information of each frame corresponding to the sample data can be classified and processed to obtain a static semantic point cloud and a dynamic semantic point cloud of each frame, and the static semantic point cloud and the dynamic semantic point cloud of each frame are spliced ​​to obtain a dense static semantic point cloud and a dense dynamic semantic point cloud respectively. Based on the dense static semantic point cloud and the dense dynamic semantic point cloud, a preset merging strategy is used to determine the dense semantic point cloud of each frame, and based on the preset voxel grid network of the specified area, each point cloud in the dense semantic point cloud of each frame is voxelized to obtain the point cloud occupancy of each voxel grid, so that the semantics of the point cloud in each voxel grid can be selected and processed based on the point cloud occupancy of each voxel grid to obtain the occupied grid semantic label corresponding to the sample data.

[0140] It should be noted that before step 201, each encoded voxel grid corresponding to the sample data can be obtained from the database based on the sample data, and then each encoded voxel grid can be decoded to obtain an occupied grid semantic label corresponding to the sample data.

[0141] It should be noted that part or all of the execution subject of steps 201 to 203 may be an application located in the local terminal, or may also be a functional unit such as a plug-in or software development kit (SDK) set in the application located in the local terminal, or may also be a processing engine located in a network-side server, or may also be a distributed system located on the network side, for example, a processing engine or distributed system in a model training platform on the network side, etc. This embodiment does not specifically limit this.

[0142] It is understandable that the application may be a local program (nativeApp) installed on the local terminal, or may also be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0143] In this way, the obstacle occupancy network model can be trained by utilizing the occupancy grid semantic labels of the sample data more effectively and accurately, thereby improving the robustness and performance of the trained obstacle occupancy network model.

[0144] It can be understood that the specific implementation method of determining the occupied grid semantic label corresponding to the sample data can be found in the label determination method of the aforementioned embodiment, and the corresponding specific implementation method and its specific implementation process are not repeated here.

[0145] In order to better understand the method of the embodiment of the present application, the method of the embodiment of the present application is described below in conjunction with the accompanying drawings and specific application scenarios.

[0146] Figure 3 is a flowchart of a method for determining a label provided by another embodiment of the present application. Figure 3 shown.

[0147] Step 301: Acquire at least one frame of point cloud and an image corresponding to the point cloud.

[0148] In this embodiment, at least one frame of point cloud may be point cloud data collected in a designated area. A single frame of point cloud in at least one frame of point cloud may be point cloud data collected by a laser radar scanning cycle. An image corresponding to a single frame of point cloud may be image data collected by an image acquisition device such as a camera corresponding to the point cloud data collected by a laser radar scanning cycle.

[0149] Step 302: Using the image semantic segmentation model, perform semantic segmentation processing on the image corresponding to the point cloud of each frame to obtain the image semantic segmentation result corresponding to the point cloud of each frame.

[0150] In this embodiment, for a single frame of point cloud, an image semantic segmentation model can be used to perform semantic segmentation processing on the image corresponding to the point cloud to obtain an image semantic segmentation result corresponding to the point cloud.

[0151] Step 303: Map the point cloud of each frame to the image semantic segmentation result corresponding to the point cloud of each frame, so as to use the semantic segmentation label of the corresponding pixel on the image as the semantic segmentation result of the point cloud, and use the semantic segmentation result of the point cloud as the semantic information of the point cloud.

[0152] Step 304: Use the 3D target detection model to perform detection processing on each frame of point cloud to obtain a dynamic obstacle detection result for each frame.

[0153] Step 305 : Based on the dynamic obstacle detection result of each frame, separate the dynamic semantic point cloud and the static semantic point cloud in the point cloud semantic information of each frame to obtain the static semantic point cloud and the dynamic semantic point cloud of each frame.

[0154] In this embodiment, firstly, based on the dynamic obstacle detection result of each frame, the dynamic semantic point cloud in the point cloud semantic information of each frame is extracted. Secondly, the other semantic point clouds in the point cloud semantic information of each frame after the dynamic semantic point cloud is extracted are used as the static semantic point cloud of each frame.

[0155] Step 306 : Based on the dynamic obstacle category ID of the dynamic obstacle detection result corresponding to the dynamic semantic point cloud of each frame, the dynamic semantic point cloud of each frame is spliced ​​to obtain a dense dynamic semantic point cloud.

[0156] In this embodiment, the dynamic obstacle detection result may include but is not limited to a dynamic obstacle detection frame and a dynamic obstacle category ID. The dynamic obstacle category ID may be a dynamic obstacle detection frame category ID.

[0157] Step 307 , based on the time sequence of the point cloud of each frame, the static semantic point cloud of each frame is spliced ​​to obtain a dense static semantic point cloud.

[0158] In this embodiment, the dense static semantic point cloud may be a static semantic point cloud obtained by splicing continuous frame point clouds.

[0159] In this embodiment, the time sequence of the point cloud of each frame may be the acquisition time sequence of the point cloud of each frame.

[0160] Step 308, perform the following operations for the point cloud of each frame: based on the position data of the point cloud of the current frame, convert the dense static semantic point cloud to obtain the dense static semantic point cloud of the current frame, match the dynamic obstacle detection result of the point cloud of the current frame and the dense dynamic semantic point cloud to obtain the dense dynamic semantic point cloud of the current frame, and merge the dense static semantic point cloud of the current frame and the dense dynamic semantic point cloud of the current frame to obtain the dense semantic point cloud of the current frame, so as to determine the dense semantic point cloud of each frame.

[0161] Step 309 : Based on the preset voxel grid network of the designated area, voxelize each point cloud in the dense semantic point cloud of each frame to obtain the point cloud occupancy of each voxel grid.

[0162] In this embodiment, the preset voxel grid network of the designated area includes a plurality of voxel grids.

[0163] Step 310 , in response to the point cloud occupancy of the voxel grid being point cloud occupied, based on the number of point clouds in the voxel grid and the semantics of the point clouds, a voting mechanism is used to select the semantics of the point cloud with the highest vote rate in the voxel grid as the occupied grid semantic label of the voxel grid.

[0164] In this embodiment, specifically, first, based on the preset voxel grid network of the specified area, each point cloud in the dense semantic point cloud of each frame is voxelized to obtain the position index of each point cloud. Next, based on the position index of each point cloud and the preset position index of the voxel grid, the number of point clouds in each voxel grid is determined. Next, based on the number of point clouds in each voxel grid, the point cloud occupancy of each voxel grid is obtained.

[0165] Here, the position index of each point cloud can be expressed as ,i [x,y,z], that is, the x_index, y_index, and z_index of the coordinates of the point cloud.

[0166] It can be understood that the position index of each point cloud can be calculated using the above formula (1): .

[0167] In this embodiment, the point cloud occupancy status of the voxel grid may include point cloud occupation and point cloud non-occupancy.

[0168] Exemplarily, each position of the voxelized grid is traversed, and if the number of point clouds in the voxel grid is greater than 0, it can be determined that the point cloud occupancy status of the voxel grid is point cloud occupied, otherwise, its point cloud occupancy status is point cloud unoccupied.

[0169] In this embodiment, each voxel grid can be traversed, and for each voxel grid, the point cloud occupancy of the voxel grid is responded to as the point cloud occupancy, and based on the number of point clouds in the voxel grid and the semantics of the point clouds, a voting mechanism is used to select the semantics of the point cloud with the highest vote rate in the voxel grid as the occupied grid semantic label of the voxel grid.

[0170] At this point, the trained obstacle occupancy network model can be iteratively trained based on the occupancy grid semantic labels of the voxel grid and the corresponding point cloud sample data to obtain a trained obstacle occupancy network model.

[0171] Step 311: Based on the acquisition range corresponding to the preset voxel grid network, the preset position index of each voxel grid and the occupied grid semantic label of each voxel grid, each voxel grid is encoded to obtain each encoded voxel grid, so as to store each encoded voxel grid.

[0172] Here, the preset position index of each voxel grid may be determined according to the position information of each voxel grid in the preset voxel grid network.

[0173] In this embodiment, the occupied grid semantic label index of each voxel grid can also be determined based on the occupied grid semantic label of each voxel grid, and then each voxel grid can be encoded based on the acquisition range corresponding to the preset voxel grid network, the preset position index of each voxel grid and the occupied grid semantic label index to obtain each encoded voxel grid, so as to store each encoded voxel grid.

[0174] It can be understood that, here, the aforementioned formula (2) can be used to calculate each encoded voxel grid index.

[0175] By adopting the technical solution in this embodiment, the static semantic point cloud and dynamic semantic point cloud of each frame can be determined based on the point cloud semantic information of each frame, and then the static semantic point cloud and dynamic semantic point cloud of each frame are spliced ​​respectively to obtain dense static semantic point cloud and dense dynamic semantic point cloud, and the dense static semantic point cloud and dense dynamic semantic point cloud of each frame are merged. The dense semantic point cloud obtained after the merger can be used to determine the occupied grid semantic label corresponding to the voxel grid, which can realize the automatic determination of the occupied grid semantic label, and accurately and effectively convert the semantics of the point cloud into the occupied grid semantic label of the corresponding grid, thereby improving the accuracy and reliability of the occupied grid semantic label.

[0176] In addition, by adopting the technical solution in this embodiment, the accuracy of the semantic label of the occupied grid can be effectively improved through the voting mechanism, and the storage space occupied by the grid of a single frame can be effectively reduced by encoding the grid.

[0177] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0178] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0179] Figure 4 A structural block diagram of a device for determining a label provided by an embodiment of the present application is shown. Figure 3The label determination device 400 of this embodiment may include an acquisition unit 401 , a classification unit 402 , a splicing unit 403 , a merging unit 404 , an obtaining unit 405 and a determination unit 406 . Among them, the acquisition unit 401 is used to acquire the point cloud semantic information of at least one frame of point cloud to be processed in the specified area; the classification unit 402 is used to classify the point cloud semantic information of each frame to obtain the static semantic point cloud and dynamic semantic point cloud of each frame; the splicing unit 403 is used to splice the static semantic point cloud and the dynamic semantic point cloud of each frame respectively to obtain dense static semantic point cloud and dense dynamic semantic point cloud respectively; the merging unit 404 is used to determine the dense semantic point cloud of each frame based on the dense static semantic point cloud and the dense dynamic semantic point cloud using a preset merging strategy; the acquisition unit 405 is used to voxelize each point cloud in the dense semantic point cloud of each frame based on the preset voxel grid network of the specified area to obtain the point cloud occupancy of each voxel grid in the voxel grid network; the determination unit 406 is used to select the semantics of the point cloud in each voxel grid based on the point cloud occupancy of each voxel grid to determine the occupied grid semantic label of each voxel grid.

[0180] It should be noted that part or all of the device determined by the label of this embodiment may be an application located in the local terminal, or may also be a functional unit such as a plug-in or software development kit (SDK) set in the application located in the local terminal, or may also be a processing engine located in a network-side server, or may also be a distributed system located on the network side, for example, a processing engine or distributed system in a data processing platform on the network side, etc. This embodiment does not specifically limit this.

[0181] It is understandable that the application may be a local program (nativeApp) installed on the local terminal, or may also be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0182] Optionally, in a possible implementation of the present embodiment, the classification unit 402 can be specifically used to use a preset target detection model to perform detection processing on each frame of point cloud to obtain a dynamic obstacle detection result for each frame; based on the dynamic obstacle detection result of each frame, extract the dynamic semantic point cloud in the point cloud semantic information of each frame to obtain the dynamic semantic point cloud of each frame; based on the dynamic semantic point cloud of each frame, filter the dynamic semantic point cloud in the point cloud semantic information of each frame to obtain a static semantic point cloud for each frame.

[0183] Optionally, in a possible implementation of the present embodiment, the stitching unit 403 can be used to obtain the dynamic obstacle detection result corresponding to the dynamic semantic point cloud of each frame; based on the dynamic obstacle detection result corresponding to the dynamic semantic point cloud of each frame, the dynamic semantic point cloud of each frame is stitched; based on the result of the stitching processing, a dense dynamic semantic point cloud is obtained.

[0184] Optionally, in a possible implementation of this embodiment, the stitching unit 403 can be used to stitch the static semantic point cloud of each frame based on a preset order; based on the result of the stitching process, a dense static semantic point cloud is obtained.

[0185] Optionally, in a possible implementation of the present embodiment, the merging unit 404 can be used to perform the following operations for the point cloud of each frame: obtain the position data of the point cloud of the current frame; based on the position data of the point cloud of the current frame, convert the dense static semantic point cloud to obtain the dense static semantic point cloud of the current frame; obtain the dynamic obstacle detection result of the point cloud of the current frame; match the dynamic obstacle detection result and the dense dynamic semantic point cloud to obtain the dense dynamic semantic point cloud of the current frame; merge the dense static semantic point cloud of the current frame and the dense dynamic semantic point cloud of the current frame to obtain the dense semantic point cloud of the current frame; determine the dense semantic point cloud of each frame based on the dense semantic point cloud of each current frame.

[0186] Optionally, in a possible implementation of the present embodiment, the obtaining unit 405 can be used to voxelize each point cloud in the dense semantic point cloud of each frame based on a preset voxel grid network of the designated area to obtain a position index of each point cloud; determine the number of point clouds in each voxel grid based on the position index of each point cloud and the preset position index of the voxel grid; and obtain the point cloud occupancy of each voxel grid based on the number of point clouds in each voxel grid.

[0187] Optionally, in a possible implementation of the present embodiment, the determination unit 406 may be used to obtain the number of point clouds and the semantics of the point clouds in the voxel grid in response to the point cloud occupancy status of the voxel grid being point cloud occupied; based on the number of point clouds in the voxel grid and the semantics of the point clouds, a voting mechanism is utilized to select the semantics of the point cloud with the highest vote rate in the voxel grid; and the semantics of the point cloud with the highest vote rate is determined as the occupied grid semantic label of the voxel grid.

[0188] In this embodiment, the acquisition unit can acquire the point cloud semantic information of at least one frame of point cloud to be processed in the specified area, and then the classification unit can classify the point cloud semantic information of each frame to obtain a static semantic point cloud and a dynamic semantic point cloud of each frame, and the splicing unit can splice the static semantic point cloud and the dynamic semantic point cloud of each frame to obtain a dense static semantic point cloud and a dense dynamic semantic point cloud respectively, and the merging unit can determine the dense semantic point cloud of each frame based on the dense static semantic point cloud and the dense dynamic semantic point cloud by using a preset merging strategy, and the acquisition unit can voxelize each point cloud in the dense semantic point cloud of each frame based on a preset voxel grid network of the specified area to obtain the point cloud occupancy of each voxel grid in the voxel grid network, so as to obtain the point cloud occupancy of each voxel grid in the voxel grid network. The determination unit can select and process the semantics of the point cloud in each voxel grid based on the point cloud occupancy of each voxel grid to determine the occupied grid semantic label of each voxel grid. Since the static semantic point cloud and dynamic semantic point cloud of each frame can be determined based on the point cloud semantic information of each frame, and then the static semantic point cloud and dynamic semantic point cloud of each frame are spliced ​​respectively to obtain dense static semantic point cloud and dense dynamic semantic point cloud, and the dense static semantic point cloud and dense dynamic semantic point cloud of each frame are merged, and then the dense semantic point cloud obtained after the merger can be used to determine the occupied grid semantic label corresponding to the voxel grid, which can realize the automatic determination of the occupied grid semantic label, and accurately and effectively convert the semantics of the point cloud to the occupied grid semantic label of the corresponding grid, thereby improving the accuracy and reliability of the occupied grid semantic label.

[0189] Figure 5 FIG. 1 shows a structural block diagram of a training device for an obstacle occupancy network model provided by an embodiment of the present application. Figure 5 As shown. The training device 500 of the obstacle occupancy network model of this embodiment may include an acquisition unit 501, a training unit 502, and an acquisition unit 503. Among them, the acquisition unit 501 is used to acquire sample data and an occupied grid semantic label corresponding to the sample data, wherein the occupied grid semantic label is determined by processing the sample data according to the label determination method in the aforementioned embodiment; the training unit 502 is used to perform iterative training processing on the obstacle occupancy network model to be trained based on the sample data and the occupied grid semantic label corresponding to the sample data; the acquisition unit 503 is used to obtain a trained obstacle occupancy network model in response to the training reaching a preset termination condition.

[0190] It should be noted that part or all of the label determination device of this embodiment may be an application located in the local terminal, or may also be a functional unit such as a plug-in or software development kit (SDK) set in the application located in the local terminal, or may also be a processing engine located in a network-side server, or may also be a distributed system located on the network side, for example, a processing engine or distributed system in a model training platform on the network side, etc. This embodiment does not specifically limit this.

[0191] It is understandable that the application may be a local program (nativeApp) installed on the local terminal, or may also be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0192] Optionally, in a possible implementation of the present embodiment, the acquisition unit 501 can be specifically used to classify the point cloud semantic information of each frame corresponding to the sample data to obtain a static semantic point cloud and a dynamic semantic point cloud of each frame, and splice the static semantic point cloud and the dynamic semantic point cloud of each frame to obtain a dense static semantic point cloud and a dense dynamic semantic point cloud respectively; based on the dense static semantic point cloud and the dense dynamic semantic point cloud, a preset merging strategy is used to determine the dense semantic point cloud of each frame; based on the preset voxel grid network of the specified area, each point cloud in the dense semantic point cloud of each frame is voxelized to obtain the point cloud occupancy of each voxel grid, so that based on the point cloud occupancy of each voxel grid, the semantics of the point cloud in each voxel grid can be selected and processed to obtain the occupied grid semantic label corresponding to the sample data.

[0193] In this embodiment, the sample data and the occupied grid semantic labels corresponding to the sample data can be obtained by the acquisition unit, wherein the occupied grid semantic labels are determined by processing the sample data according to the label determination method in the aforementioned embodiment, and then the training unit can perform iterative training processing on the obstacle occupancy network model to be trained based on the sample data and the occupied grid semantic labels corresponding to the sample data, and the acquisition unit obtains the trained obstacle occupancy network model in response to the training reaching a preset termination condition. Since the obstacle occupancy network model can be trained by utilizing the occupied grid semantic labels of the sample data more effectively and accurately, the robustness and performance of the trained obstacle occupancy network model can be improved.

[0194] In the technical solution of this application, the user personal information involved, such as the collection, storage, use, processing, transmission, provision and disclosure of user images and attribute data, etc., complies with the provisions of relevant laws and regulations and does not violate public order and good morals.

[0195] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0196] According to an embodiment of the present application, further, an autonomous driving vehicle including the provided electronic device is provided, and the autonomous driving vehicle may include unmanned vehicles of L2 and above levels, such as unmanned delivery vehicles, unmanned logistics vehicles, etc.

[0197] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0198] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0199] Multiple components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0200] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a method for determining a label or a training method for an obstacle occupancy network model. For example, in some embodiments, the method for determining a label or the training method for an obstacle occupancy network model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the method for determining a label described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the label determination method or the obstacle occupancy network model training method in any other appropriate manner (eg, by means of firmware).

[0201] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0202] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0203] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0204] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0205] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0206] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0207] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0208] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for determining a label, characterized in that: The method comprises: Acquire point cloud semantic information of at least one frame of point cloud to be processed in a specified area, wherein the point cloud semantic information is automatically annotated; Classify the point cloud semantic information of each frame to obtain a static semantic point cloud and a dynamic semantic point cloud of each frame; The static semantic point cloud and the dynamic semantic point cloud of each frame are spliced ​​to obtain a dense static semantic point cloud and a dense dynamic semantic point cloud respectively; Based on the dense static semantic point cloud and the dense dynamic semantic point cloud, a dense semantic point cloud of each frame is determined using a preset merging strategy; Based on the preset voxel grid network of the designated area, voxelize each point cloud in the dense semantic point cloud of each frame to obtain a position index of each point cloud; The position index of each point cloud satisfy ,i [x,y,z], where i_ub is the lower bound of the preset valid point cloud range in the x,y,z coordinate axis direction, and i_size is the length of the unit voxel grid in the x,y,z coordinate axis direction; Determining the number of point clouds within each voxel grid based on the position index of each point cloud and the preset position index of the voxel grid; Based on the number of point clouds in each voxel grid, the point cloud occupancy of each voxel grid is obtained; Based on the point cloud occupancy of each voxel grid, the semantics of the point cloud in each voxel grid is selected and processed to determine the occupied grid semantic label of each voxel grid.

2. The method according to claim 1, characterized in that The classifying and processing the point cloud semantic information of each frame to obtain a static semantic point cloud and a dynamic semantic point cloud of each frame includes: Using the preset target detection model, each frame of point cloud is detected and processed to obtain the dynamic obstacle detection results of each frame; Based on the dynamic obstacle detection result of each frame, extracting and processing the dynamic semantic point cloud in the point cloud semantic information of each frame to obtain the dynamic semantic point cloud of each frame; Based on the dynamic semantic point cloud of each frame, the dynamic semantic point cloud in the point cloud semantic information of each frame is filtered to obtain the static semantic point cloud of each frame.

3. The method according to claim 1, characterized in that The dynamic semantic point cloud of each frame is stitched to obtain a dense dynamic semantic point cloud, including: Get the dynamic obstacle detection results corresponding to the dynamic semantic point cloud of each frame; Based on the dynamic obstacle detection result corresponding to the dynamic semantic point cloud of each frame, splicing the dynamic semantic point cloud of each frame; Based on the results of the splicing process, a dense dynamic semantic point cloud is obtained.

4. The method according to claim 1, characterized in that: The static semantic point cloud of each frame is stitched to obtain a dense static semantic point cloud, including: Based on a preset order, stitching the static semantic point cloud of each frame; Based on the results of the stitching process, a dense static semantic point cloud is obtained.

5. The method according to claim 1, characterized in that The method of determining a dense semantic point cloud of each frame based on the dense static semantic point cloud and the dense dynamic semantic point cloud by using a preset merging strategy includes: The following operations are performed for each frame of point cloud: obtain the position data of the point cloud of the current frame; Based on the position data of the point cloud of the current frame, converting the dense static semantic point cloud to obtain a dense static semantic point cloud of the current frame; Get the dynamic obstacle detection results of the point cloud of the current frame; Performing matching processing on the dynamic obstacle detection result and the dense dynamic semantic point cloud to obtain a dense dynamic semantic point cloud of the current frame; Merging the dense static semantic point cloud of the current frame and the dense dynamic semantic point cloud of the current frame to obtain a dense semantic point cloud of the current frame; Based on the dense semantic point cloud of each current frame, a dense semantic point cloud of each frame is determined.

6. The method according to claim 1, characterized in that The method of selecting and processing the semantics of the point cloud in each voxel grid based on the point cloud occupancy of each voxel grid to determine the occupied grid semantic label of each voxel grid includes: In response to the point cloud occupancy of the voxel grid being point cloud occupied, obtaining the number of point clouds in the voxel grid and the semantics of the point clouds; Based on the number of point clouds in the voxel grid and the semantics of the point clouds, a voting mechanism is used to select the semantics of the point cloud with the highest vote rate in the voxel grid; The semantics of the point cloud with the highest vote rate is determined as the occupied grid semantic label of the voxel grid.

7. A method for training an obstacle occupancy network model, characterized in that: The method comprises: Acquire sample data and an occupied grid semantic label corresponding to the sample data, wherein the occupied grid semantic label is determined by processing the sample data according to the method of any one of claims 1 to 6; Based on the sample data and the occupied grid semantic labels corresponding to the sample data, performing iterative training processing on the obstacle occupancy network model to be trained; In response to the training reaching a preset termination condition, a trained obstacle occupancy network model is obtained.

8. A device for determining a label, characterized in that: The device comprises: An acquisition unit, used to acquire point cloud semantic information of at least one frame of point cloud to be processed in a specified area, wherein the point cloud semantic information is automatically annotated; A classification unit, used for classifying the point cloud semantic information of each frame to obtain a static semantic point cloud and a dynamic semantic point cloud of each frame; A splicing unit, used for splicing the static semantic point cloud and the dynamic semantic point cloud of each frame respectively, so as to obtain a dense static semantic point cloud and a dense dynamic semantic point cloud respectively; A merging unit, configured to determine a dense semantic point cloud of each frame based on the dense static semantic point cloud and the dense dynamic semantic point cloud using a preset merging strategy; An obtaining unit, configured to voxelize each point cloud in the dense semantic point cloud of each frame based on a preset voxel grid network of the designated area to obtain a position index of each point cloud; The position index of each point cloud satisfy ,i [x,y,z], where i_ub is the lower bound of the preset valid point cloud range in the x,y,z coordinate axis direction, and i_size is the length of the unit voxel grid in the x,y,z coordinate axis direction; Determining the number of point clouds within each voxel grid based on the position index of each point cloud and the preset position index of the voxel grid; Based on the number of point clouds in each voxel grid, the point cloud occupancy of each voxel grid is obtained; The determination unit is used to select and process the semantics of the point cloud in each voxel grid based on the point cloud occupancy of each voxel grid, so as to determine the occupied grid semantic label of each voxel grid.

9. A training device for an obstacle occupancy network model, characterized in that: The device comprises: an acquiring unit, configured to acquire sample data and an occupied grid semantic label corresponding to the sample data, wherein the occupied grid semantic label is determined by processing the sample data according to the method of any one of claims 1 to 6; A training unit, configured to perform iterative training processing on an obstacle occupancy network model to be trained based on the sample data and the occupancy grid semantic labels corresponding to the sample data; The obtaining unit is used to obtain the obstacle occupancy network model after training is completed in response to the training reaching a preset termination condition.

10. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

11. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

12. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images