Sample generation, neural network training, intelligent driving control method and device
By acquiring and matching obstacle position information and category information in point cloud data, accurately sample data is generated and categories are distinguished between categories and no categories of obstacles, the problem of low sample data accuracy in the existing technology is solved and autonomous driving is improved Obstacle detection accuracy and safety.
Patent Information
- Application Number
- CN202011370374.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-30
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-11-30
AI Technical Summary
In the prior art, obstacle detection methods based on deep learning and point cloud data require manual labeling of sample data, resulting in low accuracy of sample data and affecting the safety of autonomous driving.
By obtaining obstacle position information and category information in point cloud data, combining preset obstacle detection algorithms, the obstacle position matching is carried out, accurate sample data is generated, and the obstacles are divided into categories and categories without categories to enhance nerves The generalization ability of the network.
It improves the accuracy of sample data and the detection accuracy of neural networks, reduces the safety risks of autonomous driving, and better identify obstacles in the detection space.
Smart Images

Figure CN112347986B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing technology, and in particular to a method and apparatus for sample generation, neural network training, intelligent driving control, computer equipment, and storage medium. Background Art
[0002] Obstacle detection using LiDAR's three-dimensional point cloud data plays an important role in autonomous driving. As an active ranging device, LiDAR can obtain the three-dimensional point coordinates of objects around the vehicle. The point cloud data formed by aggregating these data can be used to infer information such as the size and position of the object.
[0003] When performing obstacle detection based on deep learning and point cloud data, it is necessary to pre-label the obstacle categories of sample point cloud data; the sample data generated by the current sample labeling method has low accuracy. Summary of the invention
[0004] The embodiments of the present disclosure at least provide a method and apparatus for sample generation, neural network training, and intelligent driving control, a computer device, and a storage medium.
[0005] In a first aspect, an embodiment of the present disclosure provides a sample generation method, comprising: acquiring point cloud data obtained by scanning a target scene; acquiring first obstacle position information and annotation category information of each first obstacle annotated in the point cloud data; and, based on a preset obstacle detection algorithm, detecting second obstacle position information of each second obstacle from the point cloud data; performing position matching on the first obstacle and the second obstacle based on the first obstacle position information and the second obstacle position information, and generating sample data based on a result of the position matching and the annotation category information of the first obstacle.
[0006] In this way, the first obstacle and the second obstacle are position-matched through the first obstacle position information of the first obstacle marked in the point cloud data and the second obstacle position information of the second obstacle detected by the obstacle detection algorithm, and sample data is generated based on the result of the position matching and the marked category information of the first obstacle, thereby achieving accurate marking of the obstacles and improving the accuracy of the generated sample data.
[0007] In a possible implementation, the sample data includes: sample point cloud data and category information corresponding to each obstacle under a plurality of obstacles; wherein the plurality of obstacles include classified obstacles and unclassified obstacles; the category information of the classified obstacles is used to identify the specific category of the classified obstacles; and the category information of the unclassified obstacles is used to identify that the unclassified obstacles have no specific category.
[0008] In this way, by dividing the sample data into classified obstacles and unclassified obstacles, the trained network can not only identify classified obstacles, but also classify all obstacles that cannot be identified into specific categories as unclassified obstacles, thereby better identifying obstacles in the detection space, improving the generalization ability of the neural network, and reducing the safety risks of autonomous driving.
[0009] In a possible implementation, the generating of sample data based on the result of the position matching and the labeling category information of the first obstacle includes: determining the target point cloud data of the target obstacle with successful position matching from the point cloud data; generating the sample data based on the target point cloud data of the target obstacle and the labeling category information of the first point cloud data corresponding to the target obstacle.
[0010] In this way, by matching the first obstacle with the second obstacle, sample data is generated only for the successfully matched target obstacle, thereby being able to obtain sample data with higher accuracy.
[0011] In a possible implementation, the acquiring target point cloud data of the target obstacle whose position is successfully matched from the point cloud data includes: determining the target obstacle whose position is successfully matched based on the first obstacle position information and the second obstacle position information; and acquiring the target point cloud data of the target obstacle from the point cloud data based on the determined second obstacle position information of the target obstacle whose position is successfully matched.
[0012] In a possible implementation, the obstacle position information includes: bounding box information of at least one obstacle in each frame of the point cloud data; the bounding box information includes: position coordinates of each vertex of the bounding box in the scene corresponding to the point cloud data; wherein the obstacle position information includes: the first obstacle position information and the second obstacle position information; the obstacle includes: the first obstacle and the second obstacle.
[0013] In a possible implementation, the performing position matching on the first obstacle and the second obstacle based on the first obstacle position information and the second obstacle position information includes: generating a first projection of each first obstacle on a reference plane based on the first obstacle position information of each first obstacle in the point cloud data; and generating a second projection of each second obstacle on the reference plane based on the second obstacle position information of each second obstacle in the point cloud data; and performing position matching based on the first projection of the first obstacle on the reference plane and the second projection of the second obstacle on the reference plane.
[0014] In a possible implementation, the performing position matching based on the first projection of the first obstacle on the reference plane and the second projection of the second obstacle on the reference plane includes: determining a minimum enclosing rectangle corresponding to each second obstacle based on the second projection of each second obstacle on the reference plane; and determining an obstacle matching result for each second obstacle based on the position of the minimum enclosing rectangle of each second obstacle in the reference plane and the position of the first projection of each first obstacle in the reference plane.
[0015] In a possible implementation, the determining the obstacle matching result of each second obstacle based on the position of the minimum enclosing rectangle of each second obstacle in the reference plane and the position of the first projection of each first obstacle in the reference plane includes: judging whether there is a first obstacle whose overlap between the corresponding first projection and the minimum enclosing rectangle of each second obstacle is greater than a preset overlap threshold based on the position of the minimum enclosing rectangle of each second obstacle in the reference plane; if so, determining that the first obstacle matches the second obstacle.
[0016] In this way, the first obstacle and the second obstacle can be matched in a relatively simple manner, which is conducive to quickly generating sample data.
[0017] In a possible implementation, the sample generation method further includes: determining the number of point cloud points that can be enclosed in the minimum enclosing rectangle of each second obstacle; for a second obstacle whose number of point cloud points is less than a preset number threshold, the obstacle matching result corresponding to the second obstacle is a matching failure.
[0018] In this way, misdetected or smaller obstacles can be screened out from the sample.
[0019] In a second aspect, an embodiment of the present disclosure provides a method for generating a neural network, wherein the neural network is used for obstacle detection; the generation method comprises: using sample data generated by the sample generation method described in any one of the first aspects to train a neural network to be trained, so as to obtain a neural network for detecting obstacles; the sample data comprises: sample point cloud data of obstacles, and category information of the obstacles.
[0020] In this way, since the generated sample data has higher accuracy, the neural network for detecting obstacles trained based on the sample data with higher accuracy also has higher accuracy.
[0021] In an optional embodiment, the neural network includes: an encoding network, a feature extraction network, and a classification network; the training of the neural network to be trained to obtain a neural network for detecting obstacles includes: using the encoding network to encode the sample point cloud data to obtain volume element data corresponding to the sample point cloud data; using the feature extraction network to perform feature extraction on the volume element data to obtain feature data corresponding to the sample point cloud data; using the classification network to classify the feature data to obtain a classification result corresponding to the sample data; based on the category information and the classification result corresponding to the sample data, the encoding network, the feature extraction network, and the classification network are trained to obtain a neural network for detecting obstacles.
[0022] In an optional embodiment, the sample point cloud data includes: position information of multiple point cloud points in the point cloud space corresponding to the obstacle; before encoding the sample point cloud data using the coding network to obtain volume element data corresponding to the sample point cloud data, it also includes: dividing the point cloud space into multiple three-dimensional grids; determining target point cloud points for each three-dimensional grid in the multiple three-dimensional grids according to the position information of the multiple point cloud points in the point cloud space corresponding to the obstacle; wherein the number of target point cloud points in each three-dimensional grid is less than a preset number; based on the target point cloud points of each three-dimensional grid, grid point cloud data of each three-dimensional grid is constructed; encoding the sample point cloud data using the coding network to obtain volume element data corresponding to the sample point cloud data includes: encoding the grid point cloud data of each three-dimensional grid using the coding network to obtain coding data corresponding to each three-dimensional grid; and constructing the volume element data of the sample point cloud data based on the coding data corresponding to each three-dimensional grid.
[0023] In this way, by encoding the target point cloud points in different three-dimensional grids respectively, it is more conducive to synchronous processing of different three-dimensional grids and improves the speed of data processing.
[0024] In an optional embodiment, determining the target point cloud points for each of the multiple three-dimensional grids according to the position information of the multiple point cloud points in the point cloud space corresponding to the obstacle includes: determining the number of point cloud points falling into each of the three-dimensional grids according to the position information of the multiple point clouds in the point cloud space; when the number of point cloud points falling into any three-dimensional grid is less than or equal to the preset number, determining the point cloud points falling into any three-dimensional grid as the target point cloud points of any three-dimensional grid; when the number of point cloud points falling into any three-dimensional grid is greater than the preset number, determining a preset number of point cloud points from the point cloud points falling into any three-dimensional network as the target point cloud points of any three-dimensional grid.
[0025] This can help reduce the amount of sample data that needs to be encoded, reduce the amount of processing when encoding sample data, reduce data processing time, increase data processing speed, and be more conducive to rapid training of neural networks.
[0026] In a third aspect, an embodiment of the present disclosure also provides an intelligent driving control method, comprising: acquiring road point cloud data; processing the road point cloud data using a neural network generated based on any generation method of the second aspect to obtain an obstacle detection result of the road point cloud data; and controlling the intelligent driving equipment that acquires the road point cloud data based on the obstacle detection result.
[0027] In a fourth aspect, an embodiment of the present disclosure also provides a sample generation device, comprising: a first acquisition module, used to acquire point cloud data obtained by scanning a target scene; a second acquisition module, used to acquire first obstacle position information and annotation category information of each first obstacle annotated in the point cloud data; and, based on a preset obstacle detection algorithm, detecting second obstacle position information of each second obstacle from the point cloud data; wherein the annotation category information corresponding to each first obstacle is used to characterize whether each first obstacle has a specific category or does not have a specific category; a matching module, used to perform position matching on the first obstacle and the second obstacle based on the first obstacle position information and the second obstacle position information, and generate sample data based on the result of the position matching and the annotation category information of the first obstacle.
[0028] In a fifth aspect, an embodiment of the present disclosure further provides a device for generating a neural network, wherein the neural network is used for obstacle detection; the generating device comprises: a training module, which is used to train a neural network to be trained using sample data generated by the sample generation method described in any one of the first aspects, so as to obtain a neural network for detecting obstacles; the sample data comprises: sample point cloud data of obstacles, and category information of the obstacles.
[0029] In the sixth aspect, the embodiment of the present disclosure also provides an intelligent driving control device, including: a second acquisition module, used to acquire road point cloud data; a detection module, used to process the road point cloud data using a neural network generated based on any generation method of the second aspect, to obtain an obstacle detection result of the road point cloud data; and a control module, used to control the intelligent driving equipment that acquires the road point cloud data according to the obstacle detection result.
[0030] In a seventh aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps in the first aspect or any possible implementation of the first aspect are performed;
[0031] Alternatively, perform the steps in the above second aspect, or any possible implementation of the second aspect;
[0032] Alternatively, execute the steps in the implementation manner of the third aspect above.
[0033] In an eighth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the first aspect or any possible implementation of the first aspect are executed;
[0034] Alternatively, perform the steps in the above second aspect, or any possible implementation of the second aspect;
[0035] Alternatively, execute the steps in the implementation manner of the third aspect above.
[0036] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present disclosure and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0038] Figure 1 A flow chart of a sample generation method provided by an embodiment of the present disclosure is shown;
[0039] Figure 2 A flowchart showing a specific method for obtaining category information corresponding to a target obstacle provided by an embodiment of the present disclosure;
[0040] Figure 3 A flowchart showing a specific method for matching the positions of a first obstacle and a second obstacle provided by an embodiment of the present disclosure is shown;
[0041] Figure 4 A flowchart of a specific method for training a neural network provided by an embodiment of the present disclosure is shown;
[0042] Figure 5 A flow chart of an intelligent driving control method provided by an embodiment of the present disclosure is shown;
[0043] Figure 6 A schematic diagram of a sample generating device provided by an embodiment of the present disclosure is shown;
[0044] Figure 7 A schematic diagram of an intelligent driving control device provided by an embodiment of the present disclosure is shown;
[0045] Figure 8 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.
[0047] Obstacle detection is related to the safety of autonomous driving and plays a pivotal role in autonomous driving. Obstacle detection method based on deep learning and three-dimensional point cloud data. This method trains a deep learning model through a training set consisting of a large number of sample point cloud data, and performs obstacle detection based on the trained deep learning model. When training the deep learning model, this method needs to pre-label the obstacle category of the sample point cloud data; currently, point cloud data is usually labeled manually to generate sample data, and then the sample data is used to train the deep learning model. However, the accuracy of the sample data obtained by manual labeling is low, which causes the accuracy of the deep learning model to be relatively low, posing a greater safety hazard to autonomous driving.
[0048] In addition, currently, there is also a method for detecting obstacles without categories, such as Lidar Freespace. This method does not require pre-labeling of point cloud data, and can directly determine the location information of each obstacle in the radar scanning space based on point cloud data. Although this method can comprehensively detect each obstacle, it cannot identify the specific category of the obstacle; for autonomous driving, different driving strategies need to be adopted for different categories of obstacles, so simply knowing the location of the obstacle is often not sufficient for autonomous driving.
[0049] Based on the above research, the present disclosure provides a sample generation method, which performs position matching on the first obstacle and the second obstacle through the first obstacle position information of the first obstacle marked in the point cloud data and the second obstacle position information of the second obstacle detected by the obstacle detection algorithm, and generates sample data based on the position matching result and the marking category information of the first obstacle, thereby achieving accurate marking of the obstacles and improving the accuracy of the generated sample data.
[0050] In addition, the disclosed example divides sample data into categorized obstacles and uncategorized obstacles, so that the trained network can not only identify categorized obstacles, but also classify all obstacles that cannot be identified into specific categories as uncategorized obstacles, thereby better identifying obstacles in the detection space, improving the generalization ability of the neural network, and reducing the safety risks of autonomous driving.
[0051] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.
[0052] The technical solutions in the present disclosure will be clearly and completely described below in conjunction with the drawings in the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.
[0053] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0054] To facilitate understanding of the present embodiment, a sample generation method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the sample generation method provided in the embodiment of the present disclosure is generally an electronic device with certain computing capabilities, and the electronic device includes, for example: a terminal device or a server or other processing device, and the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the sample generation method can be implemented by a processor calling a computer-readable instruction stored in a memory.
[0055] The sample generation method provided by the embodiment of the present disclosure is described in detail below.
[0056] See also Figure 1 FIG. 1 is a flow chart of a sample generation method provided by an embodiment of the present disclosure, wherein the method comprises steps S101 to S103, wherein:
[0057] S101: Acquire point cloud data obtained by scanning a target scene.
[0058] Here, the sample data generated by the sample generation method provided in the embodiment of the present disclosure can be used to train the neural network for obstacle detection in the field of autonomous driving, and can also be used to train the neural network for obstacle detection in other scenarios, such as the smart warehousing field and other fields that require obstacle detection based on point cloud data.
[0059] Taking the application of the sample generation method provided by the embodiment of the present disclosure to the field of autonomous driving as an example, the target scene may include a road scene; the point cloud data may include at least one frame of point cloud data; when the point cloud data has multiple frames, the different point cloud data in the multiple frames of point cloud data may come from the same road scene or from different road scenes. The point cloud data of the same road scene, for example, is the point cloud data of multiple frames continuously acquired in the process of controlling the vehicle to travel on the road. The point cloud data of different scenes, for example, is the point cloud data acquired in the process of controlling the vehicle to travel on different roads at different times.
[0060] In any frame of point cloud data, the number and types of obstacles included are uncertain.
[0061] In order to increase the diversity of samples, multi-frame point cloud data can be obtained from different roads in multiple time periods and multiple regions.
[0062] S102: Acquire first obstacle position information and annotation category information of each first obstacle annotated in the point cloud data; and, based on a preset obstacle detection algorithm, detect and obtain second obstacle position information of each second obstacle from the point cloud data.
[0063] In the specific implementation:
[0064] (1) The first obstacle position information of each first obstacle in the point cloud data may be marked from the point cloud data by manual marking, for example.
[0065] Exemplarily, when annotating the first obstacle, the annotation may be, for example, manually constructing a bounding box for each first obstacle to annotate each first obstacle from the point cloud data; each annotated bounding box includes a plurality of point cloud points constituting the first obstacle. That is, the point cloud points constituting the first obstacle are enclosed in the enclosing space determined by the corresponding bounding box; each first obstacle corresponds to a bounding box. The generated first obstacle position information includes, for example, coordinate information of each vertex of the bounding box in the point cloud space corresponding to the point cloud data.
[0066] In another embodiment, the first obstacle position information may further include connection relationship information between vertices of the bounding box.
[0067] When marking the first obstacle from the point cloud data, the marking category information of the first obstacle is also determined.
[0068] In another embodiment of the present disclosure, the generated sample data includes: sample point cloud data and category information corresponding to each obstacle under multiple obstacles.
[0069] Among them, the multiple obstacles include classified obstacles and unclassified obstacles; the category information of the classified obstacles is used to identify the specific category of the classified obstacles; the category information of the unclassified obstacles is used to identify that the unclassified obstacles have no specific category.
[0070] In order to obtain obstacles with categories and obstacles without categories, when the first obstacle is labeled, the labeled category information determined for the first obstacle can indicate that the corresponding first obstacle has a specific category; if in the subsequent S103, the first obstacle is determined as a target obstacle, the target obstacle is a classified obstacle, and the category information corresponding to the target obstacle is consistent with the labeled category information corresponding to the first obstacle, and the specific category of the target obstacle can be identified. Conversely, when the first obstacle is labeled by category, the labeled category information determined for the first obstacle can indicate that the corresponding first obstacle has no specific category; if in the subsequent S103, the first obstacle is determined as a target obstacle, the determined target obstacle is an obstacle without a category, and the category information corresponding to the target obstacle is consistent with the labeled category information corresponding to the first obstacle, and the target obstacle has no specific category.
[0071] In this way, the sample data is divided into classified obstacles and unclassified obstacles, so that the trained network can not only identify classified obstacles, but also classify all obstacles that cannot be identified into specific categories as unclassified obstacles, thereby better identifying obstacles in the detection space, improving the generalization ability of the neural network, and reducing the safety risks of autonomous driving.
[0072] (2) A preset obstacle detection algorithm, such as any of the classless obstacle detection methods such as Lidar Freespace; through the obstacle detection algorithm, the second obstacle position information of each second obstacle in the point cloud data can be identified.
[0073] Here, for the same frame of point cloud data, the first obstacle manually determined and the second obstacle identified by the obstacle detection algorithm may completely match, partially match, or not match at all.
[0074] Here, complete matching means that for any first obstacle determined manually, there is a second obstacle matching it; or for any second obstacle, there is a first obstacle matching it.
[0075] Partial matching means that among the manually determined first obstacles, some of the first obstacles can be matched to the corresponding second obstacles, while some of the first obstacles cannot be matched to the corresponding second obstacles; or, among the second obstacles determined by the obstacle detection algorithm, some of the second obstacles can be matched to the first obstacles, while some of the second obstacles cannot be matched to the corresponding first obstacles.
[0076] Complete mismatch means that among the manually determined first obstacles, no first obstacle has a matching second obstacle; or among the second obstacles determined by the obstacle detection algorithm, no second obstacle has a matching first obstacle.
[0077] Based on the preset obstacle detection algorithm, the second obstacle position information of the second obstacle determined can also be in the form of a bounding box. Generally, according to the specific shape of the second obstacle, the bounding box determined for the second obstacle by the obstacle detection algorithm is, for example, an irregular polyhedron.
[0078] Among them, when the second obstacle position information of each second obstacle is detected from the point cloud data based on the preset obstacle detection algorithm, the second obstacle position information of the second obstacle obtained is more precise than manual labeling because it is automatically detected based on the point cloud data, and therefore has higher precision.
[0079] For example, when the same obstacle in the point cloud data is manually labeled and labeled based on a preset obstacle detection algorithm, manual labeling can generally only roughly determine a bounding box for the obstacle, and the manually labeled bounding box includes many point cloud points that do not belong to the obstacle; while the labeling based on the obstacle detection algorithm can determine a bounding box for the obstacle along the outline of the obstacle, which includes fewer point cloud points that do not belong to the obstacle.
[0080] S103: Based on the first obstacle position information and the second obstacle position information, position matching is performed on the first obstacle and the second obstacle, and sample data is generated based on the position matching result and the labeling category information of the first obstacle.
[0081] In a specific implementation, the first obstacle and the second obstacle are position-matched based on the first obstacle position information of the first obstacle and the second obstacle position information of the second obstacle, in order to determine the first obstacle and the second obstacle belonging to the same obstacle from the point cloud data.
[0082] When obtaining the target point cloud data of the target obstacle whose position is successfully matched from the point cloud data, for example, the target obstacle whose position is successfully matched can be determined based on the first obstacle position information of the first obstacle and the second obstacle position information of the second obstacle; and then, based on the determined second obstacle position information of the target obstacle whose position is successfully matched, the target point cloud data of the target obstacle whose position is successfully matched is obtained from the point cloud data.
[0083] If a first obstacle is successfully matched to a second obstacle, it is determined that the first obstacle and the second obstacle correspond to the same obstacle in the point cloud data; the same obstacle corresponding to the first obstacle and the second obstacle is determined as a target obstacle, and then the target point cloud data of the target obstacle is obtained from the frame point cloud data.
[0084] Here, it should be noted that when acquiring the target point cloud data of the target obstacle, for example, the second obstacle position information of the second obstacle can be used to extract the target point cloud data corresponding to the target obstacle from the point cloud data.
[0085] See also Figure 2 As shown, the embodiment of the present disclosure also provides a specific method for obtaining category information corresponding to a target obstacle, including:
[0086] S201: Obtain labeling category information of a first obstacle corresponding to a target obstacle.
[0087] Here, the labeling category information is, for example, labeling category information manually labeled for each target obstacle in the point cloud data.
[0088] Specifically, when manually annotating the first obstacle position information for the first obstacle in the point cloud data, corresponding annotation category information may be marked for each first obstacle.
[0089] Exemplarily, if the first obstacle is a classified obstacle, the corresponding labeled category information is the specific name of the category of the first obstacle; if the first obstacle is an unclassified obstacle, the corresponding labeled category information is "other", indicating that obstacles that the neural network cannot identify as specific categories are classified as "other"; or the first obstacle may not be marked with labeled category information.
[0090] After the target obstacle is determined, the annotated category information annotated for the first obstacle corresponding to the target obstacle may be directly determined as the category information corresponding to the target obstacle.
[0091] S202: If the annotated category information of the first obstacle corresponding to the target obstacle contains the specific category of the first obstacle, use the specific category as the category information corresponding to the target obstacle.
[0092] Here, if the target obstacle corresponds to the first obstacle in the labeled category information, the target obstacle is directly determined as an obstacle without a category. In this case, for example, its category information can be determined as "others".
[0093] For example, for a certain frame of point cloud data, when the frame of point cloud data is manually labeled, the first obstacles determined are a1, a2, a3, and a4; obstacle position information corresponding to the first obstacles a1, a2, a3, and a4 is generated; the labeled category information of the first obstacle a1 is "small vehicle"; the labeled category information of the first obstacle a2 is "large vehicle": the first obstacle a3 is an object dropped on the road, and the labeled category information is "other"; the first obstacle a4 is an animal walking on the road, and its category information is not marked.
[0094] Based on a preset obstacle detection algorithm, the second obstacles determined from the frame point cloud data are b1, b2, b3, b4, and b5, and obstacle position information corresponding to the second obstacles b1, b2, b3, b4, and b5 is generated.
[0095] Match a1 with b1 to b6 in sequence; if a1 and b2 match successfully, the same obstacle corresponding to a1 and b2 is determined as the target obstacle, and since the labeled category information of a1 is "large vehicle", the category information of the target obstacle determined based on a1 and b2 is "large vehicle".
[0096] Match a2 with b1 to b6 in sequence; if the match fails, end the operation on a2.
[0097] Match a3 with b1 to b6 in sequence; if a3 and b4 match successfully, the same obstacle corresponding to a3 and b4 is determined as the target obstacle, and since the labeled category information of a3 is "others", the category information of the target obstacle determined based on a3 and b4 is "others".
[0098] Match a4 with b1 to b6 in sequence; if a4 and b5 are matched successfully, the same obstacle corresponding to a3 and b4 is determined as the target obstacle, and since there is no labeled category information of a4, the category information of the target obstacle determined based on a4 and b5 is "other".
[0099] In this example, even if some second obstacles are successfully matched with the first obstacles, they are matched again with other first obstacles to reduce the possibility of erroneous matching due to obstacles being too close in the matching process.
[0100] In another example, if some second obstacles are successfully matched with the first obstacles, they may not be matched with other first obstacles again to improve the matching efficiency.
[0101] See also Figure 3 The embodiment of the present disclosure further provides a specific method for performing position matching on the first obstacle and the second obstacle based on the first obstacle position information and the second obstacle position information, including:
[0102] S301: generating a first projection of each first obstacle on a reference plane based on first obstacle position information of each first obstacle in the point cloud data;
[0103] S302: generating a second projection of each second obstacle on the reference plane based on second obstacle position information of each second obstacle in the point cloud data;
[0104] S303: Perform position matching based on a first projection of the first obstacle on a reference plane and a second projection of the second obstacle on the reference plane.
[0105] Here, the reference plane can be set according to actual needs.
[0106] A method for determining a first projection of a first obstacle on a reference plane includes, for example, first determining projection points of each vertex on a bounding box corresponding to obstacle position information of the first obstacle on the reference plane; then determining a connection relationship between the projection points based on connection relationship information between different vertices; then connecting the projection points based on the connection relationship between the projection points; and determining an area enclosed by the connected projection points as the corresponding projection.
[0107] The method for determining the second projection is similar to the method for determining the first projection, and will not be described in detail here.
[0108] After the first projection and the second projection are determined, obstacle matching is performed based on the first projection and the second projection, for example, the following method may be used:
[0109] Based on the second projection of each second obstacle on the reference plane, determine the minimum enclosing rectangle corresponding to each second obstacle respectively; based on the position of the minimum enclosing rectangle of each second obstacle in the reference plane and the position of the first projection of each first obstacle in the reference plane, determine the obstacle matching result of each second obstacle.
[0110] Here, in the point cloud data obtained by radar, the detailed point cloud on the side of the obstacle close to the radar will be detected by the radar, while a lot of information will be lost on the side of the obstacle far away from the radar. In order to complete the missing information of the point cloud data, the minimum enclosing rectangle of the second projection can be determined first, and then the matching degree between the first obstacle and the second obstacle can be determined by the overlap between the minimum enclosing rectangle and the first projection.
[0111] Specifically, for example, based on the position of the minimum enclosing rectangle of each second obstacle in the reference plane and the position of the first projection of the first obstacle in the reference plane, it can be determined whether there is a first obstacle whose corresponding first projection has an overlap with the minimum enclosing rectangle of the second obstacle greater than a preset overlap threshold.
[0112] Here, for example, the overlapping area between the minimum enclosing rectangle and the first projection may be first determined; then the ratio between the overlapping area and the area of the first projection may be calculated, and the ratio may be determined as the overlap degree between the minimum enclosing rectangle and the first projection.
[0113] If the overlap is greater than a preset overlap threshold, the first obstacle and the second obstacle are matched successfully. If the overlap is less than or equal to the preset overlap threshold, the first obstacle and the second obstacle are matched unsuccessfully.
[0114] The preset overlap threshold is, for example, 0.5, 0.6, etc. It can be specifically set according to actual needs.
[0115] In addition, since the obstacle detection algorithm can determine most of the second obstacles in the point cloud data, some of them may have little impact on autonomous driving. For these smaller obstacles, there is no need to consider them. Therefore, in another embodiment of the present disclosure, it can also include: determining the number of point cloud points that can be enclosed in the minimum enclosing rectangle of each second obstacle; for the second obstacle whose number of point cloud points is less than the preset number threshold, the obstacle matching result corresponding to the second obstacle is a matching failure. Therefore, through this process, the smaller second obstacles are filtered out, and the matching speed of the first obstacle and the second obstacle is improved.
[0116] Through the above process, sample data can be generated based on the result of the position matching degree and the category labeling information of the first obstacle.
[0117] The disclosed embodiment obtains the first obstacle position information and the marking category information of each first obstacle marked in the point cloud data, and detects the second obstacle position information of each second obstacle from the point cloud data based on a preset obstacle detection algorithm, then performs position matching on the first obstacle and the second obstacle based on the first obstacle position information and the second obstacle position information, and generates sample data based on the result of the position matching and the marking category information of the first obstacle. The method combines the high accuracy of type marking during manual marking with the high accuracy of point cloud data determined during detection based on a preset obstacle detection algorithm, thereby enabling accurate marking of the types and positions of obstacles and improving the accuracy of the generated sample data.
[0118] The embodiment of the present disclosure also provides a method for generating a neural network, wherein the neural network is used for obstacle detection; the generating method comprises:
[0119] Using the sample data generated by the sample generation method provided by any embodiment of the present disclosure, a neural network to be trained is trained to obtain a neural network for detecting obstacles;
[0120] The sample data includes: sample point cloud data of obstacles and category information of the obstacles.
[0121] The example disclosed herein takes the application of the neural network generation method to the field of autonomous driving as an example. The category information of obstacles is the specific category of obstacles that the neural network can specifically identify, such as: small vehicles (cars, trains, off-road vehicles), large vehicles (trucks, buses), pedestrians, bicycles (bicycles, motorcycles), etc. The obstacles represented by the above sample data are categorized obstacles, and the category information corresponding to the categorized obstacles is, for example, the name of the category to which the obstacle belongs;
[0122] In another embodiment of the present disclosure, the sample data also includes sample data corresponding to unclassified obstacles. Among them, unclassified obstacles; for example, obstacles that are not often found on the road, such as animals (cats, dogs), roadblocks, and vehicle falling objects, can all be determined as unclassified obstacles; the category information corresponding to unclassified obstacles, such as "others", indicates that the neural network cannot identify the specific category of the obstacle.
[0123] The embodiments of the present disclosure provide a specific structure of a neural network, and the neural network includes, for example: an encoding network, a feature extraction network, and a classification network.
[0124] Specifically, the encoding network is used to encode the sample point cloud data in the sample data, and convert the sample point cloud data into volume element data with a unified format.
[0125] The feature extraction network is used to extract features from each sample point cloud data based on the volume element data to obtain feature data corresponding to the sample point cloud data.
[0126] The classification network is used to classify obstacles represented by the sample data based on the extracted feature data, and obtain classification results corresponding to the obstacles.
[0127] Specifically, see Figure 4 As shown, the embodiment of the present disclosure provides a specific method for training a neural network, including:
[0128] S401: Encode the sample point cloud data using the encoding network to obtain volume element data corresponding to the sample point cloud data.
[0129] Here, the sample point cloud data includes: position information of a plurality of point cloud points in the point cloud space corresponding to the obstacle.
[0130] When encoding sample point cloud data using an encoding network, for example, the following method may be used:
[0131] The point cloud space is divided into a plurality of three-dimensional grids; according to the position information of the plurality of point cloud points in the point cloud space corresponding to the obstacle, target point cloud points are determined for each of the plurality of three-dimensional grids; wherein the number of target point cloud points in each three-dimensional grid is less than a preset number; based on the target point cloud points of each three-dimensional grid, grid point cloud data of each three-dimensional grid is constructed; the grid point cloud data of each three-dimensional grid is encoded by using the encoding network to obtain the encoding data corresponding to each three-dimensional grid; based on the encoding data corresponding to each three-dimensional grid, volume element data of the point cloud data is constructed.
[0132] In a specific implementation, before dividing the point cloud space corresponding to different sample point cloud data into multiple three-dimensional grids, the point cloud space corresponding to the sample point cloud data can also be adjusted to the same size. Then, when dividing the point cloud space corresponding to the sample point cloud data into three-dimensional grids, the number of three-dimensional grids corresponding to different sample point cloud data is the same, and the size of the obtained three-dimensional grids is also the same.
[0133] For different sample point cloud data, the distribution of each point cloud point is also different; therefore, the number of point cloud points that eventually fall into different grids is also different. In order to further reduce the amount of calculation of subsequent encoding operations, the number of point cloud points that fall into each of the three-dimensional grids is determined based on the position information of the multiple point clouds in the point cloud space; when the number of point cloud points that fall into any three-dimensional grid is less than or equal to the preset number, the point cloud points that fall into any three-dimensional grid are determined as the target point cloud points of any three-dimensional grid; when the number of point cloud points that fall into any three-dimensional grid is greater than the preset number, a preset number of point cloud points are determined from the point cloud points that fall into any three-dimensional network as the target point cloud points of any three-dimensional grid.
[0134] When the number of point cloud points falling into any three-dimensional grid is greater than the preset number, a preset number of point cloud points are determined from the point cloud points falling into the arbitrary three-dimensional grid as the target point cloud points of the arbitrary three-dimensional grid. Here, the target point cloud points in any three-dimensional grid can be randomly determined from the point cloud points falling into the arbitrary three-dimensional grid, or the target point cloud points can be selected as evenly as possible according to the positions of the point cloud points falling into the arbitrary three-dimensional grid in the three-dimensional grid.
[0135] When the encoding network is used to encode the point cloud data corresponding to each obstacle, for example, the encoding network in the VoxelNet model of the end-to-end learning model for point cloud-based 3D object detection (End-to-End Learning for Point CloudBased 3Dimensions Object Detection) can be used to implement it.
[0136] Exemplarily, the point cloud space corresponding to the sample point cloud data is divided into N×N×N three-dimensional grids; in each three-dimensional grid, at most P target point cloud points are allocated to form the grid point cloud data of each three-dimensional grid; the grid point cloud data corresponding to each three-dimensional grid is encoded using a coding network to obtain K-dimensional encoded data corresponding to each three-dimensional grid. N×N×N three-dimensional grids can obtain N×N×N×K-dimensional volume element data.
[0137] S402: Using the feature extraction network, extract features from the volume element data to obtain feature data corresponding to the sample point cloud data.
[0138] In a specific implementation, the feature extraction network includes, for example, a convolutional neural network with stacked multi-level convolutional layers. When the feature extraction network is used to extract features from the volume element data of the point cloud data corresponding to each obstacle, for example, a multi-level convolution operation is performed on the volume element data of the point cloud data.
[0139] For example, after performing feature extraction operations on N×N×N×K-dimensional volume element data using a feature extraction network, M×M×M×K'-dimensional feature data is obtained, where M represents the side length of the volume element after convolution, and K' represents the length of the feature data.
[0140] S403: Using the classification network, classify the feature data to obtain a classification result corresponding to the sample data.
[0141] Here, the classification network includes, for example, stacked multi-level fully connected layers, which perform multi-level dimensionality reduction on feature data, and then perform activation processing on the data obtained by the multi-level dimensionality reduction based on a preset activation function to obtain a classification result of the point cloud data.
[0142] In another embodiment, before using the classification network to classify the feature data, the feature data of M×M×M×K′ dimensions may be converted into M 3 ×K' dimension as a vector; then the one-dimensional vector is input into the classification network to obtain the classification result of the point cloud data.
[0143] S404: Based on the category information and the classification results corresponding to the sample data, the encoding network, the feature extraction network, and the classification network are trained to obtain a neural network for detecting obstacles.
[0144] Here, for example, the classification result corresponding to the sample data and the category information are used to determine the cross entropy loss of the classification; the gradient value is calculated by the cross entropy loss function, and the stochastic gradient descent algorithm (SGD) is used to obtain the back propagation value, and back propagation is performed to adjust the parameters of the encoding network, the feature extraction network, and the classification network.
[0145] The parameter adjustment process is repeated multiple times until the cross entropy loss value cannot be reduced any further, and the obtained encoding network, the feature extraction network, and the classification network are determined as the obstacle detection neural network.
[0146] The embodiment of the present disclosure utilizes the sample data generated by the sample generation method provided by the embodiment of the present disclosure to train a neural network. The generation of the sample data has higher accuracy, and thus the obtained neural network also has higher accuracy.
[0147] In addition, the sample data in the embodiments of the present disclosure include sample data of categorized obstacles and sample data of uncategorized obstacles, so that the trained neural network can not only identify categorized obstacles, but also classify all obstacles that cannot be identified into specific categories as uncategorized obstacles, thereby better identifying obstacles in the detection space, improving the generalization ability of the neural network, and reducing the safety risks of autonomous driving.
[0148] See also Figure 5 As shown, the embodiment of the present disclosure also provides an intelligent driving control method, including:
[0149] S501: Acquire road point cloud data;
[0150] S502: Processing the road point cloud data using a neural network generated by any neural network generation method in the embodiments of the present disclosure to obtain an obstacle detection result of the road point cloud data;
[0151] S503: Controlling the intelligent driving device that obtains the road point cloud data according to the obstacle detection result.
[0152] In a specific implementation, intelligent driving equipment includes, for example: autonomous driving vehicles, vehicles equipped with advanced driver assistance systems, robots, etc.
[0153] The neural network generated by the neural network generation method provided by the embodiment of the present disclosure can not only identify categorized obstacles, but also identify a large number of uncategorized obstacles, and is therefore more suitable for the field of autonomous driving and improves the safety of autonomous driving.
[0154] At the same time, the neural network generated by the neural network generation method provided by the embodiment of the present disclosure can identify the categories of different obstacles, so that the neural network for detecting obstacles trained by the neural network generation method provided by the embodiment of the present disclosure can be well applied to the field of autonomous driving.
[0155] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.
[0156] Based on the same inventive concept, the embodiments of the present disclosure also provide a device for generating a neural network corresponding to the method for generating a neural network. Since the principle of solving the problem by the device in the embodiments of the present disclosure is similar to the method for generating a neural network in the embodiments of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0157] See also Figure 6As shown, an embodiment of the present disclosure provides a schematic diagram of a sample generation device, the sample generation device comprising:
[0158] A first acquisition module 61 is used to acquire point cloud data obtained by scanning a target scene;
[0159] The second acquisition module 62 is used to acquire the first obstacle position information and the marked category information of each first obstacle marked in the point cloud data; and, based on a preset obstacle detection algorithm, to detect and obtain the second obstacle position information of each second obstacle from the point cloud data; wherein the marked category information corresponding to each first obstacle is used to characterize whether each first obstacle has a specific category or does not have a specific category;
[0160] The matching module 63 is used to perform position matching on the first obstacle and the second obstacle based on the first obstacle position information and the second obstacle position information, and generate sample data based on the result of the position matching and the labeling category information of the first obstacle.
[0161] In a possible implementation manner, the sample data includes: sample point cloud data and category information corresponding to each obstacle under multiple obstacles;
[0162] Among them, the multiple obstacles include classified obstacles and unclassified obstacles; the category information of the classified obstacles is used to identify the specific category of the classified obstacles; the category information of the unclassified obstacles is used to identify that the unclassified obstacles have no specific category.
[0163] In a possible implementation manner, when generating sample data based on the position matching result and the labeling category information of the first obstacle, the matching module 63 is configured to:
[0164] Determine target point cloud data of the target obstacle whose position is successfully matched from the point cloud data;
[0165] The sample data is generated based on the target point cloud data of the target obstacle and the labeling category information of the first point cloud data corresponding to the target obstacle.
[0166] In a possible implementation manner, the matching module 63, when acquiring the target point cloud data of the target obstacle whose position is successfully matched from the point cloud data, is used to:
[0167] Determining a target obstacle whose position is successfully matched based on the first obstacle position information and the second obstacle position information;
[0168] Based on the second obstacle position information of the target obstacle whose position matching is successful, target point cloud data of the target obstacle is acquired from the point cloud data.
[0169] In a possible implementation, the obstacle position information includes: bounding box information of at least one obstacle in each frame of the point cloud data; the bounding box information includes: position coordinates of each vertex of the bounding box in the scene corresponding to the point cloud data;
[0170] Wherein, the obstacle position information includes: the first obstacle position information and the second obstacle position information;
[0171] The obstacle includes: the first obstacle and the second obstacle.
[0172] In a possible implementation manner, the matching module 63, when performing position matching on the first obstacle and the second obstacle based on the first obstacle position information and the second obstacle position information, is configured to:
[0173] Based on the first obstacle position information of each first obstacle in the point cloud data, generating a first projection of each first obstacle on a reference plane;
[0174] and generating a second projection of each second obstacle on the reference plane based on the second obstacle position information of each second obstacle in the point cloud data;
[0175] Position matching is performed based on a first projection of the first obstacle on a reference plane and a second projection of the second obstacle on the reference plane.
[0176] In a possible implementation manner, the matching module 63, when performing position matching based on the first projection of the first obstacle on the reference plane and the second projection of the second obstacle on the reference plane, is configured to:
[0177] Determine, based on a second projection of each second obstacle on the reference plane, a minimum enclosing rectangle corresponding to each second obstacle;
[0178] An obstacle matching result of each second obstacle is determined based on the position of the minimum enclosing rectangle of each second obstacle in the reference plane and the position of the first projection of each first obstacle in the reference plane.
[0179] In a possible implementation manner, the matching module 63, when determining the obstacle matching result of each second obstacle based on the position of the minimum enclosing rectangle of each second obstacle in the reference plane and the position of the first projection of each first obstacle in the reference plane, is configured to:
[0180] Based on the position of the minimum enclosing rectangle of each second obstacle in the reference plane and the position of the first projection of each first obstacle in the reference plane, determining whether there is a first obstacle whose overlap degree between the corresponding first projection and the minimum enclosing rectangle of each second obstacle is greater than a preset overlap degree threshold;
[0181] If so, it is determined that the first obstacle matches the second obstacle.
[0182] In a possible implementation manner, the matching module 63 is further used to: determine the number of point cloud points that can be enclosed by the minimum enclosing rectangle of each second obstacle;
[0183] For a second obstacle whose number of point cloud points is less than a preset number threshold, the obstacle matching result corresponding to the second obstacle is matching failure.
[0184] A schematic diagram of a neural network generation device provided in an embodiment of the present disclosure, wherein the neural network is used for obstacle detection; the generation device comprises:
[0185] A training module, used to train a neural network to be trained using sample data generated by the sample generation method described in any embodiment of the present disclosure, to obtain a neural network for detecting obstacles;
[0186] The sample data includes: sample point cloud data of obstacles and category information of the obstacles.
[0187] In a possible implementation, the neural network includes: an encoding network, a feature extraction network, and a classification network;
[0188] The training module, when training the neural network to be trained to obtain the neural network for detecting obstacles, is used to:
[0189] Encoding the sample point cloud data using the encoding network to obtain volume element data corresponding to the sample point cloud data;
[0190] Using the feature extraction network, extracting features from the volume element data to obtain feature data corresponding to the sample point cloud data;
[0191] Using the classification network, classify the feature data to obtain a classification result corresponding to the sample data;
[0192] Based on the category information and the classification results corresponding to the sample data, the encoding network, the feature extraction network, and the classification network are trained to obtain a neural network for detecting obstacles.
[0193] In a possible implementation manner, the sample point cloud data includes: position information of a plurality of point cloud points in a point cloud space corresponding to the obstacle;
[0194] The training module, before encoding the sample point cloud data using the encoding network to obtain volume element data corresponding to the sample point cloud data, is further used to:
[0195] Dividing the point cloud space into a plurality of three-dimensional grids;
[0196] Determine a target point cloud point for each of the multiple three-dimensional grids according to the position information of the multiple point cloud points in the point cloud space corresponding to the obstacle; wherein the number of target point cloud points in each three-dimensional grid is less than a preset number;
[0197] Based on the target point cloud points of each three-dimensional grid, constructing grid point cloud data of each three-dimensional grid;
[0198] The training module, when encoding the sample point cloud data using the encoding network to obtain volume element data corresponding to the sample point cloud data, is used to:
[0199] Encoding the grid point cloud data of each three-dimensional grid using the encoding network to obtain encoding data corresponding to each three-dimensional grid;
[0200] Based on the coded data corresponding to each three-dimensional grid, volume element data of the sample point cloud data is constructed.
[0201] The training module, when determining a target point cloud point for each of the plurality of three-dimensional grids according to the position information of the plurality of point cloud points in the point cloud space corresponding to the obstacle, is used to:
[0202] Determine the number of point cloud points falling into each of the three-dimensional grids according to the position information of the multiple point clouds in the point cloud space respectively;
[0203] When the number of point cloud points falling into any three-dimensional grid is less than or equal to the preset number, the point cloud points falling into any three-dimensional grid are determined as target point cloud points of any three-dimensional grid;
[0204] When the number of point cloud points falling into any three-dimensional grid is greater than the preset number, a preset number of point cloud points are determined from the point cloud points falling into any three-dimensional grid as target point cloud points of any three-dimensional grid.
[0205] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.
[0206] See also Figure 7 As shown, the embodiment of the present disclosure also provides an intelligent driving control device, including:
[0207] The third acquisition module 71 is used to acquire road point cloud data;
[0208] A detection module 72, configured to process the road point cloud data using a neural network generated by any neural network generation method according to an embodiment of the present disclosure, to obtain an obstacle detection result of the road point cloud data;
[0209] The control module 73 is used to control the intelligent driving device that obtains the road point cloud data according to the obstacle detection result.
[0210] The present disclosure also provides an electronic device 800, such as Figure 8 FIG. 8 is a schematic diagram of the structure of an electronic device 800 provided in an embodiment of the present disclosure, including:
[0211] Processor 81, memory 82, and bus 83; memory 82 is used to store execution instructions, including memory 821 and external memory 822; memory 821 here is also called internal memory, which is used to temporarily store operation data in processor 81 and data exchanged with external memory 822 such as hard disk. Processor 81 exchanges data with external memory 822 through memory 821. When the electronic device 800 is running, the processor 81 communicates with the memory 82 through bus 83, so that the processor 81 executes the following instructions in user mode:
[0212] Obtaining point cloud data obtained by scanning the target scene;
[0213] Acquire first obstacle position information and marking category information of each first obstacle marked in the point cloud data; and, based on a preset obstacle detection algorithm, detect and obtain second obstacle position information of each second obstacle from the point cloud data;
[0214] Based on the first obstacle position information and the second obstacle position information, position matching is performed on the first obstacle and the second obstacle, and sample data is generated based on the result of the position matching and the labeling category information of the first obstacle;
[0215] Or execute the following command:
[0216] Using the sample data generated by the sample generation method described in any embodiment of the present disclosure, training the neural network to be trained to obtain a neural network for detecting obstacles;
[0217] The sample data includes: sample point cloud data of obstacles and category information of the obstacles;
[0218] Or execute the following command:
[0219] Obtain road point cloud data;
[0220] Processing the road point cloud data using a neural network generated by the neural network generation method provided by an embodiment of the present disclosure to obtain an obstacle detection result of the road point cloud data;
[0221] The intelligent driving device that obtains the road point cloud data is controlled according to the obstacle detection result.
[0222] The embodiments of the present disclosure also provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program executes the steps of the method for generating a neural network described in the above method embodiments, or executes the steps of the intelligent driving control method described in the above method embodiments.
[0223] The computer program product of the route planning method provided in the embodiments of the present disclosure includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the steps of generating the neural network described in the above method embodiments, or to execute the steps of the intelligent driving control method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.
[0224] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.
[0225] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0226] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0227] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0228] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A sample generation method, characterized in that: include: Acquire point cloud data obtained by scanning the target scene; Acquire first obstacle position information and marking category information of each first obstacle marked in the point cloud data; and, based on a preset obstacle detection algorithm, detecting and obtaining second obstacle position information of each second obstacle from the point cloud data; Based on the first obstacle position information and the second obstacle position information, position matching is performed on the first obstacle and the second obstacle, and sample data is generated based on the result of the position matching and the labeling category information of the first obstacle; Wherein, performing position matching on the first obstacle and the second obstacle based on the first obstacle position information and the second obstacle position information includes: Based on the first obstacle position information of each first obstacle in the point cloud data, generating a first projection of each first obstacle on a reference plane; and generating a second projection of each second obstacle on the reference plane based on the second obstacle position information of each second obstacle in the point cloud data; Position matching is performed based on a first projection of the first obstacle on a reference plane and a second projection of the second obstacle on the reference plane.
2. The sample generation method according to claim 1, characterized in that: The sample data includes: sample point cloud data and category information corresponding to each obstacle under multiple obstacles; Among them, the multiple obstacles include classified obstacles and unclassified obstacles; the category information of the classified obstacles is used to identify the specific category of the classified obstacles; the category information of the unclassified obstacles is used to identify that the unclassified obstacles have no specific category.
3. The sample generation method according to claim 1 or 2, characterized in that: The generating sample data based on the result of the position matching and the labeling category information of the first obstacle includes: Determine target point cloud data of the target obstacle whose position is successfully matched from the point cloud data; The sample data is generated based on the target point cloud data of the target obstacle and the labeling category information of the first point cloud data corresponding to the target obstacle.
4. The sample generation method according to claim 3, characterized in that: The step of acquiring target point cloud data of a target obstacle whose position is successfully matched from the point cloud data comprises: Determining a target obstacle whose position is successfully matched based on the first obstacle position information and the second obstacle position information; Based on the second obstacle position information of the target obstacle whose position matching is successful, target point cloud data of the target obstacle is acquired from the point cloud data.
5. The sample generation method according to any one of claims 1 to 4, characterized in that: The obstacle position information includes: the bounding box information of at least one obstacle in each frame of the point cloud data; the bounding box information includes: the position coordinates of each vertex of the bounding box in the scene corresponding to the point cloud data; Wherein, the obstacle position information includes: the first obstacle position information and the second obstacle position information; The obstacle includes: the first obstacle and the second obstacle.
6. The sample generation method according to claim 1, characterized in that: The performing position matching based on a first projection of the first obstacle on a reference plane and a second projection of the second obstacle on the reference plane includes: Determine, based on a second projection of each second obstacle on the reference plane, a minimum enclosing rectangle corresponding to each second obstacle; An obstacle matching result of each second obstacle is determined based on the position of the minimum enclosing rectangle of each second obstacle in the reference plane and the position of the first projection of each first obstacle in the reference plane.
7. The sample generation method according to claim 6, characterized in that: The determining the obstacle matching result of each second obstacle based on the position of the minimum enclosing rectangle of each second obstacle in the reference plane and the position of the first projection of each first obstacle in the reference plane includes: Based on the position of the minimum enclosing rectangle of each second obstacle in the reference plane and the position of the first projection of each first obstacle in the reference plane, determining whether there is a first obstacle whose overlap degree between the corresponding first projection and the minimum enclosing rectangle of each second obstacle is greater than a preset overlap degree threshold; If so, it is determined that the first obstacle matches the second obstacle.
8. The sample generation method according to claim 7, characterized in that: The sample generation method further includes: Determine the number of point cloud points that can be enclosed in the minimum enclosing rectangle of each second obstacle; For a second obstacle whose number of point cloud points is less than a preset number threshold, the obstacle matching result corresponding to the second obstacle is matching failure.
9. A method for generating a neural network, characterized in that: The neural network is used for obstacle detection; the generation method comprises: Using the sample data generated by the sample generation method according to any one of claims 1 to 8, training the neural network to be trained to obtain a neural network for detecting obstacles; The sample data includes: sample point cloud data of obstacles and category information of the obstacles.
10. The generation method according to claim 9, characterized in that: The neural network includes: an encoding network, a feature extraction network, and a classification network; The step of training the neural network to be trained to obtain a neural network for detecting obstacles includes: Encoding the sample point cloud data using the encoding network to obtain volume element data corresponding to the sample point cloud data; Using the feature extraction network, extracting features from the volume element data to obtain feature data corresponding to the sample point cloud data; Using the classification network, classify the feature data to obtain a classification result corresponding to the sample data; Based on the category information and the classification results corresponding to the sample data, the encoding network, the feature extraction network, and the classification network are trained to obtain a neural network for detecting obstacles.
11. The generation method according to claim 10, characterized in that: The sample point cloud data includes: position information of a plurality of point cloud points in the point cloud space corresponding to the obstacle; Before encoding the sample point cloud data using the encoding network to obtain volume element data corresponding to the sample point cloud data, the method further includes: Dividing the point cloud space into a plurality of three-dimensional grids; Determine a target point cloud point for each of the multiple three-dimensional grids according to the position information of the multiple point cloud points in the point cloud space corresponding to the obstacle; wherein the number of target point cloud points in each three-dimensional grid is less than a preset number; Based on the target point cloud points of each three-dimensional grid, constructing grid point cloud data of each three-dimensional grid; The step of encoding the sample point cloud data using the encoding network to obtain volume element data corresponding to the sample point cloud data includes: Encoding the grid point cloud data of each three-dimensional grid using the encoding network to obtain encoding data corresponding to each three-dimensional grid; Based on the coded data corresponding to each three-dimensional grid, volume element data of the sample point cloud data is constructed.
12. The generation method according to claim 11, characterized in that: The step of determining a target point cloud point for each of the plurality of three-dimensional grids according to the position information of the plurality of point cloud points in the point cloud space corresponding to the obstacle comprises: Determine the number of point cloud points falling into each of the three-dimensional grids according to the position information of the multiple point clouds in the point cloud space respectively; When the number of point cloud points falling into any three-dimensional grid is less than or equal to the preset number, the point cloud points falling into any three-dimensional grid are determined as target point cloud points of any three-dimensional grid; When the number of point cloud points falling into any three-dimensional grid is greater than the preset number, a preset number of point cloud points are determined from the point cloud points falling into any three-dimensional grid as target point cloud points of any three-dimensional grid.
13. An intelligent driving control method, characterized in that: include: Obtain road point cloud data; Processing the road point cloud data using a neural network generated by any generation method according to claims 9 to 12 to obtain an obstacle detection result of the road point cloud data; The intelligent driving device that obtains the road point cloud data is controlled according to the obstacle detection result.
14. A sample generating device, characterized in that: include: A first acquisition module is used to acquire point cloud data obtained by scanning a target scene; A second acquisition module, used to acquire first obstacle position information and marked category information of each first obstacle marked in the point cloud data; and, based on a preset obstacle detection algorithm, detecting and obtaining second obstacle position information of each second obstacle from the point cloud data; wherein the labeled category information corresponding to each first obstacle is used to characterize whether each first obstacle has a specific category or does not have a specific category; A matching module, configured to perform position matching on the first obstacle and the second obstacle based on the first obstacle position information and the second obstacle position information, and generate sample data based on a result of the position matching and the labeling category information of the first obstacle; Wherein, the matching module, when performing position matching on the first obstacle and the second obstacle based on the first obstacle position information and the second obstacle position information, is used to: Based on the first obstacle position information of each first obstacle in the point cloud data, generating a first projection of each first obstacle on a reference plane; and generating a second projection of each second obstacle on the reference plane based on the second obstacle position information of each second obstacle in the point cloud data; Position matching is performed based on a first projection of the first obstacle on a reference plane and a second projection of the second obstacle on the reference plane.
15. A device for generating a neural network, characterized in that: The neural network is used for obstacle detection; the generating device comprises: A training module, used to train a neural network to be trained using sample data generated by the sample generation method according to any one of claims 1 to 8, to obtain a neural network for detecting obstacles; The sample data includes: sample point cloud data of obstacles and category information of the obstacles.
16. An intelligent driving control device, characterized in that: include: The third acquisition module is used to acquire road point cloud data; A detection module, used to process the road point cloud data using a neural network generated based on any one of the generation methods of claims 9 to 12 to obtain an obstacle detection result of the road point cloud data; A control module is used to control the intelligent driving equipment that obtains the road point cloud data according to the obstacle detection result.
17. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of any method described in claims 1 to 13 are performed.
18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps of the method according to any one of claims 1 to 13.
