Training method and device of semantic segmentation model, semantic segmentation method and device, and equipment

Abstract

The application discloses a semantic segmentation model training method, a semantic segmentation method, a device and equipment, relates to the technical field of artificial intelligence, and particularly relates to the technical field of deep learning, semantic segmentation, automatic driving and the like. The method comprises the following steps: determining sample point cloud data corresponding to a sample object from original point cloud data based on a preset down-sampling screening mode; the sample object comprises a compulsory object and a to-be-screened object; performing dimension reduction mapping on sample points in the sample point cloud data based on a preset projection rule to obtain sample distance images corresponding to the sample points; and training a semantic segmentation model according to sample label data corresponding to the sample distance images and the sample point cloud data. Through the above technical scheme, the accuracy of semantic segmentation can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, semantic segmentation, automatic driving and the like, and specifically relates to a semantic segmentation model training method, a semantic segmentation method, a device and equipment. BACKGROUND

[0002] An automatic driving system highly depends on accurate, real-time and highly robust environment perception. A vehicle needs to accurately identify, classify and locate driving-related obstacles on the road surface, mainly including cars, pedestrians, bicycles and other obstacles. Different automatic driving solutions may have different sensor combinations, but it is undeniable that laser radar is one of the most common components. The point cloud scanned by laser radar can directly produce distance measurement and description of the environment for vehicle controllers and subsequent planning modules. At the same time, laser radar can ensure good perception under almost all lighting conditions (including day, night, glare, shadow, etc.), so it has attracted widespread attention.

[0003] The disorder and sparsity of laser radar point cloud result in that the standard convolutional neural network cannot be directly used for point cloud data. In order to solve this problem, many new methods have been proposed in recent years, which can be roughly divided into point-based, voxel-based and range image-based methods. Generally speaking, the point-based method directly acts on the original point cloud data and can obtain the highest accuracy, but at the same time, the computational complexity of this method is very high, and the model is also very complex. The voxel-based method projects the unstructured point cloud data into a structured and regular grid, so that three-dimensional sparse convolution can be used. Although this method can achieve high accuracy, the complex model design still makes it unable to achieve real-time results. The range image-based method uses a spherical projection strategy to project the unordered point cloud data into a regular image-like representation, and then uses a standard two-dimensional convolutional neural network to complete the point cloud-based semantic segmentation task. This method can provide high semantic segmentation accuracy while ensuring inference speed and achieving real-time segmentation. It should be noted that the range image is projected from the point cloud, and the inevitable information loss of projection and deprojection occurs.

[0004] Existing point cloud semantic segmentation techniques often focus on completing high-precision segmentation of all objects in the scene, while in automatic driving, the most important thing is to complete high-precision segmentation of obstacles on the road surface under the premise of ensuring real-time segmentation. SUMMARY

[0005] The present application provides a semantic segmentation model training method, a semantic segmentation method, a device and equipment to solve the problem of low semantic segmentation accuracy in the automatic driving scene.

[0006] According to one aspect of the present invention, a method for training a semantic segmentation model is provided, characterized by comprising:

[0007] Based on a preset downsampling filtering method, sample point cloud data corresponding to sample objects are determined from the original point cloud data; the sample objects include mandatory objects and objects to be filtered.

[0008] Based on preset projection rules, the sample points in the sample point cloud data are dimensionality-reduced and mapped to obtain the sample distance image corresponding to the sample points.

[0009] The semantic segmentation model is trained based on the sample label data corresponding to the sample distance image and sample point cloud data.

[0010] According to another aspect of the present invention, a semantic segmentation method is provided, characterized in that it includes:

[0011] Based on preset projection rules, the target points in the target point cloud data of the target object are dimensionality-reduced and mapped to obtain the target distance image corresponding to the target points.

[0012] A semantic segmentation model is used to predict the target distance image to obtain a two-dimensional prediction result of the target point; wherein, the semantic segmentation model is trained based on the training method of the semantic segmentation model provided in any embodiment of the present invention;

[0013] The two-dimensional prediction result of the target is transformed into three dimensions to obtain the target prediction result of the target point.

[0014] According to another aspect of the present invention, a training apparatus for a semantic segmentation model is provided, characterized in that it comprises:

[0015] The sample point cloud data determination module is used to determine the sample point cloud data corresponding to the sample object from the original point cloud data based on a preset downsampling filtering method; the sample object includes mandatory objects and objects to be filtered.

[0016] The sample distance image determination module is used to perform dimensionality reduction mapping on sample points in sample point cloud data based on preset projection rules to obtain the sample distance image corresponding to the sample points.

[0017] The semantic segmentation model training module is used to train the semantic segmentation model based on the sample label data corresponding to the sample distance image and sample point cloud data.

[0018] According to another aspect of the present invention, a semantic segmentation apparatus is provided, characterized in that it comprises:

[0019] The target distance image determination module is used to perform dimensionality reduction mapping on the target points in the target point cloud data of the target object based on a preset projection rule, so as to obtain the target distance image corresponding to the target point;

[0020] The target two-dimensional prediction result determination module is used to predict the target distance image using a semantic segmentation model to obtain the target two-dimensional prediction result of the target point; wherein, the semantic segmentation model is trained based on the semantic segmentation model training method provided in any embodiment of the present invention;

[0021] The target prediction result determination module is used to perform a three-dimensional transformation on the two-dimensional prediction result of the target to obtain the target prediction result of the target point.

[0022] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0023] At least one processor; and

[0024] A memory communicatively connected to the at least one processor; wherein,

[0025] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the training method or semantic segmentation method of the semantic segmentation model according to any embodiment of the present invention.

[0026] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the training method or semantic segmentation method of the semantic segmentation model according to any embodiment of the present invention.

[0027] The technical solution of this invention determines sample point cloud data corresponding to sample objects from raw point cloud data based on a preset downsampling filtering method. The sample objects include mandatory objects and objects to be filtered. Then, based on a preset projection rule, the sample points in the sample point cloud data are dimensionality-reduced to obtain sample distance images corresponding to the sample points. Finally, a semantic segmentation model is trained based on the sample distance images and the sample label data corresponding to the sample point cloud data. This technical solution solves the problem of inaccurate semantic segmentation caused by imbalanced datasets by filtering the raw point cloud data of the sample objects. Simultaneously, training the semantic segmentation model using sample distance images improves the accuracy of semantic segmentation.

[0028] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart of a training method for a semantic segmentation model according to Embodiment 1 of the present invention;

[0031] Figure 2 This is a flowchart of a training method for a semantic segmentation model according to Embodiment 2 of the present invention;

[0032] Figure 3 This is a flowchart of a training method for a semantic segmentation model according to Embodiment 3 of the present invention;

[0033] Figure 4 This is a flowchart of a training method for a semantic segmentation model according to Embodiment 4 of the present invention;

[0034] Figure 5 This is a flowchart of a semantic segmentation method provided in Embodiment 5 of the present invention;

[0035] Figure 6 This is a schematic diagram of the structure of a training device for a semantic segmentation model according to Embodiment Six of the present invention;

[0036] Figure 7 This is a schematic diagram of the structure of a semantic segmentation device according to Embodiment 7 of the present invention;

[0037] Figure 8 This is a schematic diagram of the structure of an electronic device that implements the training method or semantic segmentation method of the semantic segmentation model in the embodiments of the present invention. Detailed Implementation

[0038] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0039] It should be noted that the terms "first," "second," "original," "target," "sample," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0040] Furthermore, it should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of point cloud data and other related data involved in the technical solution of this invention all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0041] Example 1

[0042] Figure 1 This is a flowchart of a semantic segmentation model training method according to Embodiment 1 of the present invention. This embodiment is applicable to obstacle segmentation in autonomous driving scenarios. The method can be executed by a semantic segmentation model training device, which can be implemented in hardware and / or software and integrated into an electronic device that carries the training function of the semantic segmentation model, such as a server. Figure 1 As shown, training the semantic segmentation model in this embodiment may include:

[0043] S110. Based on the preset downsampling filtering method, determine the sample point cloud data corresponding to the sample object from the original point cloud data.

[0044] In this embodiment, raw point cloud data refers to the unprocessed point cloud data corresponding to the sample objects. The sample objects can be objects in an autonomous driving scenario, such as cars, trucks, and people. Optionally, the sample objects can include mandatory objects and objects to be screened. Mandatory objects are those that do not require point cloud data screening, i.e., those that require special attention; objects to be screened are those that require point cloud data screening. For example, if the raw point cloud data includes point cloud data from three categories: cars, people, and motorcycles, and if the point cloud data volume of cars is large while that of people and motorcycles is small, the point cloud data of cars needs to be reduced to ensure data balance. In this case, cars are objects to be screened, while people and motorcycles are mandatory objects.

[0045] The so-called sample point cloud data refers to the point cloud data corresponding to the sample object, which contains multiple frames of point cloud. That is, a frame of point cloud includes multiple sample points, and each sample point includes three-dimensional position information.

[0046] An alternative approach is to use a preset downsampling filtering method to determine the sample point cloud data corresponding to the sample objects from the original point cloud data. For example, the number of categories of sample objects in the original point cloud data can be determined first, and the objects with the fewest point cloud counts can be selected as mandatory objects, while the remaining objects can be selected as objects to be filtered. Then, the original point cloud data corresponding to the objects to be filtered is filtered to select point cloud data with the same number of point cloud counts as the mandatory objects, thus obtaining the sample point cloud data corresponding to each sample object. At this point, the number of sample point cloud data corresponding to each sample object is consistent.

[0047] S120. Based on the preset projection rules, perform dimensionality reduction mapping on the sample points in the sample point cloud data to obtain the sample distance image corresponding to the sample points.

[0048] In this embodiment, the preset projection rule refers to the projection rule for projecting three-dimensional point cloud data into one-dimensional data. For example, the preset projection rule can be a spherical projection rule.

[0049] A sample point refers to a point in the sample point cloud data. A sample distance image refers to a one-dimensional distance image corresponding to a sample point.

[0050] Specifically, for each sample point in the sample point cloud data, a dimensionality reduction mapping can be performed on the sample point based on spherical projection rules to obtain the sample distance image corresponding to that sample point. For example, the sample distance image corresponding to a sample point can be determined using the following spherical projection rules:

[0051]

[0052] Where, f = f u +f d d represents the field of view perpendicular to the sensor; d represents the depth of the sample point. H and W represent the resolution of the sample distance application, respectively. Preferably, in this application, to ensure that the semantic segmentation model can run in real time, W = 512 and H = 128. r represents the reflectance of the sample point; the final sample distance image is a data with dimension (H, W, 5), where 5 represents (x, y, z, d, r).

[0053] S130. Train the semantic segmentation model based on the sample label data corresponding to the sample distance image and sample point cloud data.

[0054] In this embodiment, the semantic segmentation model is a model used for semantic segmentation; optionally, the semantic segmentation model can be based on a neural network.

[0055] In this embodiment, the sample label data refers to the label data of the sample point cloud data. Optionally, the sample label data includes the truth box of the sample object in each frame of the point cloud, that is, the coordinates of the four vertices of the truth box; it also includes the semantic label of each sample point in the sample point cloud data corresponding to the sample object, that is, the category label.

[0056] For example, the sample label data corresponding to the sample point cloud data can be determined as follows: The size of the truth boxes in the original label data corresponding to the sample point cloud data is expanded to obtain the sample truth boxes in the sample label data; based on the category labels in the original label data, a category label is assigned to each sample point in the sample point cloud data to obtain the sample point labels in the sample label data. Specifically, the size of the truth boxes in the original label data corresponding to each sample point in the sample point cloud data can be expanded outwards. For example, the left, right, front, back, and top faces of the truth box can be expanded outwards by a first length (e.g., 15cm), and the bottom face can be contracted upwards by a second length (e.g., 10m); wherein the first and second lengths can be set by those skilled in the art according to the actual situation. Then, based on the category labels in the original label data, the category label is assigned to each sample point in the sample point cloud data to obtain the sample point labels for each point in the sample label data.

[0057] Understandably, compared to the high cost of labeling existing semantic segmentation datasets, this invention preprocesses the original labeled data and expands the ground truth boxes in the original labeled data. This ensures that every point on the surface of an object is within the ground truth box, while minimizing the inclusion of ground points. Furthermore, it assigns semantic-level labels to each sample point in the sample point cloud data, saving costs. Additionally, it can correct the purpose of obstacle detection results in autonomous driving scenarios, assisting in obstacle detection and thus making subsequent semantic segmentation model training more accurate.

[0058] In one alternative approach, the sample distance image can be input into the semantic segmentation model to obtain a predicted segmentation result. A training loss is then calculated based on the predicted segmentation result and the sample label data corresponding to the sample point cloud data. This training loss is used to train the semantic segmentation model until a training stopping condition is met, at which point model training ceases. The training stopping condition can be that the training loss stabilizes within a set range, or that the number of training iterations reaches a set number. It should be noted that the set range and the set number of iterations can be set by those skilled in the art according to actual needs.

[0059] The training loss can be calculated based on a preset loss function, using the predicted segmentation results and sample label data. It should be noted that the preset loss function can be a cross-entropy loss function, etc.

[0060] The technical solution provided by this invention determines the sample point cloud data corresponding to the sample object from the original point cloud data based on a preset downsampling filtering method. Then, based on a preset projection rule, it performs dimensionality reduction mapping on the sample points in the sample point cloud data to obtain the sample distance image corresponding to the sample point. Finally, it trains a semantic segmentation model based on the sample distance image and the sample label data corresponding to the sample point cloud data. This technical solution solves the problem of inaccurate semantic segmentation caused by imbalanced datasets by filtering the original point cloud data of the sample object. Simultaneously, training the semantic segmentation model using the sample distance image improves the accuracy of semantic segmentation.

[0061] Example 2

[0062] Figure 2 This is a flowchart of a training method for a semantic segmentation model according to Embodiment 2 of the present invention. Based on the above embodiments, this embodiment further optimizes the step of "determining the sample point cloud data corresponding to the sample object from the original point cloud data based on a preset downsampling filtering method," providing an optional implementation scheme. For example... Figure 2 As shown, the training method for the semantic segmentation model in this embodiment may include:

[0063] S210. Determine the required point cloud data corresponding to the required objects from the original point cloud data.

[0064] In this embodiment, the mandatory point cloud data refers to the point cloud data of the mandatory objects.

[0065] Specifically, mandatory point cloud data corresponding to mandatory objects can be selected from the original point cloud data based on mandatory unit filtering rules. For example, for each single frame of point cloud data, if the single frame contains mandatory objects, then the single frame is retained as mandatory point cloud data. Similarly, if the number of points corresponding to mandatory objects in a single frame exceeds a set threshold, then the single frame is also considered mandatory point cloud data.

[0066] S220. Remove the required point cloud data from the original point cloud data to obtain the point cloud data to be filtered.

[0067] In this embodiment, the point cloud data to be screened refers to the point cloud data remaining after removing the mandatory point cloud data from the original point cloud data.

[0068] Specifically, the required point cloud data can be removed from the original point cloud data to obtain the point cloud data to be filtered.

[0069] S230. Update the point cloud data to be filtered based on the proportion of the objects to be filtered in the point cloud data to be filtered.

[0070] In one optional approach, for each object to be filtered, the point cloud data corresponding to that object is determined from the point cloud data to be filtered. The ratio between the point cloud data corresponding to the object and the total point cloud data to be filtered is calculated to obtain the proportion of the object to be filtered in the point cloud data. The point cloud data to be filtered is then updated based on this proportion. For example, if the proportion is greater than a set value, the point cloud data corresponding to the object to be filtered is removed from the point cloud data to be filtered, resulting in the filtered point cloud data, i.e., the updated point cloud data. The set value can be set by those skilled in the art according to actual needs. It should be noted that if the proportion is still greater than the set value after updating the point cloud data to be filtered, the point cloud data corresponding to the object to be filtered continues to be removed until the proportion is equal to or approximately equal to the set value.

[0071] Another optional approach is to determine, for each single frame of point cloud data in the point cloud data to be filtered, the first number of sample points corresponding to the object to be filtered in the single frame of point cloud data; based on the first number and the total number of sample points in the frame of point cloud data, determine the individual proportion of the object to be filtered in the single frame of point cloud data; determine the overall proportion of the number of category boxes corresponding to the object to be filtered in the total number of category boxes in the point cloud data to be filtered; and update the point cloud data to be filtered based on the individual proportion and the overall proportion.

[0072] Specifically, for each single frame of point cloud data in the point cloud data to be filtered, for each object to be filtered, a first number of sample points corresponding to the object to be filtered in the single frame of point cloud data is determined. Then, the ratio between the first number and the total number of sample points in the single frame of point cloud data is used as the individual proportion of the object to be filtered in the single frame of point cloud data. Furthermore, the ratio between the number of category boxes corresponding to the object to be filtered and the total number of category boxes in the point cloud data to be filtered is used as the overall proportion of the number of category boxes corresponding to the object to be filtered in the total number of category boxes in the point cloud data to be filtered. Finally, if the individual proportion is greater than k times the overall proportion, P% of the point cloud frames are randomly filtered out from the point cloud data to be filtered, resulting in updated point cloud data to be filtered. Here, k is a natural number, preferably 1 in this embodiment; P is a natural number, preferably 5 in this embodiment. It should be noted that if the individual proportion of the object to be filtered in the updated point cloud data to be filtered is still greater than k times the overall proportion, point cloud frames are continuously randomly filtered out from the point cloud data to be filtered until the individual proportion of the object to be filtered is less than or equal to k times the overall proportion.

[0073] It is understandable that the adaptive downsampling method used in this embodiment to filter the sample point cloud data can ensure the data balance of the sample point cloud data, that is, the number of sample point cloud data of different sample objects is about the same, thereby ensuring the accuracy of semantic segmentation model training.

[0074] S240. Based on the required point cloud data and the updated point cloud data to be screened, determine the sample point cloud data corresponding to the sample object.

[0075] Specifically, the required point cloud data and the updated point cloud data to be filtered can be used as the sample point cloud data corresponding to the sample object.

[0076] S250. Based on preset projection rules, perform dimensionality reduction mapping on sample points in sample point cloud data to obtain sample distance images corresponding to sample points.

[0077] S260. Train the semantic segmentation model based on the sample label data corresponding to the sample distance image and sample point cloud data.

[0078] The technical solution provided by this invention involves determining the mandatory point cloud data corresponding to the mandatory objects from the original point cloud data, removing the mandatory point cloud data from the original point cloud data to obtain the point cloud data to be screened, and updating the point cloud data to be screened based on the proportion of the objects to be screened in the point cloud data. Based on the mandatory point cloud data and the updated point cloud data to be screened, the sample point cloud data corresponding to the sample objects is determined. Then, based on a preset projection rule, the sample points in the sample point cloud data are subjected to dimensionality reduction mapping to obtain the sample distance images corresponding to the sample points. Finally, the semantic segmentation model is trained based on the sample distance images and the sample label data corresponding to the sample point cloud data. This technical solution, by filtering the original point cloud data to obtain the sample point cloud data, ensures the balance of the data used to train the semantic segmentation model, effectively improves the quality of the training data, and thus improves the efficiency of model training.

[0079] Example 3

[0080] Figure 3 This is a flowchart of a training method for a semantic segmentation model according to Embodiment 3 of the present invention. Based on the above embodiments, the semantic segmentation model further includes a convolutional feature extraction network, a backbone network, and a prediction network. The convolutional feature extraction network is used to extract features from sample distance image data; optionally, the convolutional feature extraction network can be composed of at least one convolutional layer. The backbone network is used to perform secondary extraction on the features extracted by the convolutional feature extraction network, i.e., deep feature mining; optionally, the backbone network can be composed of several residual units. The prediction network is used to predict the semantic segmentation results; optionally, the prediction network can be composed of convolutional layers and fully connected layers.

[0081] Accordingly, the step of "training the semantic segmentation model based on the sample label data corresponding to the sample distance image and sample point cloud data" is further optimized, providing an optional implementation scheme. For example... Figure 3 As shown, the training method for the semantic segmentation model in this embodiment may include:

[0082] S310. Based on the preset downsampling filtering method, determine the sample point cloud data corresponding to the sample object from the original point cloud data.

[0083] The sample objects include required objects and objects to be filtered.

[0084] S320. Based on preset projection rules, perform dimensionality reduction mapping on sample points in sample point cloud data to obtain sample distance images corresponding to sample points.

[0085] S330. A convolutional feature extraction network is used to extract features from the sample distance image to obtain the first feature.

[0086] In this embodiment, the first feature refers to the feature obtained after feature extraction of the sample distance image, which can be represented in matrix or vector form.

[0087] Specifically, the sample distance image corresponding to each sample point can be input into the convolutional feature extraction network. After processing by the convolutional feature extraction network, the first feature corresponding to that sample point can be obtained.

[0088] S340. Using a backbone network, perform secondary feature extraction on the first feature to obtain the second feature.

[0089] In this embodiment, the second feature refers to the feature extracted after in-depth mining of the first feature, which can be represented in matrix or vector form.

[0090] Specifically, the first feature corresponding to each sample point can be input into the backbone network, and the backbone network can perform feature depth mining to obtain the second feature corresponding to that sample point.

[0091] S350. Using a prediction network, the second feature is predicted to obtain the sample point's two-dimensional prediction result.

[0092] In this embodiment, the two-dimensional prediction result of the sample refers to the prediction result of each sample point, that is, the semantic prediction category of the sample point.

[0093] Specifically, the second feature corresponding to each sample point can be input into the prediction network. After processing by the prediction network, the two-dimensional prediction result of that sample point can be obtained.

[0094] S360. Based on the sample two-dimensional prediction results and the sample label data corresponding to the sample point cloud data, train the semantic segmentation model.

[0095] Specifically, based on a preset loss function, the training loss can be calculated using the semantic labels in the sample label data corresponding to the sample point cloud data and the two-dimensional prediction results of the samples. The semantic segmentation model can then be trained using this training loss until the training stopping condition is met. The preset loss function can be a cross-entropy loss function, etc.

[0096] The technical solution provided in this invention determines the sample point cloud data corresponding to the sample object from the original point cloud data based on a preset downsampling screening method. Then, based on a preset projection rule, the sample points in the sample point cloud data are dimensionality-reduced and mapped to obtain the sample distance image corresponding to the sample point. A convolutional feature extraction network is then used to extract features from the sample distance image to obtain a first feature. Next, a backbone network is used to perform secondary feature extraction on the first feature to obtain a second feature. A prediction network is then used to predict the second feature to obtain a two-dimensional prediction result for the sample point. Finally, a semantic segmentation model is trained based on the two-dimensional prediction result and the sample label data corresponding to the sample point cloud data. This technical solution allows for multiple feature extractions influenced by sample distance, enabling deeper mining of the semantic information of the point cloud, thus making the semantic segmentation model more robust and achieving higher semantic segmentation accuracy.

[0097] Based on the above embodiments, as an optional aspect of the present invention, the backbone network includes at least two sub-residual networks; the at least two sub-residual networks are connected sequentially; correspondingly, the backbone network is used to perform secondary feature extraction on the first feature to obtain the second feature, including: inputting the first feature into the backbone network to obtain auxiliary features output by at least two sub-residual networks; normalizing the at least two auxiliary features to obtain at least two normalized auxiliary features; and concatenating the at least two normalized auxiliary features to obtain the second feature.

[0098] The backbone network includes at least two sub-residual networks, which are connected sequentially, meaning that the output of the previous sub-residual network is the input of the next sub-residual network; each sub-residual network includes at least one residual unit.

[0099] Specifically, the first feature can be input into the backbone network and processed sequentially by the sub-residual networks within the backbone network. That is, the first feature is input into the first sub-residual network to obtain the auxiliary feature output by that first sub-residual network. This auxiliary feature is then input into the next sub-residual network to obtain the auxiliary feature output by that next sub-residual network, and so on, until the auxiliary feature output by each sub-residual network is obtained. Next, the auxiliary feature output by each sub-residual network is normalized to obtain the normalized auxiliary feature corresponding to each sub-residual network. Finally, the normalized auxiliary features are concatenated to obtain the second feature.

[0100] Understandably, extracting features through multiple sub-residual networks can yield richer features, thereby making the training of semantic segmentation models more accurate.

[0101] Example 4

[0102] Figure 4 This is a flowchart of a training method for a semantic segmentation model according to Embodiment 4 of the present invention. Based on the above embodiments, the semantic segmentation model may further include an auxiliary prediction network; wherein the auxiliary prediction network is used to predict the auxiliary features output by the sub-residual networks in the backbone network; the optional auxiliary prediction network may consist of convolutional layers and fully connected layers.

[0103] Accordingly, the "training of the semantic segmentation model" is further optimized, providing an optional implementation scheme. For example... Figure 4 As shown, the training method for the semantic segmentation model in this embodiment may include:

[0104] S410. Based on the preset downsampling filtering method, determine the sample point cloud data corresponding to the sample object from the original point cloud data.

[0105] The sample objects include required objects and objects to be filtered.

[0106] S420. Based on preset projection rules, perform dimensionality reduction mapping on sample points in sample point cloud data to obtain sample distance images corresponding to sample points.

[0107] S430. A convolutional feature extraction network is used to extract features from the sample distance image to obtain the first feature.

[0108] S440. Using a backbone network, perform secondary feature extraction on the first feature to obtain the second feature.

[0109] S450. Using a prediction network, the second feature is predicted to obtain the sample point's two-dimensional prediction result.

[0110] S460. Select candidate auxiliary features from at least two normalized auxiliary features.

[0111] In this embodiment, candidate auxiliary features are used to assist in the training of the semantic segmentation model.

[0112] Specifically, auxiliary features can be selected from at least two normalized auxiliary features based on preset rules. For example, a set number of normalized auxiliary features can be randomly selected from at least two normalized auxiliary features as candidate auxiliary features. Alternatively, the first normalized auxiliary feature from at least one normalized auxiliary feature can be removed, that is, the normalized auxiliary feature corresponding to the auxiliary feature output by the first sub-residual network can be removed, and the other normalized auxiliary features can be used as candidate auxiliary features.

[0113] S470. An auxiliary prediction network is used to predict candidate auxiliary features to obtain auxiliary two-dimensional prediction results.

[0114] In this embodiment, the auxiliary two-dimensional prediction result refers to the two-dimensional semantic prediction result of the sample points obtained by predicting the candidate auxiliary features.

[0115] Specifically, candidate auxiliary features can be input into the auxiliary prediction network, processed by the auxiliary prediction network, and the auxiliary two-dimensional prediction result corresponding to each candidate auxiliary feature can be obtained.

[0116] S480. The semantic segmentation model is trained based on the auxiliary two-dimensional prediction results, the sample two-dimensional prediction results, and the sample label data corresponding to the sample point cloud data.

[0117] One alternative approach is to average the auxiliary two-dimensional prediction results with the sample two-dimensional prediction results. Then, based on a preset loss function, the training loss can be calculated using this average and the semantic labels in the sample label data corresponding to the sample point cloud data. This training loss can then be used to train the semantic segmentation model. The preset loss function can be a cross-entropy loss function.

[0118] Another option is to determine the auxiliary loss based on the auxiliary two-dimensional prediction results and the sample label data; determine the main loss based on the sample two-dimensional prediction results and the sample label data; determine the training loss based on the auxiliary loss and the main loss; and use the training loss to train the semantic segmentation model.

[0119] Specifically, based on a preset loss function, the auxiliary loss corresponding to each candidate auxiliary feature can be determined according to the auxiliary two-dimensional prediction results and sample label data respectively; then, based on the preset loss function, the main loss can be calculated according to the sample two-dimensional prediction results and sample label data; then, the auxiliary losses and the main loss can be weighted and summed to obtain the training loss; finally, the training loss is used to train the semantic segmentation model.

[0120] Understandably, training the model by determining an auxiliary loss can assist in the training of the semantic segmentation model, increasing its robustness and accuracy.

[0121] The technical solution provided in this invention determines the sample point cloud data corresponding to the sample object from the original point cloud data based on a preset downsampling screening method. Then, based on a preset projection rule, the sample points in the sample point cloud data are dimensionality-reduced to obtain the sample distance image corresponding to the sample point. A convolutional feature extraction network is then used to extract features from the sample distance image to obtain a first feature. Next, a backbone network is used to perform secondary feature extraction on the first feature to obtain a second feature. A prediction network is then used to predict the second feature to obtain a sample two-dimensional prediction result for the sample point. Simultaneously, candidate auxiliary features are selected from at least two normalized auxiliary features, and an auxiliary prediction network is used to predict the candidate auxiliary features to obtain an auxiliary two-dimensional prediction result. Finally, the semantic segmentation model is trained based on the auxiliary two-dimensional prediction result, the sample two-dimensional prediction result, and the sample label data corresponding to the sample point cloud data. The above technical solution, by introducing an auxiliary prediction network for semantic segmentation model training, enables the semantic segmentation model to be trained more accurately.

[0122] Example 5

[0123] Figure 5 This is a flowchart of a semantic segmentation method according to Embodiment 5 of the present invention. This embodiment is applicable to obstacle segmentation in autonomous driving scenarios. The method can be executed by a semantic segmentation device, which can be implemented in hardware and / or software and integrated into an electronic device carrying semantic segmentation functionality, such as a server. Figure 5 As shown, the semantic segmentation method in this embodiment may include:

[0124] S510. Based on preset projection rules, perform dimensionality reduction mapping on the target points in the target point cloud data of the target object to obtain the target distance image corresponding to the target points.

[0125] In this embodiment, the preset projection rule refers to the projection rule for projecting three-dimensional point cloud data into one-dimensional data. For example, the preset projection rule can be a spherical projection rule.

[0126] The target object refers to the object that needs to be semantically segmented, such as a car or a person. Target point cloud data refers to the point cloud data of the target object, including several target points.

[0127] The so-called target distance image refers to the one-dimensional distance image corresponding to the target point.

[0128] Specifically, for each target point in the target point cloud data, a dimensionality reduction mapping can be performed on the target point based on the spherical projection rule to obtain the target distance image corresponding to that target point. For example, the target point can be processed using a formula corresponding to the spherical projection rule used during model training to obtain the target distance image corresponding to the target point.

[0129] S520. Using a semantic segmentation model, the target distance image is predicted to obtain the target point's two-dimensional prediction result.

[0130] The semantic segmentation model is trained based on the training method of the semantic segmentation model provided in any of the above embodiments.

[0131] The two-dimensional prediction result of the target refers to the prediction result of each target point, that is, the semantic prediction category of the target point.

[0132] Specifically, the target distance image corresponding to each target point can be input into the semantic segmentation model, and after the model makes a prediction, the two-dimensional prediction result of the target point can be obtained.

[0133] S530. Perform a three-dimensional transformation on the two-dimensional prediction result of the target to obtain the target prediction result of the target point.

[0134] In this embodiment, the target prediction result refers to the three-dimensional semantic prediction result of the target point.

[0135] Optionally, based on a preset projection rule, the two-dimensional prediction result of the target is reverse-mapped to obtain the three-dimensional prediction result of the target point; the three-dimensional prediction result of the target is filtered to obtain a filtered result; a set number of nearest neighbors are selected from the filtered result, and the three-dimensional prediction results of the nearest neighbors are voted on; the target prediction result of the target point is determined according to the voting result. The preset projection rule is the same as the preset projection rule used during the training of the semantic segmentation model.

[0136] Among them, the target 3D prediction result refers to the 3D semantic prediction result.

[0137] Specifically, based on preset projection rules, the two-dimensional prediction results of the target can be reverse-mapped, i.e., reverse-processed (converting two-dimensional data into three-dimensional data), to obtain the three-dimensional prediction results of the target point. Then, based on the K-Nearest Neighbor (KNN) algorithm, the target prediction result of the target point can be determined from the three-dimensional prediction results. For example, Gaussian filtering can be applied to the three-dimensional prediction results, such as applying a Gaussian filter with a search space size of 7×7, a Gaussian kernel mean of 5, and a standard deviation of 2, to obtain the filtered result. It should be noted that the filtered result includes several points and the three-dimensional prediction results of each point. A set number of nearest neighbors are selected from the filtered result, and the three-dimensional prediction results of the nearest neighbors are voted on. Based on the voting results, the target prediction result of the target point is determined; that is, among these set number of neighboring points, the categories to which these set number of neighboring points belong are compared, and the category to which the majority of neighboring points belong is the final target prediction result of the target point.

[0138] Understandably, since the semantics of the target 3D prediction results obtained after reverse mapping are relatively ambiguous at the object edges, filtering the target 3D prediction results and then voting to determine the target prediction results can make the final semantic segmentation results more accurate.

[0139] The technical solution provided in this invention performs dimensionality reduction mapping on target points in the target point cloud data of a target object based on preset projection rules to obtain a target distance image corresponding to the target point. Then, a semantic segmentation model is used to predict the target distance image to obtain a two-dimensional prediction result of the target point. Finally, the two-dimensional prediction result is transformed into three dimensions to obtain the final target prediction result. The above technical solution uses a semantic segmentation model for semantic segmentation, which can improve the accuracy of semantic segmentation.

[0140] Example 6

[0141] Figure 6 This is a schematic diagram of a training device for a semantic segmentation model according to Embodiment Six of the present invention. This embodiment is applicable to obstacle segmentation in autonomous driving scenarios. The device can be implemented in hardware and / or software and can be integrated into an electronic device that carries the training function of the semantic segmentation model, such as a server. Figure 6 As shown, the training device for the semantic segmentation model in this embodiment may include:

[0142] The sample point cloud data determination module 610 is used to determine the sample point cloud data corresponding to the sample object from the original point cloud data based on a preset downsampling filtering method; the sample object includes mandatory objects and objects to be filtered.

[0143] The sample distance image determination module 620 is used to perform dimensionality reduction mapping on sample points in sample point cloud data based on preset projection rules to obtain the sample distance image corresponding to the sample points.

[0144] The semantic segmentation model training module 630 is used to train the semantic segmentation model based on the sample label data corresponding to the sample distance image and sample point cloud data.

[0145] The technical solution provided by this invention determines the sample point cloud data corresponding to the sample object from the original point cloud data based on a preset downsampling filtering method. Then, based on a preset projection rule, it performs dimensionality reduction mapping on the sample points in the sample point cloud data to obtain the sample distance image corresponding to the sample point. Finally, it trains a semantic segmentation model based on the sample distance image and the sample label data corresponding to the sample point cloud data. This technical solution solves the problem of inaccurate semantic segmentation caused by imbalanced datasets by filtering the original point cloud data of the sample object. Simultaneously, training the semantic segmentation model using the sample distance image improves the accuracy of semantic segmentation.

[0146] Optionally, the sample point cloud data determination module 610 includes:

[0147] The mandatory point cloud determination unit is used to determine the mandatory point cloud data corresponding to the mandatory objects from the original point cloud data;

[0148] The point cloud to be filtered unit is used to remove the required point cloud data from the original point cloud data to obtain the point cloud data to be filtered.

[0149] The point cloud update unit is used to update the point cloud data to be filtered based on the proportion of the objects to be filtered in the point cloud data to be filtered.

[0150] The sample point cloud data determination unit is used to determine the sample point cloud data corresponding to the sample object based on the required point cloud data and the updated point cloud data to be screened.

[0151] Optionally, the point cloud update unit is specifically used for:

[0152] For each single frame of point cloud data in the point cloud data to be filtered, determine the first number of sample points corresponding to the object to be filtered in that single frame of point cloud data.

[0153] Based on the first quantity and the total number of sample points in the point cloud data of that frame, determine the individual proportion of the object to be screened in the single frame of point cloud data;

[0154] Determine the proportion of the number of category boxes corresponding to the objects to be filtered in the total number of category boxes in the point cloud data to be filtered;

[0155] Update the point cloud data to be screened based on individual and overall proportions.

[0156] Optionally, the semantic segmentation model includes a convolutional feature extraction network, a backbone network, and a prediction network;

[0157] Correspondingly, the semantic segmentation model training module 630 includes:

[0158] The first feature determination unit is used to extract features from the sample distance image using a convolutional feature extraction network to obtain the first feature;

[0159] The second feature determination unit is used to perform secondary feature extraction on the first feature using a backbone network to obtain the second feature.

[0160] The two-dimensional prediction result determination unit is used to use a prediction network to predict the second feature and obtain the sample two-dimensional prediction result of the sample point.

[0161] The semantic segmentation model training unit is used to train the semantic segmentation model based on the sample two-dimensional prediction results and the sample label data corresponding to the sample point cloud data.

[0162] Optionally, the backbone network includes at least two sub-residual networks; the at least two sub-residual networks are connected sequentially.

[0163] Accordingly, the second feature determination unit is specifically used for:

[0164] The first feature is input into the backbone network to obtain auxiliary features output by at least two sub-residual networks;

[0165] Normalize at least two auxiliary features to obtain at least two normalized auxiliary features;

[0166] The second feature is obtained by concatenating at least two normalized auxiliary features.

[0167] Optionally, the semantic segmentation model also includes an auxiliary prediction network; correspondingly, the semantic segmentation model training module 630 also includes:

[0168] An auxiliary feature determination unit is used to select candidate auxiliary features from at least two normalized auxiliary features;

[0169] The auxiliary prediction result determination unit is used to predict candidate auxiliary features using an auxiliary prediction network to obtain auxiliary two-dimensional prediction results;

[0170] The semantic segmentation model training unit is also used to train the semantic segmentation model based on the auxiliary two-dimensional prediction results, the sample two-dimensional prediction results, and the sample label data corresponding to the sample point cloud data.

[0171] Optionally, the semantic segmentation model training unit is specifically used for:

[0172] The auxiliary loss is determined based on the auxiliary two-dimensional prediction results and sample label data;

[0173] The principal loss is determined based on the sample two-dimensional prediction results and sample label data;

[0174] The training loss is determined based on the auxiliary loss and the main loss;

[0175] The semantic segmentation model is trained using a training loss.

[0176] Optionally, the device further includes a sample label data determination module, which is specifically used for:

[0177] The true value boxes in the original label data corresponding to the sample point cloud data are expanded in size to obtain the sample true value boxes in the sample label data.

[0178] Based on the category labels in the original label data, each sample point in the sample point cloud data is assigned a category label, thus obtaining the sample point labels in the sample label data.

[0179] The training apparatus for the semantic segmentation model provided in the embodiments of the present invention can execute the training method for the semantic segmentation model provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0180] Example 7

[0181] Figure 7 This is a schematic diagram of a semantic segmentation device according to Embodiment 7 of the present invention. This embodiment is applicable to situations involving obstacle segmentation in autonomous driving scenarios. The device can be implemented in hardware and / or software and can be integrated into electronic devices that carry semantic segmentation functions, such as servers. Figure 7 As shown, the semantic segmentation apparatus of this embodiment may include:

[0182] The target distance image determination module 710 is used to perform dimensionality reduction mapping on the target points in the target point cloud data of the target object based on a preset projection rule, so as to obtain the target distance image corresponding to the target point.

[0183] The target two-dimensional prediction result determination module 720 is used to predict the target distance image using a semantic segmentation model to obtain the target two-dimensional prediction result of the target point; wherein, the semantic segmentation model is trained based on the training method of the semantic segmentation model provided in any embodiment of the present invention;

[0184] The target prediction result determination module 730 is used to perform three-dimensional transformation on the two-dimensional prediction result of the target to obtain the target prediction result of the target point.

[0185] The technical solution provided in this invention performs dimensionality reduction mapping on target points in the target point cloud data of a target object based on preset projection rules to obtain a target distance image corresponding to the target point. Then, a semantic segmentation model is used to predict the target distance image to obtain a two-dimensional prediction result of the target point. Finally, the two-dimensional prediction result is transformed into three dimensions to obtain the final target prediction result. The above technical solution uses a semantic segmentation model for semantic segmentation, which can improve the accuracy of semantic segmentation.

[0186] Optionally, the target prediction result determination module 730 is specifically used for:

[0187] Based on the preset projection rules, the two-dimensional prediction result of the target is reverse-mapped to obtain the three-dimensional prediction result of the target point.

[0188] The 3D prediction results of the target are filtered to obtain the filtered results;

[0189] Select a set number of nearest points from the filtering results, and vote on the 3D prediction results of the nearest points;

[0190] Based on the voting results, the target prediction results for the target points are determined.

[0191] The semantic segmentation apparatus provided in the embodiments of the present invention can execute the semantic segmentation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0192] Example 8

[0193] Figure 8 This is a schematic diagram of the structure of an electronic device that implements the training method or semantic segmentation method of the semantic segmentation model in the embodiments of the present invention. Figure 8 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0194] like Figure 8 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0195] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0196] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as training methods for semantic segmentation models or semantic segmentation methods.

[0197] In some embodiments, the training method or semantic segmentation method of the semantic segmentation model may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the training method or semantic segmentation method of the semantic segmentation model described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the training method or semantic segmentation method of the semantic segmentation model by any other suitable means (e.g., by means of firmware).

[0198] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0199] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0200] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0201] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0202] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0203] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0204] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0205] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for training a semantic segmentation model, the method comprising: The method comprises the following steps: determining sample point cloud data corresponding to a sample object from original point cloud data based on a preset downsampling screening mode; the sample object comprises a mandatory object and a to-be-screened object; dimensionally mapping sample points in the sample point cloud data based on a preset projection rule to obtain sample distance images corresponding to the sample points; training a semantic segmentation model according to sample distance images and sample label data corresponding to the sample point cloud data; the method of determining sample point cloud data corresponding to a sample object from original point cloud data based on a preset downsampling screening mode comprises the following steps: determining mandatory point cloud data corresponding to the mandatory object from the original point cloud data; removing the mandatory point cloud data from the original point cloud data to obtain to-be-screened point cloud data; updating the to-be-screened point cloud data according to the proportion of the to-be-screened object in the to-be-screened point cloud data; determining sample point cloud data corresponding to the sample object according to the mandatory point cloud data and the updated to-be-screened point cloud data; the method of updating the to-be-screened point cloud data according to the proportion of the to-be-screened object in the to-be-screened point cloud data comprises the following steps: for each single-frame point cloud data in the to-be-screened point cloud data, determining a first number of sample points corresponding to the to-be-screened object in the single-frame point cloud data; determining an individual proportion of the to-be-screened object in the single-frame point cloud data according to the first number and a total number of sample points in the frame of point cloud data; determining an overall proportion of the number of category boxes corresponding to the to-be-screened object in the total number of category boxes in the to-be-screened point cloud data; updating the to-be-screened point cloud data according to the individual proportion and the overall proportion; the method of updating the to-be-screened point cloud data according to the individual proportion and the overall proportion comprises the following steps: if the individual proportion is greater than k times the overall proportion, randomly screen out P% of the point cloud frames from the to-be-screened point cloud data; wherein k is a natural number and P is a natural number.

2. The method of claim 1, wherein, the semantic segmentation model comprises a convolution feature extraction network, a backbone network and a prediction network; correspondingly, the method of training a semantic segmentation model according to sample distance images and sample label data corresponding to sample point cloud data comprises the following steps: using the convolution feature extraction network to extract features from the sample distance images to obtain first features; using the backbone network to extract second features from the first features; using the prediction network to predict the second features to obtain sample two-dimensional prediction results of the sample points; training the semantic segmentation model according to the sample two-dimensional prediction results and sample label data corresponding to the sample point cloud data.

3. The method of claim 2, wherein, the backbone network comprises at least two sub-residual networks; the at least two sub-residual networks are connected in sequence; correspondingly, the method of using the backbone network to extract second features from the first features comprises the following steps: inputting the first features into the backbone network to obtain auxiliary features output by the at least two sub-residual networks; normalizing the at least two auxiliary features to obtain at least two normalized auxiliary features; Splicing the at least two normalized auxiliary features to obtain a second feature.

4. The method of claim 3, wherein, The semantic segmentation model further comprises an auxiliary prediction network; correspondingly, training the semantic segmentation model comprises: selecting a candidate auxiliary feature from the at least two normalized auxiliary features; predicting the candidate auxiliary feature by using the auxiliary prediction network to obtain an auxiliary two-dimensional prediction result; training the semantic segmentation model according to the auxiliary two-dimensional prediction result, the sample two-dimensional prediction result and sample label data corresponding to the sample point cloud data.

5. The method of claim 4, wherein, The training of the semantic segmentation model according to the auxiliary two-dimensional prediction result, the sample two-dimensional prediction result and sample label data corresponding to the sample point cloud data comprises: determining an auxiliary loss according to the auxiliary two-dimensional prediction result and the sample label data; determining a main loss according to the sample two-dimensional prediction result and the sample label data; determining a training loss according to the auxiliary loss and the main loss; training the semantic segmentation model by using the training loss.

6. The method of claim 1, wherein, Further comprising: performing size expansion on a ground truth box in original label data corresponding to sample point cloud data to obtain a sample ground truth box in sample label data; assigning a class label to each sample point in the sample point cloud data according to a class label in the original label data to obtain a sample point label in the sample label data.

7. A method of semantic segmentation, characterized in that, Comprise: performing dimensionality reduction mapping on a target point in target point cloud data of a target object based on a preset projection rule to obtain a target distance image corresponding to the target point; predicting the target distance image by using a semantic segmentation model to obtain a target two-dimensional prediction result of the target point; wherein the semantic segmentation model is trained based on the training method of the semantic segmentation model according to any one of claims 1-6; performing three-dimensional conversion on the target two-dimensional prediction result to obtain a target prediction result of the target point.

8. The method of claim 7, wherein, The three-dimensional conversion of the target two-dimensional prediction result to obtain the target prediction result of the target point comprises: performing reverse mapping on the target two-dimensional prediction result based on a preset projection rule to obtain a target three-dimensional prediction result of the target point; performing filtering on the target three-dimensional prediction result to obtain a filtering result; selecting a set number of nearest points from the filtering result and performing voting on three-dimensional prediction results of the nearest points; determining the target prediction result of the target point according to the voting result. 9.A device for training a semantic segmentation model, the device comprising: Comprise: a sample point cloud data determination module configured to determine sample point cloud data corresponding to a sample object from original point cloud data based on a preset downsampling screening manner; the sample object comprises a required object and a to-be-screened object; a sample distance image determination module configured to perform dimensionality reduction mapping on a sample point in the sample point cloud data based on a preset projection rule to obtain a sample distance image corresponding to the sample point; a semantic segmentation model training module configured to train a semantic segmentation model according to a sample distance image and sample label data corresponding to the sample point cloud data; the sample point cloud data determination module further comprises: The optional point cloud determination unit is configured to determine the optional point cloud data corresponding to the optional object from the original point cloud data; The to-be-screened point cloud determination unit is configured to remove the optional point cloud data from the original point cloud data to obtain to-be-screened point cloud data; The point cloud updating unit is configured to update the to-be-screened point cloud data according to the proportion of the to-be-screened object in the to-be-screened point cloud data; The sample point cloud data determination unit is configured to determine sample point cloud data corresponding to a sample object according to the optional point cloud data and the updated to-be-screened point cloud data; The point cloud updating unit is specifically configured to: For each single-frame point cloud data in the to-be-screened point cloud data, determine a first number of sample points corresponding to the to-be-screened object in the single-frame point cloud data; According to the first number and a total number of sample points in the frame of point cloud data, determine an individual proportion of the to-be-screened object in the single-frame point cloud data; Determine an overall proportion of the number of category boxes corresponding to the to-be-screened object in the total number of category boxes of the to-be-screened point cloud data; According to the individual proportion and the overall proportion, update the to-be-screened point cloud data; The updating of the to-be-screened point cloud data according to the individual proportion and the overall proportion includes: If the individual proportion is greater than k times the overall proportion, randomly screen out P% of the point cloud frames from the to-be-screened point cloud data; wherein k is a natural number, and P is a natural number.

10. A semantic segmentation apparatus, characterized by, The target distance image determination module is configured to perform dimensionality reduction mapping on a target point in target point cloud data of a target object based on a preset projection rule to obtain a target distance image corresponding to the target point; The target two-dimensional prediction result determination module is configured to use a semantic segmentation model to predict the target distance image to obtain a target two-dimensional prediction result of the target point; wherein the semantic segmentation model is trained based on the training method of the semantic segmentation model in any one of claims 1-6; The target prediction result determination module is configured to perform three-dimensional conversion on the target two-dimensional prediction result to obtain a target prediction result of the target point. The electronic device includes:

11. An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the training method of the semantic segmentation model in any one of claims 1-6, or the semantic segmentation method in any one of claims 7-8. The computer readable storage medium stores computer instructions for causing the processor to execute the training method of the semantic segmentation model in any one of claims 1-6, or the semantic segmentation method in any one of claims 7-8.

12. A computer-readable storage medium, characterized in that, ​

Images