Data enhancement method and device of multi-task model, training method and device of multi-task model, equipment, computer program product

Abstract

This application discloses a data augmentation method and apparatus for a multi-task model, a training method and apparatus for a multi-task model, an equipment, and a computer program product. The method includes: acquiring raw point cloud data, including the 3D bounding box of a target object and its corresponding semantic label data; dividing the 3D bounding box of the target object into multiple parts based on the raw point cloud data using a differentiated partitioning strategy, obtaining the 3D bounding boxes of multiple parts corresponding to the target object and the point cloud data contained in each part, thereby constructing a mapping relationship between the point cloud data in each part and the semantic label data; and enhancing the point cloud data contained in each part based on the mapping relationship and a preset multi-dimensional point cloud enhancement strategy. This application, through an innovative data augmentation method, ensures that the augmented data maintains accurate semantic correspondence in the multi-task model, improves the quality and richness of the training data for the multi-task model, and enhances the model's adaptability to different scenarios.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a data augmentation method and apparatus for a multi-task model, a training method and apparatus for a multi-task model, an equipment, and a computer program product. Background Technology

[0002] Autonomous driving technology is a core area of ​​the current integration of artificial intelligence and the automotive industry. Its perception system relies on various sensors (such as cameras, LiDAR, millimeter-wave radar, etc.) to understand the surrounding environment. LiDAR has become a key sensor in autonomous driving perception systems because it can provide accurate three-dimensional spatial information. It generates point cloud data by emitting laser beams and receiving reflected signals. This data contains geometric information about objects in the environment.

[0003] In LiDAR point cloud data processing, a multi-task integrated model for 3D object detection and segmentation is a key technology. The purpose of 3D object detection is to identify and locate various objects (such as vehicles, pedestrians, cyclists, etc.) in the point cloud, while the purpose of 3D segmentation is to identify the category label of each point (such as vehicle, ground, vegetation, etc.). In recent years, deep learning-based methods for 3D object detection and segmentation have made significant progress for single-task models, such as PointPillars (Lang et al., 2019), PV-RCNN (Shi et al., 2020), VoxelNet (Zhou & Tuzel, 2018), and CPGNet (Li et al., 2022). These methods typically require training on large amounts of labeled data, but labeling point cloud data is extremely costly and time-consuming.

[0004] Data augmentation is an effective technique for addressing insufficient training data by transforming existing data to increase its diversity. In the image processing domain, common data augmentation methods include rotation, cropping, and color transformation. In point cloud data processing, existing methods primarily draw upon approaches from the image processing domain, including:

[0005] 1) Global rotation, translation, and scaling;

[0006] 2) Global flip;

[0007] 3) Randomly discard point clouds;

[0008] 4) Truth sampling (copy and paste enhancement).

[0009] For 3D object detection or segmentation models, existing point cloud data augmentation methods have the following main problems:

[0010] 1) Global augmentation disrupts geometric integrity: Existing methods such as global rotation, translation, and scaling, while simple to implement, alter the overall spatial distribution of the point cloud, potentially disrupting the geometric integrity of objects. For example, performing a global rotation on a partially occluded object may produce a geometric structure that does not conform to physical laws, leading the model to learn inaccurate geometric features.

[0011] 2) Lack of physical plausibility and scene consistency: Methods such as dropout and copy-paste lack physical plausibility and may generate data that does not conform to the actual scene. For example, dropout may result in the missing parts of objects, while copy-paste may place objects in physically impossible locations (such as in the air or underground), thus compromising scene consistency.

[0012] 3) Ignoring internal structural features: Most existing methods treat objects as a whole, ignoring their internal structural features. Point clouds in different parts of the object have different semantic information and geometric features, and global processing methods cannot fully utilize this fine-grained information. For example, the wheel and body parts of a vehicle have different point cloud distribution features, but existing methods cannot distinguish between them.

[0013] 4) Limited Enhancement Effect: Existing enhancement methods generate training data with insufficient diversity, failing to fully cover various situations in real-world scenarios, especially extreme and rare scenarios. This results in insufficient generalization ability of the trained model in complex scenarios, particularly with significant performance degradation under challenging conditions such as sparse point clouds and severe occlusion.

[0014] 5) Limited application conditions: Most data augmentation is designed for single-task models, such as 3D object detection or segmentation single-task models, and is not suitable for multi-task models. Summary of the Invention

[0015] To address at least one of the technical problems mentioned above, embodiments of this application provide a data augmentation method and apparatus for a multi-task model, a training method and apparatus for a multi-task model, a device, and a computer program product, to improve the diversity and quality of point cloud data samples used for training the multi-task model.

[0016] The embodiments of this application adopt the following technical solutions:

[0017] In a first aspect, embodiments of this application provide a data augmentation method for a multi-task model, the data augmentation method for the multi-task model comprising:

[0018] Obtain raw point cloud data for training a multi-task model, wherein the raw point cloud data includes the 3D bounding box of the target object and the corresponding semantic label data;

[0019] Based on the original point cloud data, the 3D bounding box of the target object is divided into multiple parts using a differentiated partitioning strategy, thereby obtaining the 3D bounding boxes of multiple parts corresponding to the target object and the point cloud data contained in each part.

[0020] Based on the 3D bounding boxes of multiple components corresponding to the target object and the point cloud data contained in each component, a mapping relationship between the point cloud data and semantic label data in each component is constructed.

[0021] Based on the mapping relationship between point cloud data and semantic tag data in each component and the preset multi-dimensional point cloud enhancement strategy, the point cloud data contained in each component is enhanced to obtain point cloud enhanced data.

[0022] Optionally, the step of dividing the 3D bounding box of the target object into multiple parts using a differentiated partitioning strategy based on the original point cloud data, to obtain multiple parts corresponding to the target object and the point cloud data contained in each part, includes:

[0023] The category and geometric features of the target object are determined based on the original point cloud data;

[0024] Based on the category and geometric features of the target object, determine the differentiated segmentation strategy corresponding to the target object;

[0025] The 3D bounding box of the target object is divided into multiple parts using a differentiated partitioning strategy corresponding to the target object, thereby obtaining the 3D bounding boxes of the multiple parts corresponding to the target object.

[0026] Calculate the spatial relationship between each point in the 3D bounding box of the target object and the 3D bounding boxes of each component;

[0027] Based on the spatial relationship between each point in the 3D bounding box of the target object and the 3D bounding boxes of each component, the component to which each point belongs is determined, and the point cloud data contained in each component is obtained.

[0028] Optionally, the step of constructing the mapping relationship between the point cloud data and semantic label data of each component based on the 3D bounding boxes of multiple components corresponding to the target object and the point cloud data contained in each component includes:

[0029] A component point cloud set is constructed based on the point cloud data contained in each component, and the component point cloud set is used to store the coordinate data of all points contained in each component;

[0030] An index set and a semantic tag set are constructed based on the original point cloud data. The index set is used to store the original index position of each point, and the semantic tag set is used to store the semantic tag of each point.

[0031] Based on the component point cloud set, the index set, and the semantic tag set, a mapping relationship between the point cloud data and semantic tag data in each component is constructed.

[0032] Optionally, the point cloud data contained in each component is enhanced based on the mapping relationship between point cloud data and semantic tag data in each component and a preset multi-dimensional point cloud enhancement strategy, resulting in enhanced point cloud data including:

[0033] Based on a preset multi-dimensional point cloud enhancement strategy, the current configuration parameter information of the point cloud enhancement strategy is obtained and parsed to obtain the current point cloud enhancement operation and the corresponding operation parameters. The point cloud enhancement operation includes at least one of the following: component discarding operation, component swapping operation, component mixing operation, sparsification operation, noise addition operation, and coordinate perturbation operation.

[0034] The point cloud data contained in each component is enhanced by using the currently adopted point cloud enhancement operation and the corresponding operation parameters to obtain the point cloud enhancement data.

[0035] Optionally, the point cloud enhancement operation includes a component discarding operation. The point cloud data contained in each component is enhanced based on the mapping relationship between the point cloud data and semantic tag data in each component and a preset multi-dimensional point cloud enhancement strategy, resulting in enhanced point cloud data including:

[0036] Set differentiated discard parameters for each component based on its importance and corresponding semantic tag data;

[0037] The point cloud data contained in the corresponding component is discarded according to the differentiated discard parameters of each component, and the semantic label data and index data of the corresponding component are synchronously cleared according to the mapping relationship between the point cloud data and semantic label data in each component.

[0038] Optionally, the point cloud enhancement operation includes a component exchange operation. The point cloud data contained in each component is enhanced based on the mapping relationship between point cloud data and semantic tag data in each component and a preset multi-dimensional point cloud enhancement strategy, resulting in enhanced point cloud data including:

[0039] Based on the component exchange operation, the source component and target component to be exchanged are determined, and collision detection and physical rationality detection are performed.

[0040] If the detection passes, based on the mapping relationship between the point cloud data and semantic label data in each component, the point cloud data, corresponding semantic label data, and index data contained in the source component are completely exchanged with the point cloud data, corresponding semantic label data, and index data contained in the target component, and the corresponding spatial transformation information is updated.

[0041] Optionally, the point cloud enhancement operation includes a component mixing operation. Based on the mapping relationship between point cloud data and semantic label data in each component and a preset multi-dimensional point cloud enhancement strategy, the point cloud data contained in each component is enhanced to obtain enhanced point cloud data, including:

[0042] Set differentiated mixing parameters for each component based on its importance and corresponding semantic tag data;

[0043] Based on the differentiated mixing parameters, point cloud data of other instances of the same category are mixed into the corresponding position of the current component, and based on the mapping relationship between point cloud data and semantic tag data in each component, semantic tag data corresponding to point cloud data of other instances of the same category are integrated into the current component.

[0044] Optionally, the point cloud enhancement operation includes a noise addition operation. The point cloud data contained in each component is enhanced based on the mapping relationship between the point cloud data and semantic label data in each component and a preset multi-dimensional point cloud enhancement strategy, resulting in enhanced point cloud data including:

[0045] Based on the distribution characteristics of the point cloud data contained in each component, noise points are generated in the corresponding component using a preset noise model;

[0046] The generated noise points are assigned corresponding semantic label data.

[0047] Optionally, the point cloud data contained in each component is enhanced based on the mapping relationship between point cloud data and semantic tag data in each component and a preset multi-dimensional point cloud enhancement strategy, resulting in enhanced point cloud data including:

[0048] Obtain distance information between the target object and the sensor;

[0049] The point cloud enhancement operation and corresponding operation parameters are dynamically adjusted based on the distance information between the target object and the sensor, as well as the category of the target object.

[0050] Secondly, embodiments of this application also provide a method for training a multi-task model, wherein training the multi-task model includes:

[0051] Acquire training sample data for a multi-task model, wherein the training sample data includes point cloud augmented data obtained by any of the aforementioned data augmentation methods for the multi-task model;

[0052] A multi-task model is trained using the training sample data, the multi-task model including object detection and segmentation task models.

[0053] Thirdly, embodiments of this application also provide a data augmentation apparatus for a multi-task model, the data augmentation apparatus for the multi-task model comprising:

[0054] The first acquisition unit is used to acquire raw point cloud data for training a multi-task model. The raw point cloud data includes the 3D bounding box of the target object and the corresponding semantic label data.

[0055] The partitioning unit is used to divide the 3D bounding box of the target object into multiple parts based on the original point cloud data using a differentiated partitioning strategy, thereby obtaining the 3D bounding boxes of the multiple parts corresponding to the target object and the point cloud data contained in each part.

[0056] The construction unit is used to construct a mapping relationship between point cloud data and semantic label data in each component based on the 3D bounding boxes of multiple components corresponding to the target object and the point cloud data contained in each component.

[0057] The point cloud enhancement unit is used to enhance the point cloud data contained in each component based on the mapping relationship between the point cloud data and semantic tag data in each component and the preset multi-dimensional point cloud enhancement strategy, so as to obtain point cloud enhanced data.

[0058] Fourthly, embodiments of this application also provide a training apparatus for a multi-task model, characterized in that the training apparatus for the multi-task model includes:

[0059] The second acquisition unit is used to acquire training sample data of the multi-task model, wherein the training sample data includes point cloud augmented data obtained by the data augmentation method of the multi-task model described above.

[0060] A training unit is used to train a multi-task model using the training sample data, the multi-task model including object detection and segmentation task models.

[0061] Fifthly, embodiments of this application also provide an apparatus, comprising:

[0062] A processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform a training method for any of the aforementioned multitasking models.

[0063] Sixthly, embodiments of this application also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the data augmentation method of any of the aforementioned multitasking models.

[0064] The above-mentioned technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: The data augmentation method for the multi-task model in the embodiments of this application first obtains the original point cloud data for training the multi-task model. The original point cloud data includes the 3D bounding box of the target object and the corresponding semantic label data. Then, based on the original point cloud data, the 3D bounding box of the target object is divided into multiple parts using a differentiated partitioning strategy to obtain the 3D bounding boxes of multiple parts corresponding to the target object and the point cloud data contained in each part. Then, based on the 3D bounding boxes of multiple parts corresponding to the target object and the point cloud data contained in each part, a mapping relationship between the point cloud data in each part and the semantic label data is constructed. Finally, based on the mapping relationship between the point cloud data in each part and the semantic label data and the preset multi-dimensional point cloud augmentation strategy, the point cloud data contained in each part is augmented to obtain augmented point cloud data. The data augmentation method for multi-task models in this application adopts a differentiated partitioning strategy to divide the 3D bounding box of the target object into multiple semantically meaningful components and constructs a point cloud-label synchronous management mechanism. This effectively solves the problem of insufficient data or lack of data diversity in multi-task model training. Through innovative data augmentation methods, it ensures that the augmented data maintains accurate semantic correspondence in detection and segmentation tasks, improves the quality and richness of multi-task model training data, enhances the performance of multi-task models, and strengthens the model's adaptability to different scenarios and data. This enables the model to complete target detection and segmentation tasks more accurately and stably in practical applications. Attached Figure Description

[0065] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0066] Figure 1 This is a flowchart illustrating a data augmentation method for a multi-task model in an embodiment of this application.

[0067] Figure 2 This is a schematic diagram of a data augmentation process for a multi-task model in an embodiment of this application;

[0068] Figure 3 This is a flowchart illustrating a training method for a multi-task model according to an embodiment of this application.

[0069] Figure 4This is a schematic diagram of the structure of a data augmentation device for a multi-task model according to an embodiment of this application;

[0070] Figure 5 This is a schematic diagram of the structure of a training device for a multi-task model according to an embodiment of this application;

[0071] Figure 6 This is a schematic diagram of the structure of a device according to an embodiment of this application. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0073] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0074] This application provides a data augmentation method for a multi-task model, such as... Figure 1 The diagram provided illustrates a flowchart of a data augmentation method for a multi-task model according to an embodiment of this application. The data augmentation method for the multi-task model includes the following steps S110 to S140:

[0075] Step S110: Obtain the original point cloud data for training the multi-task model. The original point cloud data includes the 3D bounding box of the target object and the corresponding semantic label data.

[0076] Combination Figure 2 This document provides a schematic diagram of a data augmentation process for a multi-task model in an embodiment of this application. The multi-task model in this embodiment can be, for example, a multi-task model for object detection and segmentation. First, pre-labeled raw point cloud data is acquired. This raw point cloud data mainly contains two key pieces of information: one is the 3D bounding box information of the target object, which provides spatial definition for subsequent operations on the target object; the other is the corresponding semantic label data. For the detection task, there are instance categories of the target object; for the segmentation task, there are semantic categories of each point. This label data provides semantic guidance for subsequent data augmentation and model training.

[0077] Step S120: Based on the original point cloud data, the 3D bounding box of the target object is divided into multiple parts using a differentiated partitioning strategy, thereby obtaining the 3D bounding boxes of multiple parts corresponding to the target object and the point cloud data contained in each part.

[0078] Based on the original point cloud data, a differentiated partitioning strategy is adopted. The key here lies in the innovative geometric structure analysis method. By meticulously analyzing the 3D bounding box of the target object and considering its geometric structural features, the 3D bounding box is divided into multiple semantically meaningful components. For example, for a car target object, it can be divided into components such as the car body and wheels. After partitioning, the 3D bounding boxes of multiple components corresponding to the target object are obtained, and the point cloud data contained in each component is clearly defined, laying the foundation for subsequent operations on each component.

[0079] Step S130: Based on the 3D bounding boxes of multiple components corresponding to the target object and the point cloud data contained in each component, construct the mapping relationship between the point cloud data and semantic label data in each component.

[0080] Based on the 3D bounding boxes and point cloud data of each component obtained in step S120, a point cloud-label synchronization management mechanism is constructed. A unique identifier is established for each point in the point cloud, and a mapping relationship between the points in the point cloud and semantic labels is established. This mapping relationship ensures that the semantic labels can be updated synchronously after any augmentation operation is performed on the point cloud data, always maintaining an accurate correspondence between the point cloud and the semantic labels. This is a key technology for achieving multi-task data augmentation.

[0081] Step S140: Based on the mapping relationship between point cloud data and semantic tag data in each component and the preset multi-dimensional point cloud enhancement strategy, the point cloud data contained in each component is enhanced to obtain point cloud enhanced data.

[0082] Based on the mapping relationship constructed in step S130 and the preset multi-dimensional point cloud enhancement strategy, the point cloud data contained in each component is enhanced. This embodiment of the application pre-sets several core enhancement operations, all of which rely on the aforementioned point cloud-label synchronous update mechanism, enabling the enhanced data to be applicable to both detection and segmentation tasks, ensuring data consistency and effectiveness.

[0083] The data augmentation method for multi-task models in this application adopts a differentiated partitioning strategy to divide the 3D bounding box of the target object into multiple semantically meaningful components and constructs a point cloud-label synchronous management mechanism. This effectively solves the problem of insufficient data or lack of data diversity in multi-task model training. Through innovative data augmentation methods, it ensures that the augmented data maintains accurate semantic correspondence in detection and segmentation tasks, improves the quality and richness of multi-task model training data, enhances the performance of multi-task models, and strengthens the model's adaptability to different scenarios and data. This enables the model to complete target detection and segmentation tasks more accurately and stably in practical applications.

[0084] In some embodiments of this application, the step of dividing the 3D bounding box of the target object into multiple parts using a differentiated partitioning strategy based on the original point cloud data to obtain multiple parts corresponding to the target object and the point cloud data contained in each part includes: determining the category and geometric features of the target object based on the original point cloud data; determining the differentiated partitioning strategy corresponding to the target object based on the category and geometric features of the target object; dividing the 3D bounding box of the target object into multiple parts using the differentiated partitioning strategy corresponding to the target object to obtain the 3D bounding boxes of multiple parts corresponding to the target object; calculating the spatial relationship between each point in the 3D bounding box of the target object and the 3D bounding boxes of each part; determining the part to which each point belongs based on the spatial relationship between each point in the 3D bounding box of the target object and the 3D bounding boxes of each part, and obtaining the point cloud data contained in each part.

[0085] Based on the acquired raw point cloud data, the target object is analyzed to determine its category, such as whether it is a truck, bus, car, person, or bike. At the same time, the geometric features of the target object, such as length, width, height and other dimensional information, as well as its overall shape, are extracted. This information provides a basis for determining the subsequent differentiated classification strategy.

[0086] Based on the target object's category and geometric features, a differentiated segmentation strategy is determined to suit the specific target object. The number of components varies depending on the object category. For large objects like trucks and buses, they can be divided into 12 components, for example, by dividing the length into three equal parts, the width into two equal parts, and the height into two equal parts. For medium-sized objects like cars, they can be divided into 8 components, for example, by dividing the length, width, and height into two equal parts. For small objects like persons and bikes, they can be divided into 4 components, for example, by dividing the width and height into two equal parts. This segmentation method comprehensively considers the geometric features and semantic information of the target object, ensuring that each component has a relatively independent geometric structure.

[0087] Using a differentiated partitioning strategy corresponding to the predetermined target object, the 3D bounding box of the target object is partitioned. First, by calculating the midpoint and corner point information of each side of the bounding box, a rich geometric representation system containing the original 8 corner points and the midpoints of each side is generated. Based on this, the 3D bounding box is divided into multiple parts according to the predetermined partitioning method, resulting in the 3D bounding boxes of multiple parts corresponding to the target object.

[0088] The spatial relationship between each point in the 3D bounding box of the target object and the 3D bounding boxes of each component is calculated. An efficient detection algorithm is employed, combined with the semantic category information of the object, to comprehensively determine the component to which each point belongs. This process considers not only the spatial location of the points but also incorporates semantic information, ensuring that the segmentation results are both geometrically consistent and semantically reasonable, ultimately yielding the point cloud data contained in each component.

[0089] This application's embodiments determine the target object category and geometric features based on raw point cloud data, and employ a differentiated partitioning strategy to accurately divide the target object's 3D bounding box into multiple parts. This fully considers the characteristics of different categories of target objects, ensuring that each part has a relatively independent geometric structure. Furthermore, the partitioning process combines spatial relationships and semantic information, guaranteeing the accuracy and rationality of the partitioning results. This provides a more detailed and targeted data foundation for subsequent multi-task model data augmentation, helping to improve the multi-task model's ability to identify and process different target objects, and enhancing the model's adaptability and accuracy in practical application scenarios.

[0090] In some embodiments of this application, the step of constructing a mapping relationship between point cloud data and semantic tag data in each component based on the 3D bounding boxes of multiple components corresponding to the target object and the point cloud data contained in each component includes: constructing a component point cloud set based on the point cloud data contained in each component, wherein the component point cloud set is used to store the coordinate data of all points contained in each component; constructing an index set and a semantic tag set based on the original point cloud data, wherein the index set is used to store the original index position of each point and the semantic tag set is used to store the semantic tag of each point; and constructing a mapping relationship between point cloud data and semantic tag data in each component based on the component point cloud set, the index set, and the semantic tag set.

[0091] This application introduces a point cloud index tracking and label synchronization management mechanism, which is a key technology for achieving multi-task data augmentation. By establishing a unique identifier and semantic label mapping relationship for each point cloud point, it ensures that the accurate correspondence between the point cloud and the semantic label is maintained after any augmentation operation. Specifically, the following three parallel data structures are maintained:

[0092] (1) Component point cloud set: For the point cloud data contained in each component, a component point cloud set is created. This set is specifically used to store the coordinate data of all points contained in each component. In this way, the point cloud data of different components are classified and stored separately, which facilitates the independent processing and analysis of the point cloud data of each component in the future.

[0093] (2) Index set: The index set is constructed based on the original point cloud data. The purpose of the index set is to store the original index position of each point in the original point cloud array. This index position is like the "identity number" of each point, which can accurately locate its position in the original data.

[0094] (3) Semantic label set: A semantic label set is constructed based on the original point cloud data. The semantic label set is used to store the semantic label corresponding to each point. The semantic label clarifies the semantic category information represented by each point in tasks such as target detection and segmentation, such as a part of the target object, background point, etc.

[0095] Based on the previously maintained component point cloud set, index set, and semantic tag set, a mapping relationship between the point cloud data and semantic tag data in each component is essentially established. Specifically, by using the original index positions recorded in the index set, each point in the component point cloud set can be associated with its corresponding semantic tag in the semantic tag set, thus establishing a unique identity and semantic tag mapping relationship for each point in each component. This mapping relationship ensures that any subsequent augmentation operations on the point cloud data can be performed synchronously based on this mapping relationship, maintaining the accurate correspondence between the point cloud and the semantic tag.

[0096] This application's embodiments construct a component point cloud set, an index set, and a semantic tag set. Based on these three sets, it innovatively introduces a point cloud index tracking and tag synchronization management mechanism, successfully establishing a mapping relationship between point cloud data and semantic tag data in each component. This mechanism assigns a unique identifier to each point cloud point and establishes a precise mapping with the semantic tag. In subsequent enhancement operations, regardless of any modifications made to the point cloud data, the semantic information can be synchronously updated through the index mapping. This completely solves the problem of data and tag synchronization issues in traditional methods, ensuring data consistency and enabling the enhanced data to simultaneously meet the needs of multiple tasks such as object detection and segmentation.

[0097] In some embodiments of this application, the step of enhancing the point cloud data contained in each component based on the mapping relationship between the point cloud data and semantic tag data in each component and a preset multi-dimensional point cloud enhancement strategy to obtain enhanced point cloud data includes: obtaining and parsing the current configuration parameter information of the point cloud enhancement strategy based on the preset multi-dimensional point cloud enhancement strategy to obtain the currently adopted point cloud enhancement operation and the corresponding operation parameters. The point cloud enhancement operation includes at least one of component discarding operation, component swapping operation, component mixing operation, sparsification operation, noise addition operation, and coordinate perturbation operation; and using the currently adopted point cloud enhancement operation and the corresponding operation parameters to enhance the point cloud data contained in each component to obtain the enhanced point cloud data.

[0098] Continue to refer to Figure 2 In this application embodiment, multiple dimensions of enhancement operation types are pre-defined, such as component dropout, component swap, component mix, sparsification, noise addition, and coordinate jitter. Based on the pre-defined multiple dimensions of enhancement operation types, a flexible parameter parsing mechanism is designed to support the free combination and probability control of multiple enhancement strategies.

[0099] Specifically, the current configuration parameters of the point cloud enhancement strategy are first obtained. Intuitive string encoding can be used to represent complex enhancement strategies; for example, "dropout_p05_swa_p10" means dropping one part with a 50% probability and performing one part swap with a 100% probability. The parameter parser first decomposes the input string into multiple parameter blocks by delimiters, then identifies the operation type and probability value in each parameter block. Simultaneously, reasonable default values ​​are set for each enhancement operation. These default values ​​are used when some parameters are not explicitly specified. Conditional probability settings are also supported, allowing dynamic adjustment of enhancement parameters based on object category, distance, or other features. Parameter validation functionality is also provided to ensure all parameter values ​​are within the valid range.

[0100] By parsing the configuration parameter information, the current point cloud enhancement operation and its corresponding parameters can be obtained. For example, it can be determined what probability operations such as component discarding and swapping are performed, as well as the specific number of operations and other related parameters.

[0101] Using the determined point cloud augmentation operations and corresponding parameters, the point cloud data contained in each component is augmented. When multiple augmentation operations are used simultaneously, they can be executed in the following designed order: Dropout → Swap → Mix → Sparse → Noise → Jitter. That is, Dropout is performed first to simulate occlusion, then Swap and Mix operations introduce data diversity, followed by Sparse operations to adjust point density, and finally Noise and Jitter are added to simulate sensor characteristics, thus ensuring the logical rationality and physical realism of the augmentation process. Of course, those skilled in the art can flexibly adjust the specific combination of augmentation operations and the specific order of the operations according to actual needs.

[0102] By designing a flexible parameter parsing mechanism and representing complex augmentation strategies with intuitive string encoding, the system supports the free combination and probabilistic control of multiple augmentation strategies, greatly improving the practicality and flexibility of data augmentation. By performing multi-dimensional augmentation processing on the point cloud data of each component in a reasonable order, various real-world scenarios are simulated, such as occlusion, data diversity, and sensor characteristics. This effectively enriches the point cloud data, enhances its diversity and realism, and enables the augmented point cloud data to better adapt to the training needs of multi-task models.

[0103] In some embodiments of this application, the point cloud enhancement operation includes a component discarding operation. The enhancement process for the point cloud data contained in each component, based on the mapping relationship between the point cloud data and semantic tag data in each component and a preset multi-dimensional point cloud enhancement strategy, to obtain enhanced point cloud data includes: setting differentiated discarding parameters for each component according to the importance of each component and the corresponding semantic tag data; discarding the point cloud data contained in the corresponding component according to the differentiated discarding parameters for each component; and synchronously clearing the semantic tag data and index data of the corresponding component according to the mapping relationship between the point cloud data and semantic tag data in each component.

[0104] The component dropout operation tailors different dropout parameters for each component based on its importance and corresponding semantic label data. Specifically, it analyzes and evaluates each component of a target object to determine its importance. For example, for a vehicle target object, critical functional parts like the wheel area are crucial for overall vehicle identification and functional judgment, so a lower dropout probability is set; while non-critical parts, such as non-feature areas on the sides of the vehicle, can be dropped with a higher probability. In this way, different components have different probabilities of being dropped when facing the dropout operation.

[0105] Based on the differentiated discard parameters set for each component, the point cloud data contained in the corresponding component is discarded. Specific components are randomly selected according to these parameters, and then the point cloud data in these components is completely removed to simulate partial occlusion situations that may occur in real-world scenes. For example, in a real-world scene, a vehicle may be partially occluded by other objects; this operation can generate a similar data augmentation effect.

[0106] While performing point cloud data discarding operations, the semantic label data and index data of the corresponding components are simultaneously cleared based on the mapping relationship between point cloud data and semantic label data in each component. Because there is a close relationship between point cloud data, semantic label data, and index data, when point cloud data is removed, the corresponding semantic label and index data also become meaningless. Clearing these data can ensure the integrity and consistency of the entire data structure and avoid data chaos.

[0107] By setting differentiated discard parameters based on component importance and semantic labels, the point cloud data of components is discarded in a targeted manner, and related semantic labels and index data are cleared simultaneously. This successfully simulates the local occlusion situation in real-world scenarios, enriches the diversity of point cloud data, and improves the model's adaptability and robustness to local occlusion situations.

[0108] In some embodiments of this application, the point cloud enhancement operation includes a component exchange operation. The enhancement process for the point cloud data contained in each component, based on the mapping relationship between the point cloud data and semantic tag data in each component and a preset multi-dimensional point cloud enhancement strategy, to obtain enhanced point cloud data includes: determining the source component and target component to be exchanged based on the component exchange operation and performing collision detection and physical rationality detection; if the detection passes, based on the mapping relationship between the point cloud data and semantic tag data in each component, completely exchanging the point cloud data, corresponding semantic tag data, and index data contained in the source component with the point cloud data, corresponding semantic tag data, and index data contained in the target component, and updating the corresponding spatial transformation information.

[0109] Based on the requirement of component swapping operations, the source and target components to be swapped are determined among different instances of the same category. For example, if the point cloud data of multiple cars is being processed, the wheel of one car can be selected as the source component, and the wheel at the same position of another car can be selected as the target component.

[0110] After identifying the source and target components, collision detection and physical plausibility checks are performed. Collision detection aims to determine whether the two components will overlap or conflict spatially after being swapped; physical plausibility checks verify whether the swapped scenario conforms to actual physical laws, such as whether the overall structure of the swapped components is reasonable and whether it violates normal physical forms. Only when both checks pass will subsequent swapping operations continue.

[0111] If the detection passes, based on the mapping relationship between point cloud data and semantic label data in each component, the point cloud data, corresponding semantic label data, and index data contained in the source component are completely exchanged with the point cloud data, corresponding semantic label data, and index data contained in the target component. This means that not only has the location of the point cloud data changed, but its corresponding semantic label and index information are also exchanged synchronously, ensuring data consistency and relevance.

[0112] After data exchange is completed, the corresponding spatial transformation information is updated. Spatial transformation information includes coordinate translation, rotation, and scaling parameters. By updating these parameters, it is ensured that the swapped components maintain the same spatial posture as the current object, so that the overall scene after the swap remains spatially reasonable and natural.

[0113] By performing component swapping operations between different instances of the same category, and conducting collision detection and physical plausibility checks before the operation, the plausibility and realism of the scene after the swap are ensured. By completely swapping the point cloud data, semantic label data, and index data of the source and target components, and updating the spatial transformation information, the diversity of point cloud data is effectively enriched while maintaining semantic plausibility. This provides more training samples with different combinations and scenarios for multi-task models.

[0114] In some embodiments of this application, the point cloud enhancement operation includes a component mixing operation. The enhancement process, based on the mapping relationship between point cloud data and semantic tag data in each component and a preset multi-dimensional point cloud enhancement strategy, to obtain enhanced point cloud data includes: setting differentiated mixing parameters for each component according to its importance and corresponding semantic tag data; mixing point cloud data from other instances of the same category into the corresponding position of the current component according to the differentiated mixing parameters; and integrating the semantic tag data corresponding to the point cloud data of other instances of the same category into the current component according to the mapping relationship between point cloud data and semantic tag data in each component.

[0115] For component mixing operations, differentiated mixing parameters are set for each component based on its importance and corresponding semantic label data. A comprehensive analysis of each component of the target object is performed to assess its importance. For example, for human point cloud data, key parts such as the head and limb joints, which are extremely important for human recognition and pose determination, will have lower mixing ratio parameters; while for some relatively less important parts, such as some non-critical areas of the arms or legs, higher mixing ratio parameters are set. At the same time, point cloud density also affects the mixing parameter settings; areas with high point cloud density may have a lower mixing ratio to avoid over-mixing and data distortion.

[0116] Based on the set differential blending parameters, suitable point cloud data is selected from other instances of the same category and blended into the corresponding position of the current part. An intelligent sampling strategy is employed to intelligently select point cloud data from instances of the same category according to the blending ratio, ensuring that the blended data increases diversity without destroying the characteristics of the original data. For example, when processing multiple human point cloud data sets, the point cloud data of the arm portion of one human body is blended into the corresponding arm position of another human body according to the set blending ratio.

[0117] While mixing point cloud data, based on the mapping relationship between point cloud data and semantic label data in each component, an index mapping mechanism is used to integrate the semantic label data corresponding to point cloud data of other instances of the same category into the current component. This ensures that each newly added point cloud data has the correct semantic identifier, guaranteeing the consistency and accuracy of the data at the semantic level.

[0118] By setting differentiated mixing parameters based on component importance and semantic labels, an intelligent sampling strategy is adopted to mix point cloud data from other instances of the same category into the corresponding position of the current component, and semantic label data is integrated simultaneously, effectively increasing the diversity and complexity of point cloud data.

[0119] In some embodiments of this application, the point cloud enhancement operation includes a sparsity operation. The enhancement process for the point cloud data contained in each component, based on the mapping relationship between the point cloud data and semantic tag data in each component and a preset multi-dimensional point cloud enhancement strategy, to obtain the enhanced point cloud data includes: setting differentiated sampling parameters for each component according to the importance of each component and the corresponding semantic tag data; sampling the corresponding component according to the differentiated sampling parameters of each component; and synchronously updating the corresponding semantic tag data and index data according to the mapping relationship between the point cloud data and semantic tag data in each component.

[0120] For the sparsity operation, the sampling parameters are first tailored to each component based on its importance and corresponding semantic label data. During the analysis, the importance of different components of the target object is carefully evaluated. For example, for a vehicle target object, key functional parts such as the wheel area are set with a lower sampling rate to retain more point cloud details because they are crucial for the overall recognition and functional judgment of the vehicle; while for some non-critical parts, such as non-feature areas on the side of the vehicle body, a higher sampling rate is set to allow for a greater degree of sparsity processing. At the same time, semantic label data is also used as a reference factor to ensure that the sampling parameter settings are consistent with the semantic features of the components.

[0121] Based on the differentiated sampling parameters set for each component, point cloud sampling is performed on the corresponding component using either random sampling or farthest-point sampling strategies. Random sampling randomly selects a certain number of points from the component's point cloud data; farthest-point sampling first selects a starting point, and then sequentially selects the points furthest from the selected points until the set number of samples is reached. These two sampling strategies reduce the amount of point cloud data within the component, simulating point cloud sparsity that may occur at different distances and resolutions. For example, when simulating observing a vehicle at a distance, the vehicle's point cloud becomes sparse; this operation can generate point cloud data for similar scenarios.

[0122] While performing the sampling operation, the corresponding semantic label data and index data are updated synchronously based on the mapping relationship between point cloud data and semantic label data in each component. Since the sampling process changes the quantity and distribution of point cloud data, the original index and label information needs to be adjusted accordingly to ensure data consistency. For example, if some points are retained after sampling, their corresponding semantic labels must be accurately matched, and the index data must also be updated according to the new point cloud distribution to ensure the accuracy of subsequent data processing and analysis.

[0123] By setting differentiated sampling parameters based on component importance and semantic labels, and employing random sampling or farthest-point sampling strategies to sparsify the component point cloud, while simultaneously updating semantic labels and index data, the model successfully simulated point cloud sparsity at different distances and resolutions. This approach generates augmented data that enriches the diversity of point cloud data, enabling the model to access point cloud scenarios with varying degrees of sparsity. Models trained with this type of augmented data exhibit stronger adaptability and robustness when facing point cloud sparsity caused by factors such as distance and occlusion in real-world scenarios.

[0124] In some embodiments of this application, the point cloud enhancement operation includes a noise addition operation. The enhancement process for the point cloud data contained in each component based on the mapping relationship between the point cloud data and semantic tag data in each component and a preset multi-dimensional point cloud enhancement strategy to obtain enhanced point cloud data includes: generating noise points in the corresponding component using a preset noise model according to the distribution characteristics of the point cloud data contained in each component; and assigning corresponding semantic tag data to the generated noise points.

[0125] For noise addition, a comprehensive analysis of the point cloud data for each component is first performed to obtain its distribution characteristics, specifically covering spatial distribution patterns and reflection intensity distribution. Spatial distribution patterns focus on the aggregation and dispersion of point clouds in three-dimensional space; for example, the point clouds of some components may be concentrated in specific areas, while others may be more uniformly distributed. Reflection intensity distribution reflects the intensity of the point cloud's reflection of sensor signals; components of different materials or structures will have different reflection intensities. Through these analyses, a deeper understanding of the characteristics of the point cloud data for each component is gained, providing a basis for subsequent noise point generation.

[0126] Based on the point cloud data distribution characteristics of each component, noise points are generated in the corresponding component using a preset noise model. Multiple noise models can be provided to adapt to different simulation scenarios, such as Gaussian noise, uniformly distributed noise, and impulse noise. Gaussian noise causes noise points to follow a normal distribution in terms of spatial location and reflection intensity around the mean of the original point cloud data of the component; uniformly distributed noise causes noise points to be generated uniformly within a specific spatial range of the component, with a uniform distribution of reflection intensity; impulse noise randomly generates some noise points with abnormally high or low intensity. By selecting an appropriate noise model, noise points that conform to different interference scenarios can be generated.

[0127] Each newly generated noise point is assigned a corresponding semantic label. Based on the mapping relationship between point cloud data and semantic label data in each component, it is ensured that the noise points have correct semantic labels, guaranteeing data consistency and accuracy in multi-task learning, enabling the model to correctly identify and process point cloud data containing noise.

[0128] By analyzing the distribution characteristics of point cloud data for each component, various preset noise models were used to generate noise points that conform to different interference scenarios. These noise points were then assigned correct semantic labels, successfully simulating the impact of sensor noise and environmental interference on point cloud data. This enriched the diversity of point cloud data and enabled the model to encounter more complex scenarios containing noise. Models trained with this type of augmented data exhibit stronger adaptability and anti-interference capabilities when facing sensor noise and environmental interference in practical applications, effectively improving the accuracy and stability of the model in tasks such as target detection and segmentation.

[0129] In some embodiments of this application, the point cloud enhancement operation includes a coordinate perturbation operation. The enhancement process for the point cloud data contained in each component, based on the mapping relationship between the point cloud data and semantic tag data in each component and a preset multi-dimensional point cloud enhancement strategy, to obtain enhanced point cloud data includes: setting perturbation parameters for the point cloud data contained in each component according to the geometric features of the point cloud data contained in each component; and performing coordinate perturbation on the point cloud data contained in the corresponding component according to the perturbation parameters of the point cloud data contained in each component.

[0130] For coordinate jitter operations, the geometric features of the point cloud data contained in each component are analyzed, and appropriate perturbation parameters are set for the point cloud data of each component based on this analysis. During the analysis, the focus is on the local geometric features of the point cloud, such as determining the flatness of a region by calculating the curvature of the point cloud surface. For flat regions, due to their relatively simple geometry, they have a higher tolerance for perturbation, so larger perturbation parameters are set; while for edge and corner regions, these areas contain key geometric features in the point cloud data and are crucial for recognizing the shape and structure of objects, smaller perturbation parameters are set to avoid excessive perturbation that could damage these key features.

[0131] Based on the perturbation parameters set for each component, the coordinates of the point cloud data contained in the corresponding component are perturbed. The perturbation amount and direction are adaptively adjusted according to the local geometric features of the point cloud. In specific implementation, for each point cloud point, a perturbation amount is randomly generated from the set perturbation parameter range according to the geometric features of its region, and a random perturbation direction is determined. Then, the perturbation amount is applied to the coordinates of the point cloud point according to the perturbation direction, thereby achieving a small random perturbation of the point cloud coordinates.

[0132] During the coordinate perturbation operation, the point indexes and semantic labels are kept strictly unchanged; only the coordinates of the perturbed points are adjusted. The point indexes identify the position and order of each point in the original data, while the semantic labels specify the component category and semantic information to which each point belongs. Ensuring this information remains unchanged guarantees that the correlation and consistency of the data are not affected during subsequent data processing and model training, allowing the model to accurately identify and process the point cloud data after coordinate perturbation.

[0133] By analyzing the geometric features of the point cloud data for each component, perturbation parameters are adaptively set, and the point cloud coordinates are subjected to minor random perturbations while maintaining the point indices and labels. This processing method generates augmented data that simulates measurement errors and minute object movements that may occur in real-world scenarios, enriching the diversity of point cloud data. Models trained with this type of augmented data are better able to adapt to and handle various complex scenarios with subtle variations in real-world applications, effectively enhancing the model's robustness.

[0134] In some embodiments of this application, the step of enhancing the point cloud data contained in each component based on the mapping relationship between the point cloud data and semantic tag data in each component and the preset multi-dimensional point cloud enhancement strategy to obtain point cloud enhanced data includes: obtaining distance information between the target object and the sensor; and dynamically adjusting the currently adopted point cloud enhancement operation and the corresponding operation parameters according to the distance information between the target object and the sensor and the category of the target object.

[0135] The various combinations of enhancement operations and the intensity of enhancement in the aforementioned embodiments can be dynamically adjusted based on feature information such as the distance between the target object and the sensor, and the category of the target object. The distance between the target object and the sensor can be obtained through a sensor (such as a LiDAR or depth camera). The sensor emits and receives reflected signals, and the straight-line distance between the target object and the sensor is accurately calculated based on parameters such as signal propagation time and angle, providing crucial reference for subsequent point cloud enhancement operations. The category of the target object can be directly obtained based on the semantic category information of the originally labeled instances.

[0136] Based on the distance information between the target object and the sensor, as well as the category of the target object, the current point cloud enhancement operation and its corresponding parameters are dynamically adjusted. For example, a distance sensitivity mechanism can be set for each point cloud enhancement operation. For instance, when the target object is far from the sensor, to avoid information distortion caused by over-enhancement, a more conservative enhancement strategy can be adopted, such as reducing the mixing ratio in the component mixing operation, reducing the sampling rate in the sparsity operation, reducing the noise intensity in the noise addition operation, and reducing the perturbation amount in the coordinate perturbation operation. When the target object is close to the sensor, due to its rich point cloud information, a larger enhancement can be applied, such as increasing the component mixing ratio, increasing the sparsity sampling rate, increasing the noise intensity, and increasing the perturbation amount.

[0137] By acquiring distance information between the target object and the sensor, and combining this with information such as the target object category, the point cloud augmentation operation and its parameters are dynamically adjusted. This achieves intelligent and refined processing of point cloud data augmentation, generating augmented data that better matches the actual scene based on target objects of different distances and categories, avoiding over- or under-augmentation issues. Models trained with this type of augmented data can better adapt to and handle various complex situations when facing target objects of different distances and categories in practical applications, effectively improving the model's accuracy, robustness, and generalization ability in tasks such as target detection, segmentation, and recognition.

[0138] This application also provides a method for training a multi-task model, such as... Figure 3 The diagram shows a flowchart of a training method for a multi-task model according to an embodiment of this application. The training of the multi-task model includes the following steps S310 to S320:

[0139] Step S310: Obtain training sample data for the multi-task model, wherein the training sample data includes point cloud augmentation data obtained by the data augmentation method of the multi-task model described above.

[0140] Step S320: Train a multi-task model using the training sample data. The multi-task model includes object detection and segmentation task models.

[0141] The point cloud augmentation data obtained from the aforementioned embodiments can be used for training multi-task models for object detection and segmentation.

[0142] Based on the aforementioned data augmentation method for the multi-task model, the original point cloud data is processed to obtain augmented point cloud data. Specifically, various point cloud augmentation operations are applied comprehensively to different parts of the target object to obtain rich and diverse augmented point cloud data, which is used as part of the training sample data for the multi-task model. At the same time, some original point cloud data can be appropriately retained to form a complete training sample dataset.

[0143] The acquired training sample data is input into a multi-task model, which includes models for object detection and segmentation. During training, the model simultaneously learns relevant features and patterns for both object detection and segmentation tasks. For object detection, the model learns how to accurately identify the location and category of objects from point cloud data; for segmentation, the model learns to accurately classify each point in the point cloud data into its corresponding semantic category. By iteratively optimizing the model's parameters, the loss function value on the training sample data is gradually reduced, thereby improving the model's performance in object detection and segmentation tasks. Appropriate optimization algorithms (such as stochastic gradient descent) and loss functions (such as cross-entropy loss) are used during training to guide the model's training direction and evaluate its performance.

[0144] By first generating rich and diverse point cloud augmentation data using multi-dimensional point cloud augmentation strategies, and then combining this data with the original data to form training sample data, a multi-task model encompassing object detection and segmentation tasks is trained using this data. This approach makes the training sample data more diverse and representative, effectively covering various point cloud conditions that may occur in real-world scenarios, including different sparsity levels, noise levels, and geometric feature variations. The multi-task model trained with these samples can more accurately detect the location and category of target objects when faced with complex and varied point cloud data in practical applications, and can more precisely segment the point cloud data into different semantic categories, significantly improving the model's accuracy, robustness, and generalization ability in object detection and segmentation tasks.

[0145] This application also provides a data enhancement device 400 for a multi-task model, such as... Figure 4 The diagram provided illustrates the structure of a data augmentation device for a multi-task model according to an embodiment of this application. The data augmentation device 400 for the multi-task model includes:

[0146] The first acquisition unit 410 is used to acquire raw point cloud data for training a multi-task model. The raw point cloud data includes the 3D bounding box of the target object and the corresponding semantic label data.

[0147] The partitioning unit 420 is used to divide the 3D bounding box of the target object into multiple parts based on the original point cloud data using a differentiated partitioning strategy, so as to obtain the 3D bounding boxes of multiple parts corresponding to the target object and the point cloud data contained in each part.

[0148] The construction unit 430 is used to construct a mapping relationship between point cloud data and semantic label data in each component based on the 3D bounding boxes of multiple components corresponding to the target object and the point cloud data contained in each component.

[0149] The point cloud enhancement unit 440 is used to enhance the point cloud data contained in each component based on the mapping relationship between the point cloud data and semantic tag data in each component and the preset multi-dimensional point cloud enhancement strategy, so as to obtain point cloud enhanced data.

[0150] In some embodiments of this application, the partitioning unit 420 is specifically used for: determining the category and geometric features of the target object based on the original point cloud data; determining a differentiated partitioning strategy corresponding to the target object based on the category and geometric features of the target object; dividing the 3D bounding box of the target object into multiple parts using the differentiated partitioning strategy corresponding to the target object, thereby obtaining 3D bounding boxes of multiple parts corresponding to the target object; calculating the spatial relationship between each point in the 3D bounding box of the target object and the 3D bounding boxes of each part; determining the part to which each point belongs based on the spatial relationship between each point in the 3D bounding box of the target object and the 3D bounding boxes of each part, thereby obtaining the point cloud data contained in each part.

[0151] In some embodiments of this application, the construction unit 430 is specifically used to: construct a component point cloud set based on the point cloud data contained in each component, wherein the component point cloud set is used to store the coordinate data of all points contained in each component; construct an index set and a semantic tag set based on the original point cloud data, wherein the index set is used to store the original index position of each point and the semantic tag set is used to store the semantic tag of each point; and construct a mapping relationship between the point cloud data and semantic tag data in each component based on the component point cloud set, the index set, and the semantic tag set.

[0152] In some embodiments of this application, the point cloud enhancement unit 440 is specifically used to: obtain and parse the current configuration parameter information of the point cloud enhancement strategy based on a preset multi-dimensional point cloud enhancement strategy, and obtain the currently adopted point cloud enhancement operation and the corresponding operation parameters. The point cloud enhancement operation includes at least one of component discarding operation, component swapping operation, component mixing operation, sparsification operation, noise addition operation, and coordinate perturbation operation; and enhance the point cloud data contained in each component using the currently adopted point cloud enhancement operation and the corresponding operation parameters to obtain the point cloud enhancement data.

[0153] In some embodiments of this application, the point cloud enhancement operation includes a component discarding operation. The point cloud enhancement unit 440 is specifically used to: set differentiated discarding parameters for each component based on the importance of each component and the corresponding semantic tag data; discard the point cloud data contained in the corresponding component according to the differentiated discarding parameters for each component; and synchronously clear the semantic tag data and index data of the corresponding component according to the mapping relationship between the point cloud data and semantic tag data in each component.

[0154] In some embodiments of this application, the point cloud enhancement operation includes a component exchange operation. The point cloud enhancement unit 440 is specifically used to: determine the source component and target component to be exchanged based on the component exchange operation and perform collision detection and physical rationality detection; if the detection passes, based on the mapping relationship between the point cloud data and semantic tag data in each component, completely exchange the point cloud data, corresponding semantic tag data, and index data contained in the source component with the point cloud data, corresponding semantic tag data, and index data contained in the target component, and update the corresponding spatial transformation information.

[0155] In some embodiments of this application, the point cloud enhancement operation includes a component mixing operation. The point cloud enhancement unit 440 is specifically used to: set differentiated mixing parameters for each component based on the importance of each component and the corresponding semantic tag data; mix point cloud data of other instances of the same category into the corresponding position of the current component according to the differentiated mixing parameters; and integrate the semantic tag data corresponding to the point cloud data of other instances of the same category into the current component according to the mapping relationship between the point cloud data and semantic tag data in each component.

[0156] In some embodiments of this application, the point cloud enhancement operation includes a sparsification operation, and the point cloud enhancement unit 440 is specifically used to: set differentiated sampling parameters for each component according to the importance of each component and the corresponding semantic label data; sample the corresponding component according to the differentiated sampling parameters of each component; and synchronously update the corresponding semantic label data and index data according to the mapping relationship between the point cloud data and semantic label data in each component.

[0157] In some embodiments of this application, the point cloud enhancement operation includes a noise addition operation, and the point cloud enhancement unit 440 is specifically used to: generate noise points in the corresponding component using a preset noise model based on the distribution characteristics of the point cloud data contained in each component; and assign corresponding semantic label data to the generated noise points.

[0158] In some embodiments of this application, the point cloud enhancement operation includes a coordinate perturbation operation, and the point cloud enhancement unit 440 is specifically used to: set perturbation parameters for the point cloud data contained in each component according to the geometric features of the point cloud data contained in each component; and perform coordinate perturbation on the point cloud data contained in the corresponding component according to the perturbation parameters of the point cloud data contained in each component.

[0159] In some embodiments of this application, the point cloud enhancement unit 440 is specifically used to: acquire distance information between the target object and the sensor; and dynamically adjust the currently performed point cloud enhancement operation and corresponding operation parameters based on the distance information between the target object and the sensor and the category of the target object.

[0160] It is understood that the data augmentation device for the multi-task model described above can implement each step of the data augmentation method for the multi-task model provided in the foregoing embodiments. The relevant explanations of the data augmentation method for the multi-task model are applicable to the data augmentation device for the multi-task model, and will not be repeated here.

[0161] This application embodiment also provides a training device 500 for a multi-task model, such as... Figure 5 The diagram shows a structural schematic of a training device for a multi-task model according to an embodiment of this application. The training device 500 for the multi-task model includes:

[0162] The second acquisition unit 510 is used to acquire training sample data of the multi-task model, wherein the training sample data includes point cloud augmentation data obtained by the data augmentation method of the multi-task model described above.

[0163] Training unit 520 is used to train a multi-task model using the training sample data, the multi-task model including object detection and segmentation task models.

[0164] It is understood that the training device for the multi-task model described above can implement each step of the training method for the multi-task model provided in the foregoing embodiments. The relevant explanations of the training method for the multi-task model are applicable to the training device for the multi-task model, and will not be repeated here.

[0165] Figure 6 This is a schematic diagram of the structure of a device according to an embodiment of this application. For example... Figure 6 As shown, the device includes one or more processors (or processing units), and may also include one or more memories coupled to the processors, and may also include a communication module coupled to the processors.

[0166] A communication module can be used to communicate with other devices or apparatuses, such as sending or receiving data and / or signals. A communication module may have at least one communication module for communication. A communication module may include any interface necessary for communicating with other devices. Exemplarily, a communication module may be a transceiver, circuit, bus, module, or other type of communication module.

[0167] The processor may include, but is not limited to, one or more of the following: a general-purpose computer, a special-purpose computer, a microcontroller, a digital signal processor (DSP), or a controller-based multi-core controller architecture. The device may have multiple processors, such as application-specific integrated circuit (ASIC) chips, which are time-dependent on a clock synchronized with the main processor.

[0168] The memory may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, at least one of the following: read-only memory (ROM), electrically programmable read-only memory (EPROM), flash memory, hard disk, compact disc (CD), digital video disc (DVD), or other magnetic and / or optical storage. Examples of volatile memories include, but are not limited to, at least one of the following: random access memory (RAM), or other volatile memories that do not persist during the duration of a power outage.

[0169] A computer program consists of computer-executable instructions that are executed by an associated processor. Programs can be stored in ROM. A processor can perform any appropriate action and processing by loading the program into RAM.

[0170] Possible implementations of this application can be achieved through a program, enabling the communication device to execute any of the processes discussed in the foregoing embodiments. Possible implementations of this application can also be achieved through hardware or a combination of software and hardware.

[0171] In some implementations, the program may be tangibly contained in a computer-readable storage medium, which may include in a device (such as in memory) or other storage device accessible by the device. The program may be loaded from the computer-readable storage medium into RAM for execution. The computer-readable storage medium may include any type of tangible non-volatile memory, such as ROM, EPROM, flash memory, hard disk, CD, DVD, etc.

[0172] This application also provides a computer-readable storage medium storing computer instructions or program code thereon, which, when executed by a processor, causes the processor to perform the methods and functions involved in any of the above embodiments. A computer-readable medium can be any tangible medium that contains or stores a program for or relating to an instruction execution system, apparatus, or device. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. More detailed examples of computer-readable storage media include electrical connections with one or more wires, magnetic media (e.g., disks, floppy disks, hard disks, magnetic tapes, magnetic storage devices), optical media (e.g., optical storage devices, DVDs), semiconductor media (e.g., solid-state drives), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), or any suitable combination thereof.

[0173] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. Embodiments of this application also provide at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. This computer program product includes one or more computer-executable instructions, such as instructions included in a program module, which execute in a device on a target real or virtual processor to perform the processes, methods, and functions involved in any of the above embodiments. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0174] This application also proposes a computer program product, including a computer program or instructions that, when run on a computer, cause the computer to perform the processes, methods, and functions described in the above embodiments. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided as needed. The machine-executable instructions for the program modules can be executed locally or in a distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0175] Generally, the various embodiments of this application can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software, which can be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of this disclosure are shown and described as block diagrams, flowcharts, or represented using some other illustration, it should be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0176] It should be noted that although embodiments of this application have been described above with reference to the accompanying drawings, these embodiments are not independent of each other, and they can be combined to obtain other embodiments. The methods, situations, categories, and classifications of embodiments in this application are only for the convenience of description and should not constitute a special limitation. Various methods, categories, situations, and features in embodiments can be combined with each other if logically consistent. The various embodiments of this application can be arbitrarily combined to achieve different technical effects. The embodiments of this application will not list various combinations.

[0177] Furthermore, although the operation of the methods of this disclosure is described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps. It should also be noted that the features and functions of two or more devices according to this disclosure may be embodied in one device. Conversely, the features and functions of one device described above may be further divided and embodied by multiple devices.

[0178] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0179] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data augmentation method for a multi-task model, characterized in that, The data augmentation method for the multi-task model includes: Obtain raw point cloud data for training a multi-task model, wherein the raw point cloud data includes the 3D bounding box of the target object and the corresponding semantic label data; Based on the original point cloud data, the 3D bounding box of the target object is divided into multiple parts using a differentiated partitioning strategy, thereby obtaining the 3D bounding boxes of multiple parts corresponding to the target object and the point cloud data contained in each part. Based on the 3D bounding boxes of multiple components corresponding to the target object and the point cloud data contained in each component, a mapping relationship between the point cloud data and semantic label data in each component is constructed. Based on the mapping relationship between point cloud data and semantic tag data in each component and the preset multi-dimensional point cloud enhancement strategy, the point cloud data contained in each component is enhanced to obtain point cloud enhanced data.

2. The data augmentation method for the multi-task model according to claim 1, characterized in that, The step of dividing the 3D bounding box of the target object into multiple parts using a differentiated partitioning strategy based on the original point cloud data, and obtaining the multiple parts corresponding to the target object and the point cloud data contained in each part, includes: The category and geometric features of the target object are determined based on the original point cloud data; Based on the category and geometric features of the target object, determine the differentiated segmentation strategy corresponding to the target object; The 3D bounding box of the target object is divided into multiple parts using a differentiated partitioning strategy corresponding to the target object, thereby obtaining the 3D bounding boxes of the multiple parts corresponding to the target object. Calculate the spatial relationship between each point in the 3D bounding box of the target object and the 3D bounding boxes of each component; Based on the spatial relationship between each point in the 3D bounding box of the target object and the 3D bounding boxes of each component, the component to which each point belongs is determined, and the point cloud data contained in each component is obtained.

3. The data augmentation method for the multi-task model according to claim 1, characterized in that, The step of constructing the mapping relationship between the point cloud data and semantic label data in each component based on the 3D bounding boxes of multiple components corresponding to the target object and the point cloud data contained in each component includes: A component point cloud set is constructed based on the point cloud data contained in each component, and the component point cloud set is used to store the coordinate data of all points contained in each component; An index set and a semantic tag set are constructed based on the original point cloud data. The index set is used to store the original index position of each point, and the semantic tag set is used to store the semantic tag of each point. Based on the component point cloud set, the index set, and the semantic tag set, a mapping relationship between the point cloud data and semantic tag data in each component is constructed.

4. The data augmentation method for the multi-task model according to claim 1, characterized in that, Based on the mapping relationship between point cloud data and semantic tag data in each component and a preset multi-dimensional point cloud enhancement strategy, the point cloud data contained in each component is enhanced to obtain enhanced point cloud data, including: Based on a preset multi-dimensional point cloud enhancement strategy, the current configuration parameter information of the point cloud enhancement strategy is obtained and parsed to obtain the current point cloud enhancement operation and the corresponding operation parameters. The point cloud enhancement operation includes at least one of the following: component discarding operation, component swapping operation, component mixing operation, sparsification operation, noise addition operation, and coordinate perturbation operation. The point cloud data contained in each component is enhanced by using the currently adopted point cloud enhancement operation and the corresponding operation parameters to obtain the point cloud enhancement data.

5. The data augmentation method for the multi-task model according to claim 4, characterized in that, The point cloud enhancement operation includes a component discarding operation. Based on the mapping relationship between point cloud data and semantic tag data in each component and a preset multi-dimensional point cloud enhancement strategy, the point cloud data contained in each component is enhanced to obtain enhanced point cloud data, including: Set differentiated discard parameters for each component based on its importance and corresponding semantic tag data; The point cloud data contained in the corresponding component is discarded according to the differentiated discard parameters of each component, and the semantic label data and index data of the corresponding component are synchronously cleared according to the mapping relationship between the point cloud data and semantic label data in each component.

6. The data augmentation method for the multi-task model according to claim 4, characterized in that, The point cloud enhancement operation includes a component swapping operation. Based on the mapping relationship between point cloud data and semantic tag data in each component and a preset multi-dimensional point cloud enhancement strategy, the point cloud data contained in each component is enhanced to obtain enhanced point cloud data, including: Based on the component exchange operation, the source component and target component to be exchanged are determined, and collision detection and physical rationality detection are performed. If the detection passes, based on the mapping relationship between the point cloud data and semantic label data in each component, the point cloud data, corresponding semantic label data, and index data contained in the source component are completely exchanged with the point cloud data, corresponding semantic label data, and index data contained in the target component, and the corresponding spatial transformation information is updated.

7. The data augmentation method for the multi-task model according to claim 4, characterized in that, The point cloud enhancement operation includes a component hybrid operation. Based on the mapping relationship between point cloud data and semantic tag data in each component and a preset multi-dimensional point cloud enhancement strategy, the point cloud data contained in each component is enhanced to obtain point cloud enhanced data, including: Set differentiated mixing parameters for each component based on its importance and corresponding semantic tag data; Based on the differentiated mixing parameters, point cloud data of other instances of the same category are mixed into the corresponding position of the current component, and based on the mapping relationship between point cloud data and semantic tag data in each component, semantic tag data corresponding to point cloud data of other instances of the same category are integrated into the current component.

8. The data augmentation method for the multi-task model according to claim 1, characterized in that, The point cloud enhancement operation includes a noise addition operation. Based on the mapping relationship between the point cloud data and semantic label data in each component and a preset multi-dimensional point cloud enhancement strategy, the point cloud data contained in each component is enhanced to obtain the enhanced point cloud data, including: Based on the distribution characteristics of the point cloud data contained in each component, noise points are generated in the corresponding component using a preset noise model; The generated noise points are assigned corresponding semantic label data.

9. The data augmentation method for the multi-task model according to claim 4, characterized in that, Based on the mapping relationship between point cloud data and semantic tag data in each component and a preset multi-dimensional point cloud enhancement strategy, the point cloud data contained in each component is enhanced to obtain enhanced point cloud data, including: Obtain distance information between the target object and the sensor; The point cloud enhancement operation and corresponding operation parameters are dynamically adjusted based on the distance information between the target object and the sensor, as well as the category of the target object.

10. A training method for a multi-task model, characterized in that, The training of the multi-task model includes: Acquire training sample data for a multi-task model, wherein the training sample data includes point cloud augmented data obtained by the data augmentation method for the multi-task model according to any one of claims 1 to 9; A multi-task model is trained using the training sample data, the multi-task model including object detection and segmentation task models.

11. A data augmentation device for a multi-task model, characterized in that, The data augmentation device for the multi-task model includes: The first acquisition unit is used to acquire raw point cloud data for training a multi-task model. The raw point cloud data includes the 3D bounding box of the target object and the corresponding semantic label data. The partitioning unit is used to divide the 3D bounding box of the target object into multiple parts based on the original point cloud data using a differentiated partitioning strategy, thereby obtaining the 3D bounding boxes of the multiple parts corresponding to the target object and the point cloud data contained in each part. The construction unit is used to construct a mapping relationship between point cloud data and semantic label data in each component based on the 3D bounding boxes of multiple components corresponding to the target object and the point cloud data contained in each component. The point cloud enhancement unit is used to enhance the point cloud data contained in each component based on the mapping relationship between the point cloud data and semantic tag data in each component and the preset multi-dimensional point cloud enhancement strategy, so as to obtain point cloud enhanced data.

12. A training device for a multi-task model, characterized in that, The training device for the multi-task model includes: The second acquisition unit is used to acquire training sample data of the multi-task model, wherein the training sample data includes point cloud augmentation data obtained by the data augmentation method of the multi-task model according to any one of claims 1 to 9; A training unit is used to train a multi-task model using the training sample data, the multi-task model including object detection and segmentation task models.

13. An apparatus comprising: processor; And a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the data augmentation method of any one of the multitasking models of claims 1 to 9, and the training method of the multitasking model of claim 10.

14. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the data augmentation method of any one of the multi-task models of claims 1 to 9, and the training method of the multi-task model of claim 10.

Images