Sample amplification method, device, equipment, medium and program product
By generating a travelable area in autonomous driving and filtering out insertable detection target data, changing the background to generate amplified samples, the problem of low generalization of the target detection model in the prior art is solved, and the generalization and accuracy of the model are improved.
Patent Information
- Application Number
- CN202510372142.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-26
- Publication Date
- 2025-07-18
AI Technical Summary
In the prior art, sample amplification methods through geometric transformation or color transformation lead to low generalization of the target detection model.
By acquiring the reference image in the reference sample, the data to be inserted mapped in the travelable area are filtered out from the detection target data, and amplified point clouds and amplified images are generated based on the data to be inserted and the reference sample, changing the background of the detection target.
The generalization and accuracy of the object detection model are improved, making the amplified samples more in line with the actual scenarios, and the generalization ability of the model is enhanced.
Smart Images
Figure CN120339744A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent driving technology, and particularly relates to a method, device, equipment, medium and program product for sample amplification. Background Art
[0002] With the continuous development of technology, vehicles can also achieve autonomous driving. A lidar and a camera are installed in the vehicle, and the acquired point cloud and image are input into a target detection model to obtain the position and category of the target, and then the vehicle driving is controlled. In order to train a target detection model, a large number of samples are required, and the number of samples of some targets is small, so sample amplification is needed.
[0003] In the prior art, for the method of sample amplification, usually, geometric transformations or color transformations such as rotation, flipping, scaling, cropping, translation, brightness adjustment, contrast adjustment, and color jitter are performed on the images in a sample, and geometric transformations such as rotation and translation are performed on the point cloud in the sample to form new samples and complete sample expansion.
[0004] However, by means of geometric transformation or color transformation, the background in the point cloud and image will not be changed, resulting in low generalization of the target detection model trained using the amplified samples. Summary of the Invention
[0005] The sample amplification method, device, equipment, medium and program product provided in the embodiments of the present application are used to solve the problem that the generalization of the target detection model trained using the amplified samples is low due to sample expansion by means of geometric transformation or color transformation in the prior art.
[0006] In a first aspect, an embodiment of the present application provides a sample amplification method, including:
[0007] Obtain a reference sample and at least one detection target data, where the reference sample includes a reference point cloud and a reference image corresponding to each shooting position, and each detection target data includes a detection target point cloud and a detection target image corresponding to each shooting position;
[0008] Generate a drivable area according to each reference image;
[0009] Select insertion data from the at least one detection target data, where the detection target point clouds are all mapped in the drivable area;
[0010] Generate an amplified point cloud and an amplified image corresponding to each shooting position according to each insertion data and the reference sample;
[0011] Use the amplified point cloud and each amplified image as amplified samples.
[0012] In a possible implementation, generating the augmented point cloud and the augmented image corresponding to each of the shooting positions according to each of the data to be inserted and the reference sample includes:
[0013] Inserting the detection target point cloud in each of the data to be inserted into the reference point cloud to obtain the augmented point cloud;
[0014] Performing the following processing on the reference image corresponding to each of the shooting positions:
[0015] For each of the data to be inserted, taking the mapped position area of the detection target point cloud in the data to be inserted in the reference image as the position to be covered of the data to be inserted;
[0016] For each of the data to be inserted, covering the detection target image corresponding to the shooting position in the data to be inserted to the position to be covered of the data to be inserted to obtain the augmented image corresponding to the shooting position.
[0017] In a possible implementation, generating the drivable area according to each of the reference images includes:
[0018] Stitching each of the reference images to obtain a bird's-eye view;
[0019] Performing drivable area detection on the bird's-eye view to obtain the drivable area.
[0020] In a possible implementation, before using the augmented point cloud and each of the augmented images as augmented samples, the method further includes:
[0021] Determining the detection target point cloud observation area in the augmented point cloud according to the vehicle position and the detection target point cloud of each of the data to be inserted;
[0022] Removing the reference point cloud in the detection target point cloud observation area to obtain an updated augmented point cloud;
[0023] Using the augmented point cloud and each of the augmented images as augmented samples includes:
[0024] Using the updated augmented point cloud and each of the augmented images as augmented samples.
[0025] In a possible implementation, before obtaining the reference sample and at least one detection target data, the method further includes:
[0026] Obtaining a sample to be segmented and annotation data, where the sample to be segmented includes a point cloud to be segmented and a segmented image corresponding to each of the shooting positions, and the annotation data includes three-dimensional detection box data;
[0027] For each image to be segmented corresponding to the shooting position, according to the 3D detection box data and the contour detection model, the detection target image corresponding to the shooting position is segmented from the image to be segmented;
[0028] The point cloud in the point cloud to be segmented that belongs to the 3D detection box corresponding to the 3D detection box data is used as the detection target point cloud;
[0029] According to the detection target point cloud and the detection target image corresponding to each shooting position, the detection target data is generated.
[0030] In a possible implementation manner, the segmenting the detection target image corresponding to the shooting position from the image to be segmented according to the 3D detection box data and the contour detection model includes:
[0031] Taking the area mapped by the 3D detection box corresponding to the 3D detection box data in the image to be segmented as the target area;
[0032] Segmenting the image of the target area in the image to be segmented to obtain an image of the target with background;
[0033] Inputting the image of the target with background into the contour detection model to obtain a target contour area;
[0034] Segmenting the image of the target contour area in the image of the target with background to obtain the detection target image corresponding to the shooting position.
[0035] In a second aspect, an embodiment of the present application provides a sample amplification device, including:
[0036] An acquisition module, configured to acquire a reference sample and at least one detection target data, where the reference sample includes a reference point cloud and a reference image corresponding to each shooting position, and each detection target data includes a detection target point cloud and a detection target image corresponding to each shooting position;
[0037] A processing module, configured to:
[0038] Generate a drivable area according to each reference image;
[0039] Screen out the data to be inserted in which the detection target point clouds are all mapped in the drivable area from the at least one detection target data;
[0040] Generate an amplified point cloud and an amplified image corresponding to each shooting position according to each data to be inserted and the reference sample;
[0041] An amplification module, configured to use the amplified point cloud and each amplified image as an amplified sample.
[0042] In a third aspect, an embodiment of the present application provides an electronic device, including:
[0043] a processor, a memory, and a communication interface;
[0044] The memory is used to store executable instructions of the processor;
[0045] Wherein, the processor is configured to execute the sample amplification method according to any one of the first aspects by executing the executable instructions.
[0046] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the sample amplification method according to any one of the first aspects.
[0047] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it is used to implement the sample amplification method according to any one of the first aspects.
[0048] The sample amplification method, device, equipment, medium, and program product provided by the embodiments of the present application generate a drivable area from the reference image in the acquired reference sample, and then screen out the data to be inserted in which the detected target point cloud is mapped in the drivable area from the acquired detection target data. The detection target data also includes a detection target image. Furthermore, according to each data to be inserted and the reference sample in the reference sample, an amplified point cloud and an amplified image are generated, and the amplified point cloud and the amplified image are used as amplified samples. This solution screens out the data to be inserted in which the detected target point cloud is mapped in the drivable area, and then generates amplified samples according to the data to be inserted and the reference sample, so as to change the background of the detection target and effectively improve the generalization of the target detection model. Description of the Drawings
[0049] The drawings here are incorporated into the description and form a part of this description, showing embodiments consistent with the present application, and are used together with the description to explain the principles of the present application.
[0050] Figure 1a It is a schematic flowchart of Embodiment 1 of the sample amplification method provided by the present application;
[0051] Figure 1b It is a schematic diagram of a three-dimensional detection frame provided by the present application;
[0052] Figure 2a It is a schematic flowchart of Embodiment 2 of the sample amplification method provided by the present application;
[0053] Figure 2b It is a schematic diagram of a sub-observation area provided by the present application;
[0054] Figure 3a Schematic flow chart of the third embodiment of the sample amplification method provided by this application;
[0055] Figure 3b Schematic diagram of the process of obtaining the detection target image provided by this application;
[0056] Figure 4 Schematic structural diagram of the sample amplification device embodiment provided by this application;
[0057] Figure 5 Schematic structural diagram of an electronic device provided by this application.
[0058] Through the above-mentioned drawings, the specific embodiments of this application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of this application in any way, but to illustrate the concept of this application to those skilled in the art by referring to specific embodiments. Detailed implementation manners
[0059] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with this application. On the contrary, they are merely examples of devices and methods consistent with some aspects of this application as detailed in the appended claims.
[0060] The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and above-mentioned drawings of this application are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.
[0061] During the process of autonomous driving of a vehicle, the vehicle will collect point clouds through the lidar in the vehicle and collect images through the camera, and then input the point clouds and images into the target detection model to obtain the position and category of the target, and then the vehicle driving can be controlled according to the position and category of the target. In order to train the target detection model, a large number of samples are required, and the number of samples of some targets is small, so sample amplification is needed.
[0062] In the prior art, for the method of sample amplification, usually, geometric transformations or color transformations such as rotation, flipping, scaling, cropping, translation, brightness adjustment, contrast adjustment, and color jitter are performed on the images in a sample, and geometric transformations such as rotation and translation are performed on the point clouds in the sample to form new samples, thus completing sample expansion. However, this method does not change the background in the point clouds and images, which will lead to the problem of low generalization of the target detection model trained using the amplified samples.
[0063] In view of the problems existing in the prior art, during the research on the sample amplification method, the inventors found that in order to change the background of the targets in the samples to improve the generalization of the target detection model, the targets can be taken out from the point clouds and images and then put into new point clouds and images. Further, in order to determine the rationality of the placement positions, the point clouds and images of the targets that can be placed in the drivable area are selected. Based on the above inventive concept, the sample amplification scheme in this application is designed.
[0064] The execution subject of the sample amplification method in this application can be a computer, or a server, a terminal device, an in-vehicle terminal, etc. This application does not limit it, and the following will take a computer as an example for illustration.
[0065] The following is an example to illustrate the application scenario of the sample amplification method provided in this application.
[0066] Exemplarily, in this application scenario, samples of multiple detection targets are required for model training to obtain a target detection model. In order to improve the accuracy and generalization of the target detection model, for each detection target, a large number of samples are required. For detection target A, there are only a small number of samples, and sample amplification is needed.
[0067] The computer can obtain the samples and annotation data of detection target A, and this sample is used as the sample to be segmented. Then, the detection target point cloud is segmented from the point cloud to be segmented in the sample to be segmented, and the detection target images are respectively segmented from the images to be segmented corresponding to each shooting position in the sample to be segmented. The detection target images only include detection target A. According to the detection target point cloud and the detection target images corresponding to each shooting position, detection target data is generated.
[0068] When performing sample amplification, the computer needs to first obtain a reference sample and at least one piece of detection target data, and then generate a drivable area according to each reference image.
[0069] Then, the data to be inserted in which the detection target point clouds are all mapped in the drivable area is screened out from at least one piece of detection target data.
[0070] According to each data to be inserted and the reference sample, amplified point clouds and amplified images corresponding to each shooting position are generated; the amplified point clouds and each amplified image are used as amplified samples.
[0071] Subsequent computers can use the amplified sample and the original sample for model training to obtain a target detection model.
[0072] It should be noted that the above scenario is only an example of an application scenario provided by the embodiments of the present application. The embodiments of the present application do not limit the actual forms of various devices included in this scenario. In the specific application of the solution, it can be set according to actual needs.
[0073] Next, the technical solution of the present application will be described in detail through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.
[0074] Figure 1a FIG. is a schematic flow chart of the first embodiment of the sample amplification method provided by the present application. The embodiments of the present application illustrate the situation where a computer generates an amplified sample according to detection target data and a reference sample. The method in this embodiment can be implemented by software, hardware, or a combination of software and hardware. Such as Figure 1a shown, the sample amplification method specifically includes the following steps:
[0075] S101: Obtain a reference sample and at least one piece of detection target data.
[0076] In this step, in order to change the background of the detection target, it is necessary to obtain at least one piece of detection target data of the reference sample so as to insert the detection target into the reference sample.
[0077] The reference sample includes a reference point cloud and a reference image corresponding to each shooting position. Each piece of detection target data includes a detection target point cloud and a detection target image corresponding to each shooting position. The shooting position is the installation position of the camera on the vehicle.
[0078] It should be noted that the detection target data also includes annotation data, and the annotation data includes three-dimensional detection box data and target types. The three-dimensional detection box data is the data of a three-dimensional detection box, and the three-dimensional detection box is a cuboid. Exemplarily, Figure 1b FIG. is a schematic diagram of the three-dimensional detection box provided by the present application. Such as Figure 1b shown, the three-dimensional detection box data includes the coordinates of the center point a of the bottom surface of the three-dimensional detection box, the length L of the three-dimensional detection box, the width W of the three-dimensional detection box, the height H of the three-dimensional detection box, and the orientation angle of the three-dimensional detection box.
[0079] S102: Generate a drivable area according to each reference image.
[0080] In this step, after the computer obtains the reference samples, in order to generate more reasonable amplified samples, it is necessary to generate a drivable area based on each reference image for subsequent screening of the detection target data.
[0081] Specifically, splice each reference image to obtain a bird's-eye view; then perform drivable area detection on the bird's-eye view to obtain the drivable area.
[0082] S103: Screen out the data to be inserted from at least one detection target data where the detection target point cloud is mapped within the drivable area.
[0083] In this step, after the computer obtains the drivable area, since the vehicle can only drive within the drivable area and does not pay attention to the detection targets outside the drivable area, and some detection targets should not appear outside the drivable area in actual situations, such as lane markings, traffic lights, etc. If the detection targets outside the drivable area are put into the reference samples, it will cause the obtained amplified samples to not match the actual situation and reduce the accuracy of the target detection model. Therefore, it is necessary to screen out the data to be inserted from at least one detection target data where the detection target point cloud is mapped within the drivable area.
[0084] It should be noted that after the lidar and camera in the vehicle are installed, their installation positions in the vehicle are fixed. The coordinates of the point cloud are the coordinates in the vehicle coordinate system, and there is a mapping relationship between the image coordinate system and the vehicle coordinate system. Therefore, the point clouds obtained at different times can be mapped onto the images taken by the camera at different times. Since the bird's-eye view is composed of spliced reference images, the point cloud can also be mapped onto the bird's-eye view.
[0085] It should be noted that if the detection target point clouds in all the obtained detection target data are not mapped within the drivable area, then obtain the detection target data again.
[0086] S104: Generate amplified point clouds and amplified images corresponding to each shooting position according to each data to be inserted and the reference samples.
[0087] In this step, after the computer screens out the data to be inserted, it generates amplified point clouds and amplified images corresponding to each shooting position according to each data to be inserted and the reference samples.
[0088] Specifically, insert the detection target point cloud in each data to be inserted into the reference point cloud to obtain the amplified point cloud.
[0089] Perform the following processing on the reference image corresponding to each shooting position:
[0090] For each data to be inserted, take the mapping position area of the detection target point cloud in this data to be inserted in this reference image as the position to be covered by this data to be inserted;
[0091] For each data to be inserted, the detection target image corresponding to the shooting position in the data to be inserted is overlaid on the position to be overlaid of the data to be inserted, and an amplified image corresponding to the shooting position is obtained.
[0092] Since the detection target point clouds of the data to be inserted are all mapped in the drivable area, the detection target images in the amplified images are also in the drivable area.
[0093] S105: Use the amplified point cloud and each amplified image as amplified samples.
[0094] In this step, after the computer obtains the amplified point cloud and the amplified samples, the amplified point cloud and each amplified image are used as amplified samples.
[0095] It should be noted that since the detection target data also includes annotation data and the data to be inserted is selected from the detection target data, the annotation data of each data to be inserted can be added to the original annotation data of the benchmark sample as the annotation data of the amplified samples.
[0096] The sample amplification method provided in this embodiment generates a drivable area through the benchmark image in the obtained benchmark sample, and then selects the data to be inserted in which the detection target point clouds are all mapped in the drivable area from the obtained detection target data. The detection target data also includes detection target images. Furthermore, according to each data to be inserted and the benchmark sample in the benchmark sample, an amplified point cloud and an amplified image are generated, and the amplified point cloud and the amplified image are used as amplified samples. This solution generates amplified samples through the data to be inserted and the benchmark sample, changing the background of the detection target, which can effectively improve the generalization of the target detection model. In addition, by selecting the data to be inserted in which the detection target point clouds are all mapped in the drivable area, the amplified samples are more in line with the reality, which can improve the accuracy of the target detection model.
[0097] Figure 2a This is a schematic flowchart of the second embodiment of the sample amplification method provided by this application. On the basis of the above embodiment, this application embodiment describes the situation where the computer removes the point clouds in the amplified point cloud that occlude the detection target point cloud and the point clouds occluded by the detection target point cloud. As Figure 2a shown, the sample amplification method specifically includes the following steps:
[0098] S201: Determine the observation area of the detection target point cloud in the amplified point cloud according to the vehicle position and the detection target point cloud of each data to be inserted.
[0099] In this step, after the computer obtains the augmented point cloud, when observing the detection target point cloud starting from the vehicle position, there may be reference point clouds in front of and behind the detection target point cloud, that is, there may be point clouds that occlude the detection target point cloud and point clouds occluded by the detection target point cloud. These point clouds will affect the accuracy of the target detection model, so these point clouds need to be removed. First, according to the vehicle position and the detection target point cloud of each data to be inserted, the observation area of the detection target point cloud in the augmented point cloud is determined.
[0100] The observation area of the detection target point cloud is composed of sub-observation areas corresponding to each data to be inserted. The sub-observation area corresponding to a data to be inserted is a conical area with the vehicle position as the vertex. This conical area includes the detection target point cloud of this data to be inserted. This conical area can be the smallest conical area with the vehicle position as the vertex and including the detection target point cloud of this data to be inserted.
[0101] Exemplarily, Figure 2b is a schematic diagram of the sub-observation area provided by this application. As Figure 2b shown, the point O in the figure represents the vehicle position, the area within the cuboid is the area where the detection target point cloud is located, and the area within the quadrangular pyramid is the sub-observation area.
[0102] S202: Remove the reference point clouds in the observation area of the detection target point cloud to obtain the updated augmented point cloud.
[0103] In this step, after the computer determines the observation area of the detection target point cloud, the reference point clouds in the observation area of the detection target point cloud are the point clouds that occlude the detection target point cloud or the point clouds occluded by the detection target point cloud. These point clouds need to be removed to obtain the updated augmented point cloud.
[0104] It should be noted that since the detection target image covers the position to be covered of the reference image to obtain the augmented image, the detection target in the augmented image is not occluded.
[0105] S203: Use the updated augmented point cloud and each augmented image as augmented samples.
[0106] In this step, after the computer obtains the updated augmented point cloud, it uses the updated augmented point cloud and each augmented image as augmented samples.
[0107] The sample augmentation method provided in this embodiment can improve the quality of the augmented samples, reduce the influence of noise and background, and improve the accuracy of the target detection model by removing the point clouds that occlude the detection target point cloud and the point clouds occluded by the detection target point cloud in the augmented point cloud, and then using the updated augmented point cloud and each augmented image as augmented samples.
[0108] Figure 3a This is a schematic flowchart of the third embodiment of the sample amplification method provided by this application. On the basis of the above embodiments, this embodiment of the application explains the situation of computer-generated detection target data. As Figure 3a shown, the sample amplification method specifically includes the following steps:
[0109] S301: Obtain the sample to be segmented and annotation data.
[0110] In this step, in order to perform sample amplification subsequently, it is necessary to segment the detection target from the sample to be segmented, which requires obtaining the sample to be segmented and annotation data first.
[0111] The sample to be segmented includes the point cloud to be segmented and the images to be segmented corresponding to each shooting position, and the annotation data includes three-dimensional detection box data.
[0112] It should be noted that the annotation data also includes the target category.
[0113] S302: For the image to be segmented corresponding to each shooting position, according to the three-dimensional detection box data and the contour detection model, segment the detection target image corresponding to this shooting position from the image to be segmented.
[0114] In this step, after the computer obtains the sample to be segmented and the annotation data, it is necessary to extract the detection target from the image to be segmented, which requires, for the image to be segmented corresponding to each shooting position, segmenting the detection target image corresponding to this shooting position from the image to be segmented according to the three-dimensional detection box data and the contour detection model.
[0115] Specifically, the area mapped by the three-dimensional detection box corresponding to the three-dimensional detection box data in the image to be segmented is used as the target area.
[0116] It should be noted that the detection target is in the target area. If the annotation data includes target area data, the target area can also be determined according to the target area data.
[0117] Segment the image of the target area in the image to be segmented to obtain the target image with background.
[0118] Input the target image with background into the contour detection model to obtain the target contour area.
[0119] Segment the image of the target contour area in the target image with background to obtain the detection target image corresponding to this shooting position.
[0120] Exemplarily, Figure 3b This is a schematic diagram of the process of obtaining the detection target image provided by this application. As Figure 3bAs shown, the left image is the image to be segmented, and the area within the dashed box in the image to be segmented is the target area. The detection target in the image to be segmented is a vehicle. There is a straight lane marking in front of the vehicle, a road sign on the right, and a pothole behind. The target area contains not only the vehicle but also some straight lane markings, some road signs, and some potholes.
[0121] Segment the image of the target area in the image to be segmented to obtain the target image with background, which is the middle image in Figure 3b . The part of the target image with background other than the vehicle is the background part. Input the target image with background into the contour detection model to obtain the target contour area. Then, segment the image of the target contour area in the target image with background to obtain the detection target image corresponding to the shooting position, which is the right image in Figure 3b . The detection target image only includes the detection target, that is, the vehicle.
[0122] S303: Use the point cloud of the three-dimensional detection box corresponding to the three-dimensional detection box data in the point cloud to be segmented as the detection target point cloud.
[0123] In this step, after the computer obtains the point cloud to be segmented and the annotation data, it also needs to use the point cloud of the three-dimensional detection box corresponding to the three-dimensional detection box data in the point cloud to be segmented as the detection target point cloud.
[0124] It should be noted that the execution order of step S302 and step S303 can be: execute step S302 first, and then execute step S303; it can also be: execute step S303 first, and then execute step S302; it can also be: step S302 and step S303 are executed simultaneously. The embodiment of the present application does not limit the execution order of step S302 and step S303, which can be determined according to the actual situation.
[0125] S304: Generate detection target data based on the detection target point cloud and the detection target image corresponding to each shooting position.
[0126] In this step, after the computer obtains the detection target point cloud and the detection target image corresponding to each shooting position, it generates detection target data based on the detection target point cloud and the detection target image corresponding to each shooting position.
[0127] It should be noted that the detection target data also includes annotation data, and the annotation data includes three-dimensional detection box data and target types.
[0128] The sample amplification method provided in this embodiment can remove most of the background point clouds by extracting the point clouds within the three-dimensional detection frame, and extract the images within the target contour area, ensuring that the detected target images do not include background images. This results in less background information for the detected target data of the detection target, making it easier to fuse with the reference image and also improving the accuracy of the target detection model.
[0129] The following is an embodiment of the device of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the device embodiment of the present application, please refer to the method embodiment of the present application.
[0130] Figure 4 It is a schematic structural diagram of an embodiment of the sample amplification device provided by the present application; as Figure 4 shown, the sample amplification device 40 includes:
[0131] An acquisition module 41, configured to acquire a reference sample and at least one detected target data, where the reference sample includes reference point clouds and reference images corresponding to each shooting position, and each detected target data includes detected target point clouds and detected target images corresponding to each shooting position;
[0132] A processing module 42, configured to:
[0133] Generate a drivable area according to each reference image;
[0134] Screen out the data to be inserted whose detected target point clouds are all mapped in the drivable area from the at least one detected target data;
[0135] Generate amplified point clouds and amplified images corresponding to each shooting position according to each data to be inserted and the reference sample;
[0136] An amplification module 43, configured to use the amplified point clouds and each amplified image as amplified samples.
[0137] Further, the processing module 42 is specifically configured to:
[0138] Insert the detected target point clouds in each data to be inserted into the reference point clouds to obtain the amplified point clouds;
[0139] Perform the following processing on the reference image corresponding to each shooting position:
[0140] For each data to be inserted, use the mapping position area of the detected target point cloud in the data to be inserted in the reference image as the position to be covered by the data to be inserted;
[0141] For each piece of the data to be inserted, cover the detection target image corresponding to the shooting position in the data to be inserted to the position to be covered in the data to be inserted, so as to obtain the amplified image corresponding to the shooting position.
[0142] Furthermore, the processing module 42 is specifically further configured to:
[0143] Stitch each of the reference images to obtain an aerial view;
[0144] Perform drivable area detection on the aerial view to obtain the drivable area.
[0145] Furthermore, before using the amplified point cloud and each of the amplified images as amplified samples, the processing module 42 is further configured to:
[0146] Determine the detection target point cloud observation area in the amplified point cloud according to the vehicle position and the detection target point cloud of each piece of the data to be inserted;
[0147] Remove the reference point cloud in the detection target point cloud observation area to obtain an updated amplified point cloud;
[0148] The amplification module 43 is further configured to use the updated amplified point cloud and each of the amplified images as amplified samples.
[0149] Furthermore, before obtaining the reference sample and at least one piece of detection target data, the acquisition module 41 is further configured to acquire a sample to be segmented and annotation data, the sample to be segmented includes a point cloud to be segmented and a to-be-segmented image corresponding to each of the shooting positions, and the annotation data includes three-dimensional detection frame data;
[0150] The processing module 42 is further configured to:
[0151] For the to-be-segmented image corresponding to each of the shooting positions, segment the detection target image corresponding to the shooting position from the to-be-segmented image according to the three-dimensional detection frame data and the contour detection model;
[0152] Use the point cloud of the three-dimensional detection frame corresponding to the three-dimensional detection frame data in the to-be-segmented point cloud as the detection target point cloud;
[0153] Generate the detection target data according to the detection target point cloud and the detection target image corresponding to each of the shooting positions.
[0154] Furthermore, the processing module 42 is specifically further configured to:
[0155] Use the area mapped by the three-dimensional detection frame corresponding to the three-dimensional detection frame data in the to-be-segmented image as the target area;
[0156] Segment the image of the target area in the image to be segmented to obtain a target image with a background.
[0157] Input the target image with a background into the contour detection model to obtain a target contour area.
[0158] Segment the image of the target contour area in the target image with a background to obtain a detection target image corresponding to the shooting position.
[0159] The sample amplification device provided in this embodiment is used to execute the technical solutions in any of the foregoing method embodiments, and its implementation principle and technical effects are similar, which will not be elaborated here.
[0160] Figure 5 It is a schematic structural diagram of an electronic device provided by this application. As Figure 5 shown, the electronic device 50 includes:
[0161] A processor 51, a memory 52, and a communication interface 53;
[0162] The memory 52 is used to store executable instructions of the processor 51;
[0163] Wherein, the processor 51 is configured to execute the technical solutions in any of the foregoing method embodiments by executing the executable instructions.
[0164] Optionally, the memory 52 can be either independent or integrated with the processor 51.
[0165] Optionally, when the memory 52 is a device independent of the processor 51, the electronic device 50 may further include:
[0166] A bus 54, and the memory 52 and the communication interface 53 are connected to the processor 51 through the bus 54 to complete mutual communication, and the communication interface 53 is used to communicate with other devices.
[0167] Optionally, the communication interface 53 can be specifically implemented by a transceiver. The communication interface is used to implement communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). The memory may include a random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.
[0168] The bus 54 may be a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, only a thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0169] The aforementioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0170] The electronic device is used to execute the technical solutions in any of the foregoing method embodiments, and its implementation principle and technical effects are similar, which will not be elaborated here.
[0171] The embodiment of the present application also provides a readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the technical solutions provided in any of the foregoing method embodiments are implemented.
[0172] The embodiment of the present application also provides a computer program product, including a computer program, and when the computer program is executed by a processor, it is used to implement the technical solutions provided in any of the foregoing method embodiments.
[0173] Those of ordinary skill in the art can understand that all or part of the steps of implementing the foregoing method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the foregoing method embodiments; and the foregoing storage medium includes: ROM, RAM, magnetic disk, or optical disk and other media that can store program codes.
[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for sample amplification, characterized in that, Including: Obtain a reference sample and at least one detection target data, where the reference sample includes a reference point cloud and a reference image corresponding to each shooting position, and each of the detection target data includes a detection target point cloud and a detection target image corresponding to each shooting position; Generate a drivable area based on each of the reference images; Filter out the data to be inserted from the at least one detection target data, where the detection target point clouds of the data to be inserted are all mapped in the drivable area; Generate an augmented point cloud and an augmented image corresponding to each shooting position based on each of the data to be inserted and the reference sample; Use the augmented point cloud and each of the augmented images as an augmented sample.
2. The method according to claim 1, characterized in that, The generating an augmented point cloud and an augmented image corresponding to each shooting position based on each of the data to be inserted and the reference sample includes: Insert the detection target point cloud in each of the data to be inserted into the reference point cloud to obtain the augmented point cloud; Perform the following processing on the reference image corresponding to each shooting position: For each of the data to be inserted, use the mapped position area of the detection target point cloud in the data to be inserted in the reference image as the position to be covered of the data to be inserted; For each of the data to be inserted, cover the detection target image corresponding to the shooting position in the data to be inserted to the position to be covered of the data to be inserted to obtain the augmented image corresponding to the shooting position.
3. The method according to claim 1, wherein The generating a drivable area based on each of the reference images includes: Stitch each of the reference images to obtain an aerial view; Perform drivable area detection on the aerial view to obtain the drivable area.
4. The method according to claim 1, wherein Before using the augmented point cloud and each of the augmented images as an augmented sample, the method further includes: Determine the detection target point cloud observation area in the augmented point cloud according to the vehicle position and the detection target point cloud of each of the data to be inserted; Remove the reference point cloud in the detection target point cloud observation area to obtain an updated augmented point cloud; The using the augmented point cloud and each of the augmented images as an augmented sample includes: Use the updated augmented point cloud and each of the augmented images as an augmented sample.
5. The method according to any one of claims 1 to 4, characterized in that, Before obtaining the reference sample and at least one detection target data, the method further includes: Obtain a sample to be segmented and annotation data, where the sample to be segmented includes a point cloud to be segmented and an image to be segmented corresponding to each shooting position, and the annotation data includes three-dimensional detection box data; For the image to be segmented corresponding to each shooting position, segment the detection target image corresponding to the shooting position from the image to be segmented according to the three-dimensional detection box data and a contour detection model; Use the point cloud in the three-dimensional detection box corresponding to the three-dimensional detection box data in the point cloud to be segmented as the detection target point cloud; Generate the detection target data according to the detection target point cloud and the detection target image corresponding to each shooting position.
6. The method according to claim 5, wherein The segmenting the detection target image corresponding to the shooting position from the image to be segmented according to the three-dimensional detection box data and a contour detection model includes: Take the area where the three-dimensional detection box corresponding to the three-dimensional detection box data is mapped in the image to be segmented as the target area; Segment the image of the target area in the image to be segmented to obtain a target image with background; Input the target image with background into the contour detection model to obtain a target contour area; Segment the image of the target contour area in the target image with background to obtain a detection target image corresponding to the shooting position.
7. A sample amplification device, characterized in that, Comprising: An acquisition module, configured to acquire a reference sample and at least one detection target data, where the reference sample includes a reference point cloud and a reference image corresponding to each shooting position, and each detection target data includes a detection target point cloud and a detection target image corresponding to each shooting position; A processing module, configured to: Generate a drivable area according to each reference image; Filter out the data to be inserted whose detection target point clouds are all mapped in the drivable area from the at least one detection target data; Generate an augmented point cloud and an augmented image corresponding to each shooting position according to each data to be inserted and the reference sample; An augmentation module, configured to use the augmented point cloud and each augmented image as an augmented sample.
8. An electronic device, characterized in that, Comprising: A processor, a memory, and a communication interface; The memory is configured to store executable instructions of the processor; Wherein, the processor is configured to execute the sample augmentation method according to any one of claims 1 to 6 by executing the executable instructions.
9. A readable storage medium, on which a computer program is stored, characterized in that, When the computer program is executed by the processor, it implements the sample augmentation method according to any one of claims 1 to 6.
10. A computer program product, characterized in that, Comprising a computer program, which is used to implement the sample augmentation method according to any one of claims 1 to 6 when executed by the processor.