Data extraction method and device, equipment and storage medium
By dynamically calculating the Gaussian radius of the labeled area and generating a more dense thermal map, the problem of poor accuracy of the perceptual model sample data in the prior art is solved, and the performance of the perceptual model is improved.
Patent Information
- Application Number
- CN202311764397.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-20
- Publication Date
- 2025-06-20
AI Technical Summary
In the existing autonomous driving system, when the perceptual model obtains sample data, the Gaussian radius is set to 0, which leads to too sparse supervision of the thermal map, which in turn affects the accuracy of the sample data and the performance of the perceptual model.
By obtaining the center position and size of the labeled area in the labeled image, the Gaussian radius is dynamically calculated based on the category and size of the labeled area, a denser thermal map is generated, and more accurate target sample data is extracted.
The density of depth estimation of the labeled area is improved, more depth data is extracted, and the accuracy of sample data is enhanced, thereby optimizing the performance of the trained perceptual model.
Smart Images
Figure CN120182569A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the technical field of deep learning, and particularly relates to a data extraction method, apparatus, device, and storage medium. Background Art
[0002] Perceiving the three-dimensional (3D) orientation angle and distance of a target moving on a road is one of the important performances of an autonomous driving system. Generally, a perception model is deployed in the autonomous driving system to support the above-mentioned perception ability of the autonomous driving system. The perception model can be obtained by deep learning of sample data.
[0003] A conventional way to obtain sample data may include: determining the central pixel of each region of interest in an annotated image, and using the central pixel as a key point to generate a two-dimensional (2D) heatmap, and then performing 3D regression on the high-density data indicated in the 2D heatmap, and using the regression-obtained 3D data as sample data.
[0004] Among them, the distribution and depth of the high-density data indicated by the heatmap are related to the setting of the parameters of the Gaussian distribution. In the conventional setting, only the central pixel of the region of interest is used as the key point, and the Gaussian radius of the region of interest of non-vehicle types is set to 0, resulting in too sparse supervision of the heatmap, so that the accuracy of the obtained sample data is poor, and further the performance of the trained perception model is poor. Summary of the Invention
[0005] This application provides a data extraction method, apparatus, device, and storage medium, which can timely detect whether a high-precision map is accurate to support timely repair of abnormal map areas.
[0006] The first aspect embodiment of this application provides a data extraction method, including:
[0007] Obtaining the central position and size of each annotation area in multiple annotation areas of an annotated image;
[0008] For any annotation area, generating a Gaussian radius of the annotation area according to the size of the annotation area and the pre-designed calculation parameters corresponding to the category of the annotation area;
[0009] Generating a heatmap of the annotated image based on the central positions and Gaussian radii of the respective annotation areas;
[0010] Extracting target sample data from the annotated image according to the density data included in the heatmap.
[0011] In some embodiments of the present application, generating the Gaussian radius of the marked area according to the pre-designed calculation parameters corresponding to the size and category of the marked area includes:
[0012] Determine whether the size of the marked area is greater than a preset size;
[0013] If the size of the marked area is greater than the preset size, determine the pre-designed calculation parameter corresponding to the marked area according to the category of the marked area, and the pre-designed calculation parameter is used to characterize the proportion of the overlapping part of the marked area and the adjacent marked area;
[0014] Call a preset model to calculate the Gaussian radius of the marked area based on the size of the marked area and the pre-designed calculation parameter.
[0015] In some embodiments of the present application, the category of the marked area is a vehicle, a cyclist or a pedestrian. Determining the pre-designed calculation parameter corresponding to the marked area according to the category of the marked area includes:
[0016] If the category of the marked area is a vehicle, determine that the pre-designed calculation parameter corresponding to the marked area is 0.5
[0017] If the category of the marked area is a cyclist or a pedestrian, determine that the pre-designed calculation parameter corresponding to the marked area is 0.3.
[0018] In some embodiments of the present application, if the size of the marked area is greater than the preset size, the Gaussian radius of the marked area is greater than or equal to 1 pixel and less than or equal to 5 pixels;
[0019] If the size of the marked area is not greater than the preset size, the Gaussian radius of the marked area is 0.
[0020] In some embodiments of the present application, generating the heat map of the marked image based on the central positions and Gaussian radii of the respective marked areas includes:
[0021] For any marked area, set the central position of the marked area as the density peak of the Gaussian distribution;
[0022] Determine a decreasing parameter based on the Gaussian radius of the marked area;
[0023] Calculate the density values corresponding to the positions other than the central position in the marked area according to the decreasing parameter to obtain the density data of the marked area in the heat map.
[0024] In some embodiments of the present application, extracting target sample data from the marked image according to the density data included in the heat map includes:
[0025] For any marked area, at least one density data within the Gaussian radius of the marked area is extracted, and the at least one density data includes the density data at the central position of the marked area;
[0026] A three-dimensional image corresponding to the marked area is constructed according to the at least one density data, and the image data of the three-dimensional image is the target sample data.
[0027] In some embodiments of the present application, the obtaining of the central position and size of each marked area in the multiple marked areas of the marked image includes:
[0028] Determine the size of the feature map corresponding to the marked image, and the feature map is a heat map;
[0029] According to the size of the marked image and the size of the feature map, determine the mapping relationship between the marked image and the feature map;
[0030] According to the mapping relationship, map the central pixel and the marked bounding box of each marked area in the multiple marked areas to the feature map to obtain the central position and size of each marked area.
[0031] An embodiment of the second aspect of the present application provides a data extraction device, including:
[0032] An acquisition module for acquiring the central position and size of each marked area in the multiple marked areas of the marked image;
[0033] A generation module for generating a Gaussian radius of any marked area according to the size of the marked area and the pre-designed calculation parameters corresponding to the category of the marked area;
[0034] The generation module is further configured to generate a heat map of the marked image based on the central position and Gaussian radius of each marked area;
[0035] An extraction module for extracting target sample data from the marked image according to the density data included in the heat map.
[0036] An embodiment of the third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor runs the computer program to implement the method described in the first aspect above.
[0037] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored, and the program is executed by a processor to implement the method described in the first aspect above.
[0038] The technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:
[0039] In the embodiments of the present application, the central position and size of each annotation area in a plurality of annotation areas of an annotation image are obtained. For any annotation area, a Gaussian radius of the annotation area is generated according to the size of the annotation area and pre-designed calculation parameters corresponding to the category of the annotation area. That is, in the embodiments of the present application, the Gaussian radius of some annotation areas is not directly set to 0, but the Gaussian radius is adaptively set for each annotation area according to the size of each annotation area. Furthermore, a heat map of the annotation image is generated based on the central positions and Gaussian radii of the respective annotation areas, and target sample data is extracted from the annotation image according to density data included in the heat map. Among them, since each annotation area corresponds to a Gaussian radius, during the process of generating the heat map, the Gaussian distribution range corresponding to the Gaussian radius in the annotation area can be deeply estimated, rather than only deeply estimating the central pixel of the annotation area. In this way, the density of the depth estimation for each annotation area can be improved, more depth data can be extracted, thereby improving the accuracy of the obtained sample data, and further optimizing the performance of the trained perception model.
[0040] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present application. Moreover, throughout the drawings, the same reference numerals are used to represent the same components.
[0042] In the drawings:
[0043] Figure 1 A flowchart of a data extraction method provided by an embodiment of the present application is shown;
[0044] Figure 2 A flowchart of a data extraction method provided by another embodiment of the present application is shown;
[0045] Figure 3 A schematic diagram of the scenario of a heat map provided by an embodiment of the present application is shown;
[0046] Figure 4 A schematic diagram of the structure of a data extraction device provided by an embodiment of the present application is shown;
[0047] Figure 5The structural schematic diagram of an electronic device provided by an embodiment of the present application is shown;
[0048] Figure 6 The schematic diagram of a storage medium provided by an embodiment of the present application is shown. Detailed implementation manners
[0049] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided so that the present application can be more thoroughly understood and the scope of the present application can be completely conveyed to those skilled in the art.
[0050] It should be noted that unless otherwise specified, the technical terms or scientific terms used in the present application should have the ordinary meanings understood by those skilled in the art to which the present application belongs.
[0051] First, the technical scenarios and related technical terms involved in the present application will be introduced.
[0052] The embodiments of the present application relate to an autonomous driving system. Specifically, it can be applied to the technical field of deep learning for perceiving the 3D orientation angle and distance of a target. In some embodiments, the embodiments of the present application may relate to the target detection technology based on monocular 3D (Mono 3d). Among them, monocular 3D target detection includes: inferring the 2D depth data of a target from a monocular RGB image, and then reconstructing the 3D scene of the target through the 2D depth data. Exemplarily, the SMOKE (Single-stage Monocular3D Object Detection via Keypoint Estimation) deep learning model can be used to locate the center of the target using 2D object detection, and then perform 3D attribute regression based on the 2D data to reconstruct the size, pose, and three-dimensional position of the target, etc.
[0053] In the conventional SMOKE method, for a target to be detected in an input image, the central pixel of the target image to be detected is used as a key point for supervision. Furthermore, based on the principle of Gaussian distribution, the input image is mapped into a 2D heat map, so that the depth data shown in the heat map can be used as supervision data for 3D reconstruction, and a 3D image of the target to be detected can be obtained.
[0054] A heat map is a statistical chart that displays data by coloring color blocks. The relationship and change trend between data can be shown through color blocks by a pre-specified color mapping rule. For example, larger values (such as the distance value from the target) can be represented by darker colors, and smaller values can be represented by lighter colors, etc.
[0055] The SMOKE method supervises the target to be detected based on the Gaussian distribution. Therefore, the setting of the Gaussian radius has an important impact on the supervision of the range of the target area to be detected. In the field of autonomous driving, the targets to be detected can include vehicles, cyclists, and pedestrians. For cyclists and pedestrians, since the target size is relatively small, the Gaussian radius is usually set to 0. Then, on the one hand, during the 3D reconstruction of cyclists and pedestrians, it is highly dependent on the accuracy of the central pixel. The deviation of the position or size of this central pixel will be magnified in the 3D space, resulting in a large error in the bounding box of the determined 3D image. On the other hand, for targets such as cyclists and pedestrians, setting the Gaussian radius to 0 makes the supervision relatively sparse, resulting in a relatively low accuracy of the depth data presented in the heatmap for characterizing the target features.
[0056] Based on this, in the technical solution provided by the embodiments of the present application, the Gaussian radius can be set for each annotation area according to the size of each annotation area in the annotated image, so that during the generation of the heatmap, the depth estimation can be performed on the Gaussian distribution range corresponding to the Gaussian radius in the annotation area. In this way, the density of the depth estimation for each annotation area can be improved, more depth data can be extracted, thereby improving the accuracy of the obtained sample data, and further optimizing the performance of the trained perception model.
[0057] The execution subject of this technical solution can be any electronic device that supports image processing, including in-vehicle devices, aircraft, or robots, etc. Such an electronic device can be deployed in an autonomous driving system. An SMOKE model, a Gaussian radius generation algorithm, a heatmap generation module, a heatmap visualization module, etc. can be deployed in such an electronic device to support the execution of the related functions of the embodiments of the present application.
[0058] Next, a data extraction method, device, and storage medium proposed according to an embodiment of the present application will be described with reference to the accompanying drawings. The technical solution of the present application will be described in detail with specific embodiments below. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.
[0059] See Figure 1 , Figure 1 shows the flowchart of a data extraction method provided by an embodiment of the present application. The method specifically includes the following steps:
[0060] Step S101, obtain the central position and size of each annotation area in multiple annotation areas of the annotated image.
[0061] Among them, the labeled image can be an image within the receptive field of any vehicle. The vehicles, cyclists, pedestrians, etc. in the image that are waiting to be detected can be labeled respectively. Exemplarily, any object among vehicles, cyclists, and pedestrians can be labeled using, for example, "bounding boxes" to obtain multiple labeled regions.
[0062] It should be noted that the central position and size of each labeled region in this embodiment can refer to the position characteristics and size characteristics of the central pixel of each labeled region. The size characteristics can include, for example, the width characteristics and height characteristics of the bounding box of each labeled region. The position characteristic of any labeled region can refer to the position where the central pixel of the labeled region is mapped to the heat map, and the size characteristic can refer to the size where the bounding box of the labeled region is mapped to the heat map.
[0063] Exemplarily, the electronic device can obtain the sizes of the input RGB image and the heat map (hereinafter referred to as the feature map). The RGB image is the labeled image. Then, the electronic device can determine the mapping relationship between the labeled image and the feature map according to the size of the labeled image and the size of the feature map. Among them, the mapping relationship can include the ratio of the size of the RGB image to the size of the feature map. This ratio includes the width ratio and the height ratio. Then, according to the mapping relationship, the central pixel and the labeled bounding box of each labeled region among the multiple labeled regions can be mapped to the feature map to obtain the central position and size of each labeled region, so as to obtain an empty heat map.
[0064] In some embodiments, the central position can be the coordinate position where the central pixel of the corresponding labeled region is mapped to the feature map.
[0065] Step S102, for any labeled region, generate the Gaussian radius of the labeled region according to the size of the labeled region and the pre-designed calculation parameters corresponding to the category of the labeled region.
[0066] Among them, the pre-designed calculation parameters are used to characterize the ratio of the overlapping part of the labeled region and the adjacent labeled regions, that is, the overlapping rate. The Gaussian radius of the labeled region can be the Gaussian distribution parameter when the labeled region is mapped to the heat map.
[0067] It should be noted that the Gaussian radius of each labeled region can be determined by the length from the center position of the labeled region to the bounding box. In the actual implementation scenario, the distances between the objects to be detected in the image may be relatively close. Then, during the process of labeling the bounding box, there may be a certain overlapping region between adjacent objects to be detected. Correspondingly, there is partial overlap in the Gaussian radii of the labeled regions of the two adjacent objects to be detected. To improve the accuracy of the depth data in the heat map, the embodiments of the present application can preset the calculation parameters (i.e., the overlap rate) of the Gaussian radius according to the category of the labeled region.
[0068] Among them, the categories of the labeled regions can include vehicles, cyclists, and pedestrians. The larger the object in the labeled region, the relatively larger the overlapping part of the labeled region with other labeled regions; the smaller the object in the labeled region, the relatively smaller the overlapping part of the labeled region with other labeled regions. Based on this, when the category of the labeled region is a vehicle, the preset calculation parameter corresponding to the labeled region is 0.5; when the category of the labeled region is a cyclist or a pedestrian, the preset calculation parameter corresponding to the labeled region is 0.3.
[0069] In addition, in the actual implementation scenario, different objects are at different distances from the vehicle that captures the image, resulting in different sizes of the images of each object in the labeled image. For example, some objects that are far from the vehicle that captures the image appear smaller in the labeled image, while some objects that are close to the vehicle that captures the image appear larger in the labeled image.
[0070] Based on this, in some embodiments, for any labeled region, the electronic device can determine whether the size of the labeled region is greater than a preset size. If the size of the labeled region is greater than the preset size, determine the preset calculation parameter corresponding to the labeled region according to the category of the labeled region, where the preset calculation parameter is used to represent the proportion of the overlapping part of the labeled region with the adjacent labeled region; call a preset model to calculate the Gaussian radius of the labeled region based on the size of the labeled region and the preset calculation parameter. The Gaussian radius of the labeled region can be, for example, greater than or equal to 1 pixel and less than or equal to 5 pixels.
[0071] In other embodiments, if the size of the labeled region is not greater than the preset size, the Gaussian radius of the labeled region is 0.
[0072] Exemplarily, the preset size can be flexibly set according to actual implementation requirements. For example, it can be set to 2*2 pixels.
[0073] For example, if the size of the labeled region is 5 * 8 pixels and the category of the labeled region is a vehicle, the overlap rate of the labeled region can be determined to be 0.5. Further, the Gaussian radius of the labeled region is calculated to be 3 pixels by calling the function gaussian_radius based on 5 * 8 pixels and the overlap rate of 0.5. Again, for example, if the size of the labeled region is 1 * 2 pixels, since the size of the labeled region is small, the Gaussian radius of the labeled region can be determined to be 0.
[0074] In some embodiments, the above preset model can be a category-adaptive Gaussian radius model, which can dynamically assign a Gaussian radius to each category of object according to the size and shape distribution of different categories of objects. In this way, for objects of smaller categories (such as pedestrians and cyclists), a heatmap density similar to that of large-sized objects (such as cars) can be deployed.
[0075] It can be seen that by adopting this implementation method, a Gaussian radius matching each labeled region can be adaptively generated according to the size and category of each labeled region. In this way, during the process of generating the heatmap, the Gaussian distribution range corresponding to the Gaussian radius in the labeled region can be deeply estimated, thereby improving the density of the depth estimation for each labeled region.
[0076] Step S103, generate a heatmap of the labeled image based on the central positions and Gaussian radii of the respective labeled regions.
[0077] Among them, each labeled region is mapped into the empty heatmap feature map constructed in S101 to obtain the heatmap of the labeled image.
[0078] In some embodiments, for any labeled region, the electronic device can set the central position of the labeled region as the density peak of the Gaussian distribution, and then determine a decreasing parameter based on the Gaussian radius of the labeled region; calculate the density values corresponding to the positions other than the central position in the labeled region according to the decreasing parameter to obtain the density data of the labeled region in the heatmap. Among them, the standard deviation of the Gaussian distribution can be determined by the Gaussian radius of the labeled region.
[0079] It should be noted that the density peaks of the central positions of the respective labeled regions can be determined respectively according to the sizes of the corresponding bounding boxes of the labeled regions and pre-designed calculation parameters, and the density peaks of the central positions of the respective labeled regions can be different from each other.
[0080] Among them, by adopting this implementation method, since each labeled region is correspondingly provided with a matching Gaussian radius, and this Gaussian radius is used as a parameter of the Gaussian distribution in the process of converting the heatmap, compared with the data that only supervises the central position, denser depth estimation data can be obtained through the heatmap.
[0081] Step S104, extract target sample data from the labeled image according to the density data included in the heat map.
[0082] Among them, for any labeled area, the electronic device can extract at least one density data within the Gaussian radius of the labeled area, and the at least one density data includes the density data at the center position of the labeled area. Then, a 3D image corresponding to the labeled area is constructed according to the at least one density data, and the image data of the three-dimensional image is the target sample data.
[0083] Exemplarily, the target sample data may include the sizes, poses, three-dimensional positions, etc. of various targets included in the labeled image.
[0084] In the embodiments of the present application, the center positions and sizes of the respective labeled areas in the labeled image are obtained. For any labeled area, a Gaussian radius of the labeled area is generated according to the size of the labeled area and the pre-designed calculation parameters corresponding to the category of the labeled area. That is, in the embodiments of the present application, the Gaussian radii of some labeled areas are not directly set to 0, but the Gaussian radii are adaptively set for each labeled area according to the sizes of the respective labeled areas. Furthermore, a heat map of the labeled image is generated based on the center positions and Gaussian radii of the respective labeled areas, and target sample data is extracted from the labeled image according to the density data included in the heat map. Among them, since each labeled area corresponds to a Gaussian radius, then, in the process of generating the heat map, the depth estimation can be performed on the Gaussian distribution range corresponding to the Gaussian radius in the labeled area, rather than only performing the depth estimation on the central pixel of the labeled area. In this way, the density of the depth estimation for each labeled area can be improved, more depth data can be extracted, thereby the accuracy of the obtained sample data can be improved, and further the performance of the trained perception model can be optimized.
[0085] The data extraction method involved in the embodiments of the present application will be described below with reference to examples.
[0086] See Figure 2 , Figure 2 which illustrates the flow of another data extraction method. Figure 2 The method flow illustrated, through an exemplary processing sequence of the electronic device, illustrates the technical solution of the embodiments of the present application. Figure 2 The data extraction method illustrated includes S201 - S208.
[0087] S201, obtain the sizes of the labeled image and the feature map.
[0088] Among them, the width of the annotated image can be img_w, and the height can be img_h. All types of objects to be detected in the annotated image are marked with bounding boxes, and each bounding box is, for example, a marked area. The objects to be detected include, for example, vehicles, pedestrians, and cyclists.
[0089] The width of the feature map can be feat_w, and the height can be feat_h.
[0090] S202, calculate the width scale factor and height scale factor for mapping the annotated image to the feature map.
[0091] Among them, the width scale factor for mapping the annotated image to the feature map is width_ratio, and the height scale factor for mapping the annotated image to the feature map is height_ratio. The bounding boxes of each marked area in the annotated image can be mapped to the feature map according to these two scale factors.
[0092] S203, initialize an empty heatmap.
[0093] An empty heatmap refers to a heatmap that does not contain any data. The size of this empty heatmap can be [feat_h, feat_w].
[0094] S204, for each bounding box in the annotated image, determine the position in the feature map where the center pixel of the bounding box is mapped.
[0095] S205, for each bounding box in the annotated image, calculate the height scale_box_h and width scale_box_w of the bounding box in the feature map according to width_ratio and height_ratio.
[0096] S206, calculate the Gaussian radius of the corresponding bounding box according to the scale_box_h and scale_box_w of each bounding box and the overlap rate.
[0097] Among them, the overlap rate of the Gaussian heatmap can be determined according to the target category of each bounding box. For example, for the category "vehicle", the overlap rate is 0.5; for non-vehicle categories, the overlap rate is 0.3.
[0098] In this way, the calculated Gaussian radius can have relatively high accuracy.
[0099] S207, generate a heatmap according to the corresponding center positions and Gaussian radii of each bounding box.
[0100] Exemplarily, for each bounding box (i.e., the annotated region), the density peak and the shape of the heat map corresponding to the center position of the bounding box can be determined according to the category and size of the bounding box, and the density peak and the shape of the heat map can be displayed at the corresponding position in the above-mentioned empty heat map. Furthermore, based on the Gaussian radius of the bounding box, the standard deviation of the Gaussian distribution corresponding to the bounding box is determined. After that, with the density peak as the center, the density data of other parts of the bounding box can be determined in the decreasing manner of the Gaussian distribution and displayed at the corresponding position in the above-mentioned empty heat map to generate the heat map of the annotated image.
[0101] For example, Figure 3 in the shown effect comparison diagram, for the same annotated image, Figure 3 (a) in it is the heat map generated by the conventional method, Figure 3 (b) in it is the heat map generated by the technical solution of this application. It can be seen that, compared with the heat map generated by the conventional method, the data supervised in the heat map generated by the embodiments of this application is denser.
[0102] S208. Extract the density data in the heat map to generate the 3D model of the target.
[0103] Combined with Figure 3 (b) shown in it, the parts with lighter colors are, for example, the density data. The electronic device can extract the data of these parts to perform 3D reconstruction based on these data, so as to obtain the 3D models of each target in the annotated image. Details are not described here again.
[0104] It should be understood that Figure 3 is only an exemplary visualization effect comparison diagram and does not constitute a limitation to the technical solution of this application. In actual implementation, the distribution of the density data in the heat map is associated with the annotated image and can be different from that shown in Figure 3 (b) herein. No limitation is imposed here.
[0105] In the embodiments of the present application, the central positions and sizes of the respective annotation regions in the annotation image are obtained, and for any annotation region, a Gaussian radius of the annotation region is generated according to the size of the annotation region and the pre-designed calculation parameters corresponding to the category of the annotation region. That is, in the embodiments of the present application, the Gaussian radii of some annotation regions are not directly set to 0, but the Gaussian radii are adaptively set for each annotation region according to the sizes of the respective annotation regions. Further, a heat map of the annotation image is generated based on the central positions and Gaussian radii of the respective annotation regions, and target sample data is extracted from the annotation image according to the density data included in the heat map. Among them, since each annotation region corresponds to a Gaussian radius, in the process of generating the heat map, the Gaussian distribution range corresponding to the Gaussian radius in the annotation region can be deeply estimated, rather than only deeply estimating the central pixel of the annotation region. In this way, the density of the depth estimation of each annotation region can be improved, more depth data can be extracted, so that the accuracy of the obtained sample data can be improved, and further the performance of the trained perception model can be optimized.
[0106] An embodiment of the present application further provides a data extraction device, and this data extraction device is used to execute the data extraction method provided in any of the above embodiments. As Figure 4 shown, the device includes: an acquisition module 41, a generation module 42, and an extraction module 43.
[0107] The acquisition module 41 is configured to obtain the central positions and sizes of the respective annotation regions in the annotation image;
[0108] The generation module 42 is configured to generate a Gaussian radius of any annotation region according to the size of the annotation region and the pre-designed calculation parameters corresponding to the category of the annotation region;
[0109] The generation module 42 is further configured to generate a heat map of the annotation image based on the central positions and Gaussian radii of the respective annotation regions;
[0110] The extraction module 43 is configured to extract target sample data from the annotation image according to the density data included in the heat map.
[0111] Optionally, the generation module 42 is further configured to:
[0112] Determine whether the size of the annotation region is greater than a preset size;
[0113] If the size of the annotation region is greater than the preset size, determine the pre-designed calculation parameters corresponding to the annotation region according to the category of the annotation region, and the pre-designed calculation parameters are used to characterize the proportion of the overlapping part of the annotation region and the adjacent annotation regions;
[0114] Call a preset model to calculate the Gaussian radius of the marked area based on the size of the marked area and the pre-designed calculation parameters.
[0115] Optionally, if the category of the marked area is a vehicle, a cyclist, or a pedestrian, the generation module 42 is further configured to:
[0116] If the category of the marked area is a vehicle, determine that the pre-designed calculation parameter corresponding to the marked area is 0.5
[0117] If the category of the marked area is a cyclist or a pedestrian, determine that the pre-designed calculation parameter corresponding to the marked area is 0.3.
[0118] Optionally, if the size of the marked area is greater than a preset size, the Gaussian radius of the marked area is greater than or equal to 1 pixel and less than or equal to 5 pixels; if the size of the marked area is not greater than the preset size, the Gaussian radius of the marked area is 0.
[0119] Optionally, the generation module 42 is further configured to:
[0120] For any marked area, set the central position of the marked area as the density peak of the Gaussian distribution;
[0121] Determine a decreasing parameter based on the Gaussian radius of the marked area;
[0122] Calculate the density values corresponding to the positions other than the central position in the marked area according to the decreasing parameter to obtain the density data of the marked area in the heat map.
[0123] Optionally, the extraction module 43 is further configured to:
[0124] For any marked area, extract at least one density data within the Gaussian radius of the marked area, and the at least one density data includes the density data of the central position of the marked area;
[0125] Construct a three-dimensional image corresponding to the marked area according to the at least one density data, and the image data of the three-dimensional image is the target sample data.
[0126] Optionally, the acquisition module 41 is further configured to: determine the size of the feature map corresponding to the marked image, and the feature map is a heat feature map;
[0127] Determine the mapping relationship between the marked image and the feature map according to the size of the marked image and the size of the feature map;
[0128] According to the mapping relationship, map the central pixels and annotation bounding boxes of each annotation area in the multiple annotation areas to the feature map to obtain the central positions and sizes of the respective annotation areas.
[0129] The data extraction device provided by the embodiments of the present application and the data extraction method provided by the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by them.
[0130] The embodiments of the present application also provide an electronic device to execute the above data extraction method. Please refer to Figure 5 It shows a schematic diagram of an electronic device provided by some embodiments of the present application. As Figure 5 shown, the electronic device 5 includes: a processor 500, a memory 501, a bus 502, and a communication interface 503. The processor 500, the communication interface 503, and the memory 501 are connected through the bus 502; a computer program that can run on the processor 500 is stored in the memory 501, and when the processor 500 runs the computer program, it executes the data extraction method provided by any one of the foregoing embodiments of the present application.
[0131] Among them, the memory 501 may include a high-speed random access memory (RAM: Random Access Memory), and may also include a non-volatile memory, such as at least one disk memory. Through at least one communication interface 503 (which can be wired or wireless), a communication connection between this device network element and at least one other network element can be realized, and the Internet, wide area network, local area network, metropolitan area network, etc. can be used.
[0132] The bus 502 may be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Among them, the memory 501 is used to store a program. After receiving an execution instruction, the processor 500 executes the program, and the data extraction method disclosed in any one of the foregoing embodiments of the present application can be applied to or implemented by the processor 500.
[0133] The processor 500 may be an integrated circuit chip with the ability to process signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor 500 or instructions in the form of software. The above-mentioned processor 500 may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 501, and the processor 500 reads the information in the memory 501 and combines its hardware to complete the steps of the above method.
[0134] The electronic device provided by the embodiments of the present application and the data extraction method provided by the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by them.
[0135] The embodiments of the present application also provide a computer-readable storage medium corresponding to the data extraction method provided by the foregoing embodiments. Please refer to Figure 6 , which shows that the computer-readable storage medium is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the data extraction method provided by any of the foregoing embodiments.
[0136] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here one by one.
[0137] The computer-readable storage medium provided by the above embodiments of the present application and the data extraction method provided by the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored in it.
[0138] It should be noted that:
[0139] In the specification provided herein, a large number of specific details are set forth. However, it will be understood that embodiments of the present application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.
[0140] Similarly, it should be understood that in order to streamline the present application and assist in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be construed as reflecting an intention that the claimed subject matter of the present application requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, the inventive aspects lie in less than all of the features of the single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the present application.
[0141] In addition, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features of different embodiments are meant to be within the scope of the present application and form different embodiments. For example, in the following claims, any one of the claimed embodiments can be used in any combination.
[0142] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the said claims.
Claims
1. A data extraction method, characterized in that, Including: Obtaining the central position and size of each annotation region among multiple annotation regions of the annotation image; For any annotation region, generating a Gaussian radius of the annotation region according to the size of the annotation region and pre-designed calculation parameters corresponding to the category of the annotation region; Generating a heat map of the annotation image based on the central positions and Gaussian radii of the respective annotation regions; Extracting target sample data from the annotation image according to the density data included in the heat map.
2. The method according to claim 1, characterized in that, The generating the Gaussian radius of the annotation region according to the size of the annotation region and pre-designed calculation parameters corresponding to the category of the annotation region includes: Determining whether the size of the annotation region is greater than a preset size; If the size of the annotation region is greater than the preset size, determining pre-designed calculation parameters corresponding to the annotation region according to the category of the annotation region, where the pre-designed calculation parameters are used to characterize the proportion of the overlapping part of the annotation region and adjacent annotation regions; Invoking a preset model to calculate the Gaussian radius of the annotation region based on the size of the annotation region and the pre-designed calculation parameters.
3. The method according to claim 2, characterized in that, The category of the annotation region is a vehicle, a cyclist or a pedestrian. The determining pre-designed calculation parameters corresponding to the annotation region according to the category of the annotation region includes: If the category of the annotation region is a vehicle, determining that the pre-designed calculation parameter corresponding to the annotation region is 0.5 If the category of the annotation region is a cyclist or a pedestrian, determining that the pre-designed calculation parameter corresponding to the annotation region is 0.
3.
4. The method according to any one of claims 1 - 3, characterized in that, If the size of the annotation region is greater than the preset size, the Gaussian radius of the annotation region is greater than or equal to 1 pixel and less than or equal to 5 pixels; If the size of the annotation region is not greater than the preset size, the Gaussian radius of the annotation region is 0.
5. The method according to claim 1, characterized in that, The generating the heat map of the annotation image based on the central positions and Gaussian radii of the respective annotation regions includes: For any annotation region, setting the central position of the annotation region as the density peak of the Gaussian distribution; Determining a decreasing parameter based on the Gaussian radius of the annotation region; Calculating density values corresponding to positions other than the central position in the annotation region according to the decreasing parameter to obtain density data of the annotation region in the heat map.
6. The method according to claim 1, characterized in that, The extracting target sample data from the annotation image according to the density data included in the heat map includes: For any annotation region, extracting at least one density data within the Gaussian radius of the annotation region, where the at least one density data includes density data at the central position of the annotation region; Constructing a three-dimensional image corresponding to the annotation region according to the at least one density data, where the image data of the three-dimensional image is the target sample data.
7. The method according to claim 1, characterized in that, The obtaining the central position and size of each annotation region among multiple annotation regions of the annotation image includes: Determining the size of the feature map corresponding to the annotation image, where the feature map is a heat feature map; Determining a mapping relationship between the annotation image and the feature map according to the size of the annotation image and the size of the feature map; According to the mapping relationship, map the central pixel and the annotation bounding box of each annotation region in the multiple annotation regions to the feature map to obtain the central position and size of each annotation region.
8. A data extraction device, characterized in that, The device includes: an acquisition module, configured to acquire the central position and size of each annotation region in multiple annotation regions of an annotation image; a generation module, configured to generate a Gaussian radius of an annotation region according to the size of the annotation region and pre-designed calculation parameters corresponding to the category of the annotation region for any annotation region; the generation module is further configured to generate a heat map of the annotation image based on the central positions and Gaussian radii of the respective annotation regions; an extraction module, configured to extract target sample data from the annotation image according to density data included in the heat map.
9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor runs the computer program to implement the method according to any one of claims 1-7.
10. A computer-readable storage medium, having a computer program stored thereon, wherein, The program is executed by the processor to implement the method according to any one of claims 1-7.