Data processing method and data processing device, and autonomous driving system

Abstract

The data processing method comprises: performing feature extraction on original point cloud data to obtain first point cloud feature data; performing feature extraction on original image data to obtain first image feature data; performing screening on the first point cloud feature data and the first image feature data based on preset target classification candidate information to obtain second point cloud feature data and second image feature data; fusing the second point cloud feature data and the second image feature data to obtain fusion data; and performing target identification on the fusion data to obtain a target identification result. The above scheme can guarantee the processing efficiency and accuracy of data fusion at a low implementation cost, and thus a more accurate target identification result can be obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic driving, in particular to a data processing method and device, and an automatic driving system. BACKGROUND

[0002] At present, an automatic driving system is often equipped with a sensor suite composed of different sensors, such as a point cloud acquisition device, an image acquisition device, a combined navigation device, and the like. The automatic driving system relies on the perception of the environment by the sensors, and the more accurate the environmental information obtained by the sensors, the more timely and accurate the driving decisions made by the automatic driving system, and the more the automatic driving is realized.

[0003] However, in the existing sensors for the automatic driving system, the environmental information perceived by a single sensor is limited and cannot meet the requirements of complex and variable real driving scenarios. For example, although the image acquisition device can capture rich environmental information at a relatively low implementation cost and the two-dimensional image data obtained retains relatively accurate environmental features, the image acquisition device is easily affected by environmental light, resulting in poor image quality; while the point cloud acquisition device can provide three-dimensional point cloud data with high accuracy without being affected by environmental light, the technology of the point cloud acquisition device is not mature enough at present, and the types of devices are limited. If the automatic driving system has a high accuracy requirement, the implementation cost may be greatly increased, and even after using expensive devices, the required accuracy may not be achieved.

[0004] In order to improve the environmental perception ability of the automatic driving system, the data of various sensors are processed by the skilled person in the art in multiple levels and multiple spaces for information complementation and optimization, so as to obtain fusion data with richer information, so as to facilitate more complex subsequent processing tasks such as target recognition.

[0005] In actual application, the image acquisition device and the point cloud acquisition device can complement information, and the image acquisition device and the point cloud acquisition device can be fused, and then the fusion data can be used for target recognition. However, there is a large deviation between the target recognized by the fusion data and the real target, the accuracy of the target recognition result is low, and the driving decision of the automatic driving system is affected. SUMMARY

[0006] Therefore, the present application provides a data processing method and device, and an automatic driving system, which can guarantee the processing efficiency and accuracy of data fusion at a low implementation cost, and can obtain more accurate target recognition results.

[0007] The present application provides a data processing method, comprising:

[0008] performing feature extraction on the original point cloud data to obtain first point cloud feature data;

[0009] perform feature extraction on the original image data to obtain first image feature data;

[0010] perform filtering on the three-dimensional target candidate information to obtain optimized target candidate information;

[0011] perform filtering on the three-dimensional target candidate information to obtain optimized target candidate information;

[0012] perform filtering on the three-dimensional target candidate information to obtain optimized target candidate information;

[0013] Optionally, the filtering on the three-dimensional target candidate information to obtain the optimized target candidate information comprises:

[0014] perform filtering on the three-dimensional target candidate information to obtain optimized target candidate information;

[0015] perform filtering on the three-dimensional target candidate information to obtain optimized target candidate information;

[0016] perform filtering on the three-dimensional target candidate information to obtain optimized target candidate information;

[0017] Optionally, the filtering on the three-dimensional target candidate information to obtain the optimized target candidate information comprises:

[0018] perform filtering on the three-dimensional target candidate information to obtain optimized target candidate information;

[0019] perform filtering on the three-dimensional target candidate information to obtain optimized target candidate information;

[0020] perform filtering on the three-dimensional target candidate information to obtain optimized target candidate information;

[0021] perform filtering on the three-dimensional target candidate information to obtain optimized target candidate information;

[0022] Optionally, the filtering on the three-dimensional target candidate information to obtain the optimized target candidate information comprises:

[0023] The three-dimensional target candidate information is evaluated, and the three-dimensional target candidate information is filtered based on preset evaluation conditions to obtain intermediate target candidate information; the intermediate target candidate information is used as the optimized target candidate information.

[0024] Optionally, before using the intermediate target candidate information as the optimized target candidate information, the method further includes:

[0025] The intermediate target candidate information is subjected to de-overlap processing.

[0026] Optionally, before performing de-overlap processing on the intermediate target candidate information, the method further includes:

[0027] The intermediate target candidate information is matched with the image range corresponding to the original image data, and the intermediate target candidate information is filtered based on preset matching conditions.

[0028] Optionally, matching the intermediate target candidate information with the image range corresponding to the original image data includes:

[0029] The intermediate target candidate information is converted to the pixel coordinate system of the original image data and matched with the image range corresponding to the original image data.

[0030] Optionally, before filtering the first point cloud feature data and the first image feature data based on preset target classification candidate information, the method further includes:

[0031] Based on the object category and object orientation, the target classification candidate information is set.

[0032] Optionally, the step of extracting features from the original point cloud data to obtain the first point cloud feature data includes:

[0033] The original point cloud data is divided into multiple voxel units;

[0034] Feature extraction is performed on the multiple voxel units to obtain voxel feature data;

[0035] The voxel feature data is compressed to obtain the first point cloud feature data.

[0036] Optionally, the step of extracting features from the plurality of voxel units to obtain voxel feature data includes:

[0037] Local feature extraction is performed on each of the multiple voxel units to obtain the voxel feature data.

[0038] Optionally, the step of performing local feature extraction on the plurality of voxel units to obtain the voxel feature data includes any one of the following:

[0039] Voxel feature stacking is performed on the plurality of voxel units respectively to obtain the voxel feature data;

[0040] Point data logic operations are performed on the multiple voxel units respectively to obtain the voxel feature data.

[0041] Optionally, compressing the voxel feature data includes:

[0042] The voxel feature data is compressed in a specified direction using a convolutional neural network block, which includes sparse convolutional layers and submanifold convolutional layers.

[0043] Optionally, the data processing method further includes:

[0044] Before filtering the first point cloud feature data and the first image feature data based on the preset target classification candidate information, it is determined whether there are any abnormalities in the first image feature data.

[0045] If the judgment result is that there is no abnormality, then the first point cloud feature data and the first image feature data are filtered respectively based on the preset target classification candidate information.

[0046] Optionally, the data processing method further includes:

[0047] If the judgment result indicates that there is an abnormal situation, then target recognition is performed on the first point cloud feature data to obtain the target recognition result.

[0048] Optionally, the data processing method further includes:

[0049] The target recognition result is obtained by using a fully connected neural network to perform target recognition on the fused data.

[0050] The present invention also provides a data processing device connected to an image acquisition device and a point cloud acquisition device, suitable for performing the data processing method described in any of the above embodiments, wherein the data processing device includes:

[0051] The data acquisition unit is adapted to acquire the raw image data of the image acquisition device and the raw point cloud data of the point cloud acquisition device.

[0052] The feature extraction unit is adapted to extract features from the original point cloud data to obtain the first point cloud feature data, and to extract features from the original image data to obtain the first image feature data.

[0053] The data filtering unit is adapted to filter the first point cloud feature data and the first image feature data respectively according to the preset target classification candidate information to obtain the second point cloud feature data and the second image feature data.

[0054] The data fusion unit is adapted to fuse the second point cloud feature data with the second image feature data to obtain fused data;

[0055] The target recognition unit is adapted to perform target recognition on the fused data to obtain target recognition results.

[0056] The present invention also provides an autonomous driving system, including a point cloud acquisition device, an image acquisition device, and a data processing device, wherein the data processing device is connected to the point cloud acquisition device and the image acquisition device respectively, wherein:

[0057] Point cloud acquisition equipment, suitable for acquiring raw point cloud data;

[0058] Image acquisition equipment, suitable for acquiring raw image data;

[0059] A data processing device is adapted to perform the data processing method described in any of the above embodiments to process the raw point cloud data and raw image data.

[0060] The data processing method of this invention, based on preset target classification candidate information, can filter the first point cloud feature data and the first image feature data respectively, effectively reducing the amount of data in the first point cloud feature data and the first image feature data, while retaining more accurate and useful data, reducing data redundancy, improving data quality and data processing efficiency. Furthermore, because the filtering is based on preset target classification candidate information, the correspondence between the obtained second point cloud feature data and the second image feature data is stronger. When performing target recognition, the deviation between the target identified by the fused data and the real target can be reduced, and the cross-union ratio between the identified target and the real target can be improved. Thus, without using expensive precision devices, fused data with a smaller data volume but higher accuracy can be obtained, thereby improving the accuracy of target recognition results and shortening the target recognition processing time. This can meet the requirements of autonomous driving systems for data accuracy and efficiency, and enhance autonomous driving capabilities. Attached Figure Description

[0061] To more clearly illustrate the technical solution of the present invention, the drawings used in the description of the present invention or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a flowchart of a data processing method according to an embodiment of the present invention;

[0063] Figure 2 This is a flowchart of a method for filtering first point cloud feature data in an embodiment of the present invention;

[0064] Figure 3 This is a flowchart of a method for acquiring second point cloud feature data according to an embodiment of the present invention;

[0065] Figure 4 This is a schematic diagram of an overlap removal process in an embodiment of the present invention;

[0066] Figure 5 This is a flowchart of a method for filtering first image feature data according to an embodiment of the present invention;

[0067] Figure 6 This is a flowchart of a feature extraction method for raw point cloud data in an embodiment of the present invention;

[0068] Figure 7 This is a schematic diagram of the partitioning of raw point cloud data in an embodiment of the present invention;

[0069] Figure 8 This is a schematic diagram of a voxel feature stacking process in an embodiment of the present invention;

[0070] Figure 9 This is a schematic diagram of a point data logic operation process in an embodiment of the present invention;

[0071] Figure 10 This is a schematic diagram of a voxel feature data compression process in an embodiment of the present invention;

[0072] Figure 11 This is a schematic diagram of an anchor point area in an embodiment of the present invention;

[0073] Figure 12 This is a schematic diagram of a region candidate network extracting target features in an embodiment of the present invention;

[0074] Figure 13 This is a flowchart of another data processing method in an embodiment of the present invention;

[0075] Figure 14 This is a schematic diagram of a target recognition result in an embodiment of the present invention;

[0076] Figure 15 This is a schematic diagram of a mobile carrier containing multiple image acquisition devices in an embodiment of the present invention;

[0077] Figure 16 This is a structural block diagram of a data processing device according to an embodiment of the present invention;

[0078] Figure 17 This is a structural block diagram of a data filtering unit in an embodiment of the present invention;

[0079] Figure 18 This is a structural block diagram of an autonomous driving system according to an embodiment of the present invention. Detailed Implementation

[0080] Currently, data fusion methods used for image acquisition devices and point cloud acquisition devices can be divided into pre-fusion methods and post-fusion methods.

[0081] Among them, the pre-fusion method is a method of processing at the raw data level. Specifically, it can include: projecting the raw point cloud data of the point cloud acquisition device onto the raw image data, and amplifying the pixel information of the image data from pure color information (such as red, green and blue information) into fused information of color and depth for data analysis.

[0082] Post-fusion methods are processing methods at the data result level. Specifically, they may include: processing the raw image data acquired by the image acquisition device and the raw point cloud data acquired by the point cloud acquisition device independently to obtain the target perception result, and then combining the target perception results from multiple sensors for further data analysis.

[0083] However, point cloud acquisition devices and image acquisition devices are not in the same mode. For example, the poses of point cloud acquisition devices and image acquisition devices are different; the formats and dimensions of the raw data acquired by point cloud acquisition devices and image acquisition devices are different; the raw point cloud data of point cloud acquisition devices is sparser than the raw image data of image acquisition devices, etc.

[0084] Due to modal differences, it is difficult to guarantee the accuracy of the fused data, whether using pre-fusion or post-fusion methods.

[0085] Therefore, those skilled in the art have applied neural network algorithms to data fusion methods in the hope of improving the fusion effect. However, the structure of neural network models is relatively complex, which reduces the efficiency of data processing. In addition, neural network models require a large amount of training data and a long time for parameter tuning. After increasing the time cost, the robustness of the trained neural network models has not met the expected requirements, making it impossible to adapt to complex and ever-changing autonomous driving scenarios, and even more impossible to ensure that the data fusion results meet the accuracy requirements.

[0086] In summary, current data fusion methods for image acquisition devices and point cloud acquisition devices have some problems, resulting in unsatisfactory data fusion results. The accuracy of target recognition obtained from the fused data is low, which in turn affects the driving decisions of autonomous driving systems.

[0087] To address the aforementioned technical problems, embodiments of the present invention provide a flowchart of a data processing method, as shown below. Figure 1 As shown, the method may include the following steps:

[0088] S11, extract features from the original point cloud data to obtain the first point cloud feature data.

[0089] In practical implementation, a point cloud acquisition device can collect raw point data of objects in the corresponding environment by transmitting and receiving signals. This raw point data may include three-dimensional coordinates and reflectivity. The collection of all raw point data constitutes the raw point cloud data. Specifically, the collection of raw point data acquired by the point cloud acquisition device within one sampling period constitutes one frame of raw point cloud data.

[0090] In practical applications, different point cloud feature extraction algorithms or neural networks can be used to extract features from the original point cloud data, depending on the application scenario and requirements. This embodiment of the invention does not impose any restrictions on this.

[0091] In specific implementations, based on the actual application scenarios and requirements of the embodiments of the present invention, different selection rules for raw point cloud data can be set. For example, the raw point cloud data can be selected frame by frame, thereby performing feature extraction on each frame of raw point cloud data respectively; or, for example, the raw point cloud data can be selected according to a preset frame interval value, thereby performing feature extraction on the raw point cloud data of the corresponding frame respectively. The embodiments of the present invention do not impose any limitations on this.

[0092] S12, extract features from the original image data to obtain the first image feature data.

[0093] In practical applications, different image feature extraction algorithms or neural networks can be used to extract features from the original image data, depending on the application scenario and requirements. This embodiment of the invention does not impose any restrictions on this.

[0094] In specific implementations, different selection rules for raw image data can be set according to the actual application scenarios and needs of the embodiments of the present invention. For example, the raw image data can be selected frame by frame, and feature extraction can be performed on each frame of raw image data. Alternatively, raw image data can be selected according to other selection rules. The embodiments of the present invention do not limit this.

[0095] S13, based on preset target classification candidate information, the first point cloud feature data and the first image feature data are filtered respectively to obtain the second point cloud feature data and the second image feature data.

[0096] In specific implementation, the target classification candidate information can be set according to the actual application scenario. This embodiment of the invention does not impose specific restrictions on the setting of target classification candidate information.

[0097] In one embodiment of the present invention, target classification candidate information can be set based on object category and object orientation, so that the target classification candidate information can characterize object category and object orientation.

[0098] Specifically, object categories can include: movable objects (such as cars, trucks, bicycles, and pedestrians) and relatively stationary objects (such as trees, grass, and signs); object orientation can include: front, left, right, and back views. By arranging and combining object categories and object orientations, corresponding target classification candidate information can be set based on the permutation and combination results.

[0099] Furthermore, in specific implementations, if the errors between the contour dimensions corresponding to multiple viewpoints of an object are within the allowable error range, one of the viewpoints can be selected as the object's orientation. For example, if the errors between the contour dimensions corresponding to the front and back of an object are within the allowable error range, one of the front and back of the object can be selected as the object's orientation; similarly, if the errors between the contour dimensions corresponding to the left and right sides of an object are within the allowable error range, one of the left and right sides of the object can be selected as the object's orientation.

[0100] The allowable error range can be set according to the specific scenario; the left side of the object can be the left-side view of the specified face of the object after taking one of its faces as a reference (such as the front), and the right side of the object can be the right-side view of the specified face of the object after taking one of its faces as a reference (such as the front). This embodiment of the invention does not impose any limitations on these aspects.

[0101] For example, if the object categories include cars and signs, and the object orientations include front and left side, then by arranging and combining the object categories and object orientations, four permutation and combination results can be obtained: front of car, left side of car, front of sign, and left side of sign. Then, target classification candidate information corresponding to the front of car, target classification candidate information corresponding to the left side of car, target classification candidate information corresponding to the front of sign, and target classification candidate information corresponding to the left side of sign can be set.

[0102] It is understood that the above examples are merely illustrative. In practical applications, the object category and object orientation can be selected according to actual needs, and this invention does not impose any restrictions on this.

[0103] S14, the second point cloud feature data and the second image feature data are fused to obtain fused data.

[0104] In a specific implementation, the second point cloud feature data and the second image feature data can be stacked according to a preset direction (such as the height direction), and the result is used as the fused data.

[0105] S15, target recognition is performed on the fused data to obtain target recognition results.

[0106] In practice, different target recognition candidate parameters can be set according to the actual scenario. Based on the preset target recognition candidate parameters, the fused data is used to perform target recognition to obtain the target recognition result.

[0107] The target recognition candidate parameters can be set according to at least one of the following: object category, object position, and object orientation. Specifically, the object category can include at least one of the following: car, truck, bicycle, and pedestrian; the object position can include at least one of the following: length, width, height, three-dimensional coordinates, and angle; and the object orientation can include at least one of the following: front, back, left side, and right side.

[0108] It is understood that the fused data may correspond to one or more objects in the environment; that is, the fused data may contain one or more targets. For each target, a target recognition result matching the target can be obtained based on the pre-set target recognition candidate parameters. This embodiment of the invention does not impose a specific limit on the number of targets.

[0109] In specific implementation, when the data processing method provided in this embodiment of the invention is used to fuse the second point cloud feature data with the second image feature data, a better data fusion effect can be achieved, and the accuracy of the target recognition result can be improved. The intersection-over-union (IoU) between the target identified by the data processing method provided in this embodiment of the invention and the real target can exceed 90%, that is, the deviation between the two is small. Therefore, compared with the existing data fusion methods, the data processing method provided in this embodiment of the invention can better meet the requirements of autonomous driving systems that have accuracy requirements for data fusion and target recognition, and has a wider range of applications.

[0110] As can be seen from the above, by filtering according to the preset target classification candidate information, the first point cloud feature data and the first image feature data can be filtered separately, effectively reducing the amount of data of the first point cloud feature data and the first image feature data, while retaining accurate and useful data, reducing data redundancy, improving data quality and data processing efficiency. Furthermore, because the filtering is based on the preset target classification candidate information, the correspondence between the obtained second point cloud feature data and the second image feature data is stronger. When performing target recognition, the deviation between the identified target and the real target can be reduced, and the cross-union ratio between the identified target and the real target can be improved. Thus, without using expensive precision devices, fused data with a smaller amount of data but higher accuracy can be obtained, thereby improving the accuracy of target recognition results and shortening the target recognition processing time. This can also meet the requirements of autonomous driving systems for data accuracy and efficiency, and enhance autonomous driving capabilities.

[0111] In summary, the above-mentioned solutions can ensure the processing efficiency and accuracy of data fusion at a relatively low implementation cost, thereby obtaining more accurate target identification results.

[0112] It is understood that steps S11 and S12 in the above embodiments are merely illustrative examples and are not intended to limit the feature extraction order of the original point cloud data and the original image data. In other words, in actual operation, there is no necessary order between steps S11 and S12. Steps S11 and S12 can be executed simultaneously or in a preset order. This embodiment of the invention does not impose any restrictions on this.

[0113] In specific implementation, such as Figure 2 The diagram shows a flowchart of a method for filtering first point cloud feature data. The filtering method may include:

[0114] S21, based on preset target classification candidate information, target screening is performed on the first point cloud feature data to obtain three-dimensional target candidate information of the first point cloud feature data.

[0115] In practical implementation, based on preset target classification candidate information, regions in the first point cloud feature data that may be targets can be filtered and labeled to obtain three-dimensional target candidate information of the first point cloud feature data. The three-dimensional target candidate information can characterize the distribution of target classification candidate information in the first point cloud feature data.

[0116] In practical implementation, if multiple target classification candidate information exists, the distribution of each target classification candidate information in the first point cloud feature data may not be the same, thus the number of corresponding three-dimensional target candidate information may also be different. For example, assuming there are target classification candidate information HX1 and target classification candidate information HX2, when the target classification candidate information HX1 is labeled in the first point cloud feature data, three-dimensional target candidate information QY1 and QY2 can be obtained, and when the target classification candidate information HX2 is labeled in the first point cloud feature data, three-dimensional target candidate information QY3 to QY5 can be obtained.

[0117] In practical applications, candidate region extraction algorithms can be used to label the first point cloud feature data, and this embodiment of the invention does not impose any restrictions on this.

[0118] In specific implementations, the target classification candidate information can be represented by candidate boxes; in other words, the target classification candidate information can be target classification candidate boxes. Specifically, since point cloud data is three-dimensional data, the target classification candidate boxes can be three-dimensional bounding boxes. The size of the target classification candidate boxes can be set based on different target classification candidate information. For example, the size of the target classification candidate box corresponding to the front of a car and the size of the target classification candidate box corresponding to the right side of a car can be different. This embodiment of the invention does not impose any limitations on this.

[0119] Based on the preset target classification candidate boxes, the first point cloud feature data can be labeled. The labeling of the target classification candidate boxes in the first point cloud feature data is the distribution of target classification candidate information in the first point cloud feature data, thus obtaining three-dimensional target candidate information.

[0120] In practice, since there is a discrepancy between the size of the target classification candidate box and the object range in the first point cloud feature data, the bounding box regression method can be used to perform operations such as area scaling and position shifting on the target classification candidate box in the first point cloud feature data, so that the target classification candidate box matches the object range in the first point cloud feature data more closely.

[0121] S22, the three-dimensional target candidate information is filtered to obtain optimized target candidate information.

[0122] S23, based on the optimized target candidate information, the first point cloud feature data is filtered to obtain the second point cloud feature data.

[0123] By adopting the above scheme, the first point cloud feature data is filtered to reduce data redundancy, retain more useful feature data, improve data quality, and increase data processing efficiency.

[0124] In practical implementation, the three-dimensional target candidate information can be evaluated, and filtered according to the evaluation results to obtain optimized target candidate information. Specifically, for example... Figure 3 The diagram shown is a flowchart of a method for optimizing the acquisition of target candidate information, which may include the following steps:

[0125] S31, evaluate the three-dimensional target candidate information, and filter the three-dimensional target candidate information based on preset evaluation conditions to obtain intermediate target candidate information.

[0126] In practice, target classification prediction can be performed on the first point cloud feature data corresponding to each three-dimensional target candidate information to obtain the target classification prediction results corresponding to each three-dimensional target candidate information. Then, the target classification prediction results corresponding to each three-dimensional target candidate information can be evaluated through a scoring function.

[0127] The target classification prediction result is determined by preset target classification prediction output parameters, which can be set according to the needs of the actual scenario.

[0128] For example, the target classification prediction output parameters can be set according to at least one of the object category, object position, and object orientation. Specifically, the object category can include at least one of: car, truck, bicycle, and pedestrian; the object position can include at least one of: length, width, height, three-dimensional coordinates, and angle; and the object orientation can include at least one of: front, back, left side, and right side.

[0129] Furthermore, the target classification prediction output parameters corresponding to the object's position can be represented by the position correction amount.

[0130] In practical implementation, neural networks can be used for target classification prediction. For example, a Region Proposal Network (RPN) can be used to perform target classification prediction on the first point cloud feature data.

[0131] In practice, the scoring function can be a nonlinear mapping function, such as the Sigmoid function, tanh function, ReLU function, softmax function, etc. Nonlinear mapping functions can highlight the differences between target classification prediction results.

[0132] Specifically, the target classification prediction results can be evaluated using a nonlinear mapping function, and a corresponding score can be output. Then, it is determined whether the score meets the evaluation conditions, the target classification prediction results corresponding to the scores that meet the evaluation conditions are determined, and the corresponding three-dimensional target candidate information is retained. This achieves the filtering of the three-dimensional target candidate information and obtains intermediate target candidate information.

[0133] It is understood that the evaluation criteria can be set based on the actual scenario. For example, the evaluation criteria can be a score threshold; if the score of the target classification prediction result is greater than the score threshold, then the evaluation criteria are met. The evaluation criteria can also be a ranking threshold; the target classification prediction results are ranked according to their scores, and if the ranking number of the target classification prediction result is less than the ranking threshold, then the evaluation criteria are met. This invention does not impose any limitations on this.

[0134] S32, the intermediate target candidate information is used as the optimized target candidate information.

[0135] Therefore, the scores obtained by evaluating the three-dimensional target candidate information can effectively filter the amount of three-dimensional target candidate information data and improve data processing efficiency.

[0136] In practice, there may be at least partial overlap in intermediate target candidate information, which may correspond to the same object in the environment. For example, such as Figure 4 As shown, an object 4A in the environment is represented by a black circle, and there are three partially overlapping intermediate target candidate information 411, 412 and 413.

[0137] Therefore, the intermediate target candidate information can be further optimized to reduce data redundancy. Specifically, continue to refer to... Figure 3 Before using the intermediate target candidate information as the optimization target candidate information, the method may further include:

[0138] S33, perform de-overlap processing on the intermediate target candidate information.

[0139] Continue to refer to Figure 4 The three partially overlapping intermediate target candidate information 411, 412, and 413 are de-overlapped, and intermediate target candidate information 412 is selected as the de-overlapped intermediate target candidate information. A non-maximum suppression method can be used to perform the de-overlapping processing on the intermediate target candidate information.

[0140] In practice, the field of view of the point cloud acquisition device and the image acquisition device may not be consistent. Therefore, the intermediate target candidate information can be further optimized to reduce data redundancy and continue to be referenced. Figure 3 Before performing the de-overlapping process on the intermediate target candidate information, the method may further include:

[0141] S34, match the intermediate target candidate information with the image range corresponding to the original image data, and filter the intermediate target candidate information based on preset matching conditions.

[0142] The image range is determined based on the length and width information of the original image data, which can be determined by the number of pixels of the image acquisition device.

[0143] Therefore, filtering the intermediate target candidate information and then performing deduplication can avoid misjudgment and incorrect deduplication, thereby improving data accuracy.

[0144] In practical applications, although the image acquisition device and the point cloud acquisition device are installed on the same mobile carrier, such as an unmanned vehicle, a car, or a handheld device, their positions and orientations on the mobile carrier may be different. To obtain the correspondence between them, the image acquisition device and the point cloud acquisition device can be jointly calibrated to determine the coordinate transformation parameters between them. Then, based on the coordinate transformation parameters, the intermediate target candidate information after the de-overlap processing can be transformed into the pixel coordinate system of the original image data and matched with the image range of the original image data.

[0145] Specifically, the coordinate transformation parameters can include extrinsic and intrinsic parameters. The extrinsic parameters, obtained through joint calibration, can project the intermediate target candidate information onto the coordinate system of the image acquisition device. Then, using the intrinsic parameters of the image acquisition device, the coordinate system of the intermediate target candidate information is transformed from the image acquisition device's coordinate system to the pixel coordinate system, resulting in two-dimensional intermediate target candidate information. This two-dimensional intermediate target candidate information can then be matched with the image range of the original image data.

[0146] The coordinate system of the image acquisition device can be the coordinate system formed by the corresponding field of view of the image acquisition device.

[0147] Based on the length and width information of the image data and the coordinates of the two-dimensional intermediate target candidate information, the portions of the intermediate target candidate information projected outside the image data range and the portions projected within the image data range can be determined.

[0148] It is understood that the matching conditions can be set based on the actual scenario. For example, the matching conditions could be: retaining the portion of the intermediate target candidate information projected within the image data range, and filtering the portion of the intermediate target candidate information projected outside the image data range, thereby effectively reducing the amount of data. This invention does not impose any limitations on this.

[0149] By adopting the above scheme, the raw image data is used to assist in the filtering of 3D target candidate information, avoiding too many target classification candidate information for a single object, reducing data redundancy, obtaining more accurate information, and the image acquisition equipment has a lower hardware cost compared with other precision devices, thus saving implementation costs.

[0150] In practical implementation, if the target classification candidate information is a three-dimensional bounding box, then the intermediate target candidate information converted to two-dimensional coordinates is a two-dimensional planar bounding box. Two-dimensional planar bounding boxes that are too small contain limited useful information, thus increasing the data volume. Therefore, based on the size of the two-dimensional planar bounding box and a preset size threshold, two-dimensional planar bounding boxes that do not meet the size threshold can be filtered out, thereby filtering the two-dimensional intermediate target candidate information and reducing the data volume.

[0151] In specific implementation, after obtaining the optimized target candidate information through any of the above embodiments or in combination with the above embodiments, the features of the region corresponding to the optimized target candidate information in the first point cloud feature data can be extracted to obtain the second point cloud feature data.

[0152] In specific implementation, such as Figure 5 The diagram shows a flowchart of a method for filtering first image feature data. The filtering method may include:

[0153] S51, based on preset target classification candidate information, target screening is performed on the first point cloud feature data to obtain three-dimensional target candidate information of the first point cloud feature data.

[0154] S52, the three-dimensional target candidate information is filtered to obtain optimized target candidate information.

[0155] It is understandable that steps S51 and S52 can be referred to the appendix. Figure 2 The description of related content will not be repeated here.

[0156] S53, the coordinate system of the optimized target candidate information is transformed to the pixel coordinate system of the original image data to obtain two-dimensional target candidate information.

[0157] In practice, based on the coordinate transformation parameters obtained from the joint calibration, the optimized target candidate information can be transformed into the pixel coordinate system of the original image data to obtain two-dimensional target candidate information. For details, please refer to the above description of the relevant coordinate transformations; further elaboration will not be repeated here.

[0158] S54, based on the two-dimensional target candidate information, the first image feature data is filtered to obtain the second image feature data.

[0159] In a specific implementation, after determining the corresponding region of the two-dimensional target candidate information in the first image feature data, the corresponding features can be extracted to obtain the second point cloud feature data.

[0160] Specifically, depending on the actual needs and application scenarios, an appropriate neural network can be selected to extract region features from the first image feature data. For example, a ROI Align (Region of Interest Alignment) neural network or a RoIPooling (Region of Interest Pooling) neural network can be used to extract region features from the first image feature data.

[0161] Among them, the ROI Align neural network implements a region feature extraction method based on quadratic interpolation. This method calibrates the feature position when extracting region features, which can avoid extracting features in incorrect positions.

[0162] By adopting the above scheme, the first image feature data is filtered by optimizing the target candidate information, reducing data redundancy, and enhancing the correlation between the first image feature data and the first point cloud feature data, which is beneficial to subsequent data fusion.

[0163] In practice, before fusing the second point cloud feature data with the second image feature data, the second point cloud feature data and the second image feature data can be input into a fully connected layer to ensure that the data dimensions are consistent, thereby improving the cross-union ratio of the data.

[0164] In practice, in order to reduce the amount of data processing and the complexity of the data process, the original point cloud data can be spatially divided according to a preset division method, and feature extraction can be performed on the point data within the spatial range respectively, thereby reasonably reducing the amount of data in different spatial ranges and ensuring data validity.

[0165] To enable those skilled in the art to better understand and implement the feature extraction process of the original point cloud data of the present invention, the following detailed description is provided with reference to the accompanying drawings and specific embodiments.

[0166] In one embodiment of the present invention, such as Figure 6 The diagram shows a flowchart of a feature extraction method for raw point cloud data. The method may include:

[0167] S61, the original point cloud data is divided to obtain multiple voxel units.

[0168] In a specific implementation, preset voxel grid size parameters are obtained, and the original point cloud data is spatially divided according to the voxel grid size parameters, so as to divide the original point cloud data into adjacent voxel grids.

[0169] In this embodiment, a voxel unit can be considered as a cubic space. The size parameters of the voxel unit include: voxel depth, voxel height, and voxel width. The values ​​of voxel unit depth, voxel unit height, and voxel unit width can be the same. For example, if the voxel unit depth, voxel unit height, and voxel unit width are all 0.1m, the resulting voxel unit is a cube. The values ​​of voxel unit depth, voxel unit height, and voxel unit width can also be different. For example, if the voxel unit depth, voxel unit height, and voxel unit width are 0.1m, 0.2m, and 0.3m respectively, the resulting voxel unit is a cuboid. This embodiment of the invention does not impose any restrictions on these values.

[0170] In an optional example, such as Figure 7 The image shown is a schematic diagram illustrating the partitioning of raw point cloud data. Figure 7 In this context, the original point cloud data is contained in a three-dimensional space P1, and the preset voxel unit size parameters are: voxel unit depth vD, voxel unit height vH, and voxel unit width vW. The original point cloud data is divided according to the voxel unit size parameters to obtain the following... Figure 7 The multiple voxel units shown have depth, height, and width of vD, vH, and vW, respectively. For details, please refer to [reference needed]. Figure 7 The middle voxel unit P11 is shown.

[0171] In practice, the more finely the raw point cloud data is divided, i.e., the smaller the voxel units, the higher the voxelization accuracy of the obtained raw point cloud data, and correspondingly, the amount of data that needs to be processed will also increase. During autonomous driving, nearby objects have a greater impact on driving decisions; therefore, point data of nearby objects collected by the point cloud acquisition device are more important, while point data of distant objects are less important.

[0172] Therefore, to reduce the amount of data, the original point cloud data can be filtered based on preset spatial distance parameters. Specifically, a distance selection space is established with a certain point as the center, and the original point data within the distance selection space is retained while the original point data outside the distance selection space is removed. The spatial distance parameters can include: spatial depth distance parameters, spatial height distance parameters, and spatial width distance parameters.

[0173] Optionally, a distance selection space can be established centered on the point cloud acquisition device, retaining the original point data within the distance selection space while removing the original point data outside the distance selection space.

[0174] Therefore, by using preset spatial distance parameters, more important raw point data (i.e., raw point data that are closer in distance) is retained, while less important raw point cloud data (i.e., raw point data that are farther away) is filtered out, thereby reducing the amount of raw point cloud data, which in turn reduces the number of voxel units and ensures data validity.

[0175] It is understandable that the spatial distance parameter is related to the characteristics of the point cloud acquisition device and the application scenario. For example, for urban scenarios and desert scenarios, since there are usually more objects in urban scenarios than in desert scenarios, the spatial distance parameter set for urban scenarios can be smaller than that set for desert scenarios in order to distinguish between distant and near objects. As another example, for the top and middle positions of mobile loading platforms (such as vehicles, airplanes, and other mobile tools), since there are usually fewer objects at the top position than at the middle position, the spatial distance parameter set when the point cloud acquisition device is installed in the middle position can be smaller than that set when the point cloud acquisition device is installed in the top position in order to distinguish between distant and near objects.

[0176] S62, feature extraction is performed on the multiple voxel units to obtain voxel feature data.

[0177] In specific implementations, local feature extraction can be performed on each of the multiple voxel units to obtain the voxel feature data. The local feature extraction method can be selected according to actual needs. For example, voxel feature stacking can be performed on each of the multiple voxel units to obtain the voxel feature data; or, point data logical operations can be performed on each of the multiple voxel units to obtain the voxel feature data. This specification does not limit this approach; several embodiments are provided below for illustration.

[0178] In one implementable embodiment, such as Figure 8 The diagram shown illustrates a voxel feature stacking process. Taking a single voxel unit PA as an example, the voxel unit PA includes three original point data points A1, A2, and A3. Figure 8 The three original data points A1, A2, and A3 are represented by colors of different gray levels.

[0179] (1) Calculate the geometric center point A0 between all the original point cloud data A1, A2 and A3 in the voxel unit PA, perform logical operations and splicing on each original point data and the geometric center position to obtain the amplified point data A1', A2' and A3'.

[0180] For example, the original point data A1 is [x, y, z, r], where x, y, z represent the three-dimensional coordinates, and r represents the reflectivity. The geometric center point A0 between the original point cloud data A1, A2, and A3 is [x', y', z'], where x', y', and z' represent the three-dimensional coordinates of the geometric center point. After logical operations and concatenation, the amplified point data A1' is obtained as [x, y, z, x-x', y-y', z-z', r]. Similarly, the amplified point data A2' and A3' obtained from the original point data A2 and A3 after logical operations and concatenation are not described further here.

[0181] For example, the original point data A1 is [x, y, z, r], where x, y, z represent the three-dimensional coordinates, and r represents the reflectivity. The geometric center point A0 between the original point cloud data A1, A2, and A3 is [x', y', z', r'], where x', y', z' represent the three-dimensional coordinates of the geometric center point, and r' is the reflectivity of the geometric center point. After logical operations and concatenation, the amplified point data A1' is obtained as [x, y, z, x-x', y-y', z-z', r-r']. Similarly, the amplified point data A2' and A3' obtained from the original point data A2 and A3 after logical operations and concatenation are not elaborated further here.

[0182] (2) The amplified point data A1', A2', and A3' are input into the first fully connected layer to obtain fully connected features B1, B2, and B3. In the first fully connected layer, each amplified point data is mapped to a feature space, and the amplified point data is further expanded through the first fully connected layer. The first fully connected layer may contain a first linear mapping sublayer, a first batch normalization (BN) sublayer, and a first nonlinear mapping sublayer. The first nonlinear mapping sublayer may use ReLU function, tanh function, etc. for nonlinear mapping.

[0183] (3) Fully connected features B1, B2 and B3 are aggregated element by element through the first pooling layer, that is, elements in the same feature dimension of fully connected features B1, B2 and B3 are aggregated to obtain local cumulative feature C1.

[0184] The first pooling layer processes the fully connected features point by point, that is, it performs max pooling extraction on each fully connected feature B1, B2 and B3 to extract the maximum value of each fully connected feature B1, B2 and B3 to characterize the feature information of the original point data in the voxel unit. For example, if the fully connected feature B1 is [2,8,3], then the maximum value obtained by max pooling is 8.

[0185] (4) Input the local cumulative feature C1 and the fully connected features B1, B2 and B3 into the splicing layer, and splice the local cumulative feature C1 with the fully connected features B1, B2 and B3 respectively to obtain splicing features D1, D2 and D3, so as to retain the original information of the original point data while reflecting the local maximum value information of the original point data, and increase the dimension and diversity of the output data.

[0186] (5) Input the splicing features D1, D2 and D3 into the second fully connected layer and the second pooling layer to obtain voxel feature data E.

[0187] It is understood that the above embodiments are only illustrative. In actual applications, there may be multiple voxel units, and the voxel feature data is obtained after multiple voxel units are stacked through voxel feature stacking.

[0188] By adopting the above-mentioned voxel feature stacking scheme, high-precision voxel feature data can be obtained, thereby ensuring the accuracy of subsequent fusion data and target recognition results.

[0189] In another feasible embodiment, such as Figure 9 The diagram shown illustrates a point data logic operation process. Taking a voxel unit PA as an example, the voxel unit PA includes three original point data points A1, A2, and A3. Figure 9 The three original data points A1, A2, and A3 are represented by colors of different gray levels.

[0190] Logical operations (such as mean operation, median operation, variance operation, etc.) are performed on the coordinates of each axis of the three original point data A1, A2 and A3 in the voxel unit PA to obtain the coordinates of a feature point, which is used as the voxel feature data F of the voxel unit PA.

[0191] It is understood that the above embodiments are for illustrative purposes only. In actual applications, there may be multiple voxel units, and the voxel feature data is obtained after point data logic operations are performed on the multiple voxel units.

[0192] Compared with the voxel feature stacking scheme, the above-mentioned logical operation scheme has a significantly reduced computational load, and the accuracy of the data obtained by voxel feature stacking is not much different from that of the data obtained by logical operation, so they can be regarded as equivalent. If the autonomous driving scenario has high requirements for real-time performance, the point data logical operation scheme has more advantages.

[0193] S63, the voxel feature data is compressed to obtain the first point cloud feature data.

[0194] In a specific implementation, the voxel feature data is compressed along a specified direction using a convolutional neural network block. The specified direction may include at least one of the following: height, width, and depth. The compression process may include: extracting features from the voxel feature data in the specified direction (e.g., the height direction) and filtering the voxel feature data.

[0195] Considering the sparsity of the original point cloud data, using ordinary convolutional layers would consume a significant amount of time, reducing computation speed. Therefore, the convolutional neural network block can include sparse convolution layers. Sparse convolution restricts convolution computation to occur only in regions with corresponding input (i.e., regions containing original point data), ignoring the large number of regions in the original point cloud data without original point data, thus reducing computational load.

[0196] In specific implementations, to limit the expansion of the original point data and further reduce the computational load, the convolutional neural network block may include sparse convolutional layers and submanifold convolutional layers. The number of submanifold convolutional layers may be greater than one, meaning the convolutional neural network block includes at least one submanifold convolutional layer.

[0197] Therefore, based on sparse convolution, the convolution calculation is restricted to occur only in the region that corresponds one-to-one with the preset output position, thereby limiting the expansion of the original point data and improving computational efficiency.

[0198] In practice, multiple convolutional neural network blocks can be set up to gradually extract the features of the voxel feature data in a specified direction and filter the voxel feature data, thereby avoiding the loss of useful information and effectively reducing the amount of data of the first point cloud feature data.

[0199] The region formed in a non-specified direction can be called the anchor point region. The first point cloud feature data includes: feature data of the anchor point region and feature data of the specified direction.

[0200] For example, such as Figure 10 The diagram illustrates a voxel feature data compression process. Taking a voxel feature as an example, voxel feature Q1 is input to the first convolutional neural network block 1, the output of convolutional neural network block 1 is input to the second convolutional neural network block 2, and so on, up to the nth convolutional neural network block N. The height feature information is extracted step by step and the height data is removed to obtain the first point cloud feature data Q2, which includes the feature Q21 of the anchor point region formed by the width and depth and the height feature Q22.

[0201] The voxel feature Q1 can be obtained by stacking voxel features to obtain voxel feature E (see the relevant description of voxel feature E for details), or by performing point data logical operations to obtain voxel feature F (see the relevant description of voxel feature F for details). This invention does not impose any limitations on this.

[0202] After the voxel features of multiple voxel units are compressed, the resulting feature set of the anchor point region and the feature set of the specified direction constitute the voxel feature data. The feature data of the anchor point region can be considered a two-dimensional feature map with the specified direction as the viewpoint. For example, if the specified direction is height, the feature data of the anchor point region can be considered a two-dimensional top view.

[0203] In practical implementation, features can be extracted from the original image data using a deep neural network to obtain the first image feature data. For example, a Residual Neural Network (ResNet) can be used for deep learning to obtain feature maps. Furthermore, two-dimensional convolutional layers can replace the fully connected layers and nonlinear mapping layers used for output in the ResNet network, thus making the two-dimensional convolutional layers the output layers of the ResNet network.

[0204] In specific implementation, the first point cloud feature data can be filtered according to the anchor point area. To enable those skilled in the art to better understand and implement the filtering process of the first point cloud feature data, the following detailed description is provided through specific embodiments.

[0205] In specific implementation, the first point cloud feature data can be divided according to the anchor point region, and based on the preset target classification candidate information, the first point cloud feature data can be labeled according to the anchor point region to obtain three-dimensional target candidate information.

[0206] For example, refer to Figure 11 First point cloud feature data ( Figure 11 (Not shown in the image) Corresponding to four anchor point regions MD1 to MD4, the first point cloud feature data can be divided into four parts. Based on the preset target classification candidate information HXA and HXB, the first point cloud feature set of anchor point region MD1 is labeled to obtain the 3D target candidate information ZQY1 and ZQY2 corresponding to the target classification candidate information HXA. Similarly, partial first point cloud feature data of the other three anchor point regions can be labeled to achieve the labeling of target classification candidate information for the first point cloud feature data, thus obtaining the 3D target candidate information of the first point cloud feature data.

[0207] It is understood that the above figures are for illustrative purposes only. In actual implementation, target classification candidate information can be represented by a three-dimensional bounding box, and the obtained three-dimensional target candidate information can also be represented by a three-dimensional bounding box. The three-dimensional target candidate information can be a three-dimensional target candidate bounding box.

[0208] Optionally, bounding box regression can be performed on the 3D target candidate boxes to make the 3D target candidate boxes more closely match the actual object range.

[0209] In practice, according to the anchor point region, a neural network can be used to perform target classification prediction on the first point cloud feature data corresponding to each three-dimensional target candidate information, so as to obtain the target classification prediction results of each three-dimensional target candidate information.

[0210] Specifically, such as Figure 12 The diagram illustrates a region candidate network for target feature extraction. Taking a 3D target candidate within an anchor region as an example, the first point cloud feature data corresponding to the 3D target candidate is determined. In the region candidate network, n convolutional neural network blocks (Convbloc) are used to extract the features of the point data. Each convolutional neural network block includes convolutional layers, normalization layers, and nonlinear operation layers. Then, each extracted feature is processed by a deconvolutional neural network (Deconv) to obtain stacked features with consistent dimensions. These stacked features are then processed by a function simulator, such as a 1*1 2D convolutional neural network (Conv2D), and based on preset target classification prediction output parameters, m target classification prediction results can be obtained, such as... Figure 12 The target classification prediction results shown are 1 to m. Similarly, based on the anchor point region, the target classification prediction results corresponding to all 3D target candidate information can be obtained.

[0211] The three-dimensional target candidate information is evaluated by a scoring function. For each anchor point region, only the three-dimensional target candidate information corresponding to the highest score is retained. Then, all target classification prediction results are sorted by score, and the three-dimensional target candidate information corresponding to the top k scores is retained to obtain intermediate target candidate information.

[0212] Furthermore, the intermediate target candidate information can be matched with the image range corresponding to the original image data, and the intermediate target candidate information can be filtered based on preset matching conditions; and the intermediate target candidate information can be de-overlapped. For details, please refer to the descriptions in the relevant sections above, which will not be repeated here.

[0213] To facilitate understanding by those skilled in the art, the following example uses actual figures. For instance, suppose there are 8 target classification candidate information, the feature data (i.e., the two-dimensional feature map) of the anchor region is 124*176, the anchor region is 1*1, and there are a total of 124*176 anchor regions; based on the 8 target classification candidate information, each anchor region is labeled, resulting in 124*176*8 = 174592 three-dimensional target candidate information; after obtaining the target classification prediction results of the three-dimensional target candidate information, the target classification prediction results of these 174592 three-dimensional target candidate information are evaluated according to the scoring function. For each anchor region, only the three-dimensional target candidate information with the highest score is retained, that is, only 1 / 8 of the data is retained, leaving 21824 three-dimensional target candidate information. Then, based on the scores, the top 3000 scores from largest to smallest are selected as the candidate information for three-dimensional targets. This reduces the number of candidate information for three-dimensional targets from 174,592 to 3,000, achieving efficient filtering of the candidate information for three-dimensional targets, greatly reducing the amount of data and improving the efficiency of subsequent processing.

[0214] In addition, the intermediate target candidate information can be matched with the image range corresponding to the original image data, and the intermediate target candidate information can be filtered based on preset matching conditions. Furthermore, the intermediate target candidate information can be de-overlapped, which can also reduce the amount of data, for example, reducing the number of three-dimensional target candidate information from 3000 to 200.

[0215] The intermediate target candidate information obtained after the above optimization process is used as the optimized target candidate information. Then, based on the optimized target candidate information, the first point cloud feature data is filtered to obtain the second point cloud feature data. The coordinate system of the optimized target candidate information is transformed to the pixel coordinate system of the original image data to obtain two-dimensional target candidate information. Based on the two-dimensional target candidate information, the first image feature data is filtered to obtain the second image feature data.

[0216] In specific implementation, such as Figure 13 The diagram shown is a flowchart of another data processing method, where steps SA to SE can be referenced. Figure 1 And related descriptions, which will not be repeated here. Figure 1 Compared to the method shown, Figure 13 In the method shown, before filtering the first point cloud feature data and the first image feature data based on preset target classification candidate information, the method further includes: step SF, determining whether there is any abnormality in the first image feature data; if the determination result is negative, that is, determining that there is no abnormality in the first image feature data, and continuing to step SC.

[0217] The abnormal conditions of the first image feature data can be set according to actual needs and application scenarios. For example, the abnormal conditions of the first image feature data can be: frame drops in the original image data, or the time difference between the original image data and the original point cloud image is greater than a preset time threshold. This invention does not limit these conditions.

[0218] In practice, fused data can be used for more complex data analysis tasks, such as target recognition. However, the results obtained through data fusion using existing methods are unsatisfactory, which also affects the subsequent target recognition results.

[0219] The fused data obtained by the embodiments of the present invention has higher accuracy and less data volume, which can improve data processing efficiency and meet the needs of autonomous driving systems for data accuracy and efficient processing. Even in dynamic and complex driving scenarios, it can ensure data processing efficiency and target recognition quality. Therefore, the data processing method provided by the embodiments of the present invention is more adaptable and conducive to the realization of autonomous driving.

[0220] In practical implementation, a fully connected neural network can be used to perform target recognition on the fused data to obtain the target recognition result. Because fully connected neural networks have good robustness, are applicable to more scenarios, and can handle data from various dimensions, they are beneficial for data analysis.

[0221] In practical implementation, after target recognition is performed on the fused data, the target recognition results can be visualized, and the recognized targets can be represented by three-dimensional bounding boxes. Furthermore, for the same target, different target recognition results can be represented by different three-dimensional bounding boxes, for example... Figure 14 The diagram shown illustrates one target recognition result, where the black three-dimensional frame 14A represents one target recognition result, and the gray three-dimensional frame 14B represents another target recognition result. This specification does not limit the scope of the embodiments.

[0222] In specific implementation, since the data processing method provided by the embodiments of the present invention can obtain first point cloud feature data with high accuracy, even if there are abnormalities in the first image feature data, subsequent target recognition tasks can still be carried out through the first point cloud feature data.

[0223] For example, continue to refer to Figure 13 For step SF, if the judgment result is yes, that is, it is determined that there is an abnormality in the first image feature data, then continue to step SG, perform target recognition on the first point cloud feature data, and obtain the target recognition result.

[0224] As can be seen from the above, the data processing method provided by the present invention has better robustness. Even if there are anomalies in the first image feature data, good data analysis results can be obtained using only the first point cloud feature data.

[0225] In specific implementation, the data processing method provided in the embodiments of the present invention can be implemented by a target recognition model, wherein the target recognition model includes various neural networks for implementing the data processing method of the embodiments of the present invention, and are connected according to corresponding logic.

[0226] Before using an object recognition model, it is necessary to train the model and adjust its parameters to make it converge. However, object recognition models involve a large number of parameters, requiring a large amount of training data for repeated parameter tuning. In order to control the training time of the model, the training data must be filtered, resulting in a target recognition model with weak robustness that cannot adapt to complex and ever-changing autonomous driving scenarios.

[0227] For example, an image acquisition device can change its pose on a mobile carrier as needed, and more than one image acquisition device may be installed on the mobile carrier, such as... Figure 15 As shown, image acquisition devices Y11, Y12, and Y13 are respectively installed at three locations on the mobile carrier Y1. If the pose of image acquisition devices Y11, Y12, and Y13 differs from the pose of the image acquisition devices during training, the trained target recognition model may fail to output accurate target recognition results.

[0228] Since the data processing method provided in this embodiment of the invention can reduce the amount of data in the model, training data from more scenarios can be used to train the target detection model.

[0229] Specifically, before inputting the original image data and original point cloud data into the initial target recognition model, the target recognition model is trained using preset training data, which includes: training point cloud data, training image data acquired from different poses and corresponding image acquisition device parameters, and reference recognition results.

[0230] Then, the error between the target recognition result and the reference target recognition result can be calculated using a preset loss function. When the error value is greater than the error threshold, the parameters of the target recognition model can be adjusted and trained again using training data. When the error value is less than the error threshold, the target recognition model has completed training.

[0231] The parameters of the target recognition model can be adjusted using gradient descent or backpropagation methods.

[0232] By adopting the above scheme, the amount of training data is increased, thereby enhancing the robustness of the model. During the training process, training image data collected by image acquisition devices in different poses can be randomly selected from the training dataset, so that the trained model can adapt to image acquisition devices in any pose during actual driving.

[0233] In practice, the acquisition frequencies of raw point cloud data and raw image data may differ, resulting in inconsistent acquisition time information between them. To facilitate the management of raw image data, raw point cloud data, and their processed data, the same timestamp can be used to mark raw image data and raw point cloud data within a preset time range, thereby obtaining time-synchronized raw image data and raw point cloud data, which ensures time synchronization between the first image feature data and the first point cloud feature data.

[0234] It should be noted that since good data analysis results can be obtained even with only the first point cloud feature data, the data processing method provided in this embodiment of the invention does not specifically limit whether the original point cloud data and the original image data used for feature extraction are time-synchronized.

[0235] It is understood that the above description provides multiple embodiment solutions provided by the embodiments of the present invention. The optional methods described in each embodiment solution can be combined with each other and cross-referenced without conflict, thereby extending to a variety of possible embodiment solutions. These can all be considered as the embodiment solutions disclosed and published by the present invention.

[0236] This invention also provides a data processing device corresponding to the above-described data processing method. The following detailed description, with reference to the accompanying drawings, will illustrate specific embodiments. It should be understood that the data processing device described below can be considered as a functional module required to implement the data processing method provided in this invention. In practical applications, the units and modules of the data processing device can be implemented through hardware, software, or a combination of both. The content of the data processing device described below can be referenced in correspondence with the content of the data processing method described above.

[0237] Reference Figure 16 The diagram shown is a structural block diagram of a data processing device according to an embodiment of the present invention. In this embodiment, the data processing device M1 is connected to an image acquisition device and a point cloud acquisition device. The data processing device M1 may include:

[0238] The data acquisition unit M11 is adapted to acquire the raw image data of the image acquisition device and the raw point cloud data of the point cloud acquisition device;

[0239] The feature extraction unit M12 is adapted to extract features from the original point cloud data to obtain the first point cloud feature data, and to extract features from the original image data to obtain the first image feature data.

[0240] The data filtering unit M13 is adapted to filter the first point cloud feature data and the first image feature data respectively according to the preset target classification candidate information to obtain the second point cloud feature data and the second image feature data.

[0241] The data fusion unit M14 is adapted to fuse the second point cloud feature data with the second image feature data to obtain fused data.

[0242] The target recognition unit M15 is adapted to perform target recognition on the fused data to obtain target recognition results.

[0243] By adopting the above scheme, based on the preset target classification candidate information, the first point cloud feature data and the first image feature data can be filtered respectively, effectively reducing the amount of data of the first point cloud feature data and the first image feature data, while retaining accurate and useful data, reducing data redundancy, improving data quality and data processing efficiency. Furthermore, since the filtering is based on the preset target classification candidate information, the correspondence between the obtained second point cloud feature data and the second image feature data is stronger. When performing target recognition, the deviation between the target identified by the fused data and the real target can be reduced, and the cross-union ratio between the target identified by the fused data and the real target can be improved. Thus, without using expensive precision devices, fused data with a smaller amount of data but higher accuracy can be obtained, thereby improving the accuracy of target recognition results and shortening the target recognition processing time. It can also meet the requirements of autonomous driving systems for data accuracy and efficiency, and enhance autonomous driving capabilities.

[0244] In summary, the above-mentioned solutions can ensure the processing efficiency and accuracy of data fusion at a relatively low implementation cost, thereby obtaining more accurate target identification results.

[0245] In specific implementation, such as Figure 17 As shown, the data filtering unit M13 may include:

[0246] The information acquisition subunit M131 is adapted to perform target filtering on the first point cloud feature data according to preset target classification candidate information, so as to obtain the three-dimensional target candidate information of the first point cloud feature data.

[0247] The information filtering subunit M132 is adapted to filter the three-dimensional target candidate information to obtain optimized target candidate information;

[0248] The filtering subunit M133 is adapted to filter the first point cloud feature data according to the optimization target candidate information to obtain the second point cloud feature data.

[0249] In specific implementation, such as Figure 17 As shown, the data filtering unit M13 may further include: a coordinate transformation subunit M134, adapted to transform the coordinate system of the optimized target candidate information to the pixel coordinate system of the original image data to obtain two-dimensional target candidate information;

[0250] The filtering subunit M133 can filter the first image feature data according to the two-dimensional target candidate information to obtain the second image feature data.

[0251] In specific implementation, such as Figure 17 As shown, the information filtering subunit M132 may include:

[0252] The first filtering module M1321 is adapted to evaluate the three-dimensional target candidate information and filter the three-dimensional target candidate information based on preset evaluation conditions to obtain intermediate target candidate information.

[0253] The information acquisition module M1322 is adapted to acquire the intermediate target candidate information as the optimized target candidate information.

[0254] In specific implementation, such as Figure 17 As shown, the information filtering subunit M132 may further include: a second filtering module M1323, located between the first filtering module M1321 and the information acquisition module M1322, wherein the second filtering module M1323 is adapted to perform de-overlap processing on the intermediate target candidate information.

[0255] In specific implementation, such as Figure 17 As shown, the information filtering subunit M132 may further include: a third filtering module M1324, located between the first filtering module M1321 and the second filtering module M1323. The third filtering module M1324 is adapted to match the intermediate target candidate information with the image range corresponding to the original image data, and to filter the intermediate target candidate information based on preset matching conditions.

[0256] Furthermore, such as Figure 17 As shown, the third filtering module M1324 is adapted to convert the intermediate target candidate information to the pixel coordinate system of the original image data, match it with the image range corresponding to the original image data, and filter the intermediate target candidate information based on preset matching conditions.

[0257] In specific implementation, continue to refer toFigure 16 The data acquisition unit M11 may include:

[0258] The voxel division subunit M111 is suitable for dividing the original point cloud data into multiple voxel units;

[0259] The feature extraction subunit M112 is adapted to extract features from the plurality of voxel units to obtain voxel feature data;

[0260] The data compression subunit M113 is adapted to compress the voxel feature data to obtain the first point cloud feature data.

[0261] In specific implementation, such as Figure 16 As shown, the data processing device M1 may further include:

[0262] The data anomaly judgment unit M16 is located between the feature extraction unit M12 and the data filtering unit M13, and is connected to the target recognition unit M15. It is suitable for judging whether there is an anomaly in the first image feature data.

[0263] The data filtering unit M13 is adapted to filter the first point cloud feature data and the first image feature data respectively according to preset target classification candidate information after the data anomaly judgment unit M16 determines that there is no anomaly in the first image feature data.

[0264] The target recognition unit M15 is adapted to perform target recognition on the first point cloud feature data after the data anomaly judgment unit M16 determines that there is an anomaly in the first image feature data, and obtain the target recognition result.

[0265] This invention also provides an autonomous driving system corresponding to the above-described data processing method. The following detailed description, with reference to the accompanying drawings, uses specific embodiments. It should be noted that in practical applications, the devices in the autonomous driving system described below can be implemented using hardware, software, or a combination of both. The content of the autonomous driving system described below can be referenced in conjunction with the content of the data processing method described above.

[0266] Reference Figure 18 The diagram shown is a structural block diagram of an autonomous driving system according to an embodiment of the present invention. In this embodiment, the autonomous driving system M3 may include: a point cloud acquisition device M31, an image acquisition device M32, and a data processing device M33 placed on a mobile carrier. The data processing device M33 may be connected to the point cloud acquisition device M31 and the image acquisition device M32 respectively.

[0267] The M31 point cloud acquisition device is suitable for acquiring raw point cloud data.

[0268] The M32 image acquisition device is suitable for acquiring raw image data;

[0269] The data processing device M33 is adapted to perform the data processing method provided in any of the above embodiments to process the raw point cloud data and raw image data.

[0270] The image acquisition device may include at least one of a digital camera, an infrared camera, and a thermal imaging camera. The point cloud acquisition device may include at least one of a lidar and a millimeter-wave radar.

[0271] It is understood that the connection between the data processing device and the image acquisition device can be wired or wireless. Accordingly, the data transmission between the two can be carried out through cables or wireless communication. In addition, the data processing device can directly interact with the image acquisition device, or it can indirectly interact with the image acquisition device through a communication relay platform (such as a switch). This embodiment of the invention does not limit this.

[0272] Similarly, the connection between the data processing device and the point cloud acquisition device can be wired or wireless. Correspondingly, the data transmission between the two can be carried out through cables or wireless communication. In addition, the data processing device can directly interact with the point cloud acquisition device, or it can indirectly interact with the data through a communication relay platform (such as a switch). This embodiment of the invention does not limit this.

[0273] In a specific implementation, the data processing device may include a memory and a processor. The memory may store one or more computer-executable instructions, and the processor may invoke the one or more computer-executable instructions to execute the steps of the method provided in the embodiments of the present invention.

[0274] This invention also provides a computer-readable storage medium storing computer instructions, which, when executed, can perform the steps of the methods described in any of the above embodiments of this invention. The computer-readable storage medium can be any suitable readable storage medium such as an optical disc, a hard disk drive, or a solid-state drive. The instructions stored on the computer-readable storage medium execute the methods described in any of the above embodiments; specific details can be found in the above embodiments and will not be repeated here.

[0275] The computer-readable storage medium may include, for example, any suitable type of memory cell, memory device, memory article, memory medium, storage device, storage article, storage medium and / or storage cell, such as memory, removable or non-removable medium, erasable or non-erasable medium, writable or rewritable medium, digital or analog medium, hard disk, floppy disk, optical disc read-only memory (CD-ROM), recordable optical disc (CD-R), rewritable optical disc (CD-RW), optical disc, magnetic medium, magneto-optical medium, removable memory card or disk, various types of digital universal optical disc (DVD), magnetic tape, cassette tape, etc.

[0276] Computer instructions may include any suitable type of code implemented using any appropriate high-level, low-level, object-oriented, visual, compiled, and / or interpreted programming language, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, encrypted code, etc.

[0277] It is understood that the above description provides multiple embodiment solutions provided by the embodiments of the present invention. The optional methods described in each embodiment solution can be combined with each other and cross-referenced without conflict, thereby extending to a variety of possible embodiment solutions. These can all be considered as the embodiment solutions disclosed and published by the present invention.

[0278] It should be noted that the term "an embodiment" or "embodiment" as used in this invention refers to a specific feature, structure, or characteristic that can be included in at least one implementation of the invention. Furthermore, in the description of this invention, terms such as "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with terms such as "first" and "second" may explicitly or implicitly include one or more of that feature. Moreover, terms such as "first" and "second" are used to distinguish similar objects and are not necessarily used to describe a specific order or indicate importance. It is understood that such terms can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein.

[0279] While the embodiments of the present invention have been disclosed above, the present invention is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the present invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A data processing method, characterized in that, include: Feature extraction is performed on the original point cloud data to obtain the first point cloud feature data; Feature extraction is performed on the original image data to obtain the first image feature data; Based on the preset target classification candidate information, the first point cloud feature data and the first image feature data are filtered respectively to obtain the second point cloud feature data and the second image feature data. The second point cloud feature data and the second image feature data are fused to obtain fused data; The fused data is used to perform target recognition to obtain target recognition results; The step of filtering the first image feature data based on preset target classification candidate information to obtain the second image feature data includes: Based on preset target classification candidate information, target screening is performed on the first point cloud feature data to obtain three-dimensional target candidate information of the first point cloud feature data. The three-dimensional target candidate information is filtered to obtain optimized target candidate information; The coordinate system of the optimized target candidate information is transformed to the pixel coordinate system of the original image data to obtain two-dimensional target candidate information; Based on the two-dimensional target candidate information, the first image feature data is filtered to obtain the second image feature data.

2. The data processing method according to claim 1, characterized in that, The process of filtering the first point cloud feature data based on preset target classification candidate information to obtain the second point cloud feature data includes: Based on preset target classification candidate information, target screening is performed on the first point cloud feature data to obtain three-dimensional target candidate information of the first point cloud feature data. The three-dimensional target candidate information is filtered to obtain optimized target candidate information; Based on the optimized target candidate information, the first point cloud feature data is filtered to obtain the second point cloud feature data.

3. The data processing method according to claim 1 or 2, characterized in that, The step of filtering the three-dimensional target candidate information to obtain optimized target candidate information includes: The three-dimensional target candidate information is evaluated, and based on preset evaluation conditions, the three-dimensional target candidate information is filtered to obtain intermediate target candidate information; The intermediate target candidate information is used as the optimized target candidate information.

4. The data processing method according to claim 3, characterized in that, Before using the intermediate target candidate information as the optimization target candidate information, the method further includes: The intermediate target candidate information is subjected to de-overlap processing.

5. The data processing method according to claim 4, characterized in that, Before performing de-overlap processing on the intermediate target candidate information, the following steps are also included: The intermediate target candidate information is matched with the image range corresponding to the original image data, and the intermediate target candidate information is filtered based on preset matching conditions.

6. The data processing method according to claim 5, characterized in that, The step of matching the intermediate target candidate information with the image range corresponding to the original image data includes: The intermediate target candidate information is converted to the pixel coordinate system of the original image data and matched with the image range corresponding to the original image data.

7. The data processing method according to claim 1, characterized in that, Before filtering the first point cloud feature data and the first image feature data based on preset target classification candidate information, the process further includes: Based on the object category and object orientation, the target classification candidate information is set.

8. The data processing method according to claim 1, characterized in that, The step of extracting features from the original point cloud data to obtain the first point cloud feature data includes: The original point cloud data is divided into multiple voxel units; Feature extraction is performed on the multiple voxel units to obtain voxel feature data; The voxel feature data is compressed to obtain the first point cloud feature data.

9. The data processing method according to claim 8, characterized in that, The step of extracting features from the plurality of voxel units to obtain voxel feature data includes: Local feature extraction is performed on each of the multiple voxel units to obtain the voxel feature data.

10. The data processing method according to claim 9, characterized in that, The step of performing local feature extraction on the plurality of voxel units to obtain the voxel feature data includes any one of the following: Voxel feature stacking is performed on the plurality of voxel units respectively to obtain the voxel feature data; Point data logic operations are performed on the multiple voxel units respectively to obtain the voxel feature data.

11. The data processing method according to claim 8, characterized in that, The compression of the voxel feature data includes: The voxel feature data is compressed in a specified direction using a convolutional neural network block, which includes sparse convolutional layers and submanifold convolutional layers.

12. The data processing method according to claim 1, characterized in that, Also includes: Before filtering the first point cloud feature data and the first image feature data based on the preset target classification candidate information, it is determined whether there are any abnormalities in the first image feature data. If the judgment result is that there is no abnormality, then the first point cloud feature data and the first image feature data are filtered respectively based on the preset target classification candidate information.

13. The data processing method according to claim 12, characterized in that, Also includes: If the judgment result indicates that there is an abnormal situation, then target recognition is performed on the first point cloud feature data to obtain the target recognition result.

14. The data processing method according to claim 1, characterized in that, Also includes: The target recognition result is obtained by using a fully connected neural network to perform target recognition on the fused data.

15. A data processing device, connected to an image acquisition device and a point cloud acquisition device, characterized in that, Suitable for performing the data processing method according to any one of claims 1-14, wherein the data processing apparatus comprises: The data acquisition unit is adapted to acquire the raw image data of the image acquisition device and the raw point cloud data of the point cloud acquisition device. The feature extraction unit is adapted to extract features from the original point cloud data to obtain the first point cloud feature data, and to extract features from the original image data to obtain the first image feature data. The data filtering unit is adapted to filter the first point cloud feature data and the first image feature data respectively according to the preset target classification candidate information to obtain the second point cloud feature data and the second image feature data. The data fusion unit is adapted to fuse the second point cloud feature data with the second image feature data to obtain fused data; A target recognition unit is adapted to perform target recognition on the fused data to obtain target recognition results; The data filtering unit includes: The information acquisition subunit is adapted to perform target filtering on the first point cloud feature data according to preset target classification candidate information, so as to obtain the three-dimensional target candidate information of the first point cloud feature data. The information filtering subunit is adapted to filter the three-dimensional target candidate information to obtain optimized target candidate information; The coordinate transformation subunit is adapted to transform the coordinate system of the optimized target candidate information to the pixel coordinate system of the original image data to obtain two-dimensional target candidate information; The filtering subunit is adapted to filter the first image feature data according to the two-dimensional target candidate information to obtain the second image feature data.

16. An autonomous driving system, characterized in that, It includes a point cloud acquisition device, an image acquisition device, and a data processing device, wherein the data processing device is connected to both the point cloud acquisition device and the image acquisition device, and wherein: Point cloud acquisition equipment, suitable for acquiring raw point cloud data; Image acquisition equipment, suitable for acquiring raw image data; A data processing device, adapted to perform the data processing method according to any one of claims 1-14, to process the raw point cloud data and raw image data.

Images