Data processing method, device, electronic device and readable storage medium

Abstract

The present invention provides a data processing method, device, electronic device, and readable storage medium. The method includes: obtaining a first visual image and first radar point cloud data, where the first radar point cloud data is radar point cloud data corresponding to the first visual image; inputting the first visual image and the first radar point cloud data into a trained target detection network to obtain the three-dimensional position information of a first constraint box where the first target is located on the first visual image, where the first constraint box is a three-dimensional constraint box; and determining the three-dimensional recognition information of the first target based on the three-dimensional position information. The present invention breaks through the limitations of a single sensor by fusing the visual image and radar point cloud data at the feature level. This not only estimates the size and position of the target, but also makes the target recognition result less susceptible to environmental influences, thereby improving the performance of three-dimensional target recognition.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer vision technology, and in particular to a data processing method, device, electronic device, and readable storage medium. Background Art

[0002] In recent years, deep learning technology based on neural networks has developed rapidly. In the field of object recognition, a large number of deep learning-based solutions have been proposed. Furthermore, autonomous driving technology is currently attracting widespread attention, and identifying objects in the environment is a fundamental task in autonomous driving.

[0003] Current target recognition technology primarily relies on single sensors or shallow sensor fusion. Due to the unique technical characteristics of various sensors, recognition capabilities are limited by the capabilities of a single sensor. Shallow fusion solutions, on the other hand, only fuse the processed data from each sensor. During the fusion process, the utilization of the raw sensor data is very limited, and the information contained in the raw data cannot be fully extracted. Both solutions suffer from poor target recognition performance. Summary of the Invention

[0004] Embodiments of the present invention provide a data processing method, device, electronic device, and readable storage medium to solve the problem of poor target recognition performance in the prior art.

[0005] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:

[0006] In a first aspect, an embodiment of the present invention provides a data processing method, comprising:

[0007] Acquire a first visual image and first radar point cloud data, where the first radar point cloud data is radar point cloud data corresponding to the first visual image;

[0008] Inputting the first visual image and the first radar point cloud data into a trained object detection network to obtain three-dimensional position information of a first constraint box in which the first object is located in the first visual image, where the first constraint box is a three-dimensional constraint box;

[0009] Three-dimensional identification information of the first target is determined based on the three-dimensional position information.

[0010] Before acquiring the first visual image and the first radar point cloud data, the method further includes:

[0011] Acquire a second visual image and second radar point cloud data for network training, where the second radar point cloud data is radar point cloud data corresponding to the second visual image;

[0012] Acquire a fusion feature map according to the second visual image and the second radar point cloud data;

[0013] Inputting the fused feature map into an object detection network to obtain a predicted value of a second object on the second visual image;

[0014] The target detection network is trained according to the predicted value of the second target and the true value of the second target marked in advance to obtain the trained target detection network.

[0015] The step of obtaining a fusion feature map based on the second visual image and the second radar point cloud data includes:

[0016] Acquire a deep radar feature map corresponding to the second radar point cloud data according to the second radar point cloud data;

[0017] acquiring, according to the second visual image, an image feature map corresponding to the second visual image;

[0018] The deep radar feature map and the image feature map are subjected to feature fusion to obtain a fused feature map.

[0019] The step of obtaining a deep radar feature map corresponding to the second radar point cloud data according to the second radar point cloud data includes:

[0020] performing coordinate transformation on the second radar point cloud data to obtain third radar point cloud data after coordinate transformation;

[0021] Obtaining first mapping data projected onto a two-dimensional image plane from the third radar point cloud data according to preset internal parameters, the first mapping data including a depth, a horizontal velocity, and a vertical velocity of the radar point;

[0022] Performing preset rendering processing on the first mapping data to obtain an original radar feature map corresponding to the second radar point cloud data;

[0023] The deep radar characteristic map is obtained according to the original radar characteristic map.

[0024] The acquiring, according to the second visual image, an image feature map corresponding to the second visual image includes:

[0025] performing feature extraction on the second visual image to obtain a plurality of feature maps of different scales;

[0026] Interpolating the multiple feature maps of different scales to obtain multiple feature maps of the same scale;

[0027] The multiple feature maps of the same scale are fused to obtain an image feature map corresponding to the second visual image.

[0028] The step of inputting the fused feature map into a target detection network to obtain a predicted value of the second target on the second visual image includes:

[0029] In a case where the target detection network includes multiple detection networks, inputting the fused feature map into the multiple detection networks respectively to obtain a predicted value of the second target;

[0030] Among them, the predicted value of the second target includes the predicted value of the center point of the second target on the first heat map, the predicted value of the vertex of the second target on the second heat map, the predicted offset value of the vertex relative to the center point, the predicted offset error value of the center point, the predicted offset error value of the vertex, the predicted dimension of the second target, the predicted depth of the second target, and the predicted two-dimensional size of the second target on the second visual image.

[0031] The method further comprises:

[0032] If the predicted value of the vertex of the second target exceeds the predicted two-dimensional size, the value of the target point on the second constraint box is updated to the predicted value of the vertex of the second target, wherein the second constraint box is a two-dimensional constraint box and corresponds to the predicted two-dimensional size, and the value of the target point is the point on the second constraint box that is closest to the vertex of the second target.

[0033] The step of training the target detection network according to the predicted value of the second target and the true value of the second target that is pre-labeled to obtain the trained target detection network includes:

[0034] Calculating a loss function of the target detection network based on the predicted value of the second target and the true value of the second target;

[0035] The target detection network is trained based on the loss function to obtain a trained target detection network.

[0036] Wherein, the three-dimensional position information includes vertex position information, dimension information and depth information;

[0037] The determining, based on the three-dimensional position information, the three-dimensional identification information of the first target includes:

[0038] Acquire first position information and rotation angle information of the first target according to the vertex position information, dimension information and depth information;

[0039] Determine three-dimensional recognition information of the first target, where the three-dimensional recognition information includes the dimension information, the depth information, the first position information, and the rotation angle information.

[0040] The acquiring of the first position information and the rotation angle information of the first target according to the vertex position information, the dimension information and the depth information includes:

[0041] Calculating second mapping data of the vertex projected onto a two-dimensional image plane according to the vertex position information, the dimension information, and the depth information;

[0042] The first position information and the rotation angle information are calculated based on the second mapping data, the vertex position information, the dimension information and the posture information of the first target.

[0043] In a second aspect, an embodiment of the present invention further provides a data processing device, comprising:

[0044] A first acquisition module is configured to acquire a first visual image and first radar point cloud data, wherein the first radar point cloud data is radar point cloud data corresponding to the first visual image;

[0045] a first processing module, configured to input the first visual image and the first radar point cloud data into a trained target detection network to obtain three-dimensional position information of a first constraint box within which a first target is located in the first visual image, where the first constraint box is a three-dimensional constraint box;

[0046] The second processing module is configured to determine three-dimensional identification information of the first target according to the three-dimensional position information.

[0047] In a third aspect, an embodiment of the present invention further provides an electronic device, including a processor and a transceiver, wherein the transceiver receives and sends data under the control of the processor, and the processor is configured to perform the following operations:

[0048] Acquire a first visual image and first radar point cloud data, where the first radar point cloud data is radar point cloud data corresponding to the first visual image;

[0049] Inputting the first visual image and the first radar point cloud data into a trained object detection network to obtain three-dimensional position information of a first constraint box in which the first object is located in the first visual image, where the first constraint box is a three-dimensional constraint box;

[0050] Three-dimensional identification information of the first target is determined based on the three-dimensional position information.

[0051] In a fourth aspect, an embodiment of the present invention further provides an electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the data processing method described above when executed by the processor.

[0052] In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0053] In the above scheme of the embodiment of the present invention, a first visual image and a first radar point cloud data are obtained, where the first radar point cloud data is the radar point cloud data corresponding to the first visual image; the first visual image and the first radar point cloud data are then input into the trained target detection network to obtain the three-dimensional position information of the first constraint box where the first target is located on the first visual image, where the first constraint box is a three-dimensional constraint box; finally, based on the three-dimensional position information, the three-dimensional recognition information of the first target is determined. In this way, by fusing the visual image and the radar point cloud data at the feature level, the limitation of a single sensor is broken through, and not only the size and position of the target can be estimated, but the target recognition result is also not easily affected by the environment, thereby improving the performance of three-dimensional target recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of a data processing method according to an embodiment of the present invention;

[0055] Figure 2 Schematic diagram of the training process of the target network detection network according to an embodiment of the present invention;

[0056] Figure 3 Schematic diagram of the structure of a radar feature extraction network according to an embodiment of the present invention;

[0057] Figure 4 Schematic diagram of the structure of a key point pyramid network according to an embodiment of the present invention;

[0058] Figure 5 Schematic diagram of the structure of a data processing device according to an embodiment of the present invention;

[0059] Figure 6 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0061] Traditional 3D object recognition solutions have the following disadvantages:

[0062] First, single-sensor 3D target recognition solutions are limited by the sensor's functional characteristics and have certain insurmountable shortcomings. Millimeter-wave radar can detect target distance, speed, and other information, but because it captures fewer radar points, it cannot obtain the target's outline, making it difficult to estimate the target's size.

[0063] Visual sensors, such as RGB sensors, can obtain RGB images of the environment and then estimate target information. However, a single RGB sensor cannot obtain depth information, which is extremely unfavorable for three-dimensional target recognition. At the same time, target recognition methods based on RGB images are easily affected by weather conditions such as rain and snow, and have poor recognition capabilities for small targets.

[0064] Secondly, existing information fusion solutions all fuse processed sensor information. During the processing, a large amount of original information is lost, which limits the use of sensor information during fusion and thus affects the fusion effect.

[0065] The above problems can all be effectively solved in the embodiments of the present invention.

[0066] like Figure 1 FIG. 1 is a flow chart of a data processing method according to an embodiment of the present invention. The following describes the specific implementation process of the method in detail with reference to the figure.

[0067] Step 101: Acquire a first visual image and first radar point cloud data, where the first radar point cloud data is radar point cloud data corresponding to the first visual image.

[0068] Here, a first visual image may be acquired through a visual sensor, and first radar point cloud data corresponding to the first visual image may be acquired through a millimeter-wave radar.

[0069] Optionally, the visual sensor is an RGB sensor, and accordingly, the first visual image is an RGB image, wherein the RGB image is a full-color image including three channels: R (red), G (green), and B (blue).

[0070] Step 102: Input the first visual image and the first radar point cloud data into the trained target detection network to obtain the three-dimensional position information of the first constraint box where the first target is located in the first visual image, where the first constraint box is a three-dimensional constraint box.

[0071] Optionally, the three-dimensional position information includes vertex position information, dimension information, and depth information, that is, the vertex position information, dimension information, and depth information of the first bounding box.

[0072] Here, the first target can be one object or multiple objects, depending on the first visual image, and is not specifically limited. Correspondingly, the first bounding box can be one bounding box or multiple bounding boxes.

[0073] It should be noted that the trained target detection network fuses the first visual image and the first radar point cloud data at the feature level, breaking through the limitations of a single sensor. It can not only estimate the depth information of targets that are not easily affected by the environment, but also estimate the size and position of the target, thereby improving the performance of three-dimensional target recognition.

[0074] Step 103: Determine three-dimensional identification information of the first target based on the three-dimensional position information.

[0075] Optionally, the three-dimensional recognition information includes dimension information, depth information, first position information, and rotation angle information, wherein the first position information is the two-dimensional position information of the first target, and the rotation angle information is the rotation angle information of the first target.

[0076] In this step, first, based on the vertex position information, dimensional information, and depth information of the first constraint box where the first target is located, the first position information and rotation angle information of the first target are calculated by minimizing the projection error. Finally, the dimension information, depth information, first position information, and rotation angle information obtained above are determined as the three-dimensional recognition information of the first target. This avoids directly using the target detection network to predict the first position information and rotation angle information, which are more difficult to predict, simplifying the target recognition steps and improving work efficiency. The specific implementation method can be found in the fourth embodiment.

[0077] above Figure 1 In step 102 of the implementation method shown in the figure, the first visual image and the first radar point cloud data are input into the trained target detection network to obtain the three-dimensional position information of the first constraint box where the first target is located on the first visual image. Figure 1 The method shown requires pre-training the target detection network to obtain a trained target detection network. The following details the training process of the target detection network.

[0078] In one possible implementation, Figure 2 As shown, before step 101, the method of the embodiment of the present invention may further include:

[0079] Step 201: Acquire a second visual image and second radar point cloud data for network training, where the second radar point cloud data is radar point cloud data corresponding to the second visual image.

[0080] The second visual image and the second radar point cloud data are used as training data for training the target detection network. It should be noted that the second visual image and the second radar point cloud data used for training the target detection network are large in amount and can be iteratively trained in batches.

[0081] Similarly, the second visual image can be acquired through a visual sensor, and the second radar point cloud data can be measured by a millimeter-wave radar. The collection scenarios corresponding to the training data can be diverse, such as road intersections, product inspection, and identity recognition.

[0082] Step 202: Obtain a fusion feature map based on the second visual image and the second radar point cloud data.

[0083] Among them, the second radar point cloud data is first converted into a three-channel original radar feature map, and then the radar feature extraction network is used to extract features from the original radar feature map to obtain a deep radar feature map.

[0084] Afterwards, the second visual image is passed through a feature extraction backbone network to obtain multi-scale feature maps at different levels, and then the multi-scale feature maps are fused into an image feature map of the second visual image.

[0085] Finally, the deep radar feature map and the image feature map of the second visual image are fused to obtain a fused feature map. Specifically, the deep radar feature map and the image feature map of the second visual image are connected in series to obtain a fused feature map. The specific implementation method is detailed in the embodiment part.

[0086] Step 203: Input the fused feature map into the target detection network to obtain a predicted value of the second target in the second visual image.

[0087] Optionally, the target detection network includes multiple detection networks. Different detection networks predict the required information respectively, that is, different prediction values of the second target are obtained from different dimensions. The specific implementation method is detailed in the second embodiment.

[0088] Step 204 : Train the target detection network based on the predicted value of the second target and the true value of the pre-labeled second target to obtain a trained target detection network.

[0089] It should be noted that since the predicted values of the second target are predicted values corresponding to different dimensions, the trained target detection network based on the different predicted values of the second target can improve detection precision and accuracy. Here, the specific implementation method of this step is detailed in Example 3.

[0090] Example 1

[0091] In one possible implementation, step 202 of the method of the present invention may include:

[0092] Step 2021: Obtain a deep radar feature map corresponding to the second radar point cloud data based on the second radar point cloud data.

[0093] Herein, this step 2021 may specifically include:

[0094] Step S1: performing coordinate transformation on the second radar point cloud data to obtain third radar point cloud data after coordinate transformation.

[0095] Since the millimeter-wave radar and the visual sensor (generally an RGB sensor) are in different coordinate systems, the second radar point cloud data is converted from the radar coordinate system to the RGB sensor coordinate system using the rotation matrix R and translation matrix T from the radar coordinate system to the RGB sensor coordinate system to obtain the third radar point cloud data after coordinate conversion.

[0096] Step S2: Obtain first mapping data of the third radar point cloud data projected onto a two-dimensional image plane according to preset internal parameters. The first mapping data includes the depth, horizontal velocity, and vertical velocity of the radar point.

[0097] Optionally, the preset internal parameter is an RGB sensor internal parameter.

[0098] Here, the third radar point cloud data is projected onto the two-dimensional image plane using the RGB sensor intrinsic parameters to obtain corresponding first mapping data. The first mapping data is used to represent the position of the radar point corresponding to the third radar point cloud data projected onto the two-dimensional image plane.

[0099] Step S3: performing preset rendering processing according to the first mapping data to obtain an original radar feature map corresponding to the second radar point cloud data.

[0100] Here, based on the first mapping data, the following formulas (1), (2), and (3) are used to render a circle with a radius of r to obtain the original radar feature map of the corresponding second radar point cloud data.

[0101]

[0102]

[0103]

[0104] Among them, z represents the depth of the radar point, v x Indicates the horizontal velocity of the radar point, v y Indicates the vertical velocity of the radar point; μ z and σ z is the mean and standard deviation of the depth of the target (such as a vehicle) in the training data.

[0105] It should be noted that the velocity is converted to data within the (-1, 1) range based on the size. The radius r used during rendering can be adjusted based on the actual application. When rendering with a smaller radius, the generated radar features have a smaller impact range, making feature extraction difficult. When using a larger rendering radius, radar points will overlap and affect each other.

[0106] From the above formula, we can see that the three channels used in rendering (RF i ,i∈(1,2,3)) is determined by the depth z of the radar point, the horizontal velocity v x and vertical velocity v y Decide.

[0107] Step S4: Obtain a deep radar characteristic map based on the original radar characteristic map.

[0108] Specifically, this step S4 may include: using a radar feature extraction network to extract features from the original radar feature map to obtain a deep radar feature map.

[0109] like Figure 3 As shown, the radar feature extraction network optionally consists of three parts. The first part includes a 7×7 convolutional layer and a 3×3 max pooling layer, where the convolution stride of the 7×7 convolutional layer is 3. The second part includes a residual block consisting of two 3×3 convolutional layers, where the convolution stride of the two 3×3 convolutional layers is 1, and the residual block contains a short-circuit path, which is connected to the output of the pooling layer and before the ReLU activation layer. The third part includes a residual block of a 3×3 convolutional layer, which contains a short-circuit path, and the connection position is the same as that of the second part. Each convolutional layer is followed by a batch normalization layer and a ReLU activation layer.

[0110] It should be noted that the size of the convolutional layer in the third part can be adjusted according to actual application. Optionally, the output of all neural network layers has 16 channels.

[0111] The radar data processing method of the present invention converts radar point cloud data into feature maps similar to RGB images, which can be directly processed using a convolutional neural network. During the conversion, the depth information is enhanced by the statistical information of the training dataset, making its features more prominent. The velocity information is converted to a smaller numerical value, ensuring stability during network training. The radar feature extraction network of the present invention has a simple structure and can be used to extract radar data features without significantly increasing the computational workload.

[0112] Step 2022: Acquire an image feature map corresponding to the second visual image based on the second visual image.

[0113] Optionally, this step 2022 may specifically include:

[0114] Step S11: extract features from the second visual image to obtain multiple feature maps of different scales.

[0115] Optionally, a feature extraction backbone network is used to perform feature extraction on the second visual image to obtain multiple feature maps of different scales.

[0116] In one example, the DLA-34 network is used to extract features from the second visual image, obtaining four feature maps of different sizes at different depths, with channel numbers of 32, 64, 128, and 256. These four feature maps are the sizes of the second visual image after being downsampled by 4, 8, 16, and 32 times, respectively.

[0117] Step S12: interpolate the multiple feature maps of different scales to obtain multiple feature maps of the same scale.

[0118] Step S13: fusing multiple feature maps of the same scale to obtain an image feature map corresponding to the second visual image.

[0119] It should be noted that steps S12 and S13 are implemented using a key point feature pyramid network.

[0120] like Figure 4 Figure 1 shows the network structure of the keypoint feature pyramid network. First, multiple feature maps of different scales are passed through a resizing network to obtain multiple feature maps of the same scale. This means that the feature maps of different scales are interpolated to a uniform size, such as the size of the second visual image downsampled by 4.

[0121] After that, a convolutional layer is used to adjust the number of channels of all feature maps to have the same number of channels and size. For example, a convolutional layer with a size of 1×1, a stride of 1, and an output of 256 channels is used to adjust the number of channels of all feature maps to 256.

[0122] Then, the softmax function is used to calculate the weights at corresponding positions of multiple feature maps of the same scale. Finally, the calculated weights are used to perform a weighted summation of the multiple feature maps of the same scale to obtain a feature map with a preset number of channels. Here, if the number of channels is adjusted according to the convolutional layer in the above example, the weighted summation process will result in a feature map with 256 channels.

[0123] Step 2023: Fusing the deep radar feature map and the image feature map to obtain a fused feature map.

[0124] Specifically, the deep radar feature map and the image feature map of the second visual image are concatenated in series to generate a fused feature map. For example, in the above example, the 16-channel deep radar feature map and the 256-channel image feature map are concatenated in series to generate a 272-channel fused feature map.

[0125] Example 2

[0126] In one possible implementation, step 203 of the method of the present invention may include:

[0127] Step 2031: When the target detection network includes multiple detection networks, the fused feature map is input into the multiple detection networks respectively to obtain the predicted value of the second target.

[0128] Among them, the predicted value of the second target includes the predicted value of the center point of the second target on the first heat map, the predicted value of the vertex of the second target on the second heat map, the predicted offset value of the vertex relative to the center point, the predicted offset error value of the center point, the predicted offset error value of the vertex, the predicted dimension of the second target, the predicted depth of the second target, and the predicted two-dimensional size of the second target on the second visual image.

[0129] Optionally, the target detection network includes 8 detection networks, and the 8 detection networks detect the second target from different dimensions respectively.

[0130] Correspondingly, each detection network consists of a 3×3 convolutional layer and a 1×1 convolutional layer. The stride of both convolutional layers is 1. The 3×3 convolutional layer outputs 256 channels, and the 1×1 convolutional layer has different output channels depending on the predicted data corresponding to the second target.

[0131] It should be noted that the first heat map is used to predict the center point of the second target, and the second heat map is used to predict the vertex of the second target.

[0132] First, two detection networks predict the first and second heatmaps, with outputs of one and eight channels, respectively. The networks predict the probability that each point on the fused feature map is the center point of the second object and a vertex of the 3D bounding box where the second object is located. Points with predicted values greater than a preset threshold are considered valid points.

[0133] Then, in order to match the predicted center point and vertex, a detection network is used to predict the offset of eight vertices relative to each center point, where the offset is the distance between the eight vertices of the three-dimensional constraint box where the second target is located and the center point of the second target, and its output is 8 channels.

[0134] The matching process for the center point of the second object and the vertices of its three-dimensional bounding box is as follows: First, a valid point is found on the first heat map, and the second object is considered to exist at that point. Then, the detection network is used to predict the offset of the eight vertices of the point relative to the center point. Based on the offset, the valid point closest to the offset is found on the corresponding second heat map, which is considered to be the vertex of the second object. In this way, the position of the center point of the second object and its corresponding vertex on the second visual image is obtained.

[0135] Furthermore, to improve the stability of the detection results, the embodiment of the present invention also predicts the two-dimensional size of the target in the visual image, that is, the width and height. Specifically, in one possible implementation, the method of the embodiment of the present invention may further include:

[0136] If the predicted value of the vertex of the second target exceeds the predicted two-dimensional size, the value of the target point on the second constraint box is updated to the predicted value of the vertex of the second target, where the second constraint box is a two-dimensional constraint box and corresponds to the predicted two-dimensional size, and the value of the target point is the point on the second constraint box that is closest to the vertex of the second target.

[0137] Through the processing of this implementation, all vertices that are predicted to be beyond the second constraint box are restricted to the boundary of the second constraint box, which can improve the accuracy of the predicted vertex positions.

[0138] In addition, the embodiment of the present invention also uses two detection networks to predict the offset error of the center point and the offset error of the vertex respectively to compensate for the position offset caused by downsampling. The output of the detection network is 2 channels and 16 channels respectively, corresponding to the lateral offset and longitudinal offset of the center point and eight vertices.

[0139] The embodiment of the present invention also uses two detection networks to respectively predict the dimensions (i.e., length, width, and height information) and depth information of the second target. In order to more accurately predict the depth and dimension information, the depth and length, width, and height are converted by the following formulas (4) and (5) before prediction:

[0140] z=μ z +δ z σ z (4)

[0141]

[0142] Among them, μ z and σ z are the mean and standard deviation of target depth in the training data, is the average value over the training data.

[0143] The final predicted value is δ z and [δ h ,δw ,δ l ] T , the output of the detection network is 1 channel and 3 channels respectively.

[0144] It should be noted that, for the sake of training stability, the predicted outputs of the heat map and dimensional information are activated, and the activation function uses the sigmoid function. For details, see the following formulas (6), (7), and (8).

[0145]

[0146]

[0147]

[0148] Among them, formula (6) is the expression of the sigmoid function, o H Represents the predicted output of the heat map, [o h ,o w ,o l ] T Prediction output representing dimensionality information.

[0149] Example 3

[0150] In one possible implementation, step 204 of the method of the present invention may include:

[0151] Step 2041: Calculate the loss function of the target detection network based on the predicted value of the second target and the true value of the second target.

[0152] As can be seen from the above optional implementation, the target detection network includes multiple detection networks, each of which detects the second target from different dimensions. The loss function of the target detection network includes loss functions corresponding to the multiple detection networks.

[0153] Specifically, according to the predicted value of the second target and the true value of the second target, the loss function of the corresponding detection network is calculated respectively; the loss function of each detection network is weightedly summed to obtain the loss function of the target detection network.

[0154] During network training, the difference between the predicted value and the true value of each point on the heat map is calculated based on the following formula (9), which is the loss function of the heat map.

[0155]

[0156] in, is the value of a point on the predicted heat map, H xycis the true value, (x, y, c) are the horizontal and vertical coordinates and channel values of the heat map respectively. α and β are hyperparameters used to adjust the impact of positive and negative sample training on the model.

[0157] It should be noted that the loss function of the first heat map and the loss function of the second heat map are both calculated using the above formula.

[0158] Among them, the method for generating the true value of the heat map is as follows:

[0159] At each real point (For example, the center point of the second target) is the center, and a Gaussian kernel function is generated. The region defined by σ p is the standard deviation of the training data.

[0160] Except for the heat map, all other prediction values use l1 distance to calculate the loss, as shown in formula (10):

[0161]

[0162] Where N is the total number of positive samples, and y i are the corresponding predicted and true values.

[0163] Optionally, the loss function of the target detection network is expressed by the following formula (11):

[0164]

[0165] Among them, L cen Represents the loss of the first heat map, L ver Represents the loss of the second heat map, L cor Represents the loss of the vertex relative to the center, L z represents the loss of target depth, L dim represents the loss of the target dimension, L cen-off The loss of the target center offset error, L ver-off Represents the loss of vertex offset error, L wh It represents the loss of the target two-dimensional size, and w is the weighted coefficient corresponding to each loss.

[0166] Step 2042: Train the target detection network based on the loss function to obtain a trained target detection network.

[0167] Specifically, the network parameters of the target detection network are updated based on the loss function, and then the target detection network with updated parameters is trained according to the training data for the next round of training until the number of training times reaches a preset number or the loss value is minimized, thereby obtaining the trained target detection network.

[0168] In one example, the Adam optimizer was used to train an object detection network, with a learning momentum of 0.9. To prevent overfitting, the network parameter weights were decayed to 0.01, and the initial learning rate was set to 0.002, which was then decayed over the learning process. The DLA-34 network used in the model was initialized using parameters trained on ImageNet. After training the model with a certain amount of data, the trained model was obtained.

[0169] Example 4

[0170] In a possible implementation, the three-dimensional position information includes vertex position information, dimension information, and depth information; accordingly, step 103 of the method of the present invention may include:

[0171] Step 1031 : Acquire first position information and rotation angle information of the first target according to the vertex position information, dimension information, and depth information.

[0172] Optionally, this step 1031 specifically includes:

[0173] Step S111 : calculating and obtaining second mapping data of the vertex projected onto the two-dimensional image plane according to the vertex position information, dimension information, and depth information.

[0174] After the above target detection network, the dimension information of the first constraint box where the first target is located, namely [h, w, l], can be obtained. T and depth information z, it is necessary to estimate the first position information and rotation angle information of the first target.

[0175] Optionally, the first position information and the rotation angle information of the first target may be estimated by minimizing the projection error.

[0176] First, the position coordinates of the vertex in the object coordinate system are calculated based on the vertex position information and dimension information. Specifically, the position coordinates of the vertex in the object coordinate system are calculated using the following formula (12):

[0177]

[0178] Among them, Cor i That is the i-th column of the vertex coordinate matrix Cor,

[0179] Then, the above vertices are transformed into the RGB sensor coordinate system. The transformation matrices between the RGB sensor coordinate system and the object coordinate system: the translation matrix T and the rotation matrix R can be calculated from the coordinates (x, y, z) and the rotation angle θ of the object in the RGB sensor coordinate system. For details, see the following formula (13):

[0180]

[0181] in,

[0182] Finally, the camera intrinsic parameter K is used to project the above vertices onto the two-dimensional image plane to obtain the second mapping data. The specific calculation is as follows:

[0183]

[0184] in, Indicates the second mapping data corresponding to the projection of the vertex onto the two-dimensional image plane.

[0185] Step S112: Calculate the first position information and the rotation angle information based on the second mapping data, the vertex position information, the dimension information and the posture information of the first target.

[0186] Here, the distance between the projection point corresponding to the vertex and the vertex estimated by the target detection network can be calculated as follows:

[0187]

[0188] Among them, x * ,y * ,θ * The three unknown quantities are estimated by minimizing the distance between the projection point corresponding to the vertex and the vertex estimated by the target detection network.

[0189] Specifically, the Lie algebra se3 is used to solve the problem, and the error and the Jacobian of each variable are calculated during the solution.

[0190] in,

[0191] Using Lie algebra, the Jacobian determinant obtained by differentiating the posture and dimension is:

[0192]

[0193]

[0194] Combining the above formulas, we can get the relationship between the projection point and each variable.

[0195] Among them, ε∈se3 is obtained by mapping the pose information of the target (here refers to the first target), f x and f y are the horizontal and vertical focal lengths in the RGB sensor intrinsic parameters.

[0196] After obtaining the Jacobian determinant, the Gauss-Newton or Levenberg-Marquardt optimization algorithm can be used to solve the problem. When solving the problem, other variables must remain unchanged and only (x, y, θ) must be solved.

[0197] Step 1032 : Determine three-dimensional recognition information of the first target, where the three-dimensional recognition information includes dimension information, depth information, first position information, and rotation angle information.

[0198] Among them, the three-dimensional recognition information can be specifically expressed as (x, y, z, h, w, l, θ).

[0199] The data processing method of the embodiment of the present invention has high efficiency and strong stability in three-dimensional target recognition, and has good performance in related fields such as unmanned driving.

[0200] Based on the above method, an embodiment of the present invention provides a data processing device for implementing the above method.

[0201] like Figure 5 FIG. 1 is a schematic diagram of the structure of a data processing device provided by an embodiment of the present invention. The data processing device provided by an embodiment of the present invention may include:

[0202] The first acquisition module 501 is configured to acquire a first visual image and first radar point cloud data, where the first radar point cloud data is radar point cloud data corresponding to the first visual image.

[0203] The first processing module 502 is used to input the first visual image and the first radar point cloud data into the trained target detection network to obtain the three-dimensional position information of the first constraint box where the first target is located in the first visual image, where the first constraint box is a three-dimensional constraint box.

[0204] The second processing module 503 is configured to determine three-dimensional identification information of the first target according to the three-dimensional position information.

[0205] Optionally, the data processing device further includes:

[0206] The second acquisition module is used to acquire a second visual image and second radar point cloud data for network training, where the second radar point cloud data is radar point cloud data corresponding to the second visual image.

[0207] The third acquisition module is used to acquire a fusion feature map according to the second visual image and the second radar point cloud data.

[0208] The third processing module is used to input the fused feature map into the target detection network to obtain a predicted value of the second target in the second visual image.

[0209] The network training module is used to train the target detection network according to the predicted value of the second target and the true value of the pre-marked second target to obtain a trained target detection network.

[0210] Optionally, the third acquisition module includes:

[0211] The first acquisition unit is used to acquire a deep radar feature map corresponding to the second radar point cloud data based on the second radar point cloud data.

[0212] The second acquisition unit is configured to acquire an image feature map corresponding to the second visual image based on the second visual image.

[0213] The feature fusion unit is used to fuse the deep radar feature map and the image feature map to obtain a fused feature map.

[0214] Optionally, the first acquiring unit is specifically configured to:

[0215] performing coordinate transformation on the second radar point cloud data to obtain coordinate transformed third radar point cloud data;

[0216] According to preset internal parameters, first mapping data of the third radar point cloud data projected onto the two-dimensional image plane is obtained, where the first mapping data includes the depth, horizontal speed, and vertical speed of the radar point.

[0217] A preset rendering process is performed according to the first mapping data to obtain an original radar feature map corresponding to the second radar point cloud data.

[0218] A deep radar feature map is obtained based on the original radar feature map.

[0219] Optionally, the second acquiring unit is specifically configured to:

[0220] Feature extraction is performed on the second visual image to obtain multiple feature maps of different scales.

[0221] Multiple feature maps of different scales are interpolated to obtain multiple feature maps of the same scale.

[0222] Multiple feature maps of the same scale are fused to obtain an image feature map corresponding to the second visual image.

[0223] Optionally, the third processing module includes:

[0224] The first processing unit is used to input the fused feature map into the multiple detection networks respectively to obtain the predicted value of the second target when the target detection network includes multiple detection networks; wherein the predicted value of the second target includes the predicted value of the center point of the second target on the first heat map, the predicted values of the vertices of the second target on the second heat map, the predicted offset values of the vertices relative to the center point, the predicted offset error value of the center point, the predicted offset error value of the vertex, the predicted dimension of the second target, the predicted depth of the second target, and the predicted two-dimensional size of the second target on the second visual image.

[0225] Optionally, the apparatus according to the embodiment of the present invention further includes:

[0226] A data update module is used to update the value of the target point on the second constraint box to the predicted value of the vertex of the second target when the predicted value of the vertex of the second target exceeds the predicted two-dimensional size, wherein the second constraint box is a two-dimensional constraint box and corresponds to the predicted two-dimensional size, and the value of the target point is the point on the second constraint box closest to the vertex of the second target.

[0227] Optionally, the network training module includes:

[0228] The computing unit is configured to compute a loss function of the target detection network based on the predicted value of the second target and the true value of the second target.

[0229] The network training unit is used to train the target detection network based on the loss function to obtain a trained target detection network.

[0230] Optionally, the three-dimensional position information includes vertex position information, dimension information, and depth information; accordingly, the second processing module 503 includes:

[0231] The third acquisition unit is used to acquire first position information and rotation angle information of the first target according to the vertex position information, dimension information and depth information.

[0232] The second processing unit is used to determine three-dimensional recognition information of the first target, where the three-dimensional recognition information includes dimension information, depth information, first position information, and rotation angle information.

[0233] Optionally, the third acquiring unit is specifically configured to:

[0234] According to the vertex position information, dimension information and depth information, second mapping data of the vertex projected onto the two-dimensional image plane is calculated.

[0235] The first position information and the rotation angle information are calculated based on the second mapping data, the vertex position information, the dimension information and the posture information of the first target.

[0236] The data processing device provided by the embodiment of the present invention can realize Figures 1 to 4 To avoid repetition, each process in the method embodiment will not be described here.

[0237] The data processing device provided by an embodiment of the present invention obtains a first visual image and first radar point cloud data, where the first radar point cloud data is radar point cloud data corresponding to the first visual image; then the first visual image and the first radar point cloud data are input into a trained target detection network to obtain three-dimensional position information of a first constraint box in which the first target is located on the first visual image, where the first constraint box is a three-dimensional constraint box; finally, based on the three-dimensional position information, the three-dimensional recognition information of the first target is determined. In this way, by fusing the visual image and radar point cloud data at the feature level, the limitations of a single sensor are broken through, and not only the size and position of the target can be estimated, but the target recognition result is not easily affected by the environment, thereby improving the performance of three-dimensional target recognition.

[0238] like Figure 6 As shown, an embodiment of the present invention further provides an electronic device, including a processor 600 and a transceiver 610. The transceiver receives and sends data under the control of the processor. The processor 600 is configured to perform the following operations:

[0239] Acquire a first visual image and first radar point cloud data, where the first radar point cloud data is radar point cloud data corresponding to the first visual image;

[0240] Inputting the first visual image and the first radar point cloud data into the trained target detection network to obtain three-dimensional position information of a first constraint box where the first target is located in the first visual image, where the first constraint box is a three-dimensional constraint box;

[0241] Three-dimensional identification information of the first target is determined based on the three-dimensional position information.

[0242] Optionally, the processor 600 is further configured to:

[0243] Acquire a second visual image and second radar point cloud data for network training, where the second radar point cloud data is radar point cloud data corresponding to the second visual image;

[0244] Obtaining a fusion feature map according to the second visual image and the second radar point cloud data;

[0245] Input the fused feature map into the target detection network to obtain the predicted value of the second target on the second visual image;

[0246] The target detection network is trained according to the predicted value of the second target and the true value of the pre-labeled second target to obtain a trained target detection network.

[0247] Optionally, the processor 600 is further configured to:

[0248] Obtaining a deep radar feature map corresponding to the second radar point cloud data based on the second radar point cloud data;

[0249] Acquire an image feature map corresponding to the second visual image according to the second visual image;

[0250] The deep radar feature map and the image feature map are fused to obtain a fused feature map.

[0251] Optionally, the processor 600 is further configured to:

[0252] performing coordinate transformation on the second radar point cloud data to obtain coordinate transformed third radar point cloud data;

[0253] According to preset internal parameters, first mapping data of the third radar point cloud data projected onto the two-dimensional image plane is obtained, where the first mapping data includes the depth, horizontal speed, and vertical speed of the radar point.

[0254] A preset rendering process is performed according to the first mapping data to obtain an original radar feature map corresponding to the second radar point cloud data.

[0255] A deep radar feature map is obtained based on the original radar feature map.

[0256] Optionally, the processor 600 is further configured to:

[0257] Feature extraction is performed on the second visual image to obtain multiple feature maps of different scales.

[0258] Multiple feature maps of different scales are interpolated to obtain multiple feature maps of the same scale.

[0259] Multiple feature maps of the same scale are fused to obtain an image feature map corresponding to the second visual image.

[0260] Optionally, the processor 600 is further configured to:

[0261] When the target detection network includes multiple detection networks, the fused feature maps are input into the multiple detection networks respectively to obtain the predicted value of the second target; wherein the predicted value of the second target includes the predicted value of the center point of the second target on the first heat map, the predicted values of the vertices of the second target on the second heat map, the predicted offset values of the vertices relative to the center point, the predicted offset error value of the center point, the predicted offset error value of the vertices, the predicted dimension of the second target, the predicted depth of the second target, and the predicted two-dimensional size of the second target on the second visual image.

[0262] Optionally, the processor 600 is further configured to:

[0263] If the predicted value of the vertex of the second target exceeds the predicted two-dimensional size, the value of the target point on the second constraint box is updated to the predicted value of the vertex of the second target, wherein the second constraint box is a two-dimensional constraint box and corresponds to the predicted two-dimensional size, and the value of the target point is the point on the second constraint box that is closest to the vertex of the second target.

[0264] Optionally, the processor 600 is further configured to:

[0265] Calculate the loss function of the target detection network based on the predicted value of the second target and the true value of the second target.

[0266] The target detection network is trained based on the loss function to obtain a trained target detection network.

[0267] Optionally, the three-dimensional position information includes vertex position information, dimension information, and depth information; accordingly, the processor 600 is further configured to:

[0268] First position information and rotation angle information of the first target are acquired according to the vertex position information, the dimension information, and the depth information.

[0269] Determine three-dimensional recognition information of the first target, where the three-dimensional recognition information includes dimension information, depth information, first position information, and rotation angle information.

[0270] Optionally, the processor 600 is further configured to:

[0271] According to the vertex position information, dimension information and depth information, second mapping data of the vertex projected onto the two-dimensional image plane is calculated.

[0272] The first position information and the rotation angle information are calculated based on the second mapping data, the vertex position information, the dimension information and the posture information of the first target.

[0273] The electronic device of an embodiment of the present invention obtains a first visual image and first radar point cloud data, where the first radar point cloud data is radar point cloud data corresponding to the first visual image; then the first visual image and the first radar point cloud data are input into a trained target detection network to obtain three-dimensional position information of a first constraint box in which the first target is located on the first visual image, where the first constraint box is a three-dimensional constraint box; finally, based on the three-dimensional position information, the three-dimensional recognition information of the first target is determined. In this way, by fusing the visual image and radar point cloud data at the feature level, the limitations of a single sensor are broken through, and not only the size and position of the target can be estimated, but the target recognition result is not easily affected by the environment, thereby improving the performance of three-dimensional target recognition.

[0274] An embodiment of the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and runnable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned data processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0275] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the various processes of the above-described data processing method embodiment and achieves the same technical effects. To avoid repetition, the details are not described here. The computer-readable storage medium may be, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0276] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0277] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0278] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are protected by the present invention.

Claims

1. A data processing method, characterized in that: include: Acquire a first visual image and first radar point cloud data, where the first radar point cloud data is radar point cloud data corresponding to the first visual image; Inputting the first visual image and the first radar point cloud data into a trained object detection network to obtain three-dimensional position information of a first constraint box in which the first object is located in the first visual image, where the first constraint box is a three-dimensional constraint box; determining three-dimensional identification information of the first target based on the three-dimensional position information; The three-dimensional position information includes vertex position information, dimension information and depth information; The determining, based on the three-dimensional position information, the three-dimensional identification information of the first target includes: Acquire first position information and rotation angle information of the first target according to the vertex position information, dimension information and depth information; Determining three-dimensional recognition information of the first target, the three-dimensional recognition information including the dimension information, the depth information, the first position information, and the rotation angle information; The acquiring, according to the vertex position information, the dimension information, and the depth information, the first position information and the rotation angle information of the first target includes: Calculating second mapping data of the vertex projected onto a two-dimensional image plane according to the vertex position information, the dimension information, and the depth information; The first position information and the rotation angle information are calculated based on the second mapping data, the vertex position information, the dimension information and the posture information of the first target.

2. The method according to claim 1, characterized in that Before acquiring the first visual image and the first radar point cloud data, the method further includes: Acquire a second visual image and second radar point cloud data for network training, where the second radar point cloud data is radar point cloud data corresponding to the second visual image; Acquire a fusion feature map according to the second visual image and the second radar point cloud data; Inputting the fused feature map into an object detection network to obtain a predicted value of a second object on the second visual image; The target detection network is trained according to the predicted value of the second target and the true value of the second target marked in advance to obtain the trained target detection network.

3. The method according to claim 2, characterized in that The acquiring of a fusion feature map according to the second visual image and the second radar point cloud data includes: Acquire a deep radar feature map corresponding to the second radar point cloud data according to the second radar point cloud data; acquiring, according to the second visual image, an image feature map corresponding to the second visual image; The deep radar feature map and the image feature map are subjected to feature fusion to obtain a fused feature map.

4. The method according to claim 3, characterized in that The acquiring, according to the second radar point cloud data, a deep radar feature map corresponding to the second radar point cloud data includes: performing coordinate transformation on the second radar point cloud data to obtain third radar point cloud data after coordinate transformation; Obtaining first mapping data projected onto a two-dimensional image plane from the third radar point cloud data according to preset internal parameters, the first mapping data including a depth, a horizontal velocity, and a vertical velocity of the radar point; Performing preset rendering processing on the first mapping data to obtain an original radar feature map corresponding to the second radar point cloud data; The deep radar characteristic map is obtained according to the original radar characteristic map.

5. The method according to claim 3, characterized in that The acquiring, according to the second visual image, an image feature map corresponding to the second visual image includes: performing feature extraction on the second visual image to obtain a plurality of feature maps of different scales; Interpolating the multiple feature maps of different scales to obtain multiple feature maps of the same scale; The multiple feature maps of the same scale are fused to obtain an image feature map corresponding to the second visual image.

6. The method according to claim 2, characterized in that Inputting the fused feature map into a target detection network to obtain a predicted value of a second target on the second visual image includes: In a case where the target detection network includes multiple detection networks, inputting the fused feature map into the multiple detection networks respectively to obtain a predicted value of the second target; Among them, the predicted value of the second target includes the predicted value of the center point of the second target on the first heat map, the predicted value of the vertex of the second target on the second heat map, the predicted offset value of the vertex relative to the center point, the predicted offset error value of the center point, the predicted offset error value of the vertex, the predicted dimension of the second target, the predicted depth of the second target, and the predicted two-dimensional size of the second target on the second visual image.

7. The method according to claim 6, characterized in that The method further comprises: If the predicted value of the vertex of the second target exceeds the predicted two-dimensional size, the value of the target point on the second constraint box is updated to the predicted value of the vertex of the second target, wherein the second constraint box is a two-dimensional constraint box and corresponds to the predicted two-dimensional size, and the value of the target point is the point on the second constraint box that is closest to the vertex of the second target.

8. The method according to claim 2 or 6, characterized in that The step of training the target detection network according to the predicted value of the second target and the pre-labeled true value of the second target to obtain the trained target detection network includes: Calculating a loss function of the target detection network based on the predicted value of the second target and the true value of the second target; The target detection network is trained based on the loss function to obtain a trained target detection network.

9. A data processing device, characterized in that: include: A first acquisition module is configured to acquire a first visual image and first radar point cloud data, wherein the first radar point cloud data is radar point cloud data corresponding to the first visual image; a first processing module, configured to input the first visual image and the first radar point cloud data into a trained target detection network to obtain three-dimensional position information of a first constraint box within which a first target is located in the first visual image, where the first constraint box is a three-dimensional constraint box; a second processing module, configured to determine three-dimensional identification information of the first target based on the three-dimensional position information; The three-dimensional position information includes vertex position information, dimension information and depth information; The second processing module includes: a third acquiring unit, configured to acquire first position information and rotation angle information of the first target according to the vertex position information, the dimension information, and the depth information; a second processing unit, configured to determine three-dimensional identification information of the first target, the three-dimensional identification information including dimension information, depth information, first position information, and rotation angle information; The third acquisition unit is specifically used for: Calculate and obtain second mapping data of the vertex projected onto the two-dimensional image plane according to the vertex position information, dimension information and depth information; The first position information and the rotation angle information are calculated based on the second mapping data, the vertex position information, the dimension information and the posture information of the first target.

10. An electronic device comprising a processor and a transceiver, wherein the transceiver receives and sends data under the control of the processor, characterized in that: The processor is configured to perform the following operations: Acquire a first visual image and first radar point cloud data, where the first radar point cloud data is radar point cloud data corresponding to the first visual image; Inputting the first visual image and the first radar point cloud data into a trained object detection network to obtain three-dimensional position information of a first constraint box in which the first object is located in the first visual image, where the first constraint box is a three-dimensional constraint box; determining three-dimensional identification information of the first target based on the three-dimensional position information; The three-dimensional position information includes vertex position information, dimension information, and depth information; the processor is further configured to: Acquire first position information and rotation angle information of the first target according to the vertex position information, dimension information and depth information; Determining three-dimensional identification information of the first target, the three-dimensional identification information including dimension information, depth information, first position information, and rotation angle information; The processor is also used to: Calculate and obtain second mapping data of the vertex projected onto the two-dimensional image plane according to the vertex position information, dimension information and depth information; The first position information and the rotation angle information are calculated based on the second mapping data, the vertex position information, the dimension information and the posture information of the first target.

11. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that: When the processor executes the program, the data processing method according to any one of claims 1 to 8 is implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 8 are implemented.

Images