Data processing method, target object detection method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method, a target object detection method, an electronic device and a storage medium, and the method comprises the steps: dividing a plurality of original data collected by a plurality of data collection devices according to a preset time interval, and obtaining a plurality of original data sets, each original data set comprises at least one piece of original data in each kind of original data; time alignment processing is conducted on each original data set, multiple time alignment data sets are obtained, and the time difference of any two original data in each time alignment data set is smaller than or equal to a preset time threshold value. A plurality of different types of data acquisition devices are used for acquiring environment information, and time synchronization of data acquired by the plurality of data acquisition devices is realized through time alignment operation, so that the accuracy of target detection is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data processing method, a target object detection method, an electronic device, and a storage medium. Background Art

[0002] One of the core functions of autonomous vehicles is to be able to recognize the environment in which the vehicle is located, which requires the vehicle to be able to perform real-time and accurate target detection of surrounding objects.

[0003] In related technologies, target detection methods acquire environmental information through multiple sensors and use deep learning models to analyze the information acquired by the multiple sensors to obtain target detection results.

[0004] However, because different types of sensors collect data at different frequencies and there may be time delays, the data collected by different types of sensors are not synchronized in time, which affects the target detection results.

[0005] It should be noted that the information disclosed in the background section of this application is intended only to enhance the understanding of the general background of this application, and should not be regarded as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0006] In view of this, this application provides a data processing method, a target detection method, an electronic device, and a storage medium to solve the problem in the prior art where the data collected by different types of sensors are not synchronized in time due to different collection frequencies, which may cause time delays and thus affect the target detection results.

[0007] In a first aspect, embodiments of this application provide a data processing method, including:

[0008] Multiple sets of raw data collected by multiple data acquisition devices are divided according to a preset time interval to obtain multiple sets of raw data, wherein each set of raw data includes at least one of the raw data of each type.

[0009] Each of the original data sets is time-aligned to obtain multiple time-aligned data sets, wherein the time difference between any two original data sets in each time-aligned data set is less than or equal to a preset time threshold.

[0010] In this embodiment of the application, multiple different types of data acquisition devices are used to collect environmental information, and the raw data collected by the data acquisition devices are divided and time-aligned. Even if there is a time delay in the raw data collected by multiple different types of data acquisition devices, the time synchronization of the data after time alignment helps to improve the accuracy of target detection.

[0011] In one possible implementation, the time alignment process includes:

[0012] Determine the target raw data in the original data set;

[0013] If the time difference between any other original data in the original data set and the target original data is greater than a preset time threshold, then any other original data will be deleted from the original data set.

[0014] In this embodiment, target raw data and a time threshold are pre-defined. If the time difference between any other raw data and the target raw data exceeds the time threshold, that other raw data is deleted. It can be understood that when data collected by a data acquisition device is lost, even after the raw data from that device is divided, there may still be another piece of raw data whose time difference with the target raw data exceeds the time threshold. This other raw data should not be in this raw data set, so it is deleted. This makes the time of the raw data in the raw data set more consistent, resulting in a more accurate raw data set.

[0015] In one possible implementation, the target raw data is the raw data with the smallest time information or the raw data with the largest time information in the raw data set.

[0016] In one possible implementation, the target raw data can be any raw data in the raw data set. However, if the target raw data is set to any raw data in the raw data set, the preset time threshold is difficult to calculate, or a first time threshold is needed to determine raw data whose time is greater than the target raw data, and a second time threshold is needed to determine raw data whose time is less than the target raw data, which obviously makes the judgment process too complicated. Therefore, in this embodiment, the raw data with the smallest or largest time information is directly set as the target raw data to facilitate the judgment of the time difference between each raw data and the target raw data, thus simplifying the judgment process.

[0017] In one possible implementation, the original data includes three-dimensional information, and the method further includes:

[0018] Spatial alignment processing is performed on each of the time-aligned data sets to obtain multiple spatially aligned data sets, wherein all three-dimensional information in the multiple spatially aligned data sets is located in the same spatial coordinate system.

[0019] In this embodiment, since multiple data acquisition devices are typically located at different positions on the vehicle, each device provides different 3D information. Therefore, to facilitate target detection, the 3D information from each data acquisition device needs to be unified into the same spatial coordinate system.

[0020] In one possible implementation, the space alignment process includes:

[0021] Determine the target spatial coordinate system and the spatial transformation matrix between each type of three-dimensional information in the time-aligned data set and the target spatial coordinate system;

[0022] Based on the spatial transformation matrix, each type of three-dimensional information is transformed to the target spatial coordinate system.

[0023] In this embodiment, the time-aligned data set is spatially aligned to facilitate target detection. In addition, by using a spatial transformation matrix to transform each type of 3D information to the target spatial coordinate system, multiple types of 3D information can be quickly transformed into the same spatial matrix, thus accelerating information processing.

[0024] In one possible implementation, the method further includes:

[0025] Feature extraction is performed on each of the spatially aligned datasets to obtain multiple feature information.

[0026] In this embodiment of the application, feature extraction is performed on the time- and space-aligned dataset to obtain feature information. This feature information is beneficial for detecting target objects. It can be understood that target objects can be detected more quickly by extracting feature information in advance.

[0027] In one possible implementation, the raw data further includes image information and velocity information, and the feature extraction includes:

[0028] Image feature information extracted from image information;

[0029] Extract frequency domain feature information based on velocity information;

[0030] Spatial feature information is extracted based on three-dimensional information.

[0031] In the embodiments of this application, image feature information, frequency domain feature information and spatial feature information of the target object are extracted respectively, which is beneficial to the subsequent identification of the shape and structure of the target object, thereby performing tasks such as target object detection and navigation.

[0032] Secondly, embodiments of this application provide a target object detection method, including:

[0033] The original data of the target object is processed using some of the methods described in the first aspect to obtain multiple feature information;

[0034] The target object prediction model is input with multiple of the aforementioned feature information to obtain the target object prediction result.

[0035] In the embodiments of this application, the target object prediction model is used to detect the target object, which can determine whether the target object exists. If the target object exists, the category, location and size of the target object can be accurately predicted, thereby obtaining accurate target object detection results.

[0036] Thirdly, embodiments of this application provide an electronic device, including:

[0037] At least one processor;

[0038] The memory is coupled to the processor;

[0039] The processor stores a computer program that includes instructions that, when executed by the processor, cause the electronic device to perform the methods described in the first and second aspects.

[0040] Fourthly, embodiments of this application provide a computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the methods described in the first and second aspects.

[0041] The solution provided in this application uses multiple different types of data acquisition devices to collect environmental information, and synchronizes the data collected by the multiple data acquisition devices through time alignment, thereby helping to improve the accuracy of target detection. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This application provides a schematic diagram of the working steps of a lidar.

[0044] Figure 2A flowchart illustrating a data processing method provided in this application;

[0045] Figure 3 A flowchart illustrating another data processing method provided in this application;

[0046] Figure 4 A schematic diagram of the field of view of two data acquisition devices without spatial alignment processing is provided for an embodiment of this application;

[0047] Figure 5 A schematic diagram of the field of view of two data acquisition devices after spatial alignment processing, provided as an embodiment of this application;

[0048] Figure 6 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0049] Figure 7 A schematic flowchart of a target object detection method provided in an embodiment of this application;

[0050] Figure 8 This is a schematic diagram of the structure of a target object prediction model provided in an embodiment of this application;

[0051] Figure 9 This is a schematic diagram of a target object prediction model optimization method provided in an embodiment of this application;

[0052] Figure 10 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0053] Figure 11 This is a schematic diagram of a spatial alignment process provided in an embodiment of this application;

[0054] Figure 12 A schematic flowchart of another target object detection method provided in an embodiment of this application;

[0055] Figure 13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Detailed Implementation

[0056] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0057] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0058] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0059] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0060] One of the core functions of autonomous vehicles is to be able to recognize the environment in which the vehicle is located, which requires the vehicle to be able to perform real-time and accurate target detection of surrounding objects.

[0061] The target detection method in related technologies is as follows: multiple sensors are installed on autonomous vehicles to detect environmental information. This environmental information is preprocessed and input into a deep learning model. The deep learning model analyzes the preprocessed data to obtain the target detection result.

[0062] In practical applications, multiple sensors collect data at different frequencies and there may be time delays, which can cause the data collected by different sensors to be out of sync in time, thus affecting the target detection results.

[0063] To address the aforementioned issues, this application provides a data processing method that uses multiple different types of data acquisition devices to collect environmental information and synchronizes the data collected by the multiple devices through time alignment, thereby helping to improve the accuracy of target detection. This will be described in detail below.

[0064] The data acquisition devices used in the embodiments of this application include, but are not limited to, cameras, radar, and lidar. The camera is used to acquire image information; the radar is used to acquire the speed information of the target object; and the lidar is used to acquire the three-dimensional information of the target object. Specifically, the working steps of the lidar are described in detail below.

[0065] See Figure 1 This is a schematic diagram illustrating the working steps of a lidar system provided in this application. Figure 1 As shown, it mainly includes the following steps.

[0066] Step S101: Send laser pulses in each preset direction.

[0067] When the vehicle engages autonomous driving, the LiDAR system begins to operate, sending laser pulses in preset directions. For example, if the preset directions encompass all directions within the LiDAR's acquisition range, then when the LiDAR starts operating, it will send laser pulses in every direction within its acquisition range.

[0068] Step S102: Receive the reflected laser pulse.

[0069] When a target receives a laser pulse sent by a lidar, it will reflect the laser pulse, and a portion of the reflected laser pulse will be received by the lidar.

[0070] Step S103: Calculate the time difference between the transmitted laser pulse and the received laser pulse.

[0071] When a lidar sends a laser pulse, it records the time when the lidar sends the laser pulse. Similarly, when the lidar receives a laser pulse, it also records the time when the lidar receives the laser pulse. The time difference is obtained by subtracting the time when the lidar sends the laser pulse from the time when the lidar receives the laser pulse.

[0072] Step S104: Calculate the distance based on the time difference and speed.

[0073] Since the propagation speed of laser is the same as the speed of light, the propagation speed of laser is a known quantity. Based on the time difference obtained in step S103 and the propagation speed of laser, the distance between the vehicle and the target object can be calculated.

[0074] Step S105: Record the distance and direction as a point in the three-dimensional information of the target object.

[0075] When the lidar receives a reflected laser pulse, it records the emission direction of the laser pulse, i.e., the direction of the target object. The distance between the direction of the target object and the distance calculated in step S104 is used as the information of a point in the three-dimensional information of the target object. After multiple recordings, three-dimensional point cloud data of the target object can be generated.

[0076] LiDAR can emit and receive a large number of laser pulses in a short period of time and continuously repeat the above steps S101-S105. Therefore, LiDAR can quickly acquire the three-dimensional point cloud data of the target object.

[0077] This application provides a data processing method in which multiple data acquisition devices (including LiDAR) collect environmental information, and the data collected by the multiple data acquisition devices are synchronized in time through a time alignment operation, thereby helping to improve the accuracy of target detection. This will be described in detail below.

[0078] See Figure 2This is a flowchart illustrating a data processing method provided in this application. Figure 2 As shown, it mainly includes the following steps.

[0079] Step S201: Divide the various raw data collected by multiple data acquisition devices into multiple raw data sets according to a preset time interval.

[0080] When the vehicle engages autonomous driving, the data processing device sends a start command to the data acquisition device. Upon receiving the start command, the data acquisition device begins collecting raw data and sends the collected raw data to the data processing device. Since this embodiment employs multiple data acquisition devices, and there may be more than one of each type, the raw data received by the data processing device consists of multiple sets of raw data collected by various (same or different) data acquisition devices. For example, the data processing device includes two first-type data acquisition devices and three second-type data acquisition devices. Each first-type data acquisition device collects 100 sets of first-type raw data, and each second-type data acquisition device collects 200 sets of second-type raw data. In this case, the raw data received by the data processing device consists of 200 sets of first-type raw data and 600 sets of second-type raw data collected by five data acquisition devices, where the five data acquisition devices include two types of data acquisition devices.

[0081] In one possible implementation, the raw data sent by the data acquisition devices is not arranged in chronological order; therefore, the raw data collected by each data acquisition device needs to be arranged in chronological order. In another possible implementation, some data acquisition devices may collect raw data that does not include time information or whose time information is not accurate enough to meet the requirements of the data processing device. In this case, the data processing device needs to provide time information for these raw data. Since multiple data acquisition devices start collecting data simultaneously, the start time of multiple sets of raw data is the same; that is, the time information of the first set of raw data collected by each data acquisition device is the same.

[0082] Multiple sets of raw data collected by multiple data acquisition devices are divided according to preset time intervals. Each set of raw data includes at least one piece of raw data from each type of raw data. For example, as shown in Table 1, the first type of raw data collected by the first data acquisition device includes raw data A1, raw data A2, and raw data A3, and the second type of raw data collected by the second data acquisition device includes raw data B1 to B9. In one possible implementation, the first data acquisition device collects data once every 1 millisecond, and the second data acquisition device collects data three times every 1 millisecond. The preset time interval is 1 millisecond. Therefore, the two types of raw data collected by the two data acquisition devices are divided according to 1 millisecond intervals to obtain multiple sets of raw data. Each set of raw data, as shown in Table 2, includes raw data A1 from the first type of raw data and raw data B1, B2, and B3 from the second type of raw data.

[0083] Table 1:

[0084]

[0085] Table 2:

[0086]

[0087] Step S202: Perform time alignment processing on each original dataset to obtain multiple time-aligned datasets.

[0088] Specifically, a target original data set is determined for each original data set. If the time difference between any other original data set and the target original data is greater than a preset time threshold, that other original data set is considered abnormal and deleted from the original data set. This results in multiple time-aligned data sets. It can be understood that the time difference between any two original data sets in each time-aligned data set is less than or equal to the preset time threshold. It should be noted that deleting abnormal data from the original data set does not completely delete the abnormal data; rather, it removes it from the original data set, and the programmer determines how to handle the abnormal data.

[0089] Among them, the target raw data is the raw data with the least time information or the raw data with the most time information in the raw data set, and the other raw data is any raw data in the raw data set other than the target raw data.

[0090] In one possible implementation, the target raw data is the raw data with the largest time information in the raw data set. In the raw data set described in Table 2, raw data B3 is the raw data with the largest time information, so raw data B3 is the target raw data. The time information of raw data B1, raw data B2, and raw data A1 in the raw data set is compared with the time information of raw data B3. If the time difference between raw data B1 and raw data B3 is greater than a preset time threshold, then raw data B1 is deleted from the raw data set. The new raw data set is shown in Table 3.

[0091] Table 3:

[0092]

[0093] In one possible implementation, the target raw data is the raw data with the smallest time information in the raw data set. In the raw data set described in Table 2, raw data B1 is the raw data with the smallest time information, so raw data B1 is the target raw data. The time information of raw data A1, raw data B2, and raw data B3 in the raw data set is compared with the time information of raw data B1. If the time difference between raw data B3 and raw data B1 is greater than a preset time threshold, then raw data B3 is deleted from the raw data set. The new raw data set is shown in Table 4.

[0094] Table 4:

[0095]

[0096] Additionally, it should be noted that if the time difference between any two original data points in the original dataset is less than or equal to a preset time threshold, then there is no need to delete the original data.

[0097] In summary, time alignment allows for the synchronization of data collected by multiple data acquisition devices, thereby improving the accuracy of target detection.

[0098] Since different data acquisition devices may be located at different positions on the vehicle, their fields of view for collecting information may differ. To address this issue, this application provides another data processing method, which will be described in detail below.

[0099] See Figure 3 This is a flowchart illustrating another data processing method provided in this application. Figure 3 As shown, the method is in Figure 2 The illustrated embodiment also includes the following steps.

[0100] Step S301: Perform spatial alignment processing on each time-aligned data set to obtain multiple spatially aligned data sets.

[0101] Specifically, the target spatial coordinate system is determined in all time-aligned data sets, as well as the spatial transformation matrix between each 3D information in the time-aligned data set and the target spatial coordinate system. Based on the spatial transformation matrix, each type of 3D information is transformed to the target spatial coordinate system.

[0102] See Figure 4 This is a schematic diagram of the field of view of two data acquisition devices without spatial alignment processing, provided in an embodiment of this application.

[0103] See Figure 5 This is a schematic diagram of the field of view of two data acquisition devices after spatial alignment processing, provided in an embodiment of this application.

[0104] In one possible implementation, as shown in Table 5, only the first and second types of raw data include three-dimensional information, such as... Figure 4 As shown in the figure, the spatial coordinate system 401 corresponds to the three-dimensional information (x1, y1, z1) of the first type of raw data, the spatial coordinate system 402 corresponds to the three-dimensional information (x2, y2, z2) of the second type of raw data, and the target object 403. In one possible implementation, the target spatial coordinate system can be 401. (x2, y2, z2) is transformed to 401 according to the spatial transformation matrix between 402 and 401, as follows: Figure 5 As shown, 401 and 402 coincide. Of course, those skilled in the art can set the target coordinate system to other coordinate systems according to actual needs, and this application embodiment does not limit this.

[0105] Table 5:

[0106] Types of raw data 3D information First type of raw data (x1,y1,z1) The second type of raw data (x2,y2,z2) The third type of raw data none

[0107] In one possible implementation, all data acquisition devices collect information including three-dimensional information. Then, a spatial transformation matrix is ​​determined by the relationship between the three-dimensional information of each type of raw data and the target spatial coordinate system. Based on the spatial transformation matrix, the three-dimensional information of each type of raw data is transformed to the target spatial coordinate system.

[0108] In summary, the data processing device calculates a spatial transformation matrix based on the relationship between the 3D information of different types of raw data and the target coordinate system. By using the spatial transformation matrix, all the data collected by the data acquisition device is transformed into the same spatial coordinate system, resulting in a spatially aligned data set, which helps to improve the accuracy of target detection.

[0109] Before using a target object prediction model to detect a target object, it is necessary to extract the feature information from the raw data and input the feature information into the target object prediction model. This helps improve the efficiency and accuracy of target object detection. Since the raw data is collected by different data acquisition devices, the format and characteristics of different types of raw data may vary greatly. Manually extracting features may be time-consuming and laborious. To solve the above problems, this application provides another data processing method, which is described in detail below.

[0110] See Figure 6 This is a flowchart illustrating another data processing method provided in an embodiment of this application. Figure 6 As shown, the method is in Figure 3 The illustrated embodiment also includes the following steps.

[0111] Step S601: Perform feature extraction on each spatially aligned dataset to obtain multiple feature information.

[0112] Specifically, the original data also includes image information, velocity information, and distance information. In each spatially aligned dataset, image feature information is extracted based on the image information of the original data; frequency domain feature information is extracted based on the velocity and distance information of the original data; and spatial feature information is extracted based on the three-dimensional information of the original data.

[0113] In one possible implementation, image feature information is extracted from the original image data using methods such as edge detection and color analysis. Edge detection is primarily used to identify the boundaries of objects; typical edge detection algorithms include the Sobel operator, Prewitt operator, Laplacian operator, and Canny operator. Most of these algorithms detect edges by calculating the gradient or second derivative of the image. Color analysis is typically used to identify and separate different objects in an image; specific methods may involve color histograms, color moments, and color space transformations.

[0114] In one possible implementation, spatial feature information is typically extracted from the 3D information in the original data using methods such as shape analysis and surface normal calculation. Shape analysis is generally achieved by calculating the geometric properties of the 3D information, for example, calculating the curvature of each point, or using a Python programming library (Point Globe Library, PCL) to perform principal component analysis (PCA). Surface normal calculation is an important feature describing the local shape of the 3D information; it is usually calculated by fitting a plane to the neighbors around each point.

[0115] In summary, by extracting features from each spatially aligned dataset, multiple feature information can be obtained, thereby improving the efficiency and accuracy of target detection.

[0116] When detecting a target object, a series of processing steps are required to obtain the target object prediction result. Therefore, this application provides a target object detection method, which is described in detail below.

[0117] See Figure 7 This is a schematic flowchart of a target object detection method provided in an embodiment of this application. Figure 7 As shown, the method is in Figure 6 The illustrated embodiment also includes the following steps.

[0118] Step S701: Input multiple feature information into the target object prediction model to obtain the target object prediction result.

[0119] Specifically, the target object prediction model includes convolutional layers, activation functions, pooling layers, and fully connected layers. In one possible implementation, the target object prediction model also includes a classifier and a regressor. Multiple feature information is input into the target object prediction model, passing through convolutional layers, activation functions, pooling layers, fully connected layers, and the classifier and regressor to obtain the target object prediction result.

[0120] See Figure 8 This is a schematic diagram of the structure of a target object prediction model provided in an embodiment of this application. Figure 8 As shown, the target object prediction model mainly includes the following structure: input layer 801, convolutional layer 802, activation function 803, pooling layer 804, fully connected layer 805, and output layer 806. The input layer 801 is used to receive... Figure 6The illustrated embodiment extracts image feature information, frequency domain feature information, and spatial feature information, and inputs this feature information into convolutional layer 802. Convolutional layer 802 is used to extract higher-level abstract feature information from the feature information input from input layer 801, and inputs this higher-level abstract feature information into activation function 903. For example, for the image feature information input from input layer 801, the convolutional layer may initially extract low-level feature information such as lines and edges, and after subsequent convolutional layers, it can further extract higher-level feature information such as shape and texture. Activation function 803 is used to introduce non-linear factors, converting the linear feature information input from convolutional layer 802 into non-linear feature information through the activation function. Common activation functions include ReLU and S... The pooling layer 804 is used to reduce the spatial dimension of feature information and reduce computational complexity, that is, to reduce the spatial dimension of the nonlinear feature information input to the activation function 803, obtain simplified nonlinear feature information, and at the same time help improve the generalization performance of the model. The fully connected layer 805 is used to learn the relationship between all feature information in all regions and generate the final decision. For example, the fully connected layer learns all the previous feature information to obtain the decision of "a pedestrian 10 meters in the southeast direction traveling east at a speed of 10 km / h". In one possible implementation, the target prediction module also includes an output layer 806, which includes a classifier and a regressor. The classifier is used to determine whether there is a target in each candidate region and the category of the target. The regressor is used to accurately predict the position and size of the target and generate the final target detection result.

[0121] To make the target object detection results more accurate and improve the self-learning ability of the target object prediction model, this application also provides a target object prediction module optimization method.

[0122] See Figure 9 This is a schematic diagram of a target object prediction model optimization method provided in an embodiment of this application. Figure 9 As shown, the method mainly includes the following steps.

[0123] Step S901: Select and execute the action.

[0124] Before the optimization model begins optimization, the user defines an initial state and action space through the system. The optimization model then selects and executes an action within the action space based on the user-defined initial state.

[0125] Step S902: Receive feedback and reward information from the environment.

[0126] Specifically, environmental feedback may include the vehicle's actual motion state, raw sensor readings, and information such as potential collisions or violations. Some of this information comes from the vehicle's own sensors and control system, while some may require manual annotation or input. Reward information is determined by factors such as false positives rate, false negatives rate, and detection time. False positives and false negatives rates are typically determined by comparing with labeled data, which refers to data where the correct answer is known. For example, before optimization, the optimization module has labeled which locations in an image contain vehicles and which do not. However, the optimization model cannot access this labeled data during detection. Finally, the detection results obtained by the optimization model are compared with the labeled data to derive the false positives and false negatives rates. Understandably, the lower the false positives and false negatives rates, the higher the reward information value. Deep Q-Networks (DQNs) update their value function based on the reward information. The value function describes the expected reward for different actions in each state.

[0127] Step S903: Save the state-action-reward information to the memory bank.

[0128] Specifically, the user-defined initial state, the action selected by the optimization model based on the initial state, and the received reward information are saved to the memory bank. Each state-action-reward information is a sample.

[0129] Step S904: Randomly select a batch of samples from the memory bank, calculate the expected future return, and update the Q value.

[0130] Specifically, the optimization model randomly selects multiple samples from the memory bank, calculates the expected future reward based on the initial state of the samples and the actions in the samples, and then updates the Q-value (Action-Value Function). The Q-value represents the expected reward obtained by performing a certain action in a certain state. By continuously updating the Q-value, the optimization model can find an optimal strategy that maximizes the expected reward after choosing an empty seat from any starting state.

[0131] Step S905: Determine whether the stopping condition has been met.

[0132] Specifically, the stopping condition is usually preset. It may be reaching a certain number of training rounds, the performance of the optimized model (e.g., reward information) reaching a certain threshold, or the performance of the optimized model not showing significant improvement over a period of time. Those skilled in the art can set the stopping condition to other conditions according to the actual situation, and the embodiments of this application do not limit this. If the stopping condition is met, the training ends; if the stopping condition is not met, an action is selected and executed again.

[0133] Step S906: End training.

[0134] If the stopping condition is met, the training ends.

[0135] By setting up an optimization model, the system can automatically explore and utilize environmental information to gradually improve the performance of target detection. At the same time, due to the online learning characteristics of reinforcement learning, the system can also quickly adapt to new environments and tasks, exhibiting excellent robustness and scalability.

[0136] In one possible implementation, the data acquisition equipment includes a camera system, a radar system, and a lidar system, while the data processing equipment includes a data collection module and a preprocessing module. See also Figure 10 This is a flowchart illustrating another data processing method provided in an embodiment of this application. Figure 10 As shown, it mainly includes the following steps.

[0137] Step S1001: Start the camera system.

[0138] Step S1002: Start the radar system.

[0139] Step S1003: Start the lidar system.

[0140] Step S1004: The camera system sends environmental image data to the data collection module.

[0141] Specifically, the data collection module is the module in the data processing equipment that collects data, and is used to receive raw data sent by the camera system, radar system, and lidar system. The raw data collected by the camera system is environmental image data.

[0142] Step S1005: The radar system sends distance and velocity data to the data collection module.

[0143] Specifically, the radar system sends the collected distance and velocity data of the target to the data collection module. The raw data collected by the radar system is distance and velocity data.

[0144] Step S1006: The lidar system sends three-dimensional data to the data collection module.

[0145] The raw data acquired by the lidar system is three-dimensional data. For details regarding this step, please refer to the above. Figure 1 The descriptions in the illustrated embodiments are omitted for brevity.

[0146] Step S1007: The data collection module packages all data and marks it with time information.

[0147] Specifically, the data collection module packages the image data sent by the camera system into image data packets, the distance and velocity data sent by the radar system into radar data packets, and the 3D data sent by the lidar into lidar data packets. In one possible implementation, if the raw data collected by some data acquisition devices does not include time information or the accuracy of the time information in the raw data is insufficient to meet the requirements of the data processing device, then the data processing device needs to provide time information for this raw data.

[0148] Step S1008: The data collection module sends various data packets to the preprocessing module.

[0149] Specifically, the data collection module sends the packaged data packets to the preprocessing module.

[0150] After the preprocessing module receives various data packets, it performs time alignment, spatial alignment, and feature information extraction on the data packets.

[0151] See Figure 11 This is a schematic diagram illustrating a spatial alignment process provided in an embodiment of this application. Figure 11 As shown, it mainly includes the following steps.

[0152] In one possible implementation, the raw data sent by the data acquisition device is not arranged in chronological order. Therefore, the raw data acquired by each data acquisition device needs to be arranged in chronological order before time alignment processing of the data packets.

[0153] Step S1101: Sort the camera data packets according to time information.

[0154] Step S1102: Sort the radar data packets according to time information.

[0155] Step S1103: Sort the lidar data packets according to time information.

[0156] Step S1104: Divide the camera data packets according to a preset time interval.

[0157] Step S1105: Divide the radar data packets according to a preset time interval.

[0158] Step S1106: Divide the lidar data packets according to a preset time interval.

[0159] Step S1107: Find the maximum time information in each data packet.

[0160] Step S1108: Determine whether the time information of the camera data packet is within the time threshold.

[0161] Step S1109: Determine whether the radar data packet time information is within the time threshold.

[0162] Step S1110: Determine whether the time information of the lidar data packet is within the time threshold.

[0163] Step S1111: Add the data packets that meet the criteria to the synchronization data list.

[0164] For specific details regarding the embodiments of this application, please refer to the above. Figure 2 The descriptions in the illustrated embodiments are omitted for brevity. The feature information extraction processing involved in the embodiments of this application can be found above. Figure 7 and Figure 8 The descriptions in the illustrated embodiments are omitted for brevity.

[0165] See Figure 12 This is a schematic flowchart of another target object detection method provided in an embodiment of this application. See also... Figure 12 It mainly includes the following steps.

[0166] Step S1201: The data collection module sends raw data to the preprocessing module.

[0167] For details regarding this step, please refer to the above. Figure 10 The descriptions in the illustrated embodiments are omitted for brevity.

[0168] Step S1202: The preprocessing module sends the preprocessed feature information to the deep neural network module.

[0169] Specifically, the preprocessing module performs time alignment processing, spatial alignment processing, and feature information extraction processing on the raw data, and sends the preprocessed feature information to the target object prediction model. In one possible implementation, the target object prediction model can be a deep neural network module.

[0170] Step S1203: The deep neural network module sends the target prediction result to the optimization module.

[0171] Specifically, the deep neural network module processes the feature information, as detailed above. Figure 7 and Figure 8 The descriptions in the illustrated embodiments are omitted for brevity. After processing the feature information, the deep neural network module obtains the target detection result and sends the result to the optimization module.

[0172] Step S1204: The optimization module sends the adjusted Q value to the deep neural network module.

[0173] For details regarding this step, please refer to the above. Figure 9 The descriptions in the illustrated embodiments are omitted for brevity.

[0174] Corresponding to the above embodiments, this application also provides an electronic device.

[0175] See Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 13 As shown, the electronic device 1300 may include a processor 1301, a memory 1302, and a communication unit 1303. These components communicate via one or more buses. Those skilled in the art will understand that the structure of the electronic device shown in the figure does not constitute a limitation on the embodiments of this application. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0176] The communication unit 1303 is used to establish a communication channel, enabling the electronic device to communicate with other devices. It can receive user data sent by other devices or send user data to other devices.

[0177] The processor 1301 serves as the control center of the electronic device, connecting various parts of the device via interfaces and lines. It executes software programs, instructions, and / or modules stored in the memory 1302, and calls data stored in the memory to perform various functions and / or process data. The processor may be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 1301 may consist only of a central processing unit (CPU). In this embodiment, the CPU may have a single processing core or include multiple processing cores.

[0178] The memory 1302 is used to store the execution instructions of the processor 1301. The memory 1302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0179] When the execution instructions in memory 1302 are executed by processor 1301, the electronic device 1300 is able to perform operations. Figure 1 Some or all of the steps in the illustrated embodiments.

[0180] In a specific implementation, this application embodiment also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, it may include some or all of the steps of the simulation scene generation method provided in various embodiments of this application. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0181] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0182] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0183] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0184] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0185] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments and terminal embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

Claims

1. A data processing method, characterized in that, include: Multiple sets of raw data collected by multiple data acquisition devices are divided according to a preset time interval to obtain multiple sets of raw data, wherein each set of raw data includes at least one of the raw data of each type. Each of the original data sets is time-aligned to obtain multiple time-aligned data sets, wherein the time difference between any two original data sets in each time-aligned data set is less than or equal to a preset time threshold.

2. The method according to claim 1, characterized in that, The time alignment process includes: Determine the target raw data in the original data set; If the time difference between any other original data in the original data set and the target original data is greater than a preset time threshold, then any other original data will be deleted from the original data set.

3. The method according to claim 2, characterized in that, The target raw data is either the raw data with the smallest time information or the raw data with the largest time information in the raw data set.

4. The method according to claim 1, characterized in that, The original data includes three-dimensional information, and the method further includes: Spatial alignment processing is performed on each of the time-aligned data sets to obtain multiple spatially aligned data sets, wherein all three-dimensional information in the multiple spatially aligned data sets is located in the same spatial coordinate system.

5. The method according to claim 4, characterized in that, The spatial alignment process includes: Determine the target spatial coordinate system and the spatial transformation matrix between each type of three-dimensional information in the time-aligned data set and the target spatial coordinate system; Based on the spatial transformation matrix, each type of three-dimensional information is transformed to the target spatial coordinate system.

6. The method according to claim 4, characterized in that, The method further includes: Feature extraction is performed on each of the spatially aligned datasets to obtain multiple feature information.

7. The method according to claim 6, characterized in that, The raw data also includes image information and velocity information, and the feature extraction includes: Image feature information extracted from image information; Extract frequency domain feature information based on velocity information; Spatial feature information is extracted based on three-dimensional information.

8. A method for detecting a target object, characterized in that, include: The method described in claim 6 or 7 is used to process the raw data of the target object to obtain multiple feature information; The target object prediction model is input with multiple of the aforementioned feature information to obtain the target object prediction result.

9. An electronic device, characterized in that, include: At least one processor; The memory is coupled to the processor; The processor stores a computer program that includes instructions that, when executed by the processor, cause the electronic device to perform the method described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 8.

Images