Target detection method, device, equipment and storage medium based on vehicle-road collaboration
By using the target detection network model to process point cloud data in the vehicle-road collaboration system, the problem of target classification in complex traffic scenarios is solved, accurate detection of target location and type is achieved, and detection accuracy and robustness are improved.
Patent Information
- Application Number
- CN202311814615.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-26
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-12-26
AI Technical Summary
The prior art is difficult to classify targets in complex urban traffic scenarios, and traditional clustering algorithms cannot cope with severe occlusion and complex environments between targets.
Using a vehicle-road collaboration-based object detection method, by obtaining sensor fusion data of the vehicle perception system, training the roadside object detection network model, detecting point cloud data, and using preprocessing layer, quantization layer, matrix transformation layer, and multi-layer perception Feature extraction and classification are performed by the machine and pooling layers.
It improves the accuracy and robustness of target detection, can perform accurate detection of target location and type in complex road environments, reduces the installation cost of roadside equipment, and improves the stability and generalization capabilities of the model.
Smart Images

Figure CN117893797B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a target detection method, device, equipment and storage medium based on vehicle-road collaboration. Background Art
[0002] The rapid development of social economy has brought more pressure and challenges to the urban transportation system. For example, road congestion and wrong decisions on travel routes have become thorny problems that hinder the improvement of traffic efficiency. Therefore, researchers have proposed the concept of Intelligent Transportation System (ITS), seeking to use advanced technical means to solve practical problems in traffic and provide better services for traffic participants. Monitoring equipment is an important road infrastructure in the intelligent transportation system. It can provide vehicles with surrounding road condition data, which helps driving safety and traffic efficiency.
[0003] Target detection is the most basic and important task of monitoring equipment in intelligent transportation systems. In point cloud data, the core of traditional detection methods is mostly clustering. For example, clustering algorithms are used in the scene to find backgrounds that do not move in the frame sequence, and then the targets are found based on the relative displacement between the point cloud and these backgrounds. Or the point clouds in the scene are directly clustered into clusters, and then based on some prior knowledge, such as the position and size of the target, it is determined whether these clusters belong to the targets to be detected. However, urban traffic scenes are often very complex, and there are serious occlusions between targets. It is obviously impossible to cope with these complex real-world scenes based solely on clustering algorithms and the use of prior knowledge. At the same time, the above methods can only locate the targets in the foreground in the scene, but cannot classify these targets.
[0004] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention
[0005] The main purpose of the present invention is to provide a target detection method, device, equipment and storage medium based on vehicle-road collaboration, aiming to solve the technical problem in the prior art that it is difficult to classify targets due to the complexity of road traffic.
[0006] To achieve the above object, the present invention provides a target detection method based on vehicle-road collaboration, the method comprising the following steps:
[0007] Obtain sensor fusion data collected by the vehicle perception system;
[0008] Inputting the sensor fusion data into an initial roadside object detection network model for training to obtain a roadside object detection network model;
[0009] Obtain point cloud data collected by roadside equipment in real time;
[0010] The point cloud data is detected by the roadside target detection network model to obtain a target detection classification result.
[0011] Optionally, the roadside object detection network model includes: a preprocessing layer, a quantization layer, a matrix transformation layer, a multi-layer perceptron and a pooling layer;
[0012] The step of detecting the point cloud data through the roadside target detection network model to obtain a target detection classification result includes:
[0013] Extracting point coordinates of the point cloud data through the preprocessing layer, and preprocessing the extracted data points to obtain standard data;
[0014] Performing tensor quantization on the standard data through the quantization layer to obtain a two-dimensional tensor;
[0015] Performing matrix transformation on the two-dimensional tensor through the matrix transformation layer to obtain an output matrix;
[0016] Projecting each matrix element of the output matrix into a high-dimensional space through the multi-layer perceptron to obtain high-dimensional space features;
[0017] Performing a pooling operation on the high-dimensional spatial features through the pooling layer to obtain global features;
[0018] The global features are predicted and evaluated by the multi-layer perceptron to obtain target detection classification results.
[0019] Optionally, the step of acquiring point cloud data collected in real time by roadside equipment includes:
[0020] Roadside point cloud data covering the entire road area is collected in real time by laser radar; the roadside point cloud data is packaged and pushed to the initial roadside target detection network model through a deep learning framework.
[0021] Optionally, the step of inputting the sensor fusion data into an initial roadside object detection network model for training to obtain the roadside object detection network model includes:
[0022] Extracting point coordinates of the sensor fusion data, and preprocessing the extracted data point set to obtain standard training data;
[0023] The standard training data is input into the initial roadside object detection network model for training to obtain a roadside object detection network model.
[0024] Optionally, the step of acquiring sensor fusion data collected by the vehicle perception system includes:
[0025] Obtaining sensor data collected by each vehicle sensor in the vehicle perception system;
[0026] Data fusion is performed according to the sensor data to obtain sensor fusion data.
[0027] Optionally, after the step of detecting the point cloud data by using the roadside object detection network model to obtain an object detection classification result, the step further includes:
[0028] The target detection classification result is sent to the vehicle perception system, so that the vehicle perception system performs feature fusion according to the target detection classification result and the collected real-time sensor fusion data to obtain fusion recognition data.
[0029] Optionally, the method further comprises:
[0030] The vehicle perception system inputs the fused recognition data into a vehicle-side target detection network model to obtain a vehicle-side target detection result.
[0031] In addition, to achieve the above-mentioned purpose, the present invention further proposes a target detection device based on vehicle-road collaboration, and the target detection device based on vehicle-road collaboration includes:
[0032] A training data acquisition module, used to obtain sensor fusion data collected by the vehicle perception system;
[0033] A roadside model training module, used for inputting the sensor fusion data into an initial roadside target detection network model for training to obtain a roadside target detection network model;
[0034] Roadside data collection module, used to obtain point cloud data collected by roadside equipment in real time;
[0035] The roadside target detection module is used to detect the point cloud data through the roadside target detection network model to obtain a target detection classification result.
[0036] In addition, to achieve the above-mentioned purpose, the present invention also proposes a target detection device based on vehicle-road collaboration, which includes: a memory, a processor, and a target detection program based on vehicle-road collaboration stored in the memory and executable on the processor, and the target detection program based on vehicle-road collaboration is configured to implement the steps of the target detection method based on vehicle-road collaboration as described above.
[0037] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a target detection program based on vehicle-road collaboration is stored. When the target detection program based on vehicle-road collaboration is executed by a processor, the steps of the target detection method based on vehicle-road collaboration as described above are implemented.
[0038] The present invention extracts point coordinates from sensor fusion data, pre-processes the extracted data point set, and obtains standard training data; the standard training data is input into the initial roadside target detection network model for training, and the roadside target detection network model is obtained. Since the point coordinates of the sensor fusion data are extracted, the data point set in the sensor fusion data can be determined, and the redundant points in the data point set are avoided by pre-processing the data point set; the modeling efficiency is improved by inputting the pre-processed standard training data into the initial roadside target detection network model for training, and the stability and robustness of the model are improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 It is a structural diagram of a target detection device based on vehicle-road collaboration in a hardware operating environment involved in an embodiment of the present invention;
[0040] Figure 2 A schematic diagram of a flow chart of a first embodiment of a target detection method based on vehicle-road collaboration according to the present invention;
[0041] Figure 3 A schematic diagram of a flow chart of a second embodiment of a target detection method based on vehicle-road collaboration according to the present invention;
[0042] Figure 4 A schematic diagram of a process for obtaining target detection and classification results according to the third embodiment of the present invention;
[0043] Figure 5 This is a flow chart of an application scenario of the target detection method based on vehicle-road collaboration of the present invention;
[0044] Figure 6 This is a structural block diagram of the first embodiment of the target detection device based on vehicle-road collaboration of the present invention.
[0045] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0046] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0047] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a target detection device based on vehicle-road collaboration in the hardware operating environment involved in an embodiment of the present invention.
[0048] like Figure 1 As shown, the target detection device based on vehicle-road collaboration may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (RandomAccess Memory, RAM) or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0049] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the target detection device based on vehicle-road collaboration, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.
[0050] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a target detection program based on vehicle-road collaboration.
[0051] exist Figure 1 In the target detection device based on vehicle-road collaboration shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the target detection device based on vehicle-road collaboration of the present invention can be set in the target detection device based on vehicle-road collaboration, and the target detection device based on vehicle-road collaboration calls the target detection program based on vehicle-road collaboration stored in the memory 1005 through the processor 1001, and executes the target detection method based on vehicle-road collaboration provided by an embodiment of the present invention.
[0052] The embodiment of the present invention provides a target detection method based on vehicle-road collaboration, referring to Figure 2 , Figure 2 Schematic diagram of the flow chart of the first embodiment of the target detection method based on vehicle-road collaboration of the present invention.
[0053] In this embodiment, the target detection method based on vehicle-road collaboration includes the following steps:
[0054] Step S10: Acquire sensor fusion data collected by the vehicle perception system.
[0055] It should be noted that the execution subject of the method of this embodiment can be a terminal device with data collection, data processing and program running functions, such as a single-chip microcomputer, an embedded operating system, etc., or an electronic device with the same or similar functions, such as the above-mentioned target detection device based on vehicle-road collaboration. The following takes the target detection device based on vehicle-road collaboration (hereinafter referred to as the detection device) as an example to illustrate this embodiment and the following embodiments.
[0056] It should be noted that Intelligent Vehicle Infrastructure Cooperative Systems (IVICS), referred to as vehicle-infrastructure cooperative systems, is an important part of intelligent transportation systems.
[0057] It should be understood that in the vehicle-road cooperative system, roadside equipment, such as remote sensing laser radar, cameras, etc., can be set up at the edge of the road, signal poles at intersections, etc. Through these roadside equipment, environmental information in road traffic can be collected.
[0058] It is understandable that the vehicle perception system can be set up on the vehicle side and is a system used to help the vehicle identify and perceive the surrounding environment. The vehicle perception system can usually be composed of multiple sensors, such as radar, camera, lidar, ultrasonic sensor, etc. Through these sensors, the vehicle can obtain various environmental information in the vehicle's surrounding environment, which can include trees, pedestrians, vehicles, stations, fences, street lights and other targets.
[0059] It should be understood that sensor fusion data is the data obtained by integrating and fusing the information obtained by multiple different types of sensors in the vehicle perception system. Through sensor fusion data, vehicle perception information can be obtained more comprehensively and accurately. That is, the step of obtaining sensor fusion data collected by the vehicle perception system includes: obtaining sensor data collected by each vehicle sensor in the vehicle perception system; performing data fusion based on the sensor data to obtain sensor fusion data.
[0060] In practical applications, since a single sensor may be affected by noise, environmental interference, etc., resulting in inaccurate sensor data obtained by a single sensor, the accuracy of the data can be improved by fusing the information obtained by multiple different types of sensors to obtain sensor fusion data.
[0061] It should be noted that the above-mentioned sensor fusion data of the present invention may be sensor fusion data obtained in real time during vehicle driving, or historical sensor fusion data, or sensor fusion data determined by other sensor data sets, etc. The embodiment of the present invention does not limit the source of the sensor fusion data.
[0062] It should be understood that the sensor data fusion method may adopt a Kalman filter algorithm, a particle wave algorithm, an extended Kalman filter algorithm, a weighted average method, etc., and the embodiment of the present invention does not limit this.
[0063] In a specific implementation, the detection device can obtain the sensor fusion data of the vehicle perception system. By analyzing the sensor fusion data, the environmental information of the vehicle's surrounding environment can be obtained more comprehensively and accurately.
[0064] Step S20: inputting the sensor fusion data into an initial roadside object detection network model for training to obtain a roadside object detection network model.
[0065] It should be understood that when the sensor fusion data is obtained, the sensor fusion data can be input into the initial roadside object detection network model for model training, so as to obtain the roadside object detection network model.
[0066] It is understandable that compared with roadside equipment, the sensor fusion data collected by the vehicle perception system has a wider field of view coverage, and can collect fields of view on both sides of the road and in front of the vehicle, which can make the target detection model have better generalization ability and adaptability. In addition, because the information collected by the vehicle perception system is usually real-time and dynamic, the movement trajectory and posture changes of targets such as vehicles and pedestrians can be better detected through sensor fusion data, which is more conducive to the detection of targets in traffic scenes and helps to improve the accuracy and robustness of target detection and target classification. Moreover, through vehicle-road collaboration, the installed roadside equipment can be effectively reduced, the design and maintenance costs of the intelligent transportation system can be reduced, and data acquisition is also more convenient and quick.
[0067] It should be noted that the above-mentioned roadside target detection network model is a model that can be used to perform target detection and target classification on input data. The target detection model can be constructed based on the R-CNN network model, or based on the SSD network model, or based on the Yolo network model, and the embodiment of the present invention does not impose any restrictions thereto.
[0068] In the specific implementation, the detection equipment inputs the sensor fusion data into the initial roadside target detection network model for training to obtain the target detection network model. The sensor fusion data helps to improve the accuracy and robustness of target detection and target classification.
[0069] Step S30: Acquire point cloud data collected by roadside equipment in real time;
[0070] Step S40: Detect the point cloud data using the roadside target detection network model to obtain a target detection classification result.
[0071] It is understandable that the roadside equipment can collect road traffic conditions in real time, thereby obtaining road condition information, signal light status, road signs and other road traffic environment data from the roadside perspective, and generate corresponding point cloud data. By inputting the point cloud data into the roadside target detection model for detection, the target detection classification result can be obtained.
[0072] Specifically, the step of obtaining point cloud data collected by roadside equipment in real time includes: collecting roadside point cloud data covering the entire road area in real time through lidar; the roadside point cloud data is packaged and pushed to the initial roadside target detection network model through a deep learning framework.
[0073] It needs to be explained that when the roadside point cloud data collected by the lidar is obtained, the roadside point cloud data can be packaged through a pre-written script, and the packaged roadside point cloud data can be pushed to the roadside target detection network model for detection through a deep learning framework with a Python interface.
[0074] It should be noted that the above target detection classification results can be expressed in the form of target detection classification scores. By identifying the point cloud data through the roadside target detection network model, the classification scores of each target type corresponding to the target in the collected point cloud data can be determined, thereby determining the target type included in the collected point cloud data. For example, if the collected point cloud data includes a target, the classification score of the tree type is 0.3, the classification score of the vehicle type is 0.1, and the classification score of the pedestrian type is 0.93, then the target type of the target can be determined as a pedestrian.
[0075] It can be understood that the target detection classification score outputted above can be a feature similarity value between the target and each target type, and its feature similarity value can be between [0, 1]. The closer to 0, the lower the similarity between the target and the target type, and the closer to 1, the higher the similarity between the target and the target type.
[0076] In the specific implementation, the detection equipment can obtain the point cloud data collected by the roadside equipment in real time, and detect the point cloud data through the roadside target detection network model to obtain the target detection classification results. By inputting the point cloud data collected by the roadside equipment into the roadside target detection model for detection, the accuracy of target position detection and classification can be improved, and the problem of incomplete target detection caused by complex road environments such as vehicle movement or occlusion can be avoided. During the detection process, the roadside target detection network model can also be continuously iterated and optimized according to the detection results, thereby enriching the data samples, improving the generalization and anti-interference capabilities of the model, and reducing the problems caused by the limited amount of vehicle-side data (that is, sensor fusion data) or the lack of samples in specific scenarios.
[0077] Furthermore, after the step of detecting the point cloud data through the roadside target detection network model to obtain the target detection classification result, the step further includes:
[0078] Step S50: sending the target detection classification result to the vehicle perception system, so that the vehicle perception system performs feature fusion according to the target detection classification result and the collected real-time sensor fusion data to obtain fusion recognition data.
[0079] It is understandable that when the target detection classification result is determined, the target detection classification result can be sent to the vehicle perception system, so that the vehicle perception system can perform feature fusion according to the sensor fusion data collected in real time, and further detect according to the fused fusion recognition data to obtain the vehicle side target detection result; wherein the vehicle side target detection result includes target position information and target type information. Specifically, the method also includes: the vehicle perception system inputs the fusion recognition data into the vehicle side target detection network model to obtain the vehicle side target detection result.
[0080] It should be noted that the above method of sending the target detection classification results to the vehicle perception system can be transmitted through 5G-RSU communication equipment or through other methods, and the embodiments of the present invention are not limited to this.
[0081] It should be understood that real-time sensor fusion data is also sensor fusion data collected in real time by the vehicle perception system. During the operation of the vehicle, environmental information can be collected in real time through various sensors, and the vehicle perception system can fuse the data collected in real time by various sensors to obtain real-time sensor fusion data.
[0082] It can be understood that since the target detection classification result is the classification result obtained by detecting the point cloud data collected by the roadside equipment in real time at the current time; by sending the target detection classification result to the vehicle and fusing it with the real-time sensor data obtained by the vehicle perception system, real-time fusion recognition data can be obtained. By detecting the fusion recognition data, the position of each target in the road traffic environment around the vehicle and the target type corresponding to each position target can be determined.
[0083] It should be understood that the above-mentioned vehicle-side target detection network model is a network model for determining the vehicle-side target detection result based on the sensor fusion data on the vehicle side and the target detection classification result on the road side. The embodiment of the present invention does not limit the specific structure and implementation method of the vehicle-side target detection network model, and different network models can be selected according to the needs of actual applications.
[0084] In the specific implementation, the detection equipment obtains the point cloud data collected by the roadside equipment in real time, and detects the point cloud data through the roadside target detection network model to obtain the target detection classification result. The target detection classification result can be sent to the vehicle perception system, and feature fused with the real-time sensor fusion data collected by the vehicle perception system to obtain fusion recognition data, and the target detection result on the vehicle side can be determined based on the fusion recognition data. Because it is based on vehicle-road collaboration, the roadside target detection model is built through the vehicle-side data, and the point cloud data collected on the roadside is detected through the roadside target detection model to obtain the target detection classification result, so that the vehicle can perform detection based on the target detection classification result and the real-time collected sensor fusion data, and determine the vehicle-side target detection result (including target position detection result and target type detection result), thereby improving the accuracy of target detection and realizing target positioning and target type detection in complex road environments.
[0085] The embodiment of the present invention obtains sensor fusion data collected by the vehicle perception system; inputs the sensor fusion data into the initial roadside target detection network model for training to obtain a roadside target detection network model; obtains point cloud data collected by roadside equipment in real time; detects the point cloud data through the roadside target detection network model to obtain target detection classification results. Since the modeling is based on the sensor fusion data collected by the vehicle perception system, the point cloud data collected by the roadside equipment in real time is identified, and the vehicle-road collaboration of road traffic is realized, and the environmental information of the vehicle's surrounding environment can be obtained more comprehensively and accurately, which improves the accuracy of target detection, so that in a complex road traffic environment, target type detection can be performed while detecting the target position.
[0086] refer to Figure 3 , Figure 3Schematic diagram of the flow chart of the second embodiment of the target detection method based on vehicle-road collaboration of the present invention.
[0087] Based on the above first embodiment, in this embodiment, the step of inputting the sensor fusion data into the initial roadside object detection network model for training to obtain the roadside object detection network model includes:
[0088] Step S21: extracting point coordinates of the sensor fusion data, and preprocessing the extracted data point set to obtain standard training data.
[0089] It can be understood that point coordinate extraction is the process of extracting the spatial coordinate position of a target or a point of interest from an image or sensor fusion data. Point coordinates can be used to represent the geometric position of a target in space. By extracting point coordinates, the target can be located and identified.
[0090] In one implementation, the point coordinates of the sensor fusion data may be extracted by a target tracking algorithm to obtain corresponding data points. For example, the point coordinates may be extracted by using a tracking algorithm, an edge detection algorithm, etc.
[0091] It should be noted that for the extracted data point set, preprocessing methods such as random sampling and normalization can be performed to reduce the size of the data set, avoid redundant points in the data set, and make the data more balanced and uniform. At the same time, it speeds up the subsequent modeling and improves the stability and robustness of the model.
[0092] In a specific implementation, the detection device extracts point coordinates from the sensor fusion data, pre-processes the extracted data point set, obtains standard training data, and extracts point coordinates from the sensor fusion data to determine the target position in the sensor fusion data, so that the vehicle can avoid obstacles or perform other operations based on the target location, and the detection device can further determine the type of target based on the target position.
[0093] Step S22: inputting the standard training data into the initial roadside object detection network model for training to obtain a roadside object detection network model.
[0094] It is understandable that when the standard training data is obtained, the standard training data can be input into the initial roadside object detection network model for model training, thereby obtaining the roadside object detection network model.
[0095] The embodiment of the present invention extracts point coordinates from sensor fusion data, preprocesses the extracted data point set, and obtains standard training data; the standard training data is input into the initial roadside target detection network model for training to obtain the roadside target detection network model. Since the point coordinates of the sensor fusion data are extracted, the data point set in the sensor fusion data can be determined, and the redundant points in the data point set are avoided by preprocessing the data point set; the modeling efficiency is improved by inputting the preprocessed standard training data into the initial roadside target detection network model for training, and the stability and robustness of the model are improved.
[0096] Based on the above embodiments, in order to obtain target detection and classification results, a third embodiment of the method of the present invention is proposed. Figure 4 As shown, Figure 4 A schematic diagram of a process for obtaining target detection and classification results according to the third embodiment of the present invention;
[0097] In this embodiment, the roadside object detection network model includes: a preprocessing layer, a quantization layer, a matrix transformation layer, a multi-layer perceptron and a pooling layer;
[0098] The step of detecting the point cloud data through the roadside target detection network model to obtain a target detection classification result includes:
[0099] Extracting point coordinates of the point cloud data through the preprocessing layer, and preprocessing the extracted data points to obtain standard data;
[0100] Performing tensor quantization on the standard data through the quantization layer to obtain a two-dimensional tensor;
[0101] Performing matrix transformation on the two-dimensional tensor through the matrix transformation layer to obtain an output matrix;
[0102] Projecting each matrix element of the output matrix into a high-dimensional space through the multi-layer perceptron to obtain high-dimensional space features;
[0103] Performing a pooling operation on the high-dimensional spatial features through the pooling layer to obtain global features;
[0104] The global features are predicted and evaluated by the multi-layer perceptron to obtain target detection classification results.
[0105] It can be understood that the roadside object detection network model may include a preprocessing layer, at least one quantization layer, at least one matrix transformation layer, at least one multi-layer perceptron and a pooling layer.
[0106] Specifically, the input point cloud data can be used to extract point coordinates, and then the extracted point set can be subjected to preprocessing operations such as random sampling and normalization to obtain standard data. By tensorizing the standard data, a two-dimensional tensor represented as nx3 is obtained, where n represents the number of point clouds and 3 represents the corresponding xyz coordinates. By directly processing the 3D point cloud itself with a 2D convolutional neural network, point set features can be stably extracted even if the point cloud fluctuates, is noisy or missing.
[0107] Furthermore, by performing matrix transformation on the two-dimensional tensor through the TMNet model, a matrix of n*3 x 3*3 or n*3 can be output, that is, the output matrix. By projecting each point in the output matrix into a high-dimensional space through a multi-layer perceptron, high-dimensional spatial features can be obtained. Specifically, each point in the output matrix can be projected into a 64-dimensional space through a multi-layer perceptron, and then the data projected into the 64-dimensional space can be matrix transformed to obtain a 64-dimensional matrix with a higher degree of normalization; by further mapping the 64-dimensional matrix through a multi-layer perceptron, 1024-dimensional features can be obtained, that is, high-dimensional spatial features.
[0108] It should be noted that the TMNet model of the present invention is a model that can align the features of input and different points so that the expression obtained by network learning also has this characteristic. It improves the network's ability to extract and express vehicle target features in high-dimensional feature space while taking into account the network reasoning speed.
[0109] Furthermore, the pooling layer performs a maximum pooling operation on 1024 dimensions of the high-dimensional spatial features to obtain the final global features.
[0110] It should be noted that the pooling layer of the present invention can be a maximum pooling layer. An important feature of 3D point cloud is disorder. For a point set of NNN points, it has N! N! N! input orders. To this end, the present invention solves the problem of disorder by performing symmetric operations (i.e., maxpooling operations) through symmetric functions. At the same time, the maximum pooling layer aggregates independently learned point features into global point set features, and then performs subsequent 3D recognition tasks. The prediction results of the network should be invariant to specific transformations, such as rigid transformations.
[0111] It should be explained that the disorder of point clouds can be solved by symmetric functions, so the original point cloud can be directly used as input. Compared with traditional voxel-based methods, it occupies less video memory space.
[0112] Furthermore, the global feature can be predicted and evaluated through a multi-layer perceptron to obtain the target detection classification score, that is, the target detection classification result.
[0113] This embodiment extracts point coordinates from point cloud data through a preprocessing layer, and preprocesses the extracted data points to obtain standard data; quantizes the standard data through a quantization layer to obtain a two-dimensional tensor; performs matrix transformation on the two-dimensional tensor through a matrix transformation layer to obtain an output matrix; projects each matrix element of the output matrix into a high-dimensional space through a multi-layer perceptron to obtain high-dimensional space features; performs pooling operations on high-dimensional space features through a pooling layer to obtain global features; and predicts and evaluates global features through a multi-layer perceptron to obtain target detection classification results. Since the point cloud data input through the preprocessing layer, quantization layer, matrix transformation layer, multi-layer perceptron, and pooling layer are processed, the 3D point cloud itself can be directly obtained through the 2D roadside target detection network model, and even if the point cloud fluctuates, is noisy, or missing, the point set features can be stably extracted; the symmetrical maxpooling operation is performed through the pooling layer to solve the disorder of the point cloud, thereby realizing target detection and classification through the point cloud data collected from the roadside.
[0114] like Figure 5 As shown, Figure 5 This is a flow chart of an application scenario of the target detection method based on vehicle-road collaboration of the present invention.
[0115] The vehicle perception system collects sensor fusion data: The vehicle perception system can obtain the sensor data collected by each sensor, and fuse the sensor data to obtain sensor fusion data; and send the sensor fusion data to the detection equipment.
[0116] Build and deploy a roadside target detection network model: The detection device can input the received sensor fusion data into the initial target detection network model for model training, thereby obtaining a roadside target detection network model; when a converged roadside target detection network model is obtained, the roadside target detection network model can be deployed.
[0117] Roadside equipment collects point cloud data: After the roadside target detection network model is deployed, the roadside equipment can collect point cloud data and send the point cloud data to the detection equipment. The detection equipment can package the point cloud collected from the roadside equipment and input it into the roadside target detection network model for detection, thereby obtaining the target detection classification result.
[0118] Furthermore, the detection equipment can transmit the target detection and classification results to the vehicle perception system through the 5G-RSU communication device and fuse them to obtain fused recognition data. Based on this fused recognition data, more accurate and reliable road user and obstacle position recognition and target classification can be achieved.
[0119] In addition, an embodiment of the present invention also proposes a storage medium, on which a target detection program based on vehicle-road collaboration is stored. When the target detection program based on vehicle-road collaboration is executed by a processor, the steps of the target detection method based on vehicle-road collaboration as described above are implemented.
[0120] Based on the first embodiment of the target detection method based on vehicle-road collaboration of the present invention, the first embodiment of the target detection device based on vehicle-road collaboration of the present invention is proposed. Figure 6 , Figure 6 This is a structural block diagram of the first embodiment of the target detection device based on vehicle-road collaboration of the present invention.
[0121] like Figure 6 As shown, the target detection device based on vehicle-road collaboration proposed in an embodiment of the present invention includes:
[0122] The training data acquisition module 601 is used to obtain sensor fusion data collected by the vehicle perception system;
[0123] A roadside model training module 602 is used to input the sensor fusion data into an initial roadside object detection network model for training to obtain a roadside object detection network model;
[0124] The roadside data collection module 603 is used to obtain point cloud data collected by roadside equipment in real time;
[0125] The roadside target detection module 604 is used to detect the point cloud data through the roadside target detection network model to obtain a target detection classification result.
[0126] The embodiment of the present invention extracts point coordinates from sensor fusion data, preprocesses the extracted data point set, and obtains standard training data; the standard training data is input into the initial roadside target detection network model for training to obtain the roadside target detection network model. Since the point coordinates of the sensor fusion data are extracted, the data point set in the sensor fusion data can be determined, and the redundant points in the data point set are avoided by preprocessing the data point set; the modeling efficiency is improved by inputting the preprocessed standard training data into the initial roadside target detection network model for training, and the stability and robustness of the model are improved.
[0127] Furthermore, the roadside target detection network model includes: a preprocessing layer, a quantization layer, a matrix transformation layer, a multi-layer perceptron and a pooling layer; the roadside target detection module 604 is also used to extract point coordinates of the point cloud data through the preprocessing layer, and preprocess the extracted data points to obtain standard data; tensorize the standard data through the quantization layer to obtain a two-dimensional tensor; perform matrix transformation on the two-dimensional tensor through the matrix transformation layer to obtain an output matrix; project each matrix element of the output matrix into a high-dimensional space through the multi-layer perceptron to obtain high-dimensional space features; perform pooling operations on the high-dimensional space features through the pooling layer to obtain global features; predict and evaluate the global features through the multi-layer perceptron to obtain target detection classification results.
[0128] Furthermore, the roadside data collection module 603 is also used to collect roadside point cloud data covering the entire road area in real time through lidar; the roadside point cloud data is used to be packaged and pushed to the initial roadside target detection network model through a deep learning framework.
[0129] Furthermore, the roadside model training module 602 is also used to extract point coordinates from the sensor fusion data, preprocess the extracted data point set, and obtain standard training data; input the standard training data into the initial roadside target detection network model for training to obtain a roadside target detection network model.
[0130] Furthermore, the training data acquisition module 601 is also used to obtain sensor data collected by each vehicle sensor in the vehicle perception system; perform data fusion according to the sensor data to obtain sensor fusion data.
[0131] Furthermore, the target detection device also includes: a detection result sending module; the detection result sending module is used to send the target detection classification result to the vehicle perception system, so that the vehicle perception system performs feature fusion based on the target detection classification result and the collected real-time sensor fusion data to obtain fusion recognition data.
[0132] Furthermore, the detection result sending module is also used for the vehicle perception system to input the fused recognition data into the vehicle side target detection network model to obtain the vehicle side target detection result.
[0133] Other embodiments or specific implementation methods of the target detection device based on vehicle-road collaboration of the present invention can refer to the above-mentioned method embodiments and will not be repeated here.
[0134] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0135] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0136] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0137] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A target detection method based on vehicle-road collaboration, characterized in that: The method comprises: Obtain sensor fusion data collected by the vehicle perception system; Inputting the sensor fusion data into an initial roadside object detection network model for training to obtain a roadside object detection network model; Obtain point cloud data collected by roadside equipment in real time; Detecting the point cloud data through the roadside target detection network model to obtain a target detection classification result; The roadside object detection network model includes: Preprocessing layer, quantization layer, matrix transformation layer, multi-layer perceptron and pooling layer; The step of detecting the point cloud data through the roadside target detection network model to obtain a target detection classification result includes: Extracting point coordinates of the point cloud data through the preprocessing layer, and preprocessing the extracted data points to obtain standard data; Performing tensor quantization on the standard data through the quantization layer to obtain a two-dimensional tensor; Performing matrix transformation on the two-dimensional tensor through the matrix transformation layer to obtain an output matrix; Projecting each matrix element of the output matrix into a high-dimensional space through the multi-layer perceptron to obtain high-dimensional space features; Performing a pooling operation on the high-dimensional spatial features through the pooling layer to obtain global features; The global features are predicted and evaluated by the multi-layer perceptron to obtain target detection classification results.
2. The target detection method based on vehicle-road collaboration according to claim 1, characterized in that: The step of obtaining point cloud data collected in real time by roadside equipment includes: Collect roadside point cloud data covering the entire road area in real time through LiDAR; The roadside point cloud data is used to be packaged and pushed to the initial roadside object detection network model through a deep learning framework.
3. The target detection method based on vehicle-road collaboration according to claim 1, characterized in that: The step of inputting the sensor fusion data into the initial roadside object detection network model for training to obtain the roadside object detection network model includes: Extracting point coordinates of the sensor fusion data, and preprocessing the extracted data point set to obtain standard training data; The standard training data is input into the initial roadside object detection network model for training to obtain a roadside object detection network model.
4. The target detection method based on vehicle-road collaboration according to claim 1, characterized in that: The step of obtaining sensor fusion data collected by the vehicle perception system includes: Obtaining sensor data collected by each vehicle sensor in the vehicle perception system; Data fusion is performed according to the sensor data to obtain sensor fusion data.
5. The target detection method based on vehicle-road collaboration according to claim 1, characterized in that: After the step of detecting the point cloud data by the roadside target detection network model to obtain the target detection classification result, the method further includes: The target detection classification result is sent to the vehicle perception system, so that the vehicle perception system performs feature fusion according to the target detection classification result and the collected real-time sensor fusion data to obtain fusion recognition data.
6. The target detection method based on vehicle-road collaboration according to claim 5, characterized in that: The method further comprises: The vehicle perception system inputs the fused recognition data into a vehicle-side target detection network model to obtain a vehicle-side target detection result.
7. A target detection device based on vehicle-road collaboration, characterized in that: The target detection device comprises: A training data acquisition module, used to obtain sensor fusion data collected by the vehicle perception system; A roadside model training module, used for inputting the sensor fusion data into an initial roadside target detection network model for training to obtain a roadside target detection network model; Roadside data collection module, used to obtain point cloud data collected by roadside equipment in real time; A roadside target detection module, used to detect the point cloud data through the roadside target detection network model to obtain a target detection classification result; The roadside object detection network model includes: Preprocessing layer, quantization layer, matrix transformation layer, multi-layer perceptron and pooling layer; The operation of detecting the point cloud data through the roadside target detection network model to obtain a target detection classification result includes: Extracting point coordinates of the point cloud data through the preprocessing layer, and preprocessing the extracted data points to obtain standard data; Performing tensor quantization on the standard data through the quantization layer to obtain a two-dimensional tensor; Performing matrix transformation on the two-dimensional tensor through the matrix transformation layer to obtain an output matrix; Projecting each matrix element of the output matrix into a high-dimensional space through the multi-layer perceptron to obtain high-dimensional space features; Performing a pooling operation on the high-dimensional spatial features through the pooling layer to obtain global features; The global features are predicted and evaluated by the multi-layer perceptron to obtain target detection classification results.
8. A target detection device based on vehicle-road collaboration, characterized in that: The device comprises: A memory, a processor, and a target detection program based on vehicle-road collaboration stored in the memory and executable on the processor, wherein the target detection program based on vehicle-road collaboration is configured to implement the steps of the target detection method based on vehicle-road collaboration as described in any one of claims 1 to 6.
9. A storage medium, characterized in that: The storage medium stores a target detection program based on vehicle-road collaboration, and when the target detection program based on vehicle-road collaboration is executed by the processor, the steps of the target detection method based on vehicle-road collaboration as described in any one of claims 1 to 6 are implemented.
Citation Information
Patent Citations
Target detection method and device, equipment and storage medium
CN113627562A
Three-dimensional target detection and identification method based on multi-source fusion in vehicle-road cooperation scene
CN114332494A
Intelligent traffic target fusion detection method and system based on image and point cloud
CN114359181A
Vehicle detection method and device based on roadside multi-sensor fusion and medium
CN114677446A
Three-dimensional environment target detection method based on multi-sensor fusion
CN115049821A
