Target detection method and device fusing roadside multi-mode top view features

Abstract

The invention relates to the technical field of target detection, in particular to a target detection method and device fusing roadside multi-mode top view features, and the method comprises the steps: obtaining the roadside sensing data of a to-be-detected target based on the final roadside radar sensing data and final roadside image sensing data of the to-be-detected target; generating a multi-modal top view feature of the to-be-detected target based on the roadside sensing data; and inputting the multi-modal top view features into a pre-constructed convolutional neural network to obtain a target detection result of the to-be-detected target. Therefore, the problem that the sensing range of the top view is limited in a limited area around the vehicle due to the adoption of a vehicle sensor data method in the related technology is solved. And a roadside top view method lacks a radar data processing mechanism, so that the comprehensive perception capability of the top view in a complex traffic scene is influenced, and the optimal performance is difficult to realize in practical application.

Description

Technical Field

[0001] The present application relates to the technical field of target detection, and in particular to a target detection method and device integrating roadside multi-modal top view features. Background Art

[0002] Top-down view perception technology in autonomous driving has received widespread attention in recent years. It uses sensor data to generate a top-down view, which enables accurate perception of the vehicle and the surrounding environment from a global perspective.

[0003] In related technologies, the multi-camera image information of the vehicle can be converted into a temporally and spatially consistent overhead view representation based on the Transformer architecture. The core idea is to project image information from different perspectives into a unified overhead view coordinate system through spatial transformation, and use the Transformer's self-attention mechanism to fuse multiple frames of images in the time dimension to extract the dynamic features of the environment. The performance of perception, prediction, and planning tasks can also be enhanced by fusing the vehicle's camera and lidar data, thereby improving the performance of the overhead view method compared to using only cameras. A dense overhead view network can also be designed through multiple roadside cameras to provide a wide field of view, reduce blind spots, and improve environmental perception capabilities.

[0004] However, in the related art, the method of using vehicle sensor data results in the top view's perception range being limited to a limited area around the vehicle. The roadside top view method lacks a mechanism for processing radar data, which affects the top view's comprehensive perception capabilities in complex traffic scenes, making it difficult to achieve optimal performance in practical applications, and it is in urgent need of improvement. Summary of the invention

[0005] The present application provides a target detection method and device integrating roadside multimodal top view features, in order to solve the problem that the method using vehicle sensor data in the related art results in the top view sensing range being limited to a limited area around the vehicle. The roadside top view method lacks a radar data processing mechanism, which affects the top view's comprehensive sensing capability in complex traffic scenes, making it difficult to achieve optimal performance in practical applications.

[0006] The first aspect of the present application provides a target detection method that integrates roadside multimodal overhead view features, including the following steps: obtaining roadside perception data of the target to be detected based on the final roadside radar perception data and the final roadside image perception data of the target to be detected; generating multimodal overhead view features of the target to be detected based on the roadside perception data; and inputting the multimodal overhead view features into a pre-constructed convolutional neural network to obtain the target detection result of the target to be detected.

[0007] Optionally, in one embodiment of the present application, the roadside perception data of the target to be detected is obtained based on the final roadside radar perception data and the final roadside image perception data of the target to be detected, including: obtaining the initial roadside radar perception data of the target to be detected, and preprocessing the initial roadside radar perception data to obtain the final roadside radar perception data that meets a first preset condition; obtaining the initial roadside image perception data of the target to be detected, and preprocessing the initial roadside image perception data to obtain the final roadside image perception data that meets a second preset condition; obtaining the roadside perception data based on the final roadside radar perception data and the final roadside image perception data.

[0008] Optionally, in one embodiment of the present application, the generating of the multimodal overhead view features of the target to be detected based on the roadside perception data includes: generating radar overhead view features of the final roadside radar perception data using the final roadside radar perception data in the roadside perception data; generating image overhead view features of the final roadside image perception data using the final roadside image perception data in the roadside perception data; and obtaining the multimodal overhead view features based on the radar overhead view features and the image overhead view features.

[0009] Optionally, in one embodiment of the present application, the multimodal top view feature is input into a pre-constructed convolutional neural network to obtain the target detection result of the target to be detected, including: based on the multimodal top view feature, combined with the convolutional neural network in the pre-constructed convolutional neural network, to obtain the final multimodal top view feature of the target to be detected; based on the final multimodal top view feature, combined with the classification module in the pre-constructed convolutional neural network, to obtain the target classification and positioning information of the target to be detected; based on the spatial information of the final multimodal top view feature, combined with the recognition module in the pre-constructed convolutional neural network, to obtain the target category and position information of the target to be detected; based on the target classification and positioning information and the target category and position information, the target detection result is obtained.

[0010] Optionally, in one embodiment of the present application, the multimodal top view features are input into a pre-built convolutional neural network to obtain the target detection result of the target to be detected, including: determining whether the target detection results are repeated; if the target detection results are repeated, deleting the repeated target detection results, otherwise, obtaining the target detection result of the target to be detected.

[0011] The second aspect of the present application provides a target detection device that integrates roadside multimodal overhead view features, including: a first generation module, used to obtain roadside perception data of the target to be detected based on the final roadside radar perception data and the final roadside image perception data of the target to be detected; a second generation module, used to generate multimodal overhead view features of the target to be detected based on the roadside perception data; a third generation module, used to input the multimodal overhead view features into a pre-constructed convolutional neural network to obtain the target detection result of the target to be detected.

[0012] Optionally, in one embodiment of the present application, the first generation module includes: a first acquisition unit, used to acquire initial roadside radar perception data of the target to be detected, and preprocess the initial roadside radar perception data to obtain the final roadside radar perception data that meets the first preset condition; a second acquisition unit, used to acquire initial roadside image perception data of the target to be detected, and preprocess the initial roadside image perception data to obtain the final roadside image perception data that meets the second preset condition; a first generation unit, used to obtain the roadside perception data based on the final roadside radar perception data and the final roadside image perception data.

[0013] Optionally, in one embodiment of the present application, the second generation module includes: a second generation unit, used to generate a radar overhead view feature of the final roadside radar perception data using the final roadside radar perception data in the roadside perception data; a third generation unit, used to generate an image overhead view feature of the final roadside image perception data using the final roadside image perception data in the roadside perception data; and a fourth generation unit, used to obtain the multimodal overhead view feature based on the radar overhead view feature and the image overhead view feature.

[0014] Optionally, in one embodiment of the present application, the third generation module includes: a fifth generation unit, which is used to obtain the final multimodal top view feature of the target to be detected based on the multimodal top view feature in combination with the convolutional neural network in the pre-constructed convolutional neural network; a sixth generation unit, which is used to obtain the target classification and positioning information of the target to be detected based on the final multimodal top view feature in combination with the classification module in the pre-constructed convolutional neural network; a seventh generation unit, which is used to obtain the target category and position information of the target to be detected based on the spatial information of the final multimodal top view feature in combination with the recognition module in the pre-constructed convolutional neural network; and an eighth generation unit, which is used to obtain the target detection result based on the target classification and positioning information and the target category and position information.

[0015] Optionally, in one embodiment of the present application, the third generation module includes: a judgment module, used to judge whether the target detection results are repeated; a ninth generation module, used to delete the repeated target detection results when the target detection results are repeated, otherwise, obtain the target detection results of the target to be detected.

[0016] The third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a target detection method integrating roadside multi-modal top view features as described in the above embodiment.

[0017] The fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned target detection method that integrates the multi-modal top view features of the roadside.

[0018] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, implements the above-mentioned target detection method that integrates roadside multi-modal top view features.

[0019] The embodiment of the present application can obtain roadside perception data based on the final roadside radar perception data and the final roadside image perception data of the target to be detected, and then generate a multimodal top view feature of the target to be detected, and input the multimodal top view feature into the pre-built convolutional neural network, thereby obtaining the target detection result of the target to be detected. By adopting the multimodal sensor fusion technology, the data of the multi-view camera and the radar sensor are deeply fused, and the accuracy of target detection is improved through the top view enhancement module processing. In addition, compared with the single-vehicle perception system, the roadside top view system can provide more viewing angles, so as to more accurately perceive the target behind the obstruction, provide a safer driving environment for autonomous driving vehicles, and reduce the occurrence of traffic accidents. This not only helps to formulate more scientific traffic policies, but also can provide a strong basis for future road expansion and intelligent transportation infrastructure planning, and promote the intelligent transformation of urban transportation systems. Thus, the method of using vehicle sensor data in the related technology is solved, which causes the perception range of the top view to be limited to a limited area around the vehicle. However, the method of the roadside top view lacks a mechanism for radar data processing, which affects the comprehensive perception ability of the top view in complex traffic scenes, making it difficult to achieve optimal performance in practical applications.

[0020] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0022] Figure 1 A flow chart of a target detection method integrating roadside multi-modal top view features provided in accordance with an embodiment of the present application;

[0023] Figure 2 A flowchart of the working principle of a target detection method integrating roadside multi-modal top view features provided in accordance with an embodiment of the present application;

[0024] Figure 3 A block diagram of a target detection device integrating roadside multi-modal top view features provided in accordance with an embodiment of the present application;

[0025] Figure 4 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0027] The following describes the target detection method and device integrating roadside multimodal top view features in the embodiment of the present application with reference to the accompanying drawings. With respect to the method of using vehicle sensor data mentioned in the above background technology, the perception range of the top view is limited to a limited area around the vehicle. However, the roadside top view method lacks a radar data processing mechanism, which affects the overall perception ability of the top view in complex traffic scenes, making it difficult to achieve optimal performance in practical applications. The present application provides a target detection method that integrates roadside multimodal top view features. In this method, roadside perception data can be obtained based on the final roadside radar perception data and the final roadside image perception data of the target to be detected, and then the multimodal top view features of the target to be detected are generated, and the multimodal top view features are input into a pre-built convolutional neural network, thereby obtaining the target detection result of the target to be detected. By adopting multimodal sensor fusion technology, the data of multi-view cameras and radar sensors are deeply fused, and the top view enhancement module is used to process the target detection accuracy. In addition, compared with a single vehicle-mounted perception system, the roadside top view system can provide more viewing angles, thereby more accurately perceiving the target behind the occlusion, providing a safer driving environment for autonomous driving vehicles, and reducing the occurrence of traffic accidents. This not only helps to formulate more scientific traffic policies, but also provides a strong basis for future road expansion and intelligent transportation infrastructure planning, and promotes the intelligent transformation of urban transportation systems. This solves the problem that the method of using vehicle sensor data in related technologies limits the perception range of the top view to a limited area around the vehicle. The roadside top view method lacks a mechanism for processing radar data, which affects the overall perception ability of the top view in complex traffic scenes, making it difficult to achieve optimal performance in practical applications.

[0028] Specifically, Figure 1 The present invention provides a flow chart of a target detection method that integrates roadside multi-modal top view features according to an embodiment of the present application.

[0029] like Figure 1 As shown, the target detection method integrating roadside multi-modal top view features includes the following steps:

[0030] In step S101 , roadside perception data of the target to be detected is obtained based on the final roadside radar perception data and the final roadside image perception data of the target to be detected.

[0031] It can be understood that the roadside perception data in the embodiment of the present application may include but is not limited to the final roadside radar perception data and the final roadside image perception data based on the target to be detected, and the present application does not impose any specific limitations.

[0032] Among them, the final roadside radar perception data can include but is not limited to lidar data, radar data, etc., and this application does not make specific restrictions.

[0033] The final roadside image perception data may include but is not limited to camera data, camera data, etc., and this application does not impose any specific restrictions.

[0034] In some embodiments, the embodiments of the present application can obtain roadside perception data of the target to be detected based on the final roadside radar perception data and the final roadside image perception data.

[0035] Optionally, in one embodiment of the present application, roadside perception data of the target to be detected is obtained based on the final roadside radar perception data and the final roadside image perception data of the target to be detected, including: obtaining initial roadside radar perception data of the target to be detected, and preprocessing the initial roadside radar perception data to obtain final roadside radar perception data that meets a first preset condition; obtaining initial roadside image perception data of the target to be detected, and preprocessing the initial roadside image perception data to obtain final roadside image perception data that meets a second preset condition; obtaining roadside perception data based on the final roadside radar perception data and the final roadside image perception data.

[0036] As a possible implementation method, the embodiment of the present application can pre-process the initial roadside radar perception data and the initial roadside image perception data respectively, and then obtain the final roadside radar perception data that meets the first certain condition and the final roadside image perception data that meets the second certain condition, thereby obtaining the roadside perception data. Among them, the first certain condition and the second certain condition can be set by a person skilled in the art according to actual conditions, and this application does not make specific restrictions.

[0037] Exemplarily, the embodiments of the present application can perform voxel processing on initial roadside radar perception data, such as lidar point cloud data, to obtain final roadside radar perception data that meets a first certain condition; and can pre-process initial roadside image perception data, such as multi-view camera data, to obtain final roadside image perception data that meets a second certain condition.

[0038] In step S102, a multimodal top view feature of the target to be detected is generated based on the roadside perception data.

[0039] During actual implementation, the embodiments of the present application can generate multimodal top view features of the target to be detected through roadside perception data.

[0040] Optionally, in one embodiment of the present application, multimodal overhead view features of the target to be detected are generated based on roadside perception data, including: using the final roadside radar perception data in the roadside perception data to generate radar overhead view features of the final roadside radar perception data; using the final roadside image perception data in the roadside perception data to generate image overhead view features of the final roadside image perception data; and obtaining multimodal overhead view features based on the radar overhead view features and the image overhead view features.

[0041] In some embodiments, the final roadside radar perception data of the embodiment of the present application can obtain radar top view features through radar coding.

[0042] In some embodiments, the final roadside image perception data of the embodiment of the present application can be subjected to feature extraction through a backbone network, and then the FPN network (Feature Pyramid Network) is used to generate multi-scale image features to cope with the problem of detecting objects of different sizes, thereby obtaining image features.

[0043] Furthermore, the embodiment of the present application can perform image feature enhancement and top-view feature generation after completing the image feature extraction. Among them, the embodiment of the present application first solves the training imbalance problem caused by the diversity of roadside camera installation postures through the data enhancement technology of the top-view coordinate system. Next, the mask technology is used to deal with the interference caused by the difference in the number of roadside cameras. Finally, in the camera's single camera feature processing, the system also incorporates camera rotation embedding, thereby solving the ambiguity problem in the object's direction angle, ensuring that the target's direction recognition is more accurate, and then obtaining the image top-view feature.

[0044] In some embodiments, the embodiments of the present application can perform multimodal data fusion on the radar top view features and the image top view features. The main contents are: first, through the point cloud cross-attention and image cross-attention mechanisms, the radar top view features and the image top view features are effectively integrated to obtain multimodal top view features. In addition, temporal self-attention is used to align and fuse the top view features of historical frames, thereby improving the prediction and planning capabilities. This solution aims to improve the performance of the roadside top view perception system in complex scenarios.

[0045] In step S103, the multimodal top view features are input into a pre-built convolutional neural network to obtain a target detection result of the target to be detected.

[0046] In the actual implementation process, the embodiment of the present application can utilize a pre-built convolutional neural network and combine it with multi-modal top view features to obtain the target detection result of the module to be detected.

[0047] Optionally, in one embodiment of the present application, the multimodal top view feature is input into a pre-built convolutional neural network to obtain a target detection result of the target to be detected, including: based on the multimodal top view feature, combined with the convolutional neural network in the pre-built convolutional neural network, to obtain the final multimodal top view feature of the target to be detected; based on the final multimodal top view feature, combined with the classification module in the pre-built convolutional neural network, to obtain target classification and positioning information of the target to be detected; based on the spatial information of the final multimodal top view feature, combined with the recognition module in the pre-built convolutional neural network, to obtain the target category and position information of the target to be detected; based on the target classification and positioning information and the target category and position information, the target detection result is obtained.

[0048] It will be understood by those skilled in the art that the convolutional neural network pre-constructed in the embodiment of the present application may include but is not limited to a convolutional neural network, a classification module, a recognition module, etc., and the present application does not impose any specific limitations.

[0049] Among them, in an embodiment of the present application, a convolutional neural network can be used to extract high-level features in the multimodal overhead view features, enhance the perception ability of different types of targets (such as vehicles, pedestrians, cyclists, etc.), and thus obtain the final multimodal overhead view features of the target to be detected.

[0050] The classification module can be used to classify and locate the candidate areas of the target to be detected after feature extraction, and then obtain target classification and positioning information.

[0051] The recognition module can use the spatial information in the feature map to accurately identify the category and location of the target to be detected, and then obtain the target category and location information.

[0052] Furthermore, the embodiments of the present application can obtain target detection results based on target classification and positioning information, target category and position information.

[0053] Optionally, in one embodiment of the present application, the multimodal top view features are input into a pre-built convolutional neural network to obtain target detection results of the target to be detected, including: determining whether there are duplicate target detection results; if there are duplicate target detection results, deleting the duplicate target detection results, otherwise, obtaining the target detection results of the target to be detected.

[0054] As a possible implementation method, the embodiment of the present application can delete duplicate target detection results to ensure that the identified target detection results are more accurate and provide reliable input data for subsequent path planning and decision-making.

[0055] Combine the following Figure 2As shown, the working principle of the target detection method integrating the multi-modal top view features of the road side proposed in the embodiment of the present application is introduced with a specific embodiment.

[0056] in, Figure 2 The present invention is a flowchart of the working principle of a target detection method integrating roadside multi-modal top view features provided in accordance with an embodiment of the present application.

[0057] Step S201: Roadside perception data collection module.

[0058] Step S202: Multi-modal top view feature generation module.

[0059] Among them, step S202 includes step S2021 of extracting radar and image feature modules, step S2022 of generating overhead view feature modules and step S2023 of multimodal data fusion modules.

[0060] Step S203: target detection module.

[0061] According to the target detection method for integrating the multimodal top view features of the roadside proposed in the embodiment of the present application, the roadside perception data can be obtained based on the final roadside radar perception data and the final roadside image perception data of the target to be detected, and then the multimodal top view features of the target to be detected are generated, and the multimodal top view features are input into the pre-built convolutional neural network, thereby obtaining the target detection result of the target to be detected. By adopting the multimodal sensor fusion technology, the data of the multi-view camera and the radar sensor are deeply fused, and the top view enhancement module is processed to improve the accuracy of target detection. In addition, compared with the single-vehicle perception system, the roadside top view system can provide more viewing angles, so as to more accurately perceive the target behind the obstruction, provide a safer driving environment for autonomous driving vehicles, and reduce the occurrence of traffic accidents. This not only helps to formulate more scientific traffic policies, but also can provide a strong basis for future road expansion and intelligent transportation infrastructure planning, and promote the intelligent transformation of urban transportation systems. Thus, the method of using vehicle sensor data in the related technology is solved, which causes the perception range of the top view to be limited to a limited area around the vehicle. However, the roadside overhead view method lacks a mechanism for processing radar data, which affects the overall perception capability of the overhead view in complex traffic scenes, making it difficult to achieve optimal performance in practical applications.

[0062] Next, a target detection device integrating roadside multi-modal top view features according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0063] Figure 3 A block diagram of a target detection device that integrates roadside multi-modal top view features according to an embodiment of the present application.

[0064] like Figure 3 As shown, the target detection device 30 integrating roadside multi-modal top view features includes: a first generation module 100, a second generation module 200 and a third generation module 300.

[0065] Among them, the first generating module 100 is used to obtain the roadside perception data of the target to be detected based on the final roadside radar perception data and the final roadside image perception data of the target to be detected.

[0066] The second generation module 200 is used to generate multimodal top view features of the target to be detected based on the roadside perception data.

[0067] The third generation module 300 is used to input the multimodal top view features into a pre-built convolutional neural network to obtain the target detection result of the target to be detected.

[0068] Optionally, in one embodiment of the present application, the first generating module 100 includes: a first acquiring unit, a second acquiring unit and a first generating unit.

[0069] Among them, the first acquisition unit is used to acquire initial roadside radar perception data of the target to be detected, and preprocess the initial roadside radar perception data to obtain final roadside radar perception data that meets the first preset condition.

[0070] The second acquisition unit is used to acquire initial roadside image perception data of the target to be detected, and preprocess the initial roadside image perception data to obtain final roadside image perception data that meets a second preset condition.

[0071] The first generating unit is used to obtain roadside perception data based on the final roadside radar perception data and the final roadside image perception data.

[0072] Optionally, in one embodiment of the present application, the second generation module 200 includes: a second generation unit, a third generation unit and a fourth generation unit.

[0073] Among them, the second generating unit is used to generate the radar overhead view features of the final roadside radar perception data using the final roadside radar perception data in the roadside perception data.

[0074] The third generating unit is used to generate the image top view features of the final roadside image perception data by using the final roadside image perception data in the roadside perception data.

[0075] The fourth generating unit is used to obtain a multi-modal top view feature based on the radar top view feature and the image top view feature.

[0076] Optionally, in one embodiment of the present application, the third generation module 300 includes: a fifth generation unit, a sixth generation unit, a seventh generation unit and an eighth generation unit.

[0077] Among them, the fifth generation unit is used to obtain the final multimodal top view features of the target to be detected based on the multimodal top view features combined with the convolutional neural network in the pre-built convolutional neural network.

[0078] The sixth generation unit is used to obtain target classification and positioning information of the target to be detected based on the final multimodal top view features in combination with the classification module in the pre-built convolutional neural network.

[0079] The seventh generation unit is used to obtain the target category and position information of the target to be detected based on the spatial information of the final multimodal top view features in combination with the recognition module in the pre-built convolutional neural network.

[0080] The eighth generating unit is used to obtain the target detection result based on the target classification and positioning information and the target category and position information.

[0081] Optionally, in one embodiment of the present application, the third generation module 300 includes: a judgment module and a ninth generation module.

[0082] The judgment module is used to judge whether there are duplications in the target detection results.

[0083] The ninth generation module is used to delete the repeated target detection results when there are duplicate target detection results, otherwise, obtain the target detection results of the target to be detected.

[0084] It should be noted that the aforementioned explanation of the target detection method embodiment for integrating roadside multi-modal overhead view features is also applicable to the target detection device for integrating roadside multi-modal overhead view features of this embodiment, and will not be repeated here.

[0085] According to the target detection device for integrating the multimodal top view features of the roadside proposed in the embodiment of the present application, the roadside perception data can be obtained based on the final roadside radar perception data and the final roadside image perception data of the target to be detected, and then the multimodal top view features of the target to be detected are generated, and the multimodal top view features are input into the pre-built convolutional neural network, thereby obtaining the target detection result of the target to be detected. By adopting the multimodal sensor fusion technology, the data of the multi-view camera and the radar sensor are deeply fused, and the top view enhancement module is processed to improve the accuracy of target detection. In addition, compared with the single-vehicle perception system, the roadside top view system can provide more viewing angles, so as to more accurately perceive the target behind the obstruction, provide a safer driving environment for autonomous driving vehicles, and reduce the occurrence of traffic accidents. This not only helps to formulate more scientific traffic policies, but also can provide a strong basis for future road expansion and intelligent transportation infrastructure planning, and promote the intelligent transformation of urban transportation systems. Thus, the method of using vehicle sensor data in the related technology is solved, which causes the perception range of the top view to be limited to a limited area around the vehicle. However, the roadside overhead view method lacks a mechanism for processing radar data, which affects the overall perception capability of the overhead view in complex traffic scenes, making it difficult to achieve optimal performance in practical applications.

[0086] Figure 4 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. The electronic device may include:

[0087] Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .

[0088] When the processor 402 executes the program, the target detection method integrating the roadside multi-modal top view features provided in the above embodiment is implemented.

[0089] Furthermore, the electronic device further comprises:

[0090] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0091] The memory 401 is used to store computer programs that can be executed on the processor 402 .

[0092] The memory 401 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0093] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0094] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0095] The processor 402 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0096] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the target detection method as described above that integrates roadside multi-modal top view features.

[0097] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, implements the above-mentioned target detection method that integrates roadside multi-modal top view features.

[0098] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0099] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0100] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0101] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0102] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of multiple of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0103] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0104] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0105] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A target detection method integrating roadside multi-modal top view features, characterized in that: The following steps are involved: Obtaining roadside perception data of the target to be detected based on the final roadside radar perception data and the final roadside image perception data of the target to be detected; Generating a multimodal top view feature of the target to be detected based on the roadside perception data; The multimodal top view features are input into a pre-built convolutional neural network to obtain the target detection result of the target to be detected.

2. The method according to claim 1, characterized in that The obtaining the roadside perception data of the target to be detected based on the final roadside radar perception data and the final roadside image perception data of the target to be detected includes: Acquiring initial roadside radar perception data of the target to be detected, and preprocessing the initial roadside radar perception data to obtain the final roadside radar perception data that meets a first preset condition; Acquiring initial roadside image perception data of the target to be detected, and preprocessing the initial roadside image perception data to obtain the final roadside image perception data that meets a second preset condition; The roadside perception data is obtained based on the final roadside radar perception data and the final roadside image perception data.

3. The method according to claim 2, characterized in that The generating of the multimodal top view features of the target to be detected based on the roadside perception data includes: Generating a radar overhead view feature of the final roadside radar perception data using the final roadside radar perception data in the roadside perception data; Generating an image top view feature of the final roadside image perception data using the final roadside image perception data in the roadside perception data; The multimodal top view feature is obtained based on the radar top view feature and the image top view feature.

4. The method according to claim 1, characterized in that The step of inputting the multimodal top view features into a pre-built convolutional neural network to obtain a target detection result of the target to be detected includes: Based on the multimodal top view feature, combined with the convolutional neural network in the pre-constructed convolutional neural network, to obtain the final multimodal top view feature of the target to be detected; Based on the final multimodal top view features, combined with the classification module in the pre-built convolutional neural network, to obtain target classification and positioning information of the target to be detected; Based on the spatial information of the final multimodal top view features, combined with the recognition module in the pre-built convolutional neural network, the target category and position information of the target to be detected are obtained; The target detection result is obtained based on the target classification and positioning information and the target category and position information.

5. The method according to claim 1, characterized in that: The step of inputting the multimodal top view features into a pre-built convolutional neural network to obtain a target detection result of the target to be detected includes: Determine whether the target detection result is repeated; If there are duplicate target detection results, the duplicate target detection results are deleted, otherwise, the target detection result of the target to be detected is obtained.

6. A target detection device integrating roadside multi-modal top view features, characterized in that: include: A first generating module, configured to obtain the roadside perception data of the target to be detected based on the final roadside radar perception data and the final roadside image perception data of the target to be detected; A second generating module, configured to generate a multimodal top view feature of the target to be detected based on the roadside sensing data; The third generation module is used to input the multimodal top view features into a pre-built convolutional neural network to obtain the target detection result of the target to be detected.

7. The device according to claim 6, characterized in that The first generating module comprises: a first acquisition unit, configured to acquire initial roadside radar perception data of the target to be detected, and preprocess the initial roadside radar perception data to obtain the final roadside radar perception data that meets a first preset condition; a second acquisition unit, configured to acquire initial roadside image perception data of the target to be detected, and preprocess the initial roadside image perception data to obtain the final roadside image perception data satisfying a second preset condition; The first generating unit is used to obtain the roadside perception data based on the final roadside radar perception data and the final roadside image perception data.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target detection method integrating roadside multi-modal top view features as described in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement a target detection method integrating roadside multi-modal top view features as described in any one of claims 1 to 5.

10. A computer program product, characterized in that It includes a computer program, which, when executed, is used to implement the target detection method integrating roadside multi-modal top view features as described in any one of claims 1 to 5.

Images