Perception fusion data processing method and system, computer equipment and storage medium

Abstract

The invention provides a perception fusion data processing method and system, computer equipment and a storage medium, and belongs to the technical field of data communication. The method is applied to the sensing fusion data processing system, and comprises the following steps: an interface interaction module generates a request instruction according to request information input by a user and sends the request instruction to each communication module; the fusion data receiving module downloads target sensing fusion data in real time according to the request instruction, and the sensing object presentation module reads and displays the target sensing fusion data regularly; the configuration module downloads a target point cloud file and a target stream acquisition information file according to the request instruction, the point cloud map loading module loads the target point cloud file according to the request instruction, and the video playing module downloads a target video stream according to the request instruction and the target stream acquisition information file and plays the target video stream. According to the invention, the fusion state of each sensing target in the sensing fusion data can be displayed in a customized manner, the video picture is displayed in real time, and the detection range of the radar camera is quantified.

Description

Technical Field

[0001] The present application relates to the field of data communication technology, and in particular to a perception fusion data processing method, system, computer device and storage medium. Background Art

[0002] In recent years, with the vigorous development of smart cars, autonomous driving has become an inevitable trend in the development of smart cars. The safety and stability of smart car operation depends on perfect road detection equipment, and the controllability and orderliness of the overall operation of smart cars are improved through computer vision, automatic control, data coding, data communication and data transmission technologies.

[0003] In related technologies, the road radar perception situation is often viewed through the interface display perception effect. Users directly view the radar perception situation through the original radar data and camera images uploaded by the radar camera deployed on the road infrastructure, and view the perception output situation, such as the number, type, and location of targets, by combining the radar perception data after data fusion and related structured data. Among them, the software tool for users to view the radar perception effect is mainly the Robot Visualization (RVIZ) provided by the Robot Operating System (ROS). As an open source tool for visual robot systems, RVIZ can display and debug the robot's sensor data, status information, and motion planning.

[0004] However, when viewing road radar perception results based on relevant technologies, there are problems such as the RVIZ tool does not support customized display of fusion status, does not support real-time display of video images, and cannot measure whether the radar camera detection range is reasonable. Summary of the invention

[0005] The purpose of this application is to provide a perception fusion data processing method, system, computer equipment and storage medium, which can achieve customized display of the fusion status of each perception target in the perception fusion data, display the video screen in real time, and quantify the detection range of the radar camera.

[0006] The embodiment of the present application is implemented as follows:

[0007] In a first aspect of an embodiment of the present application, a perception fusion data processing method is provided, which is applied to a perception fusion data processing system, wherein the perception fusion data processing system comprises: an interface interaction module, a fusion data receiving module, a perception object presentation module, a configuration module, a point cloud map loading module, and a video playback module, wherein the method comprises:

[0008] The interface interaction module generates a request instruction according to the request information input by the user, and sends it to the fusion data receiving module, the perception object presentation module, the configuration module, the point cloud map loading module and the video playback module. The request information includes: communication port, user identification and the address of the target edge device;

[0009] The fusion data receiving module downloads the target perception fusion data from the target edge device in real time according to the request instruction, and the perception object presentation module reads the target perception fusion data regularly and displays it;

[0010] The configuration module downloads the target point cloud file and the target stream information file from the target edge device according to the request instruction. The point cloud map loading module loads the target point cloud file according to the request instruction. The video playback module downloads and plays the target video stream from the target edge device according to the request instruction and the target stream information file.

[0011] As a possible implementation method, the fusion data receiving module downloads the target perception fusion data from the target edge device according to the request instruction, and the perception object presentation module periodically reads the target perception fusion data and displays it, including:

[0012] The fusion data receiving module sends an access request to the target edge device according to the request instruction;

[0013] The fusion data receiving module reads the target perception fusion data from the target edge device in real time, and caches the target perception fusion data into the fusion data queue in turn;

[0014] The perception object presentation module periodically reads the target perception fusion data from the fusion data queue according to the request instruction, and displays the read target perception fusion data.

[0015] As a possible implementation method, the perception object presentation module reads the target perception fusion data from the fusion data queue at a fixed time according to the request instruction, and displays the read target perception fusion data, including:

[0016] The perception object presentation module periodically reads the target perception fusion data in the fusion data queue according to the request instruction;

[0017] The perception object presentation module renders and displays each perception target according to the attribute information of the perception target contained in the read target perception fusion data. The attribute information includes: the category of the perception target, the position of the perception target and the heading angle of the perception target.

[0018] As a possible implementation method, the configuration module downloads the target point cloud file and the target stream information file from the target edge device according to the request instruction, including:

[0019] The configuration module sends an access request to the target edge device according to the request instruction, so that the target edge device sends the target point cloud file and the target flow information file to the configuration module according to the access request.

[0020] As a possible implementation method, the point cloud map loading module loads the target point cloud file obtained by the configuration module according to the request instruction, including:

[0021] The point cloud map loading module determines the associated radar camera of the target edge device according to the request instruction;

[0022] The point cloud map loading module loads the target point cloud file according to the associated radar camera, and performs grid calibration on the loaded target point cloud file to obtain the point cloud map corresponding to the target edge device.

[0023] As a possible implementation, the video playback module downloads and plays the target video stream from the target edge device according to the request instruction and the target stream information file, including:

[0024] The video playback module determines the associated radar camera of the target edge device according to the request instruction;

[0025] The video playback module sends an access request to the target edge device according to the request instruction and the target stream information file, so that the target edge device sends the target video stream corresponding to each associated radar camera to the video playback module according to the access request;

[0026] The interface interaction module sends a target playback instruction to the video playback module, and the video playback module responds to the target playback instruction to play the target video stream corresponding to the target radar camera selected by the user.

[0027] As a possible implementation, the perception fusion data processing system further includes: a log storage module, and the method further includes:

[0028] The log storage module stores the real-time operating status, data transmission abnormalities and fusion data troubleshooting results of the interface interaction module, fusion data receiving module, perception object presentation module, configuration module, point cloud map loading module and video playback module.

[0029] In a second aspect of an embodiment of the present application, a perception fusion data processing system is provided, the perception fusion data processing system comprising: an interface interaction module, a fusion data receiving module, a perception object presentation module, a configuration module, a point cloud map loading module, a video playback module and a log storage module;

[0030] The perception fusion data processing system is used to perform perception fusion data processing based on the perception fusion data processing method described in the first aspect above.

[0031] In a third aspect of an embodiment of the present application, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the perception fusion data processing method described in the first aspect is implemented.

[0032] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the perception fusion data processing method described in the first aspect is implemented.

[0033] The beneficial effects of the embodiments of the present application include:

[0034] The embodiment of the present application provides a method for processing perception fusion data, in which a user inputs request information through an interface interaction module, and the interface interaction module generates corresponding request instructions according to the request information input by the user and sends them to a fusion data receiving module, a perception object presentation module, a configuration module, a point cloud map loading module, and a video playback module, respectively, wherein the request information may be a communication port, a user identifier, and a logical address of a target edge device; the fusion data receiving module receives target perception fusion data generated by each program in the target edge device in real time under the triggering of the request instruction, and caches the target perception fusion data to a fusion data queue in sequence according to the receiving order; the perception object presentation module Under the triggering of the request instruction, the target perception fusion data is read from the fusion data queue at a fixed time, and each perception target is rendered and displayed based on the fusion status of each perception target contained in the read target perception fusion data; the configuration module receives the target point cloud file and the target flow information file of the target edge device in real time under the triggering of the request instruction; the point cloud map loading module loads the target point cloud file obtained by the configuration module under the triggering of the request instruction, and performs grid calibration on the loaded point cloud file to obtain the point cloud map; the video playback module downloads the target video stream from the target edge device in real time for playback according to the flow information file obtained by the configuration module under the triggering of the request instruction. In this way, the fusion status of each perception target in the perception fusion data can be customized and displayed, and the video screen can be displayed in real time to quantify the detection range of the radar camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1A diagram of the association relationship between an edge device and a radar camera provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of the structure of a perception fusion data processing system provided in an embodiment of the present application;

[0038] Figure 3 A flowchart of a first perception fusion data processing method provided in an embodiment of the present application;

[0039] Figure 4 A flowchart of a second perception fusion data processing method provided in an embodiment of the present application;

[0040] Figure 5 A flowchart of a third perception fusion data processing method provided in an embodiment of the present application;

[0041] Figure 6 A flowchart of a perception fusion data receiving and display thread provided in an embodiment of the present application;

[0042] Figure 7 A flowchart of a perception fusion data display thread provided in an embodiment of the present application;

[0043] Figure 8 A flowchart of a fourth perception fusion data processing method provided in an embodiment of the present application;

[0044] Fig. 9 A flowchart of a fifth perception fusion data processing method provided in an embodiment of the present application;

[0045] Fig.10 A flowchart of a perception fusion data processing thread provided in an embodiment of the present application;

[0046] Fig.11 A schematic diagram of the structure of a computer device provided in an embodiment of the present application.

[0047] Description of the drawings: 10: Perception fusion data processing system; 101: Interface interaction module; 102: Fusion data receiving module; 103: Perception object presentation module; 104: Configuration module; 105: Point cloud map loading module; 106: Video playback module; 107: Log storage module; 1101: Memory; 1102: Processor. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0049] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0050] At present, users usually directly view radar perception through the raw radar data and camera images uploaded by radar cameras deployed on road infrastructure, and view perception output by combining radar perception data after data fusion and related structured data. Among them, the software tool for users to view radar perception effects is mainly RVIZ. As an open source tool for visualizing robot systems, RVIZ can display and debug the robot's sensor data, status information, and motion planning. However, this solution has the problem that the RVIZ tool does not support customized display of fusion status, does not support real-time display of video images, and cannot measure whether the detection range of the radar camera is reasonable.

[0051] To this end, the embodiment of the present application provides a method for processing perception fusion data, in which the user inputs request information in the interface interaction module, the interface interaction module generates a request instruction according to the request information and sends it to each communication module, each communication module is started under the triggering of the request instruction, the fusion data receiving module starts the thread for receiving the perception fusion data, the perception object presentation module periodically reads the perception fusion data and displays the fusion status of the perception target contained in the perception fusion data, the configuration module receives the point cloud file and the stream information file, the point cloud map loading module loads the point cloud file and calibrates it to obtain the point cloud map, and the video playback module starts to receive the video stream and play it in real time. In this way, the fusion status of each perception target in the perception fusion data can be customized and displayed, and the video screen can be displayed in real time, and the effect of quantifying the detection range of the radar camera can be achieved.

[0052] Figure 1 For the association diagram of an edge device and a radar camera provided in this application, see Figure 1In the embodiment of the present application, an edge device can be associated with multiple radar cameras, that is, an edge device can centrally manage the raw radar data collected by multiple radar cameras. Radar cameras refer to radars and cameras, specifically including: roadside laser radars, millimeter-wave radars and cameras, etc. The raw radar data collected by each radar camera includes: raw radar data and video stream data. Edge devices are often implemented by multi-access edge computing devices (Multi-Access Edge Computing, referred to as MEC). MEC can simultaneously obtain the raw radar data of multiple radar cameras, and perform data fusion on the raw radar data of each radar camera based on the multi-access edge computing capability of MEC. The edge device stores the perception fusion data, video stream, and stream information files corresponding to the raw radar data collected in real time by each radar camera it manages. This application takes an edge device managing four radar cameras at a certain intersection as an example, but it does not mean that the edge device can only manage four radar cameras. This application does not make specific limitations on this.

[0053] Optionally, radar cameras are often deployed on fixed poles of road infrastructure to collect road traffic conditions in real time. The edge device is used to fuse the original radar data collected by each radar camera it manages, and save the fused radar data separately. The original radar data between the radar cameras managed by the edge device do not interfere with each other. Among them, the edge device obtains the original radar data of each radar camera it manages in real time through wireless communication technology, and performs data fusion processing on the original radar data of each radar camera based on its own multi-access edge computing capabilities. Each radar camera managed by the edge device is the associated radar camera of the edge device.

[0054] Optionally, in the intelligent transportation system, each edge device has its corresponding IP address, and each radar camera has its corresponding stream acquisition address. The edge device can process the original radar data of each radar camera separately based on the stream acquisition address of the original radar data of each radar camera it manages to obtain the perception fusion data corresponding to each radar camera.

[0055] Optionally, the radar cameras associated with the edge device refer to a collection of radars and cameras. The radar cameras specifically include: roadside lidar, millimeter-wave radar and cameras, etc. The radar camera can detect traffic participants at various intersections in the intelligent transportation system, and perceive the various perceived targets presented by the object presentation module. Users can directly view the position, category, driving speed, heading angle, fusion status and other information of each perceived target, and determine the perception distance of each radar camera according to the position of each perceived target on the point cloud map, thereby further improving the classification accuracy of radar-vision fusion perception.

[0056] Figure 2 This is a schematic diagram of the structure of a perception fusion data processing system provided by this application, see Figure 2A perception fusion data processing system 10 provided in an embodiment of the present application includes: an interface interaction module 101, a fusion data receiving module 102, a perception object presentation module 103, a configuration module 104, a point cloud map loading module 105, a video playback module 106 and a log storage module 107. Among them, the fusion data receiving module 102, the perception object presentation module 103, the configuration module 104, the point cloud map loading module 105 and the video playback module 106 all serve as communication modules in the perception fusion data processing system 10. The interface interaction module 101 is respectively communicated with the fusion data receiving module 102, the perception object presentation module 103, the configuration module 104, the point cloud map loading module 105 and the video playback module 106. The fusion data receiving module 102, the configuration module 104 and the video playback module 106 are all directly communicated with the edge device. The fusion data receiving module 102 is also communicated with the perception object presentation module 103. The configuration module 104 is also communicated with the point cloud map loading module 105. The interface interaction module 101, the fusion data receiving module 102, the perception object presentation module 103, the configuration module 104, the point cloud map loading module 105 and the video playback module 106 are all communicated with the log storage module 107.

[0057] Optionally, the user inputs request information in the interface interaction module 101, and the interface interaction module 101 triggers the start of each communication module according to the request information input by the user; the fusion data receiving module 102 obtains the perception fusion data in the target edge device based on the request instruction sent by the interface interaction module 101, the configuration module 104 obtains the point cloud file and the flow information file in the target edge device based on the request instruction sent by the interface interaction module 101, and the perception object presentation module 103 loads the perception fusion data received by the fusion data receiving module 102 at a fixed time based on the request instruction sent by the interface interaction module 101, and renders and displays each perception target based on the fusion state of the perception target contained in the perception fusion data; the point cloud map loading module 10 Based on the request instruction sent by the interface interaction module 101, the point cloud file obtained by the configuration module 104 is loaded, and the point cloud file is used as the background for grid calibration to obtain the point cloud map of the area where the edge device is located. The video playback module 106 obtains the video stream files of each radar camera managed by the edge device based on the request instruction of the interface interaction module 101 and the stream information file obtained by the configuration module 104 for playback; the log storage module 107 stores the operation status, data transmission status and perception fusion data troubleshooting results of the interface interaction module 101, the fusion data receiving module 102, the perception object presentation module 103, the configuration module 104, the point cloud map loading module 105 and the video playback module 106 in real time.

[0058] Optionally, the perception fusion data processing system provided in the embodiment of the present application is a system built on a cross-platform user interface application development framework. The perception fusion data processing system can dynamically display the fusion status of each traffic participant based on point cloud data, and display the posture of each traffic participant under different viewing angles of different radar cameras through the stream display of the video playback module. Among them, the display interfaces of different radar cameras can be displayed in different windows of the video playback module.

[0059] The following is a detailed explanation of the perception fusion data processing method provided in the embodiment of the present application in conjunction with the accompanying drawings.

[0060] Figure 3 This is a flowchart of a perception fusion data processing method provided in this application. This method can be applied to the above-mentioned perception fusion data processing system. Figure 3 , the embodiment of the present application provides a perception fusion data processing method, including:

[0061] S301. The interface interaction module generates a request instruction according to the request information input by the user, and sends it to the fusion data receiving module, the perception object presentation module, the configuration module, the point cloud map loading module and the video playback module. The request information includes: communication port, user ID and the address of the target edge device.

[0062] Optionally, the user inputs request information in the interface interaction module, and the request information input by the user can be one or more of a communication port, a user identifier, or an address of a target edge device. The communication port is used to indicate the identifier of the communication protocol or communication interface of the edge device that the user requests to query, such as communication port 100, etc. The user identifier is used to indicate the label of the traffic participant that the user requests to query, and the traffic participant can be a pedestrian, a car, a non-motorized vehicle, etc. The label of the traffic participant can be a car license plate number, a pedestrian label, etc. The address of the target edge device is used to indicate the IP address of the edge device that the user requests to query, and the IP address of the target edge device refers to the logical address of the edge device.

[0063] Optionally, the interface interaction module generates a request instruction based on the request information input by the user, and the interface interaction module sends the request instruction to communication modules such as the fusion data receiving module, the perception object presentation module, the configuration module, the point cloud map loading module and the video playback module respectively. The request instruction is used to trigger each communication module. The request instruction includes one or more of the logical address of the edge device specified by the user, or the identifier of the communication port of the edge device specified by the user, or the label of the traffic participant specified by the user. Based on the request instruction sent by the interface interaction module, each communication device starts its own thread, such as the fusion data receiving module starts to receive the perception fusion data sent by the edge device, and the configuration module starts to receive the point cloud file and the flow information file sent by the edge device, etc. This application does not make specific limitations on this.

[0064] S302, the fusion data receiving module downloads the target perception fusion data from the target edge device in real time according to the request instruction, and the perception object presentation module periodically reads the target perception fusion data and displays it.

[0065] Optionally, under the action of the request instruction issued by the interface interaction module, the fusion data receiving module downloads the target perception fusion data from the target edge device in real time. The target edge device is the edge device specified by the request information input by the user. The user can indicate the target edge device through the communication port of the specified edge device, or through the logical address of the specified edge device, or through the label of the specified traffic participant. This application does not make specific limitations on this.

[0066] Optionally, the fusion data receiving module determines the target edge device according to the request instruction sent by the interface interaction module, and the fusion data receiving module starts to read the perception fusion data generated by each program in the target edge device in real time according to the request instruction. The target perception fusion data is used to indicate the perception fusion data corresponding to each radar camera managed by the target edge device. The perception fusion data corresponding to each radar camera includes the traffic operation information of each perception target within the detection range of each radar camera.

[0067] Optionally, the fusion data receiving module receives the perception fusion data generated by each edge computing program in the target edge device in real time, and caches the read target perception fusion data into the fusion data queue in sequence based on the receiving order. The fusion data queue is used to indicate a fusion data sequence formed by sorting the target perception fusion data based on the reading order.

[0068] Optionally, the perception object presentation module reads the target perception fusion data from the fusion data queue in the fusion data receiving module in a regular manner based on the timing duration specified by the timer, and customizes the display of the fusion status of each perception target contained in the read target perception fusion data. Among them, the picture displayed by the perception object presentation module is often a frame of the radar camera, and the timing duration of the timer can be dynamically adjusted according to the size of the perception fusion data corresponding to each frame of the radar camera. This application does not make specific restrictions on this.

[0069] S303, the configuration module downloads the target point cloud file and the target stream information file from the target edge device according to the request instruction, the point cloud map loading module loads the target point cloud file according to the request instruction, and the video playback module downloads and plays the target video stream from the target edge device according to the request instruction and the target stream information file.

[0070] Optionally, the configuration module downloads the target point cloud file and the target stream information file from the target edge device under the trigger of the request instruction. The target point cloud file is a point cloud file constructed by the target edge device after deduplication and noise reduction based on the point cloud data within the detection range of each radar camera it manages. The target stream information file is used to indicate the stream information of each radar camera managed by the target edge device. The stream information of each radar camera includes the video stream generated by each radar camera in real time, the stream acquisition address of each radar camera, the identification of each radar camera and other information.

[0071] Optionally, the point cloud map loading module loads the target point cloud file obtained by the configuration module under the trigger of the request instruction, and the point cloud map loading module uses the point cloud data contained in the target point cloud file as the background to draw the point cloud map. The point cloud map loading module performs grid calibration on the point cloud map obtained after loading. The point cloud map based on the grid calibration can quantitatively determine the detection range of each radar camera associated with the target edge device.

[0072] Optionally, the video playback module, under the triggering of the request instruction, plays the target video stream from the target edge device based on the target stream information file obtained by the configuration module. The target video stream refers to the video images captured by each radar camera managed by the target edge device. It is worth noting that the video playback module has multiple windows, and the video playback module can simultaneously play the video images corresponding to each radar camera associated with the target edge device. This application does not specifically limit the playback window of the video playback module.

[0073] In an embodiment of the present application, a user inputs request information through an interface interaction module, and the interface interaction module generates corresponding request instructions according to the request information input by the user and sends them to the fusion data receiving module, the perception object presentation module, the configuration module, the point cloud map loading module and the video playback module respectively, wherein the request information may be a communication port, a user identifier and a logical address of a target edge device; the fusion data receiving module receives the target perception fusion data generated by each program in the target edge device in real time under the triggering of the request instruction, and caches the target perception fusion data to the fusion data queue in sequence according to the receiving order; the perception object presentation ... The target perception fusion data is read from the fusion data queue at a fixed time, and each perception target is rendered and displayed based on the fusion status of each perception target contained in the read target perception fusion data; the configuration module receives the target point cloud file and the target stream information file of the target edge device in real time under the triggering of the request instruction; the point cloud map loading module loads the target point cloud file obtained by the configuration module under the triggering of the request instruction, and performs grid calibration on the loaded point cloud file to obtain the point cloud map; the video playback module downloads the target video stream from the target edge device in real time for playback according to the stream information file obtained by the configuration module under the triggering of the request instruction. In this way, the fusion status of each perception target in the perception fusion data can be customized and displayed, and the video screen can be displayed in real time to quantify the detection range of the radar camera.

[0074] For a possible implementation, see Figure 4 The operation of step S302 may specifically be:

[0075] S401. The fusion data receiving module sends an access request to the target edge device according to the request instruction.

[0076] Optionally, the fusion data receiving module determines the communication interface and logical address of the target edge device according to the request instruction, and sends an access request to the target edge device, where the access request is used to instruct the fusion data receiving module to request to receive the perception fusion data generated by each edge computing program in the target edge device.

[0077] S402, the fusion data receiving module reads the target perception fusion data from the target edge device in real time, and caches the target perception fusion data into the fusion data queue in turn.

[0078] Optionally, the fusion data receiving module adopts TCP communication to obtain the target perception fusion data generated by each program from the target edge device through a specific communication port, and caches the obtained target perception fusion data into the fusion data queue in sequence according to the time sequence of the perception fusion data acquisition.

[0079] Optionally, the fusion data receiving module obtains the target perception fusion data from the program in the target edge device by using TCP / IP communication. The debugging interface of the fusion data receiving module obtains the target perception fusion data from the perception data fusion processing program in the target edge device through a specific communication port. When there are no traffic participants at the intersection actually perceived by each radar camera associated with the target edge device, the number of perceived targets contained in the perception fusion data received by the fusion data receiving module is 0, and the display end of the fusion data receiving module will display that the number of perceived targets contained in the current frame of target perception fusion data is 0. The fusion data receiving module clears the perception targets of the current interface, so that the user can more clearly see that there are no traffic participants at the current target intersection.

[0080] S403, the perception object presentation module periodically reads the target perception fusion data from the fusion data queue according to the request instruction, and displays the read target perception fusion data.

[0081] Optionally, the perception object presentation module periodically reads a section of target perception fusion data from the fusion data queue when triggered by a request instruction, renders each perception target contained in the read target perception fusion data, and displays the read target perception fusion data based on the rendering result.

[0082] It is worth noting that in order to better demonstrate the radar perception effect of an intersection or road section, the fusion data receiving module in the perception fusion data processing system often simultaneously accesses two or more perception fusion data. In this way, combined with the perception fusion data of the previous intersection containing information such as the category and trajectory of the perceived target, the radar detection range and detection category accuracy of the radar camera at the current intersection can be better judged, and then whether there is any overlap in the intersection detection range can be determined.

[0083] For a possible implementation, see Figure 5 The operation of step S403 may specifically be:

[0084] S501. The perception object presentation module periodically reads the target perception fusion data in the fusion data queue according to the request instruction.

[0085] Optionally, the perception object presentation module reads the target perception fusion data from the fusion data queue based on the control timing of the timer under the triggering of the request instruction. The data size of the target perception fusion data read by the perception object presentation module at regular intervals is often a complete frame of the shooting picture. The target perception fusion data contained in the fusion data queue stored in the fusion data receiving module is at least greater than or equal to the length of the target perception fusion data currently read by the perception object presentation module at regular intervals.

[0086] S502, the perception object presentation module renders and displays each perception target according to the attribute information of the perception target contained in the read target perception fusion data, where the attribute information includes: the category of the perception target, the position of the perception target, and the heading angle of the perception target.

[0087] Optionally, the perception object presentation module renders each perception target according to the attribute information of the perception target contained in the read target perception fusion data. The perception target is used to indicate the traffic participants such as pedestrians and vehicles contained in the target perception fusion data. Each perception target is rendered and displayed according to the attribute information of each perception target contained in the target perception fusion data. Among them, the attribute information of each perception target is used to indicate the category of each perception target, the location of the perception target and the heading angle of the perception target. The category of the perception target can be pedestrians, cars, etc. The location of the perception target is used to indicate the latitude and longitude information of the perception target, and the heading angle of the perception target is used to indicate the direction of the driving track of the perception target.

[0088] Optionally, different shapes and colors are used to render each perceived target based on its attribute information. For example, a rectangular block is used for a motor vehicle, and the size of the rectangular block is set according to the length, width and height of the vehicle; a cone is used for a non-motor vehicle, a cylinder is used for a pedestrian, and a sphere is used for other or unknown types; and the shape corresponding to each perceived target is colored according to the fusion state of each perceived target to indicate the different fusion states of each perceived target.

[0089] Optionally, the perception object presentation module can set a label for each perception target, and use the number, category, driving speed, longitude and latitude, distance from the origin, heading angle and other information of each perception target as label display content.

[0090] Optionally, after the target perception fusion data read in each frame is displayed, the perception object presentation module will continue to obtain the target perception fusion data again from the fusion data queue of the fusion data receiving module based on the timing duration indicated by the timer, and render and display the perception target contained in the again obtained target perception fusion data, until there is no target perception fusion data in the fusion data queue, and the display interface of the perception object presentation module no longer displays any perception target.

[0091] Figure 6 For a flowchart of a perception fusion data receiving and display thread provided by this application, see Figure 6In the perception data fusion processing method provided in the embodiment of the present application, the edge device determines the fusion state of each perception target according to the multi-sensor fusion algorithm. Based on whether the perception target is obtained by single sensor or multi-sensor fusion detection, the fusion state of the perception target can be divided into: single millimeter wave radar detection, single laser radar detection, single camera detection, millimeter wave + camera detection (denoted as CR), laser radar + camera detection (denoted as LR), laser radar + camera + millimeter wave radar (denoted as LCR) detection. The perception target is rendered in color based on the fusion state of each perception target, such as blue, red, cyan, orange, green, etc. For example, green indicates that the perception target is in a multi-sensor fusion state, such as CR, LR, and LCR fusion.

[0092] Optionally, the identification, category, longitude and latitude, driving speed, heading angle and other information of the perceived target can be displayed in the form of a label. When refreshing the perceived target is paused, the user can zoom in on the detailed information of the perceived target of interest by clicking and interacting in the human-computer interaction interface of the interface interaction module.

[0093] Optionally, for the orientations of different types of traffic participants, the perception target may be rotated according to the heading angle of the perception target with reference to the true north direction of the map to conform to the traffic movement situation of the traffic participant.

[0094] Figure 7 For a flowchart of a perception fusion data display thread provided by this application, see Figure 7 The point cloud map loading module provided in the embodiment of the present application displays each perceived target contained in the target perception fusion data currently taken out in the point cloud map according to its corresponding plane coordinates (X, Y). The reference origin of the point cloud map is generally the laser radar calibration, and the laser radar measures the physical location of the actual laser radar at the intersection to form the point cloud map. The plane coordinates of the perceived target contained in the perception fusion data are calculated by calibration based on the reference origin of the point cloud map.

[0095] In a possible implementation, the operation of step S303 may specifically be:

[0096] The configuration module sends an access request to the target edge device according to the request instruction, so that the target edge device sends the target point cloud file and the target flow information file to the configuration module according to the access request.

[0097] Optionally, the configuration module sends an access request to the target edge device when triggered by a request instruction. After receiving the access request sent by the configuration module, the target edge device sends the target point cloud file and the target flow information file to the configuration module, wherein the target point cloud file and the target flow information file can be used as configuration files for each detection device at the intersection that the user currently wants to query.

[0098] For a possible implementation, see Figure 8 The operation of step S303 may also be specifically:

[0099] S801. The point cloud map loading module determines the associated radar camera of the target edge device according to the request instruction.

[0100] Optionally, the point cloud map loading module determines a radar camera associated with the target edge device when triggered by a request instruction.

[0101] S802. The point cloud map loading module loads the target point cloud file according to the associated radar camera, and performs grid calibration on the loaded target point cloud file to obtain a point cloud map corresponding to the target edge device.

[0102] Optionally, the point cloud map loading module loads the target point cloud file in the configuration file according to the ID of the associated radar camera of the target edge device, draws the point cloud map based on the target point cloud file, and performs grid calibration on the drawn point cloud map, so as to quantify the detection range of each associated radar camera.

[0103] Optionally, the point cloud map loading module adds a ruler grid to the point cloud map drawing interface to realize grid calibration of the point cloud map, which can more intuitively determine the detection range of each radar camera. The ruler grid can be set to 10 meters per grid or 1 meter per grid. The size of the ruler grid can be dynamically adjusted, and this application does not make any specific limitations on this.

[0104] Optionally, the point cloud map loading module adds a grid line layer on the point cloud map, which can associate the grid line intervals with the point cloud map without interfering with the accuracy of the point cloud map. Among them, the grid lines take the coordinate origin of the point cloud as the origin, the horizontal distance to the right is positive, the horizontal distance to the left is negative, the vertical distance upward is positive, and the vertical distance downward is negative. The farther the grid is from the origin of the point cloud, the greater the distance value. When the perceived target is on different grid lines, the detection distance corresponding to the perceived target can be roughly determined. Based on the distance range of each perceived target, the perception range of each radar camera can be determined, and then it can be determined whether the layout of each radar camera is reasonable.

[0105] Optionally, the position of the perceived target in the point cloud map is calibrated according to the plane (x, y) coordinates of the perceived target contained in the perceived fusion data, and the point cloud map is established based on the origin of the laser radar. When the perceived object presentation module refreshes and displays new perceived fusion data, if the plane coordinates (x, y) of the perceived target change, the point cloud map loading module will place the perceived target on the new coordinates (x, y) to achieve the position update of the perceived target, presenting the effect of multiple perceived targets moving in real time. Among them, the license plate information is often displayed in the form of a label. When the perceived target is a vehicle, when the vehicle moves, the license plate and other information of the perceived target also moves on the point cloud map.

[0106] As an optional implementation method, the method of troubleshooting the multi-sensor perception effect according to the perception effect is mainly determined based on the perception range of each detection device and the perception overlap range. Generally, the perception distance of millimeter-wave radar is relatively long, which can be about 0-300 meters; the detection distance of laser radar can be about 0-200 meters; the camera perception is generally about 0-70 meters, and the radar and camera installed in the same direction will detect the same perception target within the overlapping perception range. If more perception targets are detected by a single sensor within the overlapping range, it means that there is a problem with the fusion perception. If you can see many moving perception targets in the video screen at a distance, but the display interface after perception fusion does not show traffic participants, there is a problem with the fusion perception. If there is a fusion perception effect of traffic participants before entering the camera perception area, there is a problem with the fusion perception; otherwise, there is a problem with the original radar vision data. In addition, combined with the perception range of different radars and the distance between the radar and the origin on the point cloud map, it can be judged whether the range or distance of radar installation detection is consistent with the actual nominal setting.

[0107] For a possible implementation, see Fig. 9 The operation of step S303 may also be specifically:

[0108] S901. The video playback module determines the associated radar camera of the target edge device according to the request instruction.

[0109] Optionally, the video playback module can determine the target edge device according to the request instruction, and determine the radar camera associated with the target edge device according to the logical address of the target edge device, and the associated radar camera of the target edge device is used to indicate the radar camera managed by the target edge device.

[0110] S902. The video playback module sends an access request to the target edge device according to the request instruction and the target stream information file, so that the target edge device sends the target video stream corresponding to each associated radar camera to the video playback module according to the access request.

[0111] Optionally, the video playback module obtains the target stream information file in the configuration module through the interface interaction module. When triggered by the request instruction, the video playback module sends a video stream access request to the target edge device based on the stream address indicated by the target stream information file. The target edge device sends the target video stream corresponding to the associated radar camera of the target edge device to the video playback module based on the video stream access request.

[0112] Optionally, the video playback module may acquire streams from each camera associated with the target edge device according to the target stream acquisition information file acquired by the configuration module, and play the video streams in multiple windows.

[0113] S903: The interface interaction module sends a target playback instruction to the video playback module, and the video playback module responds to the target playback instruction to play the target video stream corresponding to the target radar camera selected by the user.

[0114] Optionally, the user selects the target radar camera of interest through the interface interaction module. The interface interaction module sends a target playback instruction to the video playback module based on the user's selection operation. The target playback instruction includes the number and stream address of the target radar camera selected by the user. After receiving the target playback instruction, the video playback module separately plays the target video stream corresponding to the target radar camera selected by the user.

[0115] As an optional implementation, the above-mentioned perception fusion data processing method further includes:

[0116] The log storage module stores the real-time operating status, data transmission abnormalities and fusion data troubleshooting results of the interface interaction module, fusion data receiving module, perception object presentation module, configuration module, point cloud map loading module and video playback module.

[0117] Optionally, the log storage module is respectively connected to the interface interaction module, the fusion data receiving module, the perception object presentation module, the configuration module, the point cloud map loading module and the video playback module in communication, and the log storage module can store the real-time operation status, data transmission abnormality and fusion data investigation results of the interface interaction module, the fusion data receiving module, the perception object presentation module, the configuration module, the point cloud map loading module and the video playback module. Among them, the real-time operation status is used to indicate the thread operation process such as the interface interaction module, the fusion data receiving module, the perception object presentation module, the configuration module, the point cloud map loading module and the video playback module receiving data, transmitting data, stopping operation, etc.; the data transmission abnormality is used to indicate the abnormal situation in the data transmission of the interface interaction module, the fusion data receiving module, the perception object presentation module, the configuration module, the point cloud map loading module and the video playback module, such as the perception fusion data of no traffic participants at a certain intersection, the communication abnormality between the two modules, etc.; the fusion data investigation result is used to indicate whether the fusion data is abnormal, invalid or valid.

[0118] Fig.10 A flowchart of a perception fusion data processing thread provided by this application, see Fig.10 The perception fusion data processing thread provided in this application is mainly based on an interface program developed for an X86 machine to present the perception fusion data, and to troubleshoot fusion perception problems of the perception fusion data based on the presentation effect of the perception fusion data.

[0119] Optionally, the perception fusion data processing system provided by the present application is implemented based on a debugging tool, which includes a fusion data receiving module, a point cloud map loading module, a perception object presentation module, a video playback module, a log storage module, and an interface interaction module. The perception fusion data processing system is mainly based on TCP / IP communication to obtain the target point cloud file, target flow information file, and target perception fusion data provided by the target edge device. When the perception target is detected by multiple sensors, the radar data of the perception target will be fused.

[0120] As an optional implementation, the perception fusion data processing method provided in the embodiment of the present application is realized by combining PCL with VTK, and combined with mec-app to output perception fusion data, and display various information of each traffic participant at the intersection detected by each radar camera managed by the target edge device. The displayed information of each traffic participant includes: the category of the traffic participant, the identification (ID) of the traffic participant, the location of the traffic participant, the heading angle of the traffic participant, the running speed of the traffic participant and other information.

[0121] Optionally, the perception fusion data processing system determines the fusion state of each perception target according to the detection range of the associated radar camera of the target edge device and the detection device corresponding to each perception target in the perception fusion data, performs color rendering on each perception target according to the fusion state of each perception target, and intuitively determines whether each perception target is detected by a single detection device or by multiple detection devices according to the color information of each perception target. In addition, perception targets farther away from the intersection are generally detected by millimeter waves, which can further assist in determining the detection range of the millimeter waves, and then determine whether the installation position of the radar camera is reasonable. Among them, farther away from the intersection generally refers to a distance of more than 200 meters from the intersection, and this application does not make specific restrictions on this.

[0122] Optionally, the configuration module in the perception fusion data processing method obtains the point cloud PCD file and the streaming media configuration file from the target edge device, connects the server to the client, and the user can obtain the relevant files of each radar camera associated with the edge device without logging into the edge device. In addition, the system and user names of different edge devices are different, and the fixed user ID is not applicable to all edge devices, which can solve the problem of system compatibility of different edge devices.

[0123] The following describes the computer device and computer-readable storage medium used to execute the perception fusion data processing method provided in this application. The specific implementation process and technical effects are described above and will not be repeated below.

[0124] Fig.11 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.11 The computer device includes: a memory 1101 and a processor 1102. The memory 1101 stores a computer program that can be run on the processor 1102. When the processor 1102 executes the computer program, the steps in any of the above method embodiments are implemented.

[0125] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0126] Optionally, the present application also provides a program product, such as a computer-readable storage medium, including a program, which is used to execute any of the above-mentioned perception fusion data processing method embodiments when executed by a processor.

[0127] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

[0128] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0129] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for processing perception fusion data, characterized in that: Applied to a perception fusion data processing system, the perception fusion data processing system includes: an interface interaction module, a fusion data receiving module, a perception object presentation module, a configuration module, a point cloud map loading module and a video playback module, the method includes: The interface interaction module generates a request instruction according to the request information input by the user, and sends it to the fusion data receiving module, the perception object presentation module, the configuration module, the point cloud map loading module and the video playback module. The request information includes: communication port, user identification and address of the target edge device; The fusion data receiving module downloads the target perception fusion data from the target edge device in real time according to the request instruction, and the perception object presentation module periodically reads and displays the target perception fusion data; The configuration module downloads the target point cloud file and the target stream information file from the target edge device according to the request instruction, the point cloud map loading module loads the target point cloud file according to the request instruction, and the video playback module downloads and plays the target video stream from the target edge device according to the request instruction and the target stream information file.

2. The method for processing perception fusion data according to claim 1, characterized in that: The fusion data receiving module downloads the target perception fusion data from the target edge device according to the request instruction, and the perception object presentation module periodically reads and displays the target perception fusion data, including: The fusion data receiving module sends an access request to the target edge device according to the request instruction; The fusion data receiving module reads the target perception fusion data from the target edge device in real time, and sequentially caches the target perception fusion data into the fusion data queue; The object perception presentation module periodically reads the target perception fusion data from the fusion data queue according to the request instruction, and displays the read target perception fusion data.

3. The method for processing perception fusion data according to claim 2, characterized in that: The object perception presentation module periodically reads the target perception fusion data from the fusion data queue according to the request instruction, and displays the read target perception fusion data, including: The perceived object presentation module periodically reads the target perceived fusion data in the fusion data queue according to the request instruction; The perceived object presentation module renders and displays each perceived target according to the attribute information of the perceived target contained in the read target perception fusion data, wherein the attribute information includes: the category of the perceived target, the position of the perceived target, and the heading angle of the perceived target.

4. The method for processing perception fusion data according to claim 1, characterized in that: The configuration module downloads the target point cloud file and the target stream information file from the target edge device according to the request instruction, including: The configuration module sends an access request to the target edge device according to the request instruction, so that the target edge device sends a target point cloud file and a target flow information file to the configuration module according to the access request.

5. The method for processing perception fusion data according to claim 1, characterized in that: The point cloud map loading module loads the target point cloud file acquired by the configuration module according to the request instruction, including: The point cloud map loading module determines the associated radar camera of the target edge device according to the request instruction; The point cloud map loading module loads the target point cloud file according to the associated radar camera, and performs grid calibration on the loaded target point cloud file to obtain a point cloud map corresponding to the target edge device.

6. The method for processing perception fusion data according to claim 1, characterized in that: The video playback module downloads and plays the target video stream from the target edge device according to the request instruction and the target stream acquisition information file, including: The video playback module determines, according to the request instruction, an associated radar camera of the target edge device; The video playback module sends an access request to the target edge device according to the request instruction and the target stream information file, so that the target edge device sends the target video stream corresponding to each of the associated radar cameras to the video playback module according to the access request; The interface interaction module sends a target playback instruction to the video playback module, and the video playback module responds to the target playback instruction to play the target video stream corresponding to the target radar camera selected by the user.

7. The method for processing perception fusion data according to any one of claims 1 to 6, characterized in that: The perception fusion data processing system further includes: a log storage module, and the method further includes: The log storage module stores the real-time operating status, data transmission abnormality and fusion data troubleshooting results of the interface interaction module, the fusion data receiving module, the perceived object presentation module, the configuration module, the point cloud map loading module and the video playback module.

8. A perception fusion data processing system, characterized in that: The perception fusion data processing system includes: an interface interaction module, a fusion data receiving module, a perception object presentation module, a configuration module, a point cloud map loading module, a video playback module and a log storage module; The perception fusion data processing system is used to perform perception fusion data processing based on the method described in any one of claims 1-7.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps of the method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images