Multi-sensor target fusion processing method, system, device and medium

Abstract

This application proposes a multi-sensor target fusion processing method, system, device, and medium, including: synchronously fusing the output data of multiple sensor processing units at the arrival of each preset fusion cycle to obtain target fusion data; updating the target fusion data to each sensor processing unit, and using the sensor processing unit to data-associate the target fusion data with real-time sensor data to obtain associated data as the output data of each sensor processing unit; and outputting the target fusion data to the application end after each fusion cycle. This application can effectively improve the system's fault tolerance and the efficiency of target data fusion processing.

Description

Technical Field

[0001] The present application relates to the field of automotive perception applications, and in particular to a multi-sensor target fusion processing method, system, device and medium. Background Art

[0002] In recent years, autonomous driving research has become not only a hot topic within the scientific community, but also a key technology implementation project for both technology companies and traditional automakers. Currently, multi-sensor fusion has become a widely accepted approach for environmental object perception. Multi-sensor object fusion leverages the unique perceptual characteristics of each sensor, complementing each other to achieve comprehensive and accurate perception of environmental objects. However, the increased number of sensors requires the fusion module to process more sensor data, resulting in longer runtimes and significant latency, necessitating optimization.

[0003] Currently, multi-sensor target fusion software often uses a serial processing approach to correlate and fuse individual sensor data. Under this architecture, as the number of sensors and information increases, the time required to fully fuse all sensor information increases. This can lead to information delays and, in severe cases, the entire system can't keep up with the data fusion process, resulting in data loss or severe fusion lags and functional failures.

[0004] Application Contents

[0005] In view of the above problems existing in the prior art, the present application proposes a multi-sensor target fusion processing method, system, device and medium, which mainly solves the problems of low efficiency and delay in multi-sensor data fusion processing.

[0006] In order to achieve the above-mentioned objectives and other objectives, the technical solutions adopted in this application are as follows.

[0007] This application provides a multi-sensor target fusion processing method, including:

[0008] When each preset fusion cycle arrives, the output data of multiple sensor processing units are synchronously fused to obtain target fusion data;

[0009] Updating the target fusion data to each of the sensor processing units, and performing data association between the target fusion data and the real-time sensor data through the sensor processing unit to obtain associated data as output data of each sensor processing unit;

[0010] After each fusion cycle is completed, the target fusion data is output to the application end.

[0011] In one embodiment of the present application, the output data of multiple sensor processing units are synchronously fused to obtain target fused data, including:

[0012] If it is detected within a preset first waiting time that the real-time sensing data of any of the sensing processing units is not updated, the corresponding sensing processing unit is skipped, and the output data of the remaining sensing processing units are synchronously integrated;

[0013] The data fusion of the skipped sensor processing unit is recorded in the next fusion cycle.

[0014] In one embodiment of the present application, updating the target fusion data to each of the sensor processing units includes:

[0015] After the output data fusion is completed within the current fusion cycle, a fusion completion instruction is output;

[0016] Obtaining a fusion update request from each of the sensor processing units according to the fusion completion instruction;

[0017] Outputting the target fusion data to the sensor processing unit that issues the request according to the fusion update request.

[0018] In one embodiment of the present application, the output data of multiple sensor processing units are synchronously fused to obtain target fused data, including:

[0019] When a fusion cycle arrives, obtaining a processing completion signal from each of the sensing processing units;

[0020] When the number of processed signals reaches a preset number, the output data of each sensor processing unit is synchronously integrated.

[0021] In one embodiment of the present application, obtaining a fusion update request from each of the sensor processing units according to the fusion completion instruction includes:

[0022] If the fusion update request is not received within the preset second waiting time, the update operation of the corresponding sensor processing unit is skipped;

[0023] In the next fusion cycle, the target fusion data is used as the request response data of the sensor processing unit that has not been updated in the current fusion cycle.

[0024] In one embodiment of the present application, the sensor processing unit associates the target fusion data with the real-time sensor data, including:

[0025] Acquire edge features of one or more target objects in the target fusion data;

[0026] Delineating a related area according to the edge features, and eliminating real-time sensing data outside the related area;

[0027] The features of the real-time sensing data in the association area are compared with the edge features to obtain matching features to complete data association.

[0028] In one embodiment of the present application, the output data of multiple sensor processing units are synchronously fused to obtain target fused data, including:

[0029] The target fusion data of the previous fusion cycle is used as the initial ontology model;

[0030] Comparing the initial ontology model with the output data of each sensor processing unit to determine a fusion boundary, and marking the output data of each sensor processing unit according to the fusion boundary;

[0031] Inputting the output data outside the fusion boundary into a preset classification model to obtain edge features of different categories;

[0032] Edge feature matching is performed according to the categories of the edge features, and the matched edge features are associated to complete the fusion of corresponding edge features.

[0033] The present application also provides a multi-sensor target fusion processing system, comprising:

[0034] A fusion processing module is used to synchronously fuse the output data of multiple sensor processing units at the arrival of each preset fusion cycle to obtain target fusion data;

[0035] a sensor processing module, configured to update the target fusion data to each of the sensor processing units, and to perform data association between the target fusion data and the real-time sensor data through the sensor processing units to obtain associated data as output data of each sensor processing unit;

[0036] The fusion output module is used to output the target fusion data to the application end after each fusion cycle ends.

[0037] A computer device comprises: a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the multi-sensor target fusion processing method when executing the computer program.

[0038] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the multi-sensor target fusion processing method.

[0039] As described above, the multi-sensor target fusion processing method, system, device and medium provided in this application have the following beneficial effects.

[0040] This application obtains target fusion data by synchronously fusing the output data of multiple sensor processing units at the arrival of each preset fusion cycle; updating the target fusion data to each sensor processing unit, and using the sensor processing unit to perform data association between the target fusion data and real-time sensor data to obtain associated data as the output data of each sensor processing unit; and outputting the target fusion data to the application end after each fusion cycle. By performing synchronous data fusion after each sensor processing unit completes data association, the efficiency of data fusion processing can be effectively improved, and the real-time response rate of the fused data can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a schematic diagram of an application scenario in one embodiment of the present application.

[0042] Figure 2 This is a structural diagram of a multi-sensor fusion processing device in one embodiment of the present application.

[0043] Figure 3 Schematic diagram of the process of multi-sensor fusion processing in one embodiment of the present application.

[0044] Figure 4 Schematic diagram of the process of multi-sensor processing unit fusion judgment in one embodiment of the present application.

[0045] Figure 5 Schematic diagram of the process of updating target fusion data by the sensor processing unit in one embodiment of the present application. DETAILED DESCRIPTION

[0046] The following describes the embodiments of the present application through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present application from the content disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0047] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. Therefore, the illustrations only show components related to the present application and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0048] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0050] Because traditional vehicle sensor fusion mostly uses a parallel fusion approach for data fusion, it takes a long time to complete full sensor information fusion, which can easily lead to information delays, severe fusion lags, and data loss, resulting in low fusion efficiency and poor accuracy. Existing technologies also propose vehicle-road collaboration and its process technology. Vehicle-road collaboration refers to the sharing and interaction of information and data between vehicles and roads through wireless communications and sensor technologies. This approach improves vehicle operation efficiency and driving safety through intelligent vehicle-road coordination and vehicle-vehicle collaboration. Vehicle sensor information is a key component of vehicle-to-vehicle interaction. Vehicles equipped with various sensors continuously transmit driving status parameters to nearby vehicles and roadside infrastructure. Vehicles obtain the driving status of surrounding vehicles to guide their own driving and cooperative driving. These methods, based on interactive perception between vehicles, are signal acquisition and reconstruction methods built on sparse signal representation, non-correlation of measurement matrices, and approximation theory. These methods focus on information exchange between vehicles and do not significantly improve the perception processing of the vehicles themselves, making it difficult to address the issues of sensing accuracy and fusion latency for single-vehicle driving.

[0051] In order to solve the above problems of the prior art, the solution of the present application is described in detail below with reference to specific embodiments.

[0052] The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, smart cars, etc. The following describes exemplary applications of the device provided in the embodiments of the present application. The device provided in the embodiments of the present application can be implemented as various types of user terminals such as smart phones, smart watches, laptops, tablet computers, desktop computers, set-top boxes, mobile devices (for example, mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), intelligent voice interaction devices, smart home appliances, and vehicle-mounted terminals, and can also be implemented as servers. Below, an exemplary application when the device is implemented as a server will be described.

[0053] In one embodiment, a plurality of sensing devices may be installed on the vehicle body, and the sensing devices may include a combination of one or more of temperature sensors, acceleration sensors, laser sensors, infrared sensors, etc. Each sensing device may be connected to a corresponding sensor processing unit, and the sensing device transmits the collected real-time data to the connected sensor processing unit for data preprocessing and data association. After the sensor processing unit completes the data processing, it outputs the data to the data fusion module for sensor data fusion. Finally, the data fusion module outputs the fused data to the application end. The application end can visualize the fused data in the form of images or other forms, and can also control the vehicle to perform corresponding actions based on the fused data, such as accelerating, bypassing obstacles, turning, etc., and can also perform fault prediction or troubleshooting based on the fused data. The specific application of fused data is not limited here.

[0054] See also Figure 1 , Figure 1 The figure is a schematic diagram of an application scenario of a multi-sensor target fusion processing system in one embodiment of the present application. The sensing device is usually installed on the vehicle body, and the sensor processing unit can be installed on the vehicle body close to the sensing device to avoid data loss or data delay caused by long-distance data transmission. Of course, under the premise of ensuring the stability of the data transmission network 100, the sensor processing unit can also be set on the server 200. A communication connection is established between the sensing device and the sensor processing unit through the mobile network to complete the uploading of sensor data. The data fusion module can be set on the server 200 to receive the output data of each sensor processing unit, and synchronously fuse the output data of each sensor processing unit according to a preset fusion cycle, thereby improving the efficiency of the fusion processing.

[0055] In one embodiment, server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0056] In one embodiment, the sensor processing unit and data fusion module may also be provided in a terminal, which may be an in-vehicle terminal. After receiving the real-time sensor data collected by the corresponding sensor device, each sensor processing unit concurrently performs preprocessing and data association of its own real-time sensor data with the fusion data of the previous fusion cycle of the data fusion module, outputs the data processing results to the data fusion module to complete the data fusion of the current fusion cycle, and displays the data fusion results in real time on the in-vehicle display terminal so that occupants can refer to the fusion data and perform corresponding operations. In another embodiment, the terminal may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, smart voice interaction device, smart home appliance, in-vehicle terminal, etc., but is not limited to these.

[0057] In one embodiment, the sensor processing unit and the data fusion module respectively include at least one processor chip and one memory, and the processing chip integrates corresponding processing and fusion algorithms to realize the functions of the sensor processing unit and the data fusion module, and stores the processing and fusion results in corresponding memories respectively.

[0058] See also Figure 2 , Figure 2 is a schematic diagram of the structure of the terminal 400 provided in an embodiment of the present application, Figure 2 The terminal 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0059] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0060] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0061] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 410.

[0062] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0063] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0064] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0065] A network communication module 452 for reaching other computing devices via one or more (wired or wireless) network interfaces 420 , exemplary network interfaces 420 including Bluetooth, WiFi, and USB;

[0066] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0067] The input processing module 454 is configured to detect one or more user inputs or interactions from one of the one or more input devices 432 and to translate the detected inputs or interactions.

[0068] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2 A multi-sensor target fusion processing system 455 stored in the memory 450 is shown, which can be software in the form of programs and plug-ins, including the following software modules: a fusion processing module 4551, a sensor processing module 4552 and a fusion output module 4553. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented.

[0069] The functions of each module will be described below.

[0070] In other embodiments, the system provided in the embodiments of the present application can be implemented in hardware. As an example, the system provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the multi-sensor target fusion processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0071] In some embodiments, the terminal or server can implement the multi-sensor target fusion processing method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a social application APP or a message sharing APP; it can also be a small program, that is, a program that can be run only by downloading it into a browser environment; it can also be a small program or web client program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plug-in.

[0072] The multi-sensor target fusion processing method provided in the embodiment of the present application will be described below in combination with the exemplary application and implementation of the device provided in the embodiment of the present application.

[0073] See also Figure 3 , Figure 3 The flowchart of the multi-sensor target fusion processing method in one embodiment of the present application is shown in FIG. The multi-sensor target fusion processing method provided in the present application includes the following steps.

[0074] Step S101: When each preset fusion cycle arrives, the output data of multiple sensor processing units are synchronously fused to obtain target fusion data.

[0075] To improve fusion processing efficiency and the fault tolerance of the multi-sensor target fusion processing system, the sensor processing units can be configured as a parallel structure. Each sensor processing unit is connected to its own sensor device, receives real-time sensor data from each sensor device for parallel processing, and outputs the processed data to the data fusion module for unified data fusion. Adding or reducing sensor processing units will not significantly affect the overall data fusion processing efficiency.

[0076] See also Figure 4 In one embodiment, step S101 synchronously fuses the output data of multiple sensor processing units to obtain target fused data, including the following steps:

[0077] Step S1011: if it is detected within a preset first waiting time that the real-time sensing data of any one of the sensing processing units is not updated, the corresponding sensing processing unit is skipped, and the output data of the remaining sensing processing units are synchronously integrated;

[0078] Step S1012: Record the data fusion of the skipped sensor processing unit into the next fusion cycle.

[0079] In one embodiment, when the data fusion module's data fusion cycle arrives, the module generates corresponding instructions and outputs them to each sensor processing unit. Each sensor processing unit outputs response data corresponding to the instruction, and the module determines whether the corresponding sensor processing unit has updated its sensor data based on the response data. For example, the response data may include a sensor data update timestamp or other update identifier. In another embodiment, the instruction generated by the module may also include a heartbeat frame. The heartbeat frame is used to determine whether the corresponding sensor processing unit has preset heartbeat feedback. If not, it indicates that the corresponding sensor processing unit is operating abnormally and the sensor processing unit may be skipped. If normal heartbeat feedback is received, the connection between the module and the corresponding sensor processing unit is maintained. The output data of the sensor processing unit in normal operation is selected for data fusion. In practical applications, a comprehensive judgment may be made based on the heartbeat and sensor data update identifier to ensure accuracy. For skipped sensor processing units, their updated sensor data may be used as output data in the next fusion cycle.

[0080] In one embodiment, step S101 includes synchronously fusing the output data of multiple sensor processing units to obtain target fused data, and further includes the following steps:

[0081] Step S1013, when the fusion cycle arrives, obtaining the processing completion signal of each sensing processing unit;

[0082] Step S1014: When the number of processed signals reaches a preset number, the output data of each sensor processing unit is synchronously integrated.

[0083] In one embodiment, before acquiring the output data of each sensor processing unit, a data read request can be triggered based on whether each sensor processing unit has completed processing. Specifically, after each sensor processing unit completes its own data processing and, in response to the aforementioned instructions from the data fusion module, determines that the corresponding sensor processing unit is properly connected and updating, it can generate a processing completion signal and send this processing completion signal to the back-end data fusion module. Data fusion is initiated only after the data fusion module has received the processing completion signals from all sensor processing units. The "all sensor processing units" here do not include the sensor processing units skipped in the aforementioned steps; they only represent sensor processing units in normal operation. In another embodiment, due to fusion cycle constraints, a waiting time can be set for each sensor processing unit. For example, the first waiting time in the aforementioned step indicates that if a corresponding sensor processing unit has not completed data processing within the waiting time, that sensor processing unit can be skipped and the output data of the remaining sensor processing units can be synchronously fused. At the next fusion cycle, the data processed by the skipped sensor processing unit is fed back to the data fusion module as output data. The length of the waiting time can be set according to the actual application scenario requirements and is not limited here. The preset number in step S1014 may include two situations. The first situation is that all sensor processing units have completed the sensor data processing and updating normally. At this time, the preset number can be the total number of sensor processing units. The second situation is to count the number of sensor processing units skipped in the current fusion cycle by counting methods such as counters. At this time, the preset number is the difference between the total number of sensor processing units and the number of skipped sensor processing units. After the data fusion module confirms that it has received the preset number of processing completion signals fed back by the sensor processing units, it sends a data read request to each sensor processing unit to obtain the output data of each sensor processing unit for data fusion.

[0084] Step S102: updating the target fusion data to each of the sensor processing units, and performing data association between the target fusion data and the real-time sensor data through the sensor processing units to obtain associated data as output data of each sensor processing unit.

[0085] In one embodiment, the target fusion module sends the target fusion data to each sensor processing unit after each fusion cycle, so that each sensor processing unit can perform parallel data association based on its own real-time sensor data. The sensor processing units do not interfere with each other when performing data association. Unified data fusion is performed based on the associated data of each sensor processing unit, which reduces the amount of calculation, simplifies the fusion operation, and improves the fusion efficiency.

[0086] See also Figure 5 In one embodiment, in step S102, updating the target fusion data to each of the sensor processing units includes the following steps:

[0087] Step S1021: After the output data fusion is completed within the current fusion cycle, a fusion completion instruction is output;

[0088] Step S1022: obtaining a fusion update request of each sensor processing unit according to the fusion completion instruction;

[0089] Step S1023: output the target fusion data to the sensor processing unit that issues the request according to the fusion update request.

[0090] In one embodiment, each sensor processing unit triggers an update of its own fused data based on the completion of fusion by the data fusion module. Specifically, after the data fusion module completes data fusion within a fusion cycle, it outputs a fusion completion instruction to each sensor processing unit. Upon receiving the fusion completion instruction, each sensor processing unit generates a fusion update request, obtains the target fused data from the data fusion module based on the fusion update request, and updates the target fused data into its own storage container.

[0091] In one embodiment, obtaining the fusion update request of each sensor processing unit according to the fusion completion instruction in step S1022 includes the following steps:

[0092] Step S10221: If the fusion update request is not received within the preset second waiting time, the update operation of the corresponding sensor processing unit is skipped;

[0093] Step S10222: In the next fusion cycle, the target fusion data is used as the request response data of the sensor processing unit that has not been updated in the current fusion cycle.

[0094] In one embodiment, the data fusion module can count the number of fusion update requests received. When the number of fusion update requests is less than the number of sensor processing units, the sensor processing unit that has not issued a fusion update request is determined based on the identifier of the sensor processing unit contained in the fusion update request, and the timing is started. If it is determined that the fusion update request of the corresponding sensor processing unit has not been received after the second waiting time, the update operation of the sensor processing unit is skipped, and the fusion data is updated only for the remaining normally connected sensor processing units. When the next fusion cycle arrives, heartbeat data can be sent to the sensor processing unit that has not issued a fusion update request to detect whether the sensor processing unit is connectable. After the connection is established, the target fusion data of the previous fusion cycle is output to the sensor processing unit for fusion data update.

[0095] In one embodiment, in step S102, the sensor processing unit performs data association on the target fusion data and the real-time sensor data, including the following steps:

[0096] Step S1024, obtaining edge features of one or more target objects in the target fusion data;

[0097] Step S1025, defining a correlation region based on the edge features, and removing real-time sensing data outside the correlation region;

[0098] Step S1026 , comparing the features of the real-time sensing data within the association area with the edge features to obtain matching features to complete data association.

[0099] In one embodiment, taking vehicle environment perception as an example, after completing the target fusion data update, each sensor processing unit can segment one or more target objects from the target fusion data and extract edge features of the target objects. Target object segmentation can be performed using a preset segmentation model trained using a neural network. The target fusion data is then classified using this segmentation model to obtain one or more target objects. Edge features are extracted based on the obtained target object data. The specific feature extraction method can employ a convolutional network, etc., which is not limited here. To associate the target object data with the real-time sensor data received by the sensor processing unit, an association region is defined based on the edge features. For example, the region containing the edge features is expanded by a specified size, and the edge feature region and the expanded region are combined to form the association region. Because real-time sensor information from sensor devices can overlap significantly at adjacent moments, defining an association region and excluding real-time sensor data outside of the association region can effectively reduce the computational complexity of data association and optimize the efficiency of subsequent data fusion. Features extracted from the real-time sensor data within the association region are compared with the edge features for similarity. If the similarity reaches a set threshold, the two are determined to match, and the matching features are associated and marked as associated. In another embodiment, the real-time sensing data and target fusion data can be pre-processed at the sensor processing unit, such as removing redundant data and interference point sets, to improve the accuracy of data association. After data association is completed, the associated data is used as the output of the corresponding sensor processing unit.

[0100] In one embodiment, step S101 further includes the following steps:

[0101] Step S1015, using the target fusion data of the previous fusion cycle as the initial ontology model;

[0102] Step S1016, comparing the initial ontology model with the output data of each sensor processing unit, determining a fusion boundary, and marking the output data of each sensor processing unit according to the fusion boundary;

[0103] Step S1017, inputting the data of the output data outside the fusion boundary into a preset classification model to obtain edge features of different categories;

[0104] Step S1018: performing edge feature matching according to the categories of the edge features, and associating the matched edge features to complete the fusion of corresponding edge features.

[0105] In one embodiment, after the data fusion module completes the fusion based on the output data of multiple sensor processing units in each fusion cycle, a fused ontology model can be obtained, and the ontology model includes the target fusion data obtained in the previous fusion cycles. The target fusion data of the previous fusion cycle is used as the initial ontology model, and the initial ontology model is compared with the output data of each sensor processing unit to determine a fusion boundary. The data outside the fusion boundary of the initial ontology model is input into the pre-trained classification model to obtain the edge feature classification of the associated data output by each sensor processing unit. If the edge feature is a left edge feature or a right edge feature, the edge feature matching is performed according to the relative position of the edge feature to complete the edge feature fusion of the current fusion cycle.

[0106] Step S103: After each fusion cycle is completed, the target fusion data is output to the application end.

[0107] In one embodiment, the application can receive updates of the target fusion data and display it in real time, allowing relevant personnel to analyze and make decisions about the vehicle's operating environment and status based on the displayed fusion data. This can include troubleshooting, operational decisions, etc. The application can be an in-vehicle terminal or a remotely connected terminal device.

[0108] In another embodiment, the specific process of the multi-sensor target fusion processing method is as follows:

[0109] S1, the single sensor processing unit waits for the conditional variables processed by the data fusion module to be processed.

[0110] S2: A single sensor receives the updated information after information fusion, and then queries the sensor update container to check whether there is an update and sets the waiting time to waitTime, where:

[0111] waitTime=Tfusion-fusionLastUseT;

[0112] Where: Tfusion is the set fusion operation cycle, which is generally a fixed value; fusionLastUseT is the actual operation time of the data fusion in the previous cycle;

[0113] If there is no sensor data update in this fusion cycle, the processing is skipped directly, and the sensor update status flag is set to the data fusion for processing;

[0114] In step S3, each single sensor processing module obtains the updated data of the sensor, the vehicle status data, and the fused data for customized processing. This generally includes preprocessing of the single sensor data and data association. It then updates the sensor association information used for information fusion and finally updates the processed conditional variables, which are defined as follows:

[0115] sensorProcessStateUpdate[MaxS];

[0116] Among them, MaxS is the number of single sensor processing units in the fusion system, so each single sensor processing unit corresponds to a conditional variable that has completed single sensor processing.

[0117] S4, the data fusion module waits for sensorProcessStateUpdate[MaxS] to be ready and then obtains the latest sensor-related data for unified data fusion.

[0118] Set a maximum waiting time for each sensor processing. If the timeout is exceeded, the fusion of the sensor information is skipped and put into the next fusion cycle for processing.

[0119] S5, after performing customized information fusion processing, calculate waitTime, update fusion target information, and determine whether each sensor processing unit has completed the update of fusion target information.

[0120] According to the description of the above multi-sensor target fusion processing method, the embodiment of the present application provides a multi-sensor target fusion processing device, such as Figure 1 The device runs a multi-sensor target fusion processing system, which includes: a fusion processing module 4551, which is used to synchronously fuse the output data of multiple sensor processing units at the arrival of each preset fusion cycle to obtain target fusion data; a sensor processing module 4552, which is used to update the target fusion data to each sensor processing unit, and to associate the target fusion data with real-time sensor data through the sensor processing unit to obtain associated data as the output data of each sensor processing unit; and a fusion output module 4553, which is used to output the target fusion data to the application end after each fusion cycle.

[0121] In one embodiment, the fusion processing module 4551 is also used to synchronously fuse the output data of multiple sensor processing units to obtain target fusion data, including: if it is detected that the real-time sensor data of any sensor processing unit is not updated within a preset first waiting time, the corresponding sensor processing unit is skipped and the output data of the remaining sensor processing units are synchronously fused; and the data fusion of the skipped sensor processing units is recorded in the next fusion cycle.

[0122] In one embodiment, the sensor processing module 4552 is also used to update the target fusion data to each of the sensor processing units, including: after completing the output data fusion within the current fusion cycle, outputting a fusion completion instruction; obtaining a fusion update request from each of the sensor processing units according to the fusion completion instruction; and outputting the target fusion data to the sensor processing unit that issued the request according to the fusion update request.

[0123] In one embodiment, the fusion processing module 4551 is also used to synchronously fuse the output data of multiple sensor processing units to obtain target fusion data, including: when the fusion cycle arrives, obtaining the processing completion signal of each of the sensor processing units; when the number of the processing completion signals reaches a preset number, synchronously fusing the output data of each of the sensor processing units.

[0124] In one embodiment, the sensor processing module 4552 is also used to obtain a fusion update request of each of the sensor processing units according to the fusion completion instruction, including: if the fusion update request is not received within a preset second waiting time, skipping the update operation of the corresponding sensor processing unit; and using the target fusion data as the request response data of the sensor processing unit that has not been updated in the current fusion cycle in the next fusion cycle.

[0125] In one embodiment, the sensor processing module 4552 is also used to perform data association between the target fusion data and the real-time sensor data through the sensor processing unit, including: obtaining edge features of one or more target objects in the target fusion data; defining an association area based on the edge features, and eliminating real-time sensor data outside the association area; comparing the features of the real-time sensor data in the association area with the edge features to obtain matching features to complete data association.

[0126] In one embodiment, the fusion processing module 4551 is also used to synchronously fuse the output data of multiple sensor processing units to obtain target fusion data, including: using the target fusion data of the previous fusion cycle as the initial ontology model; comparing the initial ontology model with the output data of each sensor processing unit to determine the fusion boundary, and marking the output data of each sensor processing unit according to the fusion boundary; inputting the output data outside the fusion boundary into a preset classification model to obtain edge features of different categories; performing edge feature matching according to the category of the edge feature, and associating the matched edge features to complete the fusion of corresponding edge features.

[0127] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0128] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0129] As an example, executable instructions may, but do not necessarily, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (for example, files storing one or more modules, subroutines, or code portions).

[0130] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0131] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical concepts disclosed in this application shall be covered by the claims of this application.

Claims

1. A multi-sensor target fusion processing method, characterized in that: include: When each preset fusion cycle arrives, the output data of multiple sensor processing units are synchronously fused to obtain target fusion data; Synchronously fusing the output data of multiple sensor processing units to obtain target fusion data, including: using the target fusion data of the previous fusion cycle as an initial ontology model; comparing the initial ontology model with the output data of each sensor processing unit to determine a fusion boundary, and marking the output data of each sensor processing unit according to the fusion boundary; inputting the data of the output data outside the fusion boundary into a preset classification model to obtain edge features of different categories; performing edge feature matching according to the category of the edge features, and associating the matched edge features to complete the fusion of corresponding edge features; Updating the target fusion data to each of the sensor processing units, and performing data association between the target fusion data and the real-time sensor data through the sensor processing unit to obtain associated data as output data of each sensor processing unit; After each fusion cycle is completed, the target fusion data is output to the application end.

2. The multi-sensor target fusion processing method according to claim 1, characterized in that: Synchronously fuse the output data of multiple sensor processing units to obtain target fusion data, including: If it is detected within a preset first waiting time that the real-time sensing data of any of the sensing processing units is not updated, the corresponding sensing processing unit is skipped, and the output data of the remaining sensing processing units are synchronously integrated; The data fusion of the skipped sensor processing unit is recorded in the next fusion cycle.

3. The multi-sensor target fusion processing method according to claim 1, characterized in that: Updating the target fusion data to each of the sensor processing units includes: After the output data fusion is completed within the current fusion cycle, a fusion completion instruction is output; Obtaining a fusion update request from each of the sensor processing units according to the fusion completion instruction; Outputting the target fusion data to the sensor processing unit that issues the request according to the fusion update request.

4. The multi-sensor target fusion processing method according to claim 1, characterized in that: Synchronously fuse the output data of multiple sensor processing units to obtain target fusion data, including: When a fusion cycle arrives, obtaining a processing completion signal from each of the sensing processing units; When the number of processed signals reaches a preset number, the output data of each sensor processing unit is synchronously integrated.

5. The multi-sensor target fusion processing method according to claim 3, characterized in that: Acquiring a fusion update request of each of the sensor processing units according to the fusion completion instruction includes: If the fusion update request is not received within the preset second waiting time, the update operation of the corresponding sensor processing unit is skipped; In the next fusion cycle, the target fusion data is used as the request response data of the sensor processing unit that has not been updated in the current fusion cycle.

6. The multi-sensor target fusion processing method according to claim 1, characterized in that: The sensor processing unit performs data association on the target fusion data and the real-time sensor data, including: Acquire edge features of one or more target objects in the target fusion data; Delineating a related area according to the edge features, and eliminating real-time sensing data outside the related area; The features of the real-time sensing data in the association area are compared with the edge features to obtain matching features to complete data association.

7. A multi-sensor target fusion processing system using the multi-sensor target fusion processing method according to any one of claims 1 to 6, characterized in that: include: A fusion processing module is used to synchronously fuse the output data of multiple sensor processing units at the arrival of each preset fusion cycle to obtain target fusion data; a sensor processing module, configured to update the target fusion data to each of the sensor processing units, and to perform data association between the target fusion data and the real-time sensor data through the sensor processing units to obtain associated data as output data of each sensor processing unit; The fusion output module is used to output the target fusion data to the application end after each fusion cycle ends.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images