Sensing fusion result acquisition method, readable storage medium, and smart device

Abstract

The present application relates to the technical field of autonomous driving. Specifically provided are a sensing fusion result acquisition method, a sensing fusion processing method, a readable storage medium, and a smart device, which aim to solve the problem of how to realize rapid stitching and combination of multimodal asynchronous sensing data. To this end, in the present application, after a target detection result acquired by a sensor that is provided on a smart device is paired with positioning data, a target detection pairing result is obtained, target detection pairing results of a plurality of sensors are stored in a feature queue for storing sensing features, and a sensing target tracking result is obtained on the basis of the target detection pairing results in the feature queue and on the basis of a neural network model. By means of the configuration mode, the present application can realize rapid combination and stitching of target detection results of a plurality of sensors by means of a feature queue, thereby realizing the generation of a sensing target tracking result on the basis of multimodal asynchronous time-series data.

Description

Perception fusion result acquisition method, readable storage medium and intelligent device This application claims priority to Chinese patent application CN 202410340412.3, filed on March 25, 2024, entitled “Method for obtaining perception fusion results, readable storage medium and intelligent device”. The entire contents of the above Chinese patent application are incorporated into this application by reference. Technical Field

[0001] The present application relates to the field of autonomous driving technology, and specifically provides a method for obtaining perception fusion results, a perception fusion processing method, a readable storage medium, and an intelligent device. Background Art

[0002] Object tracking in autonomous driving has made significant progress in recent years, particularly in neural networks, multi-sensor fusion, and time series information processing. For example, this involves combining LiDAR data with image data and leveraging time series information. In this regard, the Transformer, a powerful sequence modeling tool, uses its self-attention mechanism to capture long-range object dependencies across diverse sensor data and time series information, playing a key role in multimodal data fusion.

[0003] However, when deploying the Transformer in a real vehicle, due to the complexity of data calculations, the large number of parameters, and the characteristics of multimodal asynchronous time series data, specialized engineering links are required to ensure execution speed and efficiency. However, since the Transformer is a new architecture in the field of autonomous driving and hardware conditions are also limited, how to quickly splice and combine multimodal asynchronous time series data to enable the Transformer neural network to be applied in real vehicle scenarios remains a challenge in this field.

[0004] Accordingly, this field requires a new solution for obtaining perception fusion results to solve the above problems. Summary of the Invention

[0005] In order to overcome the above-mentioned defects, the present application is proposed to provide a solution or at least partially solve the problem of how to achieve rapid splicing and combination of multimodal asynchronous perception data.

[0006] In a first aspect, the present application provides a method for obtaining a perception fusion result, the method being applied to a smart device provided with a plurality of sensors, the method comprising:

[0007] For each sensor, pairing a target detection result obtained based on the sensor with positioning data corresponding to the smart device to obtain a target detection pairing result corresponding to the sensor;

[0008] The target detection pairing results of multiple sensors are stored in a feature queue for storing perception fusion results to obtain the perception fusion results.

[0009] In one technical solution of the above-mentioned method for obtaining a perception fusion result, pairing the target detection result obtained by the sensor with the positioning data corresponding to the smart device to obtain the target detection pairing result corresponding to the sensor includes:

[0010] Putting the target detection result obtained by the sensor into the cache queue corresponding to the sensor;

[0011] For each target detection result in the cache queue, the target detection result is paired with the positioning data closest to the time when the target detection result is generated to obtain the target detection pairing result.

[0012] In one technical solution of the above-mentioned method for obtaining a perception fusion result, storing the target detection pairing results of multiple sensors in a feature queue for storing perception fusion results to obtain the perception fusion result includes:

[0013] When the target detection pairing results of the multiple sensors are updated, storing the target detection pairing results in a feature queue for storing perception features according to a preset event thread manager;

[0014] The event thread manager asynchronously manages the target detection pairing results of the sensors at different frequencies.

[0015] In a second aspect, a perception fusion processing method is provided, characterized in that the method is applied to a smart device, the smart device being provided with a plurality of sensors, and the method comprising:

[0016] For each sensor, pairing a target detection result obtained based on the sensor with positioning data corresponding to the smart device to obtain a target detection pairing result corresponding to the sensor;

[0017] Storing the target detection pairing results of the plurality of sensors in a feature queue for storing perception features;

[0018] The target detection pairing results in the feature queue are input into a preset neural network model to obtain the perception target tracking results.

[0019] In one technical solution of the above-mentioned perception fusion processing method, pairing the target detection result obtained by the sensor with the positioning data corresponding to the smart device to obtain the target detection pairing result corresponding to the sensor includes:

[0020] Putting the target detection result obtained by the sensor into the cache queue corresponding to the sensor;

[0021] For each target detection result in the cache queue, the target detection result is paired with the positioning data closest to the time when the target detection result is generated to obtain the target detection pairing result.

[0022] In one technical solution of the above-mentioned perception fusion processing method, storing the target detection pairing results of multiple sensors into a feature queue for storing perception features includes:

[0023] When the target detection pairing results of the multiple sensors are updated, storing the target detection pairing results in a feature queue for storing perception features according to a preset event thread manager;

[0024] The event thread manager asynchronously manages the target detection pairing results of the sensors at different frequencies.

[0025] In one technical solution of the above-mentioned perception fusion processing method, storing the target detection pairing result in a feature queue for storing perception features includes:

[0026] Performing data analysis on the target detection pairing result to obtain a target data analysis result;

[0027] The target data parsing result is stored in a feature queue for storing perception features.

[0028] In one technical solution of the above-mentioned perception fusion processing method, storing the target data analysis result in a feature queue for storing perception features includes:

[0029] Performing sliding window sampling on the target data parsing results in the feature queue;

[0030] The parsing results of the sampled target data are stored in a feature queue for storing perception features.

[0031] In one technical solution of the above-mentioned perception fusion processing method, the sliding window sampling of the target data analysis results in the feature queue includes:

[0032] The detection target parsing result is sampled using a time-based sliding window.

[0033] In one technical solution of the above-mentioned perception fusion processing method, the sliding window sampling of the target data analysis results in the feature queue includes:

[0034] The detection target parsing result is subjected to distance-based sliding window sampling.

[0035] In a technical solution of the above-mentioned perception fusion processing method, the event thread manager prioritizes preset sensors and asynchronously manages the target detection pairing results of the sensors of different frequencies.

[0036] In a technical solution of the above-mentioned perception fusion processing method, the event thread manager asynchronously manages the target detection pairing results of the sensors of different frequencies with time priority.

[0037] In one technical solution of the above-mentioned perception fusion processing method, the method further includes:

[0038] Obtaining the sensor type and generation time of the target detection pairing result contained in the feature queue;

[0039] The obtained sensor type and generation time are synchronized to the cloud.

[0040] In a third aspect, a computer-readable storage medium is provided, which stores a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute the perception fusion result acquisition method described in any one of the technical solutions of the above-mentioned perception fusion result acquisition method.

[0041] In a fourth aspect, a smart device is provided, comprising:

[0042] at least one processor;

[0043] and, a memory communicatively coupled to the at least one processor;

[0044] Wherein, a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the method for obtaining the perception fusion result described in any one of the technical solutions of the above-mentioned method for obtaining the perception fusion result is implemented.

[0045] The above one or more technical solutions of this application have at least one or more of the following beneficial effects:

[0046] In the technical solution for implementing the present application, the present application pairs the target detection results obtained by the sensors provided on the smart device with the positioning data to obtain the target detection pairing results, stores the target detection pairing results of multiple sensors in a feature queue for storing perception features, and inputs the target detection pairing results in the feature queue into a preset neural network model to obtain the perception target tracking results. Through the above configuration, the present application can realize the rapid combination and splicing of the target detection results of multiple sensors through the feature queue, so that the feature queue can provide multi-modal asynchronous time series data for the neural network model, and then realize the generation of perception target tracking results based on the multi-modal asynchronous time series data based on the neural network model, so as to better apply the perception target tracking results in actual application scenarios for real-time reasoning, prediction and other functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The disclosure of this application will become more easily understood with reference to the accompanying drawings. Those skilled in the art will readily appreciate that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Among them:

[0048] FIG1 is a flow chart showing the main steps of a method for obtaining a perception fusion result according to an embodiment of the present application;

[0049] FIG2 is a schematic flow chart of the main steps of a perception fusion processing method according to an embodiment of the present application;

[0050] FIG3 is a schematic block diagram of a main implementation method of a perception fusion processing method according to an implementation of an embodiment of the present application;

[0051] FIG4 is a schematic diagram of backlog data when the target detection pairing result is placed in a cache queue according to an implementation of an embodiment of the present application;

[0052] FIG5 is a schematic diagram of a process in which an event thread manager stores target detection pairing results into a feature queue according to an implementation of an embodiment of the present application;

[0053] FIG6 is a schematic diagram of performing sliding window sampling on the detection target analysis results according to an implementation of an embodiment of the present application;

[0054] FIG7 is a schematic diagram of ensuring data consistency between a smart device and a cloud according to an implementation of an embodiment of the present application;

[0055] FIG8 is a schematic diagram of a connection relationship between a memory and a processor of a smart device according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the scope of protection of the present application.

[0057] In the description of this application, "module" and "processor" may include hardware, software, or a combination of both. A module may include hardware circuitry, various suitable sensors, communication ports, and memory. It may also include software components, such as program code, or a combination of software and hardware. A processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. A processor has data and / or signal processing capabilities. A processor may be implemented in software, hardware, or a combination of both. Non-transitory computer-readable storage media include any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" refers to all possible combinations of A and B, such as only A, only B, or both A and B. The terms "at least one of A or B" or "at least one of A and B" have similar meanings to "A and / or B" and may include only A, only B, or both A and B. The singular forms "a" and "the" may also include the plural forms.

[0058] An automated driving system (ADS) is a system that continuously performs all dynamic driving tasks (DDT) within its operational domain design (ODD). Specifically, the system is only allowed to fully assume the task of autonomous vehicle control under specified appropriate driving scenarios. When the vehicle meets the ODD conditions, the system is activated, replacing the human driver as the vehicle's primary driver. The DDT refers to the continuous lateral (left and right steering) and longitudinal motion control (acceleration, deceleration, and constant speed) of the vehicle, as well as the detection and response to objects and events in the vehicle's driving environment. The ODD refers to the conditions under which the automated driving system can operate safely. These conditions can include geographic location, road type, speed range, weather, time of day, and national and local traffic laws and regulations.

[0059] Referring to Figure 1 , Figure 1 is a schematic flow diagram illustrating the main steps of a method for obtaining a perception fusion result according to an embodiment of the present application. As shown in Figure 1 , the method for obtaining a perception fusion result in this embodiment of the present application is applied to a smart device equipped with multiple sensors. The method primarily includes the following steps S101 through S103 .

[0060] Step S101: For each sensor, pair the target detection result obtained by the sensor with the positioning data corresponding to the smart device to obtain the target detection pairing result corresponding to the sensor.

[0061] In this embodiment, for each sensor configured on the smart device, the target detection result obtained by the sensor can be paired with the corresponding positioning data to obtain the sensor's target detection pairing result. Specifically, when a target detection result is generated, the smart device's most recent positioning data (Ego Motion) can be queried and the target detection result can be paired with the positioning data to obtain the target detection pairing result.

[0062] In one embodiment, the smart device may be a driving device, a smart car, a robot, or the like.

[0063] In one embodiment, the sensor may include a camera, a lidar, a millimeter-wave radar, etc.

[0064] In one embodiment, a separate thread can be set up for each sensor's target detection result to receive a message about the target detection result. Please refer to Figure 2, which is a schematic block diagram of the main implementation of a method for obtaining perception fusion results according to an embodiment of the present application. As shown in Figure 2, the thread can be a trigger. A trigger for the target detection result of the lidar can be set for the lidar, and a trigger for the target detection result of the camera can be set for the camera.

[0065] Step S102: Storing the target detection pairing results of multiple sensors into a feature queue for storing perception features to obtain a perception fusion result.

[0066] In this embodiment, the target detection pairing results of multiple sensors can be stored in a feature queue in time sequence to obtain a perception fusion result.

[0067] In one embodiment, as shown in FIG2 , an event thread manager may be provided to store the target detection pairing results in a feature queue. A time thread manager may be provided to asynchronously manage target detection pairing results of different frequencies.

[0068] Among them, the frequency of the event thread manager is higher than the generation frequency of the target detection results of each sensor. For example, the generation frequency of the target detection results of the lidar is 10hz, the generation frequency of the target detection results of the camera is 30hz, and the frequency of the event thread manager is 40hz. In this way, it is possible to ensure that the output frequency of the event thread manager is consistent with the original generation frequency of the target detection results, thereby avoiding the loss of target detection pairing results in the event thread manager. Referring to Figure 4, Figure 4 is a schematic diagram of the process of the event thread manager storing the target detection pairing results into the feature queue according to an embodiment of the present application. As shown in Figure 4, the camera frame captured by the camera passes through target detection to obtain the target detection result of the camera (output frequency is 30hz). After the target detection result is paired with the positioning data, it enters the event thread manager. After the event thread manager performs buffer pop (Buffer Pop), the frequency of the target detection pairing result of the camera output is also 30hz for downstream use. The lidar frames collected by the lidar are subjected to target detection to obtain the target detection results of the lidar (output frequency is 10hz). After the target detection results are paired with the positioning data, they enter the event thread manager. The event thread manager can perform buffer pop (Buffer Pop) according to the timing, and the frequency of the output lidar target detection pairing results is also 10hz for downstream use.

[0069] In one embodiment, when target detection pairing results of multiple sensors are updated, the target detection pairing results of sensors with different frequencies may be asynchronously managed by an event thread manager.

[0070] In one embodiment, the event thread manager may asynchronously manage target pairing results of different frequencies based on time priority, that is, the target detection pairing result generated earlier will be stored in the feature queue first.

[0071] In another implementation, the event thread manager can asynchronously manage target pairing results of different frequencies based on preset sensor priorities. Specifically, when target detection pairing results from multiple sensors are updated, the target detection pairing results from a particular sensor can be prioritized for storage in the feature queue. For example, if a lidar's target detection results are more accurate, a lidar-prioritized asynchronous management strategy can be configured.

[0072] In one embodiment, the event line manager may also apply a combined asynchronous management strategy, such as an asynchronous management strategy based on both time priority and preset sensor priority. Specifically, if multiple sensors have different update times, the asynchronous management strategy based on time priority may be used. If two sensors have the same update time, the asynchronous management strategy based on preset sensor priority may be used.

[0073] In one implementation, the perception fusion results can be applied to neural network models such as Transformer, and the feature queue can follow the first-in-first-out principle, so that the perception fusion results can be applied to real-time reasoning of smart devices based on time sequence, such as obstacle prediction, real-time construction of road network topology, etc.

[0074] Based on the above steps S101-S102, the embodiment of the present application pairs the target detection results obtained by the sensors set on the smart device with the positioning data to obtain the target detection pairing results, and stores the target detection pairing results of multiple sensors in the feature queue used to store the perception fusion results, thereby obtaining the perception fusion results. Through the above configuration, the embodiment of the present application can realize the rapid combination and splicing of the target detection results of multiple sensors through the feature queue, thereby realizing the generation of perception fusion results based on multi-modal asynchronous time series data, so that the perception fusion results can be better applied to the real-time reasoning, prediction and other functions of neural network models such as Transformer in real vehicle scenarios.

[0075] Furthermore, the present application also provides a perception fusion processing method.

[0076] Referring to Figure 2, Figure 2 is a schematic flow diagram of the main steps of a perception fusion processing method according to an embodiment of the present application. As shown in Figure 2, the perception fusion processing method in this embodiment of the present application is applied to a smart device equipped with multiple sensors. The perception fusion method mainly includes the following steps S201-S203.

[0077] Step S201: For each sensor, pair the target detection result obtained based on the sensor with the positioning data corresponding to the smart device to obtain the target detection pairing result corresponding to the sensor.

[0078] In this embodiment, for each sensor configured on the smart device, the target detection result obtained by the sensor can be paired with the corresponding positioning data to obtain the sensor's target detection pairing result. Specifically, when a target detection result is generated, the smart device's most recent positioning data (Ego Motion) can be queried and the target detection result can be paired with the positioning data to obtain the target detection pairing result.

[0079] Step S202: storing the target detection pairing results of the multiple sensors into a feature queue for storing perception features.

[0080] In this embodiment, the target detection pairing results of multiple sensors may be stored in a feature queue for storing perception features in a time sequence.

[0081] In one embodiment, as shown in FIG2 , an event thread manager may be provided to store the target detection pairing results in the feature queue. The event thread manager can asynchronously manage target detection pairing results of different frequencies.

[0082] Among them, the frequency of the event thread manager is higher than the generation frequency of the target detection results of each sensor. For example, the generation frequency of the target detection results of the lidar is 10hz, the generation frequency of the target detection results of the camera is 30hz, and the frequency of the event thread manager is 40hz. In this way, it is possible to ensure that the output frequency of the event thread manager is consistent with the original generation frequency of the target detection results, thereby avoiding the loss of target detection pairing results in the event thread manager. Referring to Figure 4, Figure 4 is a schematic diagram of the process of the event thread manager storing the target detection pairing results into the feature queue according to an embodiment of the present application. As shown in Figure 4, the camera frame captured by the camera passes through target detection to obtain the target detection result of the camera (output frequency is 30hz). After the target detection result is paired with the positioning data, it enters the event thread manager. After the event thread manager performs buffer pop (Buffer Pop), the frequency of the target detection pairing result of the camera output is also 30hz for downstream use. The lidar frames collected by the lidar are subjected to target detection to obtain the target detection results of the lidar (output frequency is 10hz). After the target detection results are paired with the positioning data, they enter the event thread manager. The event thread manager can perform buffer pop (Buffer Pop) according to the timing, and the frequency of the output lidar target detection pairing results is also 10hz for downstream use.

[0083] In one embodiment, when target detection pairing results of multiple sensors are updated, the target detection pairing results of sensors with different frequencies may be asynchronously managed by an event thread manager.

[0084] In one embodiment, the event thread manager can asynchronously manage target pairing results of different frequencies based on time priority. That is, the target detection pairing result generated earlier will be stored in the feature queue first.

[0085] In another implementation, the event thread manager can asynchronously manage target pairing results of different frequencies based on preset sensor priorities. Specifically, when target detection pairing results from multiple sensors are updated, the target detection pairing results from a particular sensor can be prioritized for storage in the feature queue. For example, if a lidar's target detection results are more accurate, an asynchronous management strategy can prioritize the lidar.

[0086] In one embodiment, the event thread manager may also apply a combined asynchronous management strategy, such as an asynchronous management strategy based on both time priority and preset sensor priority. Specifically, if multiple sensors have different update times, the asynchronous management strategy based on time priority may be used. If two sensors have the same update time, the asynchronous management strategy based on preset sensor priority may be used.

[0087] Step S203: Input the target detection pairing results in the feature queue into the preset neural network model to obtain the perception target tracking results.

[0088] In this embodiment, the target detection pairing results in the feature queue can be used as input data for the neural network model, thereby obtaining a perception target tracking result based on the neural network model. The perception target tracking result is a time series fusion result of the perception target generated based on multimodal asynchronous time series data.

[0089] In one embodiment, the neural network model may be a model based on the Transformer architecture.

[0090] In one embodiment, the feature queue can follow the first-in-first-out principle, so that the target detection pairing results are input into the neural network model based on the time sequence to obtain the perception target tracking results.

[0091] In one embodiment, the perception target tracking results can be applied to real-time reasoning of smart devices, such as obstacle prediction, real-time construction of road network topology, etc.

[0092] Based on the above steps S201-S203, the embodiment of the present application pairs the target detection results obtained by the sensors set on the smart device with the positioning data to obtain the target detection pairing results, stores the target detection pairing results of multiple sensors in a feature queue for storing perception features, and inputs the target detection pairing results in the feature queue into a preset neural network model to obtain the perception target tracking results. Through the above configuration method, the embodiment of the present application can realize the rapid combination and splicing of the target detection results of multiple sensors through the feature queue, so that the feature queue can provide multi-modal asynchronous time series data for the neural network model, and then realize the generation of perception target tracking results based on the multi-modal asynchronous time series data based on the neural network model, so as to better apply the perception target tracking results in actual application scenarios for real-time reasoning, prediction and other functions.

[0093] The following further describes step S201 and step S202.

[0094] In one implementation of the embodiment of the present application, step S201 may further include the following steps S2011 and S2012:

[0095] Step S2011: putting the target detection result obtained by the sensor into the buffer queue corresponding to the sensor.

[0096] Step S2012: For each target detection result in the cache queue, pair the target detection result with the positioning data closest to the time when the target detection result is generated to obtain a target detection pairing result.

[0097] In this embodiment, as shown in FIG2 , a buffer queue may be set in the trigger of each sensor. When a target detection result is generated, the target detection result may be stored in the buffer queue in a time sequence. After each target detection result in the buffer queue is paired with the positioning data, the target detection pairing result is obtained, and the buffer is popped in a time sequence.

[0098] The setting of the cache queue can ensure the integrity of the target detection results generated by each sensor and prevent message blocking. For example, when the downstream model reasoning times out, multiple frames of target detection results can be backlogged in the cache queue. Among them, the cache queue can follow the first-in-first-out principle, thereby ensuring the temporal consistency of the target detection results during the application process. Please refer to Figure 3, which is a schematic diagram of the backlog data when the target detection pairing results are placed in the cache queue according to an implementation of an embodiment of the present application. As shown in the first and second figures in Figure 3, at time Tk and Tk+1, since there is no backlog data, each frame of target detection results is converted into target detection pairing results in sequence and stored in the feature queue. As shown in the third figure in Figure 3, due to problems such as scheduling or model timeout, the target detection result at time Tk+3 cannot be sent out. In order not to lose data, the target detection result at time Tk+3 can be temporarily stored in the cache queue for the next time it is sent out. As shown in the fourth figure in Figure 3, if data is accumulated at times Tk+2 and Tk+3 due to scheduling or model timeout, these backlogged data are stored in the cache queue and will not be lost. There is still hope that they can be converted into target detection pairing results and stored in the feature queue for downstream use.

[0099] In one implementation of the embodiment of the present application, step S202 may include the following steps S2021 and S2022:

[0100] Step S2021: performing data analysis on the target detection pairing result to obtain a target data analysis result.

[0101] In this embodiment, as shown in FIG2 , the target detection pairing result may be first subjected to data analysis to obtain a target data analysis result.

[0102] In one implementation, the target data parsing result may be in tensor format.

[0103] Step S2022: Storing the target data analysis result in a feature queue for storing perception features.

[0104] In this embodiment, as shown in FIG2 , the target parsing result can be placed in a feature queue for triggering by a downstream model (ie, a neural network-based model) to obtain an output result, ie, a perception target tracking result.

[0105] In one embodiment, step S2022 may be further configured to: perform sliding window sampling on the target data parsing results in the feature queue, and store the sampled target data parsing results in the feature queue for storing perception features.

[0106] In this embodiment, as shown in FIG2 , due to the different requirements of the scenarios and models for the application perception fusion results, the target data analysis results in the feature queue can be sampled using a sliding window based on different requirements, thereby obtaining appropriate target data analysis results to be input into the neural network model.

[0107] Please refer to Figure 5, which is a schematic diagram of sliding window sampling of the detection target analysis results according to an embodiment of the present application. As shown in Figure 5, the target data analysis results in the feature queue can be sampled by sliding window, and after sampling, they are input into the downstream model (i.e., the neural network model). Among them, the target data analysis results are obtained based on the target detection results of the camera (BEV OD), the target detection results of the laser radar (Lidar OD) and the positioning data of the smart device (Ego Motion).

[0108] In one embodiment, the target data analysis results can be sampled using a time-based sliding window. For example, if the 30 Hz output frequency of the lidar is too high, the target data analysis results can be sampled using a time-based sliding window to reduce the frequency to 10 Hz.

[0109] In another implementation, distance-based sliding window sampling can be performed on the target data analysis results. For example, if the downstream model needs to be expanded to include static input elements, and the number of model input frames is more positively correlated with the travel distance of the smart device itself, distance-based sliding window sampling can be used.

[0110] In one embodiment, the specific parameters and configuration of the sliding window sampling can be configured through a config file.

[0111] In one implementation of the embodiment of the present application, the present application further includes the following step S204:

[0112] Step S204: Obtain the sensor type and generation time of the target detection pairing result contained in the feature queue; and synchronize the obtained sensor type and generation time to the cloud.

[0113] In this embodiment, the data of the smart device can be recorded and uploaded to the cloud for data analysis and functional training and upgrading of algorithms and models. Please refer to Figure 6, which is a schematic diagram of ensuring data consistency between the smart device and the cloud according to an embodiment of the present application. As shown in Figure 6, when the function of the neural network model is executed on the smart device, due to scheduling or model timeout, even if the cache queue in the trigger can accumulate target detection results, the cache queue still has an upper limit, and the event thread manager cannot record the execution process. There is a possibility that the cache queue is full and some target detection results are lost, which further leads to the situation where the data actually used by the smart device is inconsistent with the data transmitted back to the cloud. To eliminate this problem, the sensor type and generation time corresponding to the target detection pairing result after sliding window sampling can be recorded and synchronized to the cloud. When reproducing the data of the smart device on the cloud, this record can be directly read, thereby ensuring the consistency of the data between the smart device and the cloud. In Figure 6, Lidar OD represents the target detection result of the lidar, BEV OD represents the target detection result of the camera, T1, T2, T3, T4, Tk-1, and Tk represent the generation time, and Pose represents the positioning data of the smart device.

[0114] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of the present application, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present application.

[0115] It will be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment of the present application can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium can include: any entity or device, medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium that can carry the computer program code. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0116] Furthermore, the present application also provides a computer-readable storage medium. In a computer-readable storage medium embodiment according to the present application, the computer-readable storage medium can be configured to store a program for executing the perception fusion result acquisition method of the above-mentioned method embodiment, and the program can be loaded and run by the processor to implement the above-mentioned perception fusion result acquisition method. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-temporary computer-readable storage medium.

[0117] Furthermore, the present application also provides an intelligent device. In an embodiment of an intelligent device according to the present application, the intelligent device may include at least one processor; and a memory communicatively connected to the at least one processor; wherein a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the method described in any of the above embodiments is implemented. The intelligent device of the present application may include driving equipment, smart cars, robots and other devices. Referring to Figure 8, Figure 8 is a schematic diagram of the connection relationship between the memory and the processor of an intelligent device according to an embodiment of the present application. As shown in Figure 8, the memory and the processor of the intelligent device are communicatively connected via a bus.

[0118] In some embodiments of the present application, the smart device further includes at least one sensor for sensing information. The sensor is communicatively coupled to any of the types of processors described herein. Optionally, the smart device further includes an autonomous driving system for guiding the smart device to drive autonomously or with assistance. The processor communicates with the sensor and / or autonomous driving system to perform the method described in any of the above embodiments.

[0119] Furthermore, it should be understood that since the configuration of each module is merely for the purpose of illustrating the functional units of the apparatus of the present application, the physical devices corresponding to these modules may be the processor itself, or a portion of the software in the processor, a portion of the hardware, or a combination of software and hardware. Therefore, the number of modules in the figure is merely illustrative.

[0120] Those skilled in the art will appreciate that the various modules in the device can be adaptively split or merged. Such splitting or merging of specific modules will not cause the technical solution to deviate from the principles of this application. Therefore, the technical solutions after splitting or merging will fall within the scope of protection of this application.

[0121] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, and based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.

[0122] The user personal information processed by this application will vary depending on the specific product / service scenario and will be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. The applicant will treat the user's personal information and its processing with a high degree of diligence.

[0123] This application attaches great importance to the security of user personal information and has taken reasonable and feasible security protection measures that comply with industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

[0124] Thus far, the technical solutions of the present application have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present application.

Claims

1. A method for obtaining perception fusion results, characterized in that: The method is applied to a smart device, wherein a plurality of sensors are provided on the smart device, and the method comprises: For each sensor, pairing a target detection result obtained based on the sensor with positioning data corresponding to the smart device to obtain a target detection pairing result corresponding to the sensor; The target detection pairing results of multiple sensors are stored in a feature queue for storing perception features to obtain the perception fusion result.

2. The method for obtaining perception fusion results according to claim 1, characterized in that: The pairing of the target detection result obtained based on the sensor with the positioning data corresponding to the smart device to obtain the target detection pairing result corresponding to the sensor includes: Putting the target detection result obtained by the sensor into the cache queue corresponding to the sensor; For each target detection result in the cache queue, the target detection result is paired with the positioning data closest to the time when the target detection result is generated to obtain the target detection pairing result.

3. The method for obtaining perception fusion results according to claim 1, characterized in that: Storing the target detection pairing results of the multiple sensors in a feature queue for storing perception fusion results to obtain the perception fusion results includes: When the target detection pairing results of the multiple sensors are updated, storing the target detection pairing results in a feature queue for storing perception features according to a preset event thread manager; The event thread manager asynchronously manages the target detection pairing results of the sensors at different frequencies.

4. A perception fusion processing method, characterized in that: The method is applied to a smart device, wherein a plurality of sensors are provided on the smart device, and the method comprises: For each sensor, pairing a target detection result obtained based on the sensor with positioning data corresponding to the smart device to obtain a target detection pairing result corresponding to the sensor; Storing the target detection pairing results of the plurality of sensors in a feature queue for storing perception features; The target detection pairing results in the feature queue are input into a preset neural network model to obtain the perception target tracking results.

5. The perception fusion processing method according to claim 4, characterized in that: The pairing of the target detection result obtained based on the sensor with the positioning data corresponding to the smart device to obtain the target detection pairing result corresponding to the sensor includes: Putting the target detection result obtained by the sensor into the cache queue corresponding to the sensor; For each target detection result in the cache queue, the target detection result is paired with the positioning data closest to the time when the target detection result is generated to obtain the target detection pairing result.

6. The perception fusion processing method according to claim 4, characterized in that: Storing the target detection pairing results of the multiple sensors in a feature queue for storing perception features includes: When the target detection pairing results of the multiple sensors are updated, storing the target detection pairing results in a feature queue for storing perception features according to a preset event thread manager; The event thread manager asynchronously manages the target detection pairing results of the sensors at different frequencies.

7. The perception fusion processing method according to claim 4 or 6, characterized in that: Storing the target detection pairing result in a feature queue for storing perception features includes: Performing data analysis on the target detection pairing result to obtain a target data analysis result; The target data parsing result is stored in a feature queue for storing perception features.

8. The perception fusion processing method according to claim 7, characterized in that: The step of storing the target data parsing result in a feature queue for storing perception features includes: Performing sliding window sampling on the target data parsing results in the feature queue; The parsing results of the sampled target data are stored in a feature queue for storing perception features.

9. The perception fusion processing method according to claim 8, characterized in that: The performing sliding window sampling on the target data parsing result in the feature queue includes: The detection target parsing result is sampled using a time-based sliding window.

10. The perception fusion processing method according to claim 8, characterized in that: The performing sliding window sampling on the target data parsing result in the feature queue includes: The detection target parsing result is subjected to distance-based sliding window sampling.

11. The perception fusion processing method according to claim 7, characterized in that: The event thread manager prioritizes preset sensors and asynchronously manages target detection pairing results of the sensors of different frequencies.

12. The perception fusion processing method according to claim 7, characterized in that: The event thread manager asynchronously manages the target detection pairing results of the sensors of different frequencies with time priority.

13. The perception fusion processing method according to claim 4, characterized in that: The method further comprises: Obtaining the sensor type and generation time of the target detection pairing result contained in the feature queue; The obtained sensor type and generation time are synchronized to the cloud.

14. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the perception fusion result acquisition method according to any one of claims 1 to 3 or the perception fusion processing method according to any one of claims 4 to 13.

15. A smart device, characterized in that: The smart device includes: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores a computer program, which, when executed by the at least one processor, implements the perception fusion result acquisition method according to any one of claims 1 to 3 or the perception fusion processing method according to any one of claims 4 to 13.

Images