A Multi-Class Sensor Data Fusion Method, Device and Electronic Equipment

Abstract

The embodiments of this specification disclose a multi-class sensor data fusion method, device, and electronic device. The multi-class sensor data fusion method includes performing time registration and space registration on detection data respectively based on a reference time; determining key feature attribute data, preprocessing the key feature attribute data, and correcting noise data in the key feature attribute data; clustering the processed key feature attribute data based on target category attributes; complementing missing data of each target data at each reference time respectively based on an evolutionary strategy; calculating the prediction data based on the Euclidean distance weight method to obtain weight values at each reference time, and calculating the final fusion values of each target object at each reference time respectively based on the weight values. The embodiments of this specification continuously optimize and correct the accuracy of data fusion through the dynamic adaptability and fault tolerance characteristics of the evolutionary strategy, and the accuracy and real-time performance of the final fusion data are relatively good.

Description

Technical Field

[0001] One or more embodiments of this specification relate to sensor data processing technologies, and in particular, to a multi-class sensor data fusion method, apparatus, and electronic device. Background Art

[0002] With the rapid development of science and technology and various sensor technologies, various integrated electronic systems have been widely used in the field of monitoring and protection. Due to the complex and changeable external detection environmental factors in the security area, the detection range and function of a single sensor have limitations and uncertainties, and only feature information of a certain aspect of the identified target can be obtained. Therefore, at present, it is necessary to comprehensively utilize the characteristics of multiple heterogeneous sensors to obtain various feature information of the target object from multiple aspects and angles, so as to improve the coverage of the system data collected in each dimension. However, the collaborative work of multiple sensors is not simply to superimpose and process the information collected by multiple types of sensors. It is necessary to utilize the redundancy and complementarity of the data of multiple sensors to achieve deep fusion of data resources, so as to improve the detection and recognition capabilities of the target object.

[0003] Since the detection of the target object by the sensor is easily affected by the environment and the measurement angle, the multi-sensor data fusion will be interfered by factors such as asynchronous measurement periods, inconsistent data rates, and inconsistent measurement attribute dimensions. Therefore, most current data fusion methods will directly and forcibly process the noise data and missing values generated by the sensors, and cannot adjust the fusion weights of the sensors in real time, resulting in poor accuracy and real-time performance of the final fusion data. Summary of the Invention

[0004] To solve the above problems, one or more embodiments of this specification describe a multi-class sensor data fusion method, apparatus, and electronic device.

[0005] According to a first aspect, a multi-class sensor data fusion method is provided, including:

[0006] Obtain the detection data collected by each sensor within a target time period, and perform time registration and spatial registration on the detection data respectively based on a reference time, where the reference time is determined based on the transmission data frame rate of the sensor;

[0007] Determine the key feature attribute data common to the registered detection data, preprocess the key feature attribute data based on prior rules, and correct the noise data in the key feature attribute data;

[0008] Cluster the processed key feature attribute data based on the target category attribute to obtain the target data corresponding to each target object;

[0009] Complete the missing data of each of the target data at each of the reference times respectively based on an evolutionary strategy to obtain prediction data;

[0010] Calculate the prediction data based on the Euclidean distance weighting method to obtain the weight values of each of the sensors at each of the reference times, and calculate the final fusion values of each of the target objects at each of the reference times respectively based on the weight values.

[0011] Preferably, the time registration and space registration of the detection data respectively based on the reference time include:

[0012] Determine a first sensor with the minimum data transmission frame rate, and determine the reference time based on the data transmission period of the first sensor;

[0013] Respectively determine the adjacent reference times closest to each of the second detection data of a second sensor, and perform piecewise linear interpolation on the time data of each of the second detection data based on the adjacent reference times to obtain time-registered data, where the second sensor is a sensor other than the first sensor;

[0014] Perform space registration on the detection data.

[0015] Preferably, the performing space registration on the detection data includes:

[0016] Take the geocentric inertial coordinate system as the reference coordinate system, convert the geodetic coordinate system data of the sensor into geocentric coordinate system data, and convert the space data of the detection data based on the geocentric coordinate system data to obtain space-registered data.

[0017] Preferably, the preprocessing of the key feature attribute data based on prior rules includes:

[0018] Perform homogeneous conversion on the measurement units of the key feature attribute data;

[0019] Calibrate the time difference of the time calibration systems of each of the sensors;

[0020] Mark the detection data with inconsistent simultaneous position attributes as different target objects;

[0021] Mark the detection data with the same position attribute and inconsistent velocity attribute as different target objects;

[0022] Mark the detection data with the same target ID as the same target object.

[0023] Preferably, the correcting the noise data in the key feature attribute data includes:

[0024] For the first key feature attribute data of any one of the sensors, calculate the first average value and standard deviation of the first key feature attribute data at each of the reference times;

[0025] Mark the first key feature attribute data whose absolute value of the difference from the first average value is greater than three times the standard deviation as noise data, and correct the noise data based on the second average value of the corresponding second key feature attribute data of the remaining sensors at the same time.

[0026] Preferably, the method for respectively completing the missing data of each target data at each reference time based on the evolutionary strategy to obtain predicted data includes:

[0027] Determine the target data to be completed for the missing data;

[0028] For any one of the reference times, randomly generate a first preset number of initial arrays based on the target data to be completed, and the number of elements in the initial array is equal to the number of missing data in the array of the target data to be completed at the next reference time;

[0029] Solve each of the initial arrays respectively based on a preset fitness function, and determine the contemporary individual optimal solution array with the smallest fitness function value;

[0030] Perform positive and negative deviations on the contemporary individual optimal solution array to generate the initial data of the next generation population, and repeat the step of solving each of the initial arrays respectively based on the preset fitness function and determining the contemporary individual optimal solution array with the smallest fitness function value until the iteration end condition is met;

[0031] Use the target array with the smallest global fitness function value as the completion data for the missing data, and repeat the step of randomly generating a first preset number of initial arrays based on the target data to be completed for any one of the reference times until all the missing data at all the reference times of all the target data to be completed are completed.

[0032] Preferably, the method for respectively calculating the final fusion value of each target object at each reference time based on the weight value includes:

[0033] Respectively obtain the non-key feature attribute data that is not completed and the third key feature attribute data whose array elements are less than the total number of sensors after completion of each target object at the reference time, and generate a first calculation array;

[0034] Calculate the third average value of the first calculation array, and calculate the first fusion value of the fourth key feature attribute data based on the weight value, where the fourth key feature attribute data is the key feature attribute data whose array elements are equal to the total number of sensors after completion;

[0035] Determine the final fusion value of each of the target objects at each of the reference times based on the sum of the third average value and the first fusion value.

[0036] According to a second aspect, there is provided a multi-class sensor data fusion device, the device comprising:

[0037] An acquisition module, configured to acquire detection data collected by each sensor within a target time period, and perform time registration and spatial registration on the detection data respectively based on a reference time, where the reference time is determined based on the transmission data frame rate of the sensor;

[0038] A correction module, configured to determine key feature attribute data common to the registered detection data, perform preprocessing on the key feature attribute data based on prior rules, and correct noise data in the key feature attribute data;

[0039] A clustering module, configured to cluster the processed key feature attribute data based on a target category attribute to obtain respective target data corresponding to each target object;

[0040] A completion module, configured to respectively complete missing data of each of the target data at each of the reference times based on an evolutionary strategy to obtain prediction data;

[0041] A calculation module, configured to calculate the prediction data based on the Euclidean distance weighting method to obtain weight values of each of the sensors at each of the reference times, and calculate the final fusion value of each of the target objects at each of the reference times based on the weight values.

[0042] According to a third aspect, there is provided an electronic device, comprising a processor and a memory;

[0043] The processor is connected to the memory;

[0044] The memory is configured to store executable program code;

[0045] The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the steps of the method provided in the first aspect or any one of the possible implementation manners of the first aspect.

[0046] According to a fourth aspect, there is provided a computer-readable storage medium, on which a computer program is stored, and instructions are stored in the computer-readable storage medium, and when the instructions are run on a computer or a processor, the computer or the processor is caused to execute the method provided in the first aspect or any one of the possible implementation manners of the first aspect.

[0047] The method and device provided by the embodiments of this specification can achieve the following: 1. By setting a fine fusion period through time registration, make full use of complementary and redundant data attributes to improve the anti-interference ability of data fusion.

[0048] 2. Continuously optimize and correct the accuracy of data fusion through the dynamic adaptability and fault tolerance characteristics of the evolutionary strategy.

[0049] 3. Flexibly solve the weight of each reference moment through the Euclidean distance weight method to increase the reliability of data fusion, and the accuracy and real-time performance of the final fused data are relatively good. Description of the Drawings

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0051] Figure 1 It is a flowchart of a multi-class sensor data fusion method in an embodiment of this specification.

[0052] Figure 2 It is a schematic structural diagram of a multi-class sensor data fusion device in an embodiment of this specification.

[0053] Figure 3 It is a schematic structural diagram of an electronic device in an embodiment of this specification. Detailed Embodiments

[0054] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0055] In the following description, the terms "first" and "second" are only for the purpose of description and cannot be construed as indicating or implying relative importance. The following description provides multiple embodiments of the present application. Different embodiments can be replaced or combined. Therefore, the present application can also be considered to include all possible combinations of the same and / or different embodiments described. Thus, if one embodiment includes features A, B, and C, and another embodiment includes features B and D, then the present application should also be considered to include embodiments containing all other possible combinations of A, B, C, and D, although such embodiments may not be explicitly described in the following content.

[0056] The following description provides examples and does not limit the scope, applicability, or examples set forth in the claims. Changes may be made to the function and arrangement of the elements described without departing from the scope of the content of this application. Various processes or components may be appropriately omitted, substituted, or added to each example. For example, the methods described may be performed in a different order than the order described, and various steps may be added, omitted, or combined. In addition, the features described for some examples may be combined into other examples.

[0057] See Figure 1 , Figure 1 is a schematic flowchart of a multi-class sensor data fusion method provided by an embodiment of this application. In the embodiment of this application, the method includes:

[0058] S101. Obtain the detection data collected by each sensor during the target time period, and perform time registration and spatial registration on the detection data respectively based on the reference time, where the reference time is determined based on the data transmission frame rate of the sensor.

[0059] The execution subject of this application may be a cloud server.

[0060] In the embodiment of this specification, to improve data processing efficiency, the cloud server will split the time into multiple target time periods in chronological order and only process the data within a certain target time period each time. The cloud server will obtain the detection data collected by a set of multiple sensors {S1, S2,..., S m} during the target time period, and perform time registration and spatial registration on the detection data respectively according to the reference time. Among them, the reference time will be determined according to the data transmission frame rate of the sensor. For example, the data acquisition time corresponding to the sensor with the minimum data transmission frame rate may be selected as the reference time, and the data of other sensors will be registered to the reference time. Specifically, time registration can be performed by means of piecewise linear interpolation, and spatial registration can be performed by means of coordinate system transformation and alignment. The methods of time registration and spatial registration may also be any other methods that can achieve registration.

[0061] Among them, the sensors may include LD, optoelectronic, electronic fence, vibration optical fiber, etc. Of course, they may also be other Internet of Things sensors, and the embodiments of this specification do not make specific limitations.

[0062] The data information collected by LD includes: target ID, timestamp, target category, target longitude, target latitude, target distance, target speed, movement direction, movement state, structural features, surface material, device type.

[0063] The data information collected by optoelectronic means includes: target ID, timestamp, target category, target longitude, target latitude, target distance, target speed, direction of movement, movement state, structural features, target color, device type.

[0064] The data information of the target collected by the electronic fence includes: target ID, timestamp, target distance, defense zone number, device type.

[0065] The data information of the target collected by the vibration optical fiber includes: target ID, timestamp, target distance, target category, longitude, latitude, defense zone number, device type.

[0066] In an implementable manner, the time registration and spatial registration of the detection data based on the reference time includes:

[0067] Determine the first sensor with the minimum data transmission frame rate, and determine the reference time based on the data transmission period of the first sensor;

[0068] Respectively determine the adjacent reference times closest to each second detection data of the second sensor, and perform piecewise linear interpolation on the time data of each second detection data based on the adjacent reference times to obtain time-registered data. The second sensor is a sensor other than the first sensor;

[0069] Perform spatial registration on the detection data.

[0070] In the embodiments of this specification, the cloud server will determine the data transmission frame rates of each sensor, and then determine the first sensor with the minimum data transmission frame rate. Among the data transmission periods of the first sensor, the moment of each data transmission is the reference time. Obviously, since data is transmitted in each period, the reference time is not unique. Therefore, the cloud server will determine which adjacent reference time is closest to each detection data corresponding to other second sensors, and perform piecewise linear interpolation on this detection data based on the adjacent reference time to achieve time registration of this detection data. Among them, the piecewise linear interpolation method approximates the value of the unknown function by connecting the line segments between adjacent data points. For two given data points (x0, y0) and (x1, y1), where x0 < x1, this method uses the linear function y = mx + c to approximate the function value between these two data points. Among them, m is the slope and c is the intercept.

[0071] Exemplarily, taking the period T of the LD device with the minimum data transmission frame rate as the reference time, taking the optoelectronic sensor as an example, assuming the period of the optoelectronic observation data is T2, where T < T2, X t1 , X t2They are the measured values of a certain target attribute of the optoelectronic device at times t1 and t2 = t1 + Δt respectively, and the fusion time is t, where t1 < t < t2. Then, according to piecewise linear interpolation, the measured value of the optoelectronic device regarding this target attribute at time t can be obtained as follows:

[0072]

[0073] In an implementable manner, the spatial registration of the detection data includes:

[0074] Taking the geocentric inertial coordinate system as the reference coordinate system, converting the geodetic coordinate system data of the sensor into geocentric coordinate system data, and converting the spatial data of the detection data based on the geocentric coordinate system data to obtain spatially registered data.

[0075] In the embodiments of this specification, the geocentric inertial coordinate system can be selected as the reference coordinate system for the spatial registration process. Taking LD as an example, according to the longitude, latitude, and altitude of LD in the geodetic coordinate system, they are converted into the geocentric coordinate system data of LD. The conversion formulas are as follows:

[0076]

[0077] X = (N + H) * cosB * cosL

[0078] Y = (N + H) * cosB * cosL

[0079] Z = [N * (1 - e 2 ) + H] * sinB

[0080] Among them, N represents the normal length, L represents the longitude, B represents the latitude, H represents the altitude, and a represents the length of the major axis of the earth.

[0081] Converting the spatial position of the target object collected by the sensor in the local geodetic coordinate system of LD into the position of the target object in the geocentric coordinate system. The formula is as follows:

[0082]

[0083] Among them, X T is the X-axis position of the target object in the geocentric coordinate system, X R is the position of LD in the geocentric coordinate system, and x T is the position of the target object in the local geodetic coordinate system of LD.

[0084] S102. Determine the common key feature attribute data of the registered detection data, preprocess the key feature attribute data based on prior rules, and correct the noise data in the key feature attribute data.

[0085] In the embodiments of this specification, after the cloud server registers each detection data, the cloud server extracts the key feature attribute data common to each detection data from the detection data. Taking the aforementioned several types of sensors as an example, the determined key feature attribute data extracted are the target distance, longitude, latitude, and speed. Then, the cloud server preprocesses the key feature attribute data according to a priori rules. The preprocessing process of the a priori rules refers to the steps of processing and preparing the data before applying the a priori rules. This process aims to make the data meet the requirements of the a priori rules so as to better apply the rules and obtain accurate results. The preprocessing process can include data cleaning, data transformation, dataset partitioning, data balancing, etc. There will also be noise data in the key feature attribute data, so the cloud server will also correct the noise data. The correction methods for the noise data can use methods such as smoothing, interpolation, and frequency domain filtering.

[0086] In an implementable manner, the preprocessing of the key feature attribute data based on the a priori rules includes:

[0087] Perform homogeneous conversion on the measurement units of the key feature attribute data;

[0088] Calibrate the time difference of the time calibration systems of each of the sensors;

[0089] Mark the detection data with inconsistent simultaneous position attributes as different target objects;

[0090] Mark the detection data with the same position attributes but inconsistent speed attributes as different target objects;

[0091] Mark the detection data with the same target ID as the same target object.

[0092] In the embodiments of this specification, the preprocessing may include: uniformly converting the distance unit, longitude and latitude unit, and speed unit of the key feature attribute data to maintain the same order of magnitude, calibrating the time difference of the time calibration systems of each sensor, marking the detection data with significantly inconsistent simultaneous position attributes as different targets, marking the detection data with the same position attributes but inconsistent speed attributes as different targets, and marking the detection data with the same target ID as the same target.

[0093] In an implementable manner, the correction of the noise data in the key feature attribute data includes:

[0094] For the first key feature attribute data of any one of the sensors, calculate the first average value and standard deviation of the first key feature attribute data at each of the reference times;

[0095] Mark the first key feature attribute data whose absolute value of the difference from the first average value is greater than three times the standard deviation as noise data, and correct the noise data based on the second average value of each second key feature attribute data corresponding to the remaining sensors at the same moment.

[0096] In the embodiments of this specification, for the data {X1, X2,..., X5} of the first key feature attribute data at each reference moment, the cloud server will calculate the first average value and the standard deviation σ. If the first average value and the standard deviation calculated from the first key feature attribute data are satisfied, the cloud server considers it to be noise data that needs to be corrected and marks it. The correction method is to obtain the second key feature attribute data of the same type corresponding to other sensors, and calculate the second average value of each second key feature attribute data corresponding to each sensor at the same moment. This second average value will be used as the correction value of the first key feature attribute data at this moment.

[0097] S103. Cluster the processed key feature attribute data based on the target category attribute to obtain the target data corresponding to each target object.

[0098] In the embodiments of this specification, each sensor may collect data of multiple target objects. To facilitate subsequent accurate filling of missing data and calculation of weight values, the cloud server needs to distinguish each target object according to the target category attribute to calculate the data of each target object respectively. Specifically, the cloud server will perform clustering analysis on the key feature attribute data through the Kmeans clustering algorithm to obtain the target data {G1, G2,..., G k}. Among them, the target category attribute can be the marked attribute during preprocessing, the target ID, the location attribute, etc.

[0099] S104. Complement the missing data of each target data at each reference moment respectively based on the evolutionary strategy to obtain predicted data.

[0100] In the embodiments of this specification, after the detection data undergoes time registration and space registration to improve the anti-interference ability of the data and correct the noise data in the data, the cloud server can use the dynamic adaptability and fault tolerance characteristics of the evolutionary strategy to complement and correct the missing data of each classified target data to obtain complete predicted data that can be used for data fusion. Through the evolutionary strategy, finally, the missing data existing in each target data at each reference moment can be complemented respectively.

[0101] The specific process of the evolutionary strategy can be divided into the following steps:

[0102] Initialize the population: First, it is necessary to initialize a population containing multiple individuals. Each individual represents a solution, which can be a set of parameters or decision variables.

[0103] Evaluate fitness: For each individual, it is necessary to evaluate the quality of its solution through a fitness function. The fitness function is defined according to the specific requirements of the problem, and the goal is to distinguish individuals with higher fitness from those with lower fitness.

[0104] Selection: Based on the fitness values of the individuals, perform a selection operation to select individuals with higher fitness. The selection operation usually uses a probabilistic selection method, that is, the probability of selecting an individual with higher fitness is greater.

[0105] Update the population: According to the selected new individuals, update the original population to form a new generation of population.

[0106] Termination condition: Repeat the above steps until the termination condition is reached, such as reaching the maximum number of iterations or finding a solution that meets the requirements.

[0107] In one implementable manner, the method for respectively completing the missing data of each of the target data at each of the reference times based on the evolutionary strategy to obtain prediction data includes:

[0108] Determine the target data to be completed for the missing data;

[0109] For any one of the reference times, randomly generate a first preset number of initial arrays based on the target data to be completed, and the number of elements in the initial array is equal to the number of missing data in the array of the target data to be completed at the next reference time;

[0110] Solve each of the initial arrays respectively based on a preset fitness function, and determine the optimal solution array of the current generation individual with the smallest fitness function value;

[0111] Perform positive and negative deviations on the optimal solution array of the current generation individual to generate the initial data of the next generation population, and repeat the step of solving each of the initial arrays respectively based on the preset fitness function to determine the optimal solution array of the current generation individual with the smallest fitness function value until the iteration end condition is met;

[0112] Use the target array with the smallest global fitness function value as the completion data for the missing data, and repeat the step of randomly generating a first preset number of initial arrays based on the target data to be completed for any one of the reference times until all the missing data at all the reference times of all the target data to be completed are completed.

[0113] In the embodiments of this specification, the cloud server will first determine the target data to be completed. For the target data at each reference time t, the cloud server will complete the missing data at the next reference time t+1.

[0114] Among them, the determination method of the target data to be completed is that the number of elements z in the array corresponding to the target data to be completed satisfies In this embodiment, m can be the total number of sensors. In addition, if the array of a certain target data meets the completion conditions at the initial reference time t0, the cloud server will first generate 50 arrays by randomly deviating 5% positively and negatively based on the average value of the target data at this time. The number of elements in the random array is the number of missing data. Then, the 50 random array elements are filled into the array of the target data to calculate its standard deviation. Finally, the array elements corresponding to the smallest standard deviation are filled into the array of the target data at the initial reference time t0.

[0115] Exemplarily, the completion process based on the evolutionary strategy can be specifically as follows:

[0116] 1) Initialize the evolutionary population: Randomly generate 50 initial arrays, the data range is the upper and lower limits of the target data at time t, that is, [Xmin, Xmax], where the random positive and negative deviation range is 5%. And set the number of elements in each initial array to be equal to the number of missing data of this key feature attribute data at the next reference time t+1. The number of evolutionary generations is 1000, and the evolutionary convergence accuracy is 10 -4 。

[0117] 2) Set the population evolution fitness function f(x) min =|σ t+1 -σ t |, respectively fill the calculated fitness function values into the initial arrays, and solve the fitness function f(x) min of each initial array. The initial array corresponding to the minimum value of the fitness function is the optimal solution array of the gth generation individual. Among them, σ t is the standard deviation of the target data of this target at time t.

[0118] 3) Randomly generate the population data of the (g+1)th generation by positive and negative deviation according to the optimal solution array solved by the gth generation individual, where the positive and negative deviation range is 5%.

[0119] 4) Repeat steps 2) and 3) until the number of iterations reaches 1000 times or the minimum value of the population fitness function is less than the convergence accuracy 10 -4 。

[0120] 5) Select the array corresponding to the minimum value of the fitness function in the final population after the iteration ends as the optimal prediction data of this target data, and use this set of data to complete the missing data.

[0121] 6) Repeat steps 1) to 5) for the target data corresponding to other key feature attributes of the target object to complete the missing data in the target data corresponding to all key feature attributes.

[0122] 7) Execute steps 1) to 6) for the target data corresponding to the key feature attributes of other target objects to complete the missing data of the k target objects {G1, G2, …, G k} at the reference time t + 1.

[0123] 8) Repeat steps 1) to 7) cyclically to complete the data corresponding to all reference times of the k target objects {G1, G2, …, G k}.

[0124] S105. Calculate the predicted data based on the Euclidean distance weighting method to obtain the weight values of each sensor at each reference time, and calculate the final fusion values of each target object at each reference time based on the weight values.

[0125] In the embodiments of the present specification, according to the predicted data obtained in the foregoing steps, the cloud server will calculate each predicted data according to the Euclidean distance weighting method to obtain the weight values of each sensor at each reference time. According to the calculated weight values, the detection data collected by different sensors at each reference time can be weighted and calculated, and finally the final fusion values of each target object at each reference time are obtained, completing the data fusion process of multiple types of sensors. Among them, the calculation formula of the Euclidean distance weighting method is as follows:

[0126]

[0127] where n is 5, is the average value of the predicted data at a certain reference time, and X i is the predicted data corresponding to a certain sensor at a certain reference time.

[0128] In an implementable manner, the calculating the final fusion values of each target object at each reference time based on the weight values includes:

[0129] Respectively obtain the non-key feature attribute data that is not completed and the third key feature attribute data with the number of array elements less than the total number of sensors after completion of each target object at the reference time to generate a first calculation array;

[0130] Calculate the third average value of the first calculation array, and calculate the first fusion value of the fourth key feature attribute data based on the weight values. The fourth key feature attribute data is the key feature attribute data with the number of array elements equal to the total number of sensors after completion;

[0131] Determine the final fusion value of each target object at each reference time based on the sum of the third average value and the first fusion value.

[0132] In the embodiments of this specification, the calculation process of the weight value is based on prediction data, and the prediction data is determined according to the key feature attribute data in the detection data. The data fusion process should consider the complete detection data, rather than only considering the key feature attribute data. Therefore, the cloud server will obtain the first calculation array and calculate the third average value Q`` of the data of each attribute type in these obtained first calculation arrays. In addition, according to the weight value, the cloud server can also calculate the first fusion value O` of the fourth key feature attribute data by weighting. Then, the cloud server can calculate the final fusion value of the target object at the reference time t. Where Q1 = Q`` + Q`. By repeating the above process, the final fusion value at each reference time can be obtained.

[0133] Next, the multi-class sensor data fusion device provided in the embodiments of this application will be introduced in detail in combination with the attached Figure 2 , and it should be noted that the attached Figure 2 The multi-class sensor data fusion device shown is used to execute the method of the embodiments of this application Figure 1 As shown, for the sake of simplicity, only the parts related to the embodiments of this application are shown. For the specific technical details not disclosed, please refer to the embodiments of this application Figure 1 As shown.

[0134] Please refer to Figure 2 , Figure 2 FIG. Figure 2 As shown, the device includes:

[0135] An acquisition module 201, configured to acquire detection data collected by each sensor within a target time period, and perform time registration and space registration on the detection data respectively based on a reference time, where the reference time is determined based on the data transmission frame rate of the sensor;

[0136] A correction module 202, configured to determine the key feature attribute data common to the registered detection data, preprocess the key feature attribute data based on prior rules, and correct the noise data in the key feature attribute data;

[0137] A clustering module 203, configured to cluster the processed key feature attribute data based on a target category attribute to obtain respective target data corresponding to each target object;

[0138] The completion module 204 is configured to complete the missing data of each of the target data at each of the reference times respectively based on an evolutionary strategy to obtain predicted data;

[0139] The calculation module 205 is configured to calculate the predicted data based on the Euclidean distance weighting method to obtain the weight values of each of the sensors at each of the reference times, and calculate the final fusion values of each of the target objects at each of the reference times respectively based on the weight values.

[0140] In an implementable manner, the acquisition module 201 is specifically configured to:

[0141] Determine a first sensor with the minimum data transmission frame rate, and determine the reference times based on the data transmission period of the first sensor;

[0142] Respectively determine the adjacent reference times closest to each of the second detection data of a second sensor, and perform piecewise linear interpolation on the time data of each of the second detection data based on the adjacent reference times to obtain time registration data, where the second sensor is a sensor other than the first sensor;

[0143] Perform spatial registration on the detection data.

[0144] In an implementable manner, the acquisition module 201 is further specifically configured to:

[0145] Take the geocentric inertial coordinate system as the reference coordinate system, convert the geodetic coordinate system data of the sensor into geocentric coordinate system data, and convert the spatial data of the detection data based on the geocentric coordinate system data to obtain spatial registration data.

[0146] In an implementable manner, the correction module 202 is specifically configured to:

[0147] Perform homogeneous conversion on the measurement units of the key feature attribute data;

[0148] Calibrate the time difference of the time calibration systems of the sensors;

[0149] Mark the detection data with inconsistent simultaneous position attributes as different target objects;

[0150] Mark the detection data with the same position attribute and inconsistent speed attributes as different target objects;

[0151] Mark the detection data with the same target ID as the same target object.

[0152] In an implementable manner, the correction module 202 is further specifically configured to:

[0153] For the first key feature attribute data of any one of the sensors, calculate the first average value and standard deviation of the first key feature attribute data at each of the reference times;

[0154] Mark the first key feature attribute data whose absolute value of the difference from the first average value is greater than three times the standard deviation as noise data, and correct the noise data based on the second average value of the corresponding second key feature attribute data of the remaining sensors at the same time.

[0155] In an implementable manner, the complementing module 204 is specifically configured to:

[0156] Determine the target data to be complemented for the missing data;

[0157] For any one of the reference times, randomly generate a first preset number of initial arrays based on the target data to be complemented, and the number of elements in the initial array is equal to the number of missing elements in the array of the target data to be complemented at the next reference time;

[0158] Solve each of the initial arrays respectively based on a preset fitness function, and determine the contemporary individual optimal solution array with the smallest fitness function value;

[0159] Perform positive and negative deviations on the contemporary individual optimal solution array to generate the initial data of the next generation population, and repeat the step of solving each of the initial arrays respectively based on the preset fitness function and determining the contemporary individual optimal solution array with the smallest fitness function value until the iteration end condition is met;

[0160] Use the target array with the smallest global fitness function value as the complemented data for the missing data, and repeat the step of randomly generating a first preset number of initial arrays based on the target data to be complemented for any one of the reference times until all the missing data at all the reference times of all the target data to be complemented are complemented.

[0161] In an implementable manner, the calculation module 205 is specifically configured to:

[0162] Respectively obtain the non-critical feature attribute data that is not complemented and the third key feature attribute data whose array elements after complementation are less than the total number of sensors of each target object at the reference time, and generate a first calculation array;

[0163] Calculate the third average value of the first calculation array, and calculate the first fusion value of the fourth key feature attribute data based on the weight value, where the fourth key feature attribute data is the key feature attribute data whose array elements after complementation are equal to the total number of sensors;

[0164] Determine the final fusion value of each of the target objects at each of the reference times based on the sum of the third average value and the first fusion value.

[0165] Those skilled in the art can clearly understand that the technical solutions of the embodiments of the present application can be implemented by means of software and / or hardware. The "units" and "modules" in this specification refer to software and / or hardware that can independently complete or cooperate with other components to complete specific functions, where the hardware can be, for example, a Field-Programmable Gate Array (FPGA), an Integrated Circuit (IC), etc.

[0166] Each processing unit and / or module of the embodiments of the present application can be implemented by an analog circuit that implements the functions described in the embodiments of the present application, or can be implemented by software that executes the functions described in the embodiments of the present application.

[0167] See Figure 3 , which shows a schematic structural diagram of an electronic device involved in the embodiments of the present application. This electronic device can be used to implement Figure 1 the method in the illustrated embodiments. As Figure 3 shown, the electronic device 300 may include: at least one central processing unit 301, at least one network interface 304, a user interface 303, a memory 305, and at least one communication bus 302.

[0168] Among them, the communication bus 302 is used to implement connection communication between these components.

[0169] Among them, the user interface 303 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 303 may further include a standard wired interface and a wireless interface.

[0170] Among them, the network interface 304 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0171] Among them, the central processing unit 301 may include one or more processing cores. The central processing unit 301 uses various interfaces and lines to connect various parts within the entire electronic device 300, and executes various functions of the terminal 300 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 305, and by invoking the data stored in the memory 305. Optionally, the central processing unit 301 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The central processing unit 301 may integrate one or a combination of several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, application programs, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communication. It can be understood that the above modem may not be integrated into the central processing unit 301 and may be implemented separately by a single chip.

[0172] Among them, the memory 305 may include random access memory (RAM) and may also include read-only memory. Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, code, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing the operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above method embodiments, etc.; the data storage area may store the data involved in the above method embodiments. Optionally, the memory 305 may also be at least one storage device located far from the aforementioned central processing unit 301. As Figure 3 shown, the memory 305, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and program instructions.

[0173] In Figure 3In the electronic device 300 shown, the user interface 303 is mainly used to provide an interface for the user to input and obtain the data input by the user. The central processing unit 301 can be used to call various sensor data fusion application programs stored in the memory 305 and specifically perform the following operations:

[0174] Obtain the detection data collected by each sensor during the target time period, and respectively perform time registration and space registration on the detection data based on the reference time, where the reference time is determined based on the transmission data frame rate of the sensor;

[0175] Determine the key feature attribute data common to the registered detection data, preprocess the key feature attribute data based on prior rules, and correct the noise data in the key feature attribute data;

[0176] Cluster the processed key feature attribute data based on the target category attribute to obtain the target data corresponding to each target object;

[0177] Complement the missing data of each target data at each reference time based on the evolutionary strategy to obtain prediction data;

[0178] Calculate the prediction data based on the Euclidean distance weight method to obtain the weight values of each sensor at each reference time, and calculate the final fusion values of each target object at each reference time based on the weight values.

[0179] This application also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the above method are implemented. Among them, the computer-readable storage medium can include, but is not limited to, any type of disk, including floppy disks, optical discs, DVDs, CD-ROMs, microdrives, and magneto-optical discs, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0180] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0181] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0182] In several embodiments provided by the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some service interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical or other form.

[0183] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0184] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0185] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. And the aforementioned memory includes: USB flash drives, read-only memory (ROM), random access memory (RAM), mobile hard disks, magnetic disks, or optical disks, etc., which can store program codes.

[0186] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable memory. The memory can include: flash drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, etc.

[0187] The foregoing are only exemplary embodiments of the present disclosure, and the scope of the present disclosure cannot be limited thereby. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure still fall within the scope covered by the present disclosure. Those skilled in the art will readily conceive of other implementations of the present disclosure after considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common general knowledge or conventional technical means in the technical field not recorded in the present disclosure. The specification and examples are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A multi-class sensor data fusion method, characterized in that The method includes: Obtaining the detection data collected by each sensor during the target time period, determining the first sensor with the minimum data transmission frame rate, and determining the reference time based on the data transmission period of the first sensor; respectively determining the adjacent reference times closest to each second detection data of the second sensor, and performing piecewise linear interpolation on the time data of each second detection data based on the adjacent reference times to obtain time registration data, where the second sensor is a sensor other than the first sensor; performing spatial registration on the detection data, and the reference time is determined based on the data transmission frame rate of the sensor; Determining the key feature attribute data common to the registered detection data, preprocessing the key feature attribute data based on prior rules, and correcting the noise data in the key feature attribute data; Clustering the processed key feature attribute data based on the target category attribute to obtain the target data corresponding to each target object; Completing the missing data of each target data at each reference time based on the evolutionary strategy to obtain prediction data; Calculating the prediction data based on the Euclidean distance weight method to obtain the weight values of each sensor at each reference time, and calculating the final fusion values of each target object at each reference time based on the weight values.

2. The method according to claim 1, characterized in that, The performing spatial registration on the detection data includes: Using the geocentric inertial coordinate system as the reference coordinate system, converting the geodetic coordinate system data of the sensor into geocentric coordinate system data, and converting the spatial data of the detection data based on the geocentric coordinate system data to obtain spatially registered data.

3. The method according to claim 1, wherein The preprocessing the key feature attribute data based on prior rules includes: Performing homogeneous conversion on the measurement units of the key feature attribute data; Calibrating the time difference of the time calibration systems of each sensor; Marking the detection data with inconsistent simultaneous position attributes as different target objects; Marking the detection data with the same position attribute but inconsistent velocity attributes as different target objects; Marking the detection data with the same target ID as the same target object.

4. The method according to claim 1, wherein The correcting the noise data in the key feature attribute data includes: For the first key feature attribute data of any sensor, calculating the first average value and standard deviation of the first key feature attribute data at each reference time; Marking the first key feature attribute data whose absolute value of the difference from the first average value is greater than three times the standard deviation as noise data, and correcting the noise data based on the second average value of the corresponding second key feature attribute data of the other sensors at the same time.

5. The method according to claim 1, wherein The completing the missing data of each target data at each reference time based on the evolutionary strategy to obtain prediction data includes: Determining the target data to be completed for the missing data; For any reference time, randomly generating a first preset number of initial arrays based on the target data to be completed, where the number of elements in the initial array is equal to the number of missing elements in the array of the target data to be completed at the next reference time; Solve each of the initial arrays respectively based on a preset fitness function, and determine the contemporary individual optimal solution array with the smallest fitness function value; Perform positive and negative deviations on the contemporary individual optimal solution array to generate the initial arrays of the next generation population, and repeat the step of solving each of the initial arrays respectively based on a preset fitness function and determining the contemporary individual optimal solution array with the smallest fitness function value until the iteration end condition is satisfied; Use the target array with the smallest global fitness function value as the complement data for the missing data, and repeat the step of randomly generating the first preset number of initial arrays based on the target data to be complemented for any of the reference times until the missing data at all the reference times of all the target data to be complemented are all complemented.

6. The method according to claim 1, wherein The calculating the final fusion value of each of the target objects at each of the reference times based on the weight values includes: Respectively obtain the uncomplemented non-critical feature attribute data of each of the target objects at the reference time and the third critical feature attribute data with the array element less than the total number of sensors after complementation, and generate a first calculation array; Calculate the third average value of the first calculation array, and calculate the first fusion value of the fourth critical feature attribute data based on the weight values, where the fourth critical feature attribute data is the critical feature attribute data with the array element equal to the total number of sensors after complementation; Determine the final fusion value of each of the target objects at each of the reference times based on the sum of the third average value and the first fusion value.

7. A multi-class sensor data fusion device, comprising: An acquisition module, configured to acquire the detection data collected by each sensor within a target time period, determine the first sensor with the smallest data sending frame rate, and determine the reference time based on the data sending period of the first sensor; Respectively determine the adjacent reference times closest to each second detection data of the second sensor, and perform piecewise linear interpolation on the time data of each second detection data based on the adjacent reference times to obtain time registration data, where the second sensor is a sensor other than the first sensor; Perform spatial registration on the detection data, and the reference time is determined based on the data sending frame rate of the sensor; A correction module, configured to determine the common critical feature attribute data of the registered detection data, perform preprocessing on the critical feature attribute data based on prior rules, and correct the noise data in the critical feature attribute data; A clustering module, configured to cluster the processed critical feature attribute data based on the target category attribute to obtain the target data corresponding to each target object; A complementation module, configured to complement the missing data of each target data at each reference time respectively based on an evolutionary strategy to obtain prediction data; A calculation module, configured to calculate the prediction data based on the Euclidean distance weight method to obtain the weight values of each sensor at each reference time, and calculate the final fusion value of each target object at each reference time based on the weight values.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, wherein the computer-readable storage medium stores instructions that, when executed on a computer or a processor, cause the computer or the processor to perform the steps of the method according to any one of claims 1-6.

Images