Target matching method and device for multi-sensing data, vehicle and medium

Abstract

The invention relates to a target matching method and device for multi-sensing data, a vehicle and a medium. The method comprises the steps that multiple pieces of sensing data are acquired; the multiple pieces of sensor data are input into a deep learning model, the deep learning model outputs the matching weight of at least one object group, the at least one object group is associated with the multiple pieces of sensor data, and each object group comprises two to-be-matched objects detected through any two pieces of sensor data in the multiple pieces of sensor data; determining a relation value of at least one object group through a strategy based on logic matching; according to the matching weight and the relation value of the at least one object group, target matching is carried out on the to-be-matched objects detected through the multiple pieces of sensor data, and a matching result is obtained; according to the method, the advantage that deep learning can continuously learn complex working conditions is combined, the advantage that a strategy based on logic matching provides data support for the deep learning method is also combined, and the accuracy of target matching can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of target matching technology, and in particular, to a method, device, vehicle and medium for target matching of multi-sensor data. Background Art

[0002] Target matching of multiple sensor data (determining whether they are the same target) is used to determine whether the targets detected by sensor data from different sensors correspond to the same physical entity.

[0003] At present, the target matching of multi-sensor data is usually carried out by calculating the weights based on the strategy of logical matching (for example, Hungarian algorithm), that is, the weights of the targets output by different sensors are calculated first, the corresponding relationship between different sensors is quantified, and then the matching algorithm is used for matching. However, when this method faces complex working conditions, there will be problems of strategy mutual exclusion or logical loopholes, which will lead to a decrease in accuracy. Summary of the invention

[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0005] The present application aims to at least solve the technical problems existing in the prior art. To this end, the present application proposes a method, device, vehicle and medium for target matching of multi-sensor data, which can improve the accuracy of matching.

[0006] A first aspect of an embodiment of the present application provides a method for target matching of multi-sensor data, the method comprising:

[0007] Acquire multiple sensor data, where different sensor data are collected by different types of sensors;

[0008] Inputting the plurality of sensor data into a deep learning model, and outputting a matching weight of at least one object group by the deep learning model, wherein the at least one object group is associated with the plurality of sensor data, each of the object groups includes two objects to be matched that are respectively detected by any two of the plurality of sensor data, at least one object to be matched between different object groups is different, and each matching weight is used to characterize the degree of similarity between two objects to be matched in the corresponding object group;

[0009] Determine the relationship value of the at least one object group through a strategy based on logic matching, each of the relationship values ​​being used to measure the association relationship between the two to-be-matched objects in the corresponding object group in terms of physical attributes;

[0010] According to the matching weight and the relationship value of the at least one object group, target matching is performed on the objects to be matched that are respectively detected by the multiple sensor data to obtain a matching result, wherein the matching result is used to indicate a target object group in the at least one object group, and the target object group is an object group that matches two of the objects to be matched with each other.

[0011] The embodiment of the present application provides a target matching method for multi-sensor data. In view of the problem that strategies based on logic matching have mutual exclusion or logic loopholes when facing complex working conditions, the method uses a deep learning method to extract the matching weight between two to-be-matched objects in an object group, and uses a strategy based on logic matching to extract the relationship value between two to-be-matched objects in an object group. The matching weight and the relationship value are used to jointly perform target matching on multiple to-be-matched objects detected respectively through multiple sensor data. This combines the advantage of deep learning that can continuously learn complex working conditions with the advantage of the strategy based on logic matching that provides data support for the deep learning method, and can improve the accuracy of target matching.

[0012] In some embodiments of the present application, performing target matching on the objects to be matched respectively detected by the plurality of sensor data according to the matching weight and the relationship value of the at least one object group to obtain a matching result includes:

[0013] Obtaining a fusion weight corresponding to the object group according to the matching weight and the relationship value of each object group;

[0014] Based on the fusion weight of the at least one object group, a target object group in which two of the to-be-matched objects in the at least one object group match each other is determined to obtain a first matching result.

[0015] In some embodiments of the present application, the relationship value is a numerical deviation, and the numerical deviation is used to measure the difference in physical properties between two matching objects in the object group to which it corresponds;

[0016] The obtaining, according to the matching weight and the relationship value of each object group, a fusion weight corresponding to the object group includes:

[0017] Acquire a first weight corresponding to the matching weight of each of the object groups and a second weight corresponding to the numerical deviation of each of the object groups;

[0018] The sum of the first weight product and the second weight product is determined as the fusion weight corresponding to the object group, the first weight product is the product of the matching weight and the first weight, the second weight product is the product of the second weight and a target ratio, the numerator of the target ratio is 1, the denominator of the target ratio includes the numerical deviation, and the sum of the second weight and the first weight is 1.

[0019] In some embodiments of the present application, the numerical deviation includes N sub-deviations, the physical attribute includes N sub-physical attributes corresponding to the N sub-deviations one by one, each of the sub-deviations is used to measure the difference between the two matching objects in the object group in the corresponding sub-physical attribute, and N is a positive integer greater than 1;

[0020] Before determining the sum of the first weight product and the second weight product as the fusion weight corresponding to the object group, the method further includes:

[0021] Obtaining a sub-weight of each of the N sub-physical attributes;

[0022] Performing weight fusion on the sub-weight of each of the N sub-physical attributes and the sub-deviation corresponding to the sub-physical attribute to obtain a fused sub-weight;

[0023] Divide 1 by the fusion sub-weight to obtain the target ratio.

[0024] In some embodiments of the present application, the N sub-physical attributes include at least two of distance, speed, area, length, angle, and identity.

[0025] In some embodiments of the present application, the deep learning model includes a weight extraction network and a target matching network, and the target matching network is used to determine the second matching result according to the matching weight;

[0026] The step of inputting the plurality of sensor data into a deep learning model, and outputting a matching weight of at least one object group by the deep learning model, comprises:

[0027] Inputting the plurality of sensor data into a deep learning model to obtain a matching weight of at least one object group output by the weight extraction network;

[0028] After performing target matching on the objects to be matched respectively detected by the plurality of sensor data according to the matching weight and the numerical relationship of the at least one object group and obtaining a matching result, the method further includes:

[0029] generating a training sample based on the plurality of sensor data and the matching result;

[0030] The deep learning model is trained based on the training samples to optimize parameters of the weight extraction network and the target matching network.

[0031] In some embodiments of the present application, after training the deep learning model based on the training samples to optimize the parameters of the weight extraction network and the target matching network, the method further includes:

[0032] When a plurality of sensor data collected for the N+1th time is obtained, if a matching condition is satisfied between the Nth first matching result and the Nth second matching result, the plurality of sensor data collected for the N+1th time is input into the deep learning model to obtain the N+1th second matching result output by the target matching network, wherein the Nth first matching result is a first result determined based on the plurality of sensor data collected for the N+1th time, and the N second matching results are second matching results output by the target matching network based on the plurality of sensor data collected for the Nth time;

[0033] The N+1 second matching results are determined as the N+1 first matching results.

[0034] A second aspect of an embodiment of the present application provides a target matching device for multi-sensor data, the device comprising:

[0035] A data acquisition unit, used to acquire a plurality of sensor data, wherein the different sensor data are acquired by different types of sensors;

[0036] A first matching unit, configured to input the plurality of sensor data into a deep learning model, and the deep learning model outputs a matching weight of at least one object group, wherein the at least one object group is associated with the plurality of sensor data, each of the object groups includes two objects to be matched that are respectively detected by any two of the plurality of sensor data, at least one object to be matched between different object groups is different, and each matching weight is used to characterize a degree of similarity between two objects to be matched in the corresponding object group;

[0037] A second matching unit, configured to determine a relationship value of the at least one object group by a strategy based on logic matching, wherein each relationship value is used to measure an association relationship between two objects to be matched in the corresponding object group in terms of physical attributes;

[0038] A target matching unit is used to perform target matching on the objects to be matched respectively detected by the multiple sensor data according to the matching weight and the relationship value of the at least one object group, and obtain a matching result, wherein the matching result is used to indicate a target object group in the at least one object group, and the target object group is an object group that matches two of the objects to be matched with each other.

[0039] The vehicle provided in the third aspect of an embodiment of the present application includes a vehicle controller and a memory for communicating with the vehicle controller; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one vehicle controller to enable the at least one vehicle controller to execute the target matching method of multi-sensor data.

[0040] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is used to enable a computer to execute the target matching method for multi-sensor data.

[0041] It can be understood that the beneficial effects of the second to fourth aspects compared with the related art are the same as the beneficial effects of the first aspect compared with the related art. Please refer to the relevant description in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0043] Figure 1 It is a flowchart of an embodiment of a method for target matching of multi-sensor data provided by the present application;

[0044] Figure 2 is a flowchart of an embodiment of the present application for implementing target matching using a deep learning model;

[0045] Figure 3 It is a flowchart of an embodiment of the present application for achieving target matching by using a deep learning model and a strategy based on logic matching. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0049] like Figure 1 An embodiment of the present application provides a method for target matching of multi-sensor data, the method comprising steps S110 to S140:

[0050] Step S110, acquiring a plurality of sensor data, where different sensor data are collected by different types of sensors;

[0051] Step S120, inputting the multiple sensor data into the deep learning model, and the deep learning model outputs a matching weight of at least one object group, wherein the at least one object group is associated with the multiple sensor data, each object group includes two objects to be matched that are respectively detected by any two sensor data in the multiple sensor data, at least one object to be matched in different object groups is different, and each matching weight is used to characterize the similarity between the two objects to be matched in the corresponding object group;

[0052] Step S130, determining a relationship value of at least one object group through a strategy based on logic matching, wherein each relationship value is used to measure the association relationship between two matching objects in the corresponding object group in terms of physical attributes;

[0053] Step S140, according to the matching weight and relationship value of at least one object group, target matching is performed on the objects to be matched detected by multiple sensor data respectively to obtain a matching result, and the matching result is used to indicate a target object group in at least one object group, and the target object group is an object group in which two objects to be matched match each other.

[0054] In step S110 of this embodiment, the sensing data is collected by sensors. Taking vehicle-mounted sensors as an example, different types of sensors include but are not limited to: laser radar, camera, millimeter wave radar. Each sensor corresponds to a type of sensing data. Different types of sensors are sensors of different modalities.

[0055] In step S120 of this embodiment, the deep learning model can be obtained by training a network such as CNN or Transformer, or it can be based on a variant network of CNN. The specific network structure is not limited here.

[0056] The object to be matched refers to the object detected from the sensor data. Taking the vehicle-mounted sensor as an example, the object to be matched can be a vehicle or obstacle on the road ahead.

[0057] Taking the vehicle-mounted sensor as an example, the object group is introduced: 3 objects to be matched are detected in the sensor data of the camera, and 5 objects to be matched are detected in the sensor data of the radar. The object group is composed of one object to be matched in the sensor data of the camera and one object to be matched in the sensor data of the radar. Theoretically, there are 15 object groups. The purpose is to determine whether there is a matching object between the object to be matched detected in the sensor data of the camera and the object to be matched detected in the sensor data of the radar. For example, among the three objects to be matched detected in the sensor data of the camera, there is one object to be matched detected in the sensor data of the camera that matches them.

[0058] By inputting multiple sensor data into a deep learning model, the matching weight of at least one object group output by the deep learning model can be obtained, wherein each object group has a corresponding matching weight, and each matching weight is used to characterize the degree of similarity between two to-be-matched objects in the corresponding object group.

[0059] The parameter setting and training process of the deep learning model will be introduced in the subsequent embodiments.

[0060] In step S130 of this embodiment, a relationship value of at least one object group is determined based on a logic matching strategy, and each relationship value is used to measure the association relationship between two to-be-matched objects in its corresponding object group in terms of physical properties, wherein each object group has a corresponding relationship value.

[0061] Strategies based on logical matching include but are not limited to: Hungarian algorithm and maximum matching algorithm.

[0062] Physical properties include but are not limited to: speed, area, length, angle. For example, an object to be matched is detected in the sensor data collected by the camera, and an object to be matched is detected in the sensor data of the laser radar to form an object group. The relationship value of the object group includes a speed relationship value, an area relationship value, a length relationship value, an angle relationship value, etc. The relationship value here can be the deviation between the two, such as speed deviation, or the similarity between the two, such as angle similarity, etc., which is not specifically limited here.

[0063] In step S140 of the present implementation, target matching is performed on the objects to be matched that are detected respectively through multiple sensor data based on the matching weights and relationship values ​​of at least one object group. The purpose is to select an object group that matches the objects to be matched with each other as the target object group from at least one object group output by the deep learning model.

[0064] The mutual matching of two objects to be matched in the object group means that the two objects to be matched are the same object. For example, if an object to be matched detected by the sensor data of the vehicle camera is the same object as the object to be matched detected in the sensor data collected by the vehicle radar, then the object group composed of the two objects to be matched is the target object group. The matching result here indicates the target object group.

[0065] The embodiments provided in this application have at least the following beneficial effects:

[0066] The embodiment of the present application provides a target matching method for multi-sensor data. In view of the problem that strategies based on logic matching have mutual exclusion or logic loopholes when facing complex working conditions, the method uses a deep learning method to extract the matching weight between two to-be-matched objects in an object group, and uses a strategy based on logic matching to extract the relationship value between two to-be-matched objects in an object group. The matching weight and the relationship value are used to jointly perform target matching on multiple to-be-matched objects detected respectively through multiple sensor data. This combines the advantage of deep learning that can continuously learn complex working conditions with the advantage of the strategy based on logic matching that provides data support for the deep learning method, and can improve the accuracy of target matching.

[0067] In some embodiments of the present application, in step S140, target matching is performed on the objects to be matched that are respectively detected by multiple sensor data according to the matching weight and relationship value of at least one object group to obtain a matching result, including the following steps S210 and S220:

[0068] Step S210, obtaining a fusion weight corresponding to each object group according to the matching weight and relationship value of each object group;

[0069] Step S220: Based on the fusion weight of at least one object group, a target object group in which two to-be-matched objects in at least one object group match each other is determined to obtain a first matching result.

[0070] In this embodiment, the relationship value is also used as a weight, and the two weights are fused to obtain a fusion weight. The fusion weight can be matched using a traditional matching method, such as the Hungarian algorithm, the KM algorithm, and the maximum matching method. In one embodiment, a weight can be assigned to the matching weight and another weight can be assigned to the relationship value. The sum of the assigned weights is 1, and then weighted fusion is performed to obtain the fusion weight.

[0071] It should be noted that each object group is set with a corresponding fusion weight. Taking the maximum matching method as an example, assuming that there are 3 objects to be matched detected in the sensor data of the camera and 5 objects to be matched detected in the sensor data of the radar, then there are 15 object groups in total; among which the fusion weight of the object group composed of the first object to be matched detected in the sensor data of the camera and the fifth object to be matched detected in the sensor data of the radar is the largest, then the first object to be matched detected in the sensor data of the camera and the fifth object to be matched detected in the sensor data of the radar are matched with each other, and so on, the second object to be matched detected in the sensor data of the camera and the third object to be matched detected in the sensor data of the camera, and finally there are 3 target object groups in the 15 object groups.

[0072] In this embodiment, by fusing two weights to obtain a fusion weight, the fusion weight represents the similarity between two objects to be matched in the object group. This can combine the advantages of deep learning and logic operations, improve the accuracy of target matching, and facilitate matching based on traditional weight matching algorithms.

[0073] It should be noted that it is also possible to match the matching weight and the relationship value separately, and then obtain the respective matching results, and finally comprehensively judge the effects of the two matching results, and select the matching result with the best effect. For example, if the effect based on logical operation is better, then the result of logical operation is selected.

[0074] In some embodiments of the present application, the relationship value is a numerical deviation, and the numerical deviation is used to measure the difference in physical properties between two matching objects in the object group to which it corresponds;

[0075] The step S210 of obtaining the fusion weight corresponding to the object group according to the matching weight and the relationship value of each object group includes steps S310 and S320:

[0076] Step S310, obtaining a first weight corresponding to a matching weight of each object group and a second weight corresponding to a numerical deviation of each object group;

[0077] Step S320, the sum of the first weight product and the second weight product is determined as the fusion weight corresponding to the object group, the first weight product is the product of the matching weight and the first weight, the second weight product is the product of the second weight and the target ratio, the numerator of the target ratio is 1, the denominator of the target ratio includes the numerical deviation, and the sum of the second weight and the first weight is 1.

[0078] Refer to the following formula:

[0079]

[0080] Among them, S i,j k0P represents the fusion weight of the object group, which is composed of the to-be-matched object i of the sensor data I and the to-be-matched object j of the sensor data J. cNN represents the first weight product, k0 is the first weight, P CNN is the matching weight output by the deep learning model, 1-k0 is the second weight, and P refers to the numerical deviation of the physical properties of the object group. In some embodiments, the physical properties may include several types, and the specific content will be introduced in subsequent embodiments.

[0081] In this embodiment, first, the matching weight P CNN Set the first weight k0, and then set the second weight for the numerical deviation P, because P CNN is a normalized value, and the second weight is used as the numerator, so that the second weight product can be normalized to make S i,j The value is between 0 and 1, which is convenient for statistics.

[0082] It should be noted that, instead of taking the target ratio as the denominator, the second weight product can be directly taken as the product of the numerical deviation and the second weight, which can also achieve the purpose.

[0083] In some embodiments of the present application, the numerical deviation includes N sub-deviations, and the physical attribute includes N sub-physical attributes corresponding to the N sub-deviations one by one, each sub-deviation is used to measure the difference between two matching objects in the object group in the corresponding sub-physical attribute, and N is a positive integer greater than 1;

[0084] Before determining the sum of the first weight product and the second weight product as the fusion weight corresponding to the object group in step S320, the method further includes steps S410 to S430:

[0085] Step S410, obtaining a sub-weight of each sub-physical attribute among the N sub-physical attributes;

[0086] Step S420, weight fusion is performed on the sub-weight of each sub-physical attribute in the N sub-physical attributes and the sub-deviation corresponding to the sub-physical attribute to obtain a fused sub-weight;

[0087] Step S430, divide 1 by the fusion sub-weight to obtain a target ratio.

[0088] In this embodiment, the physical attribute may include N sub-physical attributes, for example, including but not limited to: distance, speed, area, length, angle, and identity (ID).

[0089] Each of the N sub-physical properties includes a corresponding sub-bias, for example:

[0090]

[0091] P v =v i -v j

[0092] P wh =w i *l i -w j l j

[0093] P l = l i -l j , P g =g i -g j

[0094] P id =id i -id j

[0095] Among them, S i,j P (t) represents the fusion weight of the object group at time t, where the object group consists of two objects to be matched between different sensor data. d Represents the distance deviation between different objects to be matched, P v Represents the speed deviation between different objects to be matched, P wh Represents the area deviation between different objects to be matched, P l Indicates the length deviation between different objects to be matched, P g represents the angle deviation between different objects to be matched, P id Indicates the historical matching id deviation between different objects to be matched; k1, k2, k3, k4, k5, k6 represent sub-weights. i ,v j are the speeds of the two objects to be matched. i *l i represents the area of ​​one of the objects to be matched, w j l j represents the area of ​​another object to be matched, the area is equal to the product of length l and width w, and g is the angle. id represents the ID number. The specific values ​​of k1, k2, k3, k4, k5, and k6 can be assigned based on performance evaluation, confidence evaluation, or expert experience, or scenario evaluation.

[0096] The target ratio is:

[0097] The second weight product is:

[0098] For example, there are three objects to be matched detected by the sensor data of the camera: objects to be matched 11, 12, and 13, and there are five objects to be matched detected by the sensor data of the radar: objects to be matched 21, 22, 23, 24, and 25. Then a total of 15 fusion weights are obtained, and the following matrix is ​​constructed (using the maximum matching method):

[0099] Table 1

[0100] 0.9 0.1 0.3 0.2 0.8 0.3 0.4 0.5 0.9 0.1 0.2 0.6 0.2 0.3 0.1

[0101] It can be seen that the first column is the fusion weights corresponding to the object group consisting of the to-be-matched object 11 and the to-be-matched objects 21, 22, 23, 24 and 25, among which the first value is the largest, which proves that the object group consisting of the to-be-matched object 11 and the to-be-matched object 21 is the target object group. The second and third columns are not repeated here.

[0102] Step S410, step S420 and step S430 provided in this embodiment divide the physical attribute into multiple sub-physical attributes, introduce influencing factors of the multiple sub-physical attributes, and thus improve the accuracy of the fusion weight.

[0103] It should be noted that the purpose can be achieved using only one physical attribute, which will not be elaborated here.

[0104] In some embodiments of the present application, the deep learning model includes a weight extraction network and a target matching network, and the target matching network is used to determine the second matching result according to the matching weight;

[0105] In step S120, the plurality of sensor data are input into the deep learning model, and the deep learning model outputs the matching weight of at least one object group, including:

[0106] Inputting a plurality of sensor data into a deep learning model, obtaining matching weights of at least one object group output by a weight extraction network;

[0107] After performing target matching on the objects to be matched respectively detected by the plurality of sensor data according to the matching weight and the numerical relationship of the at least one object group in step S140 and obtaining the matching result, the method further includes steps S510 and S520:

[0108] Step S510, generating training samples based on multiple sensor data and matching results;

[0109] Step S520: training the deep learning model based on the training samples to optimize the parameters of the weight extraction network and the target matching network.

[0110] In the embodiment of the present application, the deep learning model can not only generate matching weights, but also directly determine whether there is a match. The deep learning model here includes: 1. Weight extraction network, 2. Target matching network, which is described in detail below:

[0111] The role of the weight extraction network is to output the matching weight of at least one object group.

[0112] The role of the target matching network is to determine the second matching result based on the matching weight. The second matching result here is also used to identify the target object group, because the deep learning network can set the loss function to perform different functions, such as achieving target matching.

[0113] The purpose of step S510 and step S520 is to generate training samples based on multiple sensor data and matching results, train the deep learning model based on the training samples to optimize the parameters of the weight extraction network and the target matching network, and continuously train the parameters of the weight extraction network and the target matching network to optimize the parameters, so as to finally achieve the process of finding the target object group from multiple object groups by the deep learning model alone, that is, to achieve the target matching of the objects to be matched. Figure 2 and Figure 3 ,The advantage is that because the deep learning network can be continuously optimized, the model target matching effect (i.e. accuracy) can be continuously improved through continuous training.

[0114] Currently, the necessary data support for deep learning methods is provided by strategies based on logic matching. When the model is trained multiple times, it can be separated from the data support provided by the strategies based on logic matching and directly use the deep learning model for target matching. Moreover, the target matching effect can achieve the effect of combining deep learning methods with strategies based on logic matching. This processing can reduce the complexity of calculations.

[0115] The following is a set of implementation processes for deep learning models:

[0116] (1) Collect sensor data including multiple sensors (such as corner radar, 4D millimeter wave radar, laser radar (LiDAR), camera, etc.).

[0117] This data should include basic information such as the target's ID, location (such as x, y, z coordinates), speed, acceleration, heading angle, size, etc. Lane line information (i.e. scene information) is also collected to improve the ability to understand the environmental context.

[0118] (2) Synchronize data from different sensors to ensure data consistency in the time dimension.

[0119] The sensor data of multiple consecutive frames are organized into N sequences so that the model can capture the temporal dependencies.

[0120] (3) The weight extraction network constructs the preprocessed multi-frame sensor data into a multi-dimensional tensor.

[0121] The weight extraction network extracts and fuses features, and the weight extraction network outputs the corresponding matching weights.

[0122] The target matching network identifies the target object group according to the needs of the actual project (implemented through the loss function) to complete the target matching.

[0123] The loss functions include:

[0124] Binary matching: Each object group is classified as "match" or "mismatch". Assume that the output layer of the model is a sigmoid activation function, which is used to predict whether the two objects to be matched match (the output value is between 0 and 1, indicating the probability of mismatch and match). Binary Cross-Entropy Loss can be used as the loss function.

[0125] In some embodiments, it may also be set as:

[0126] Multi-class matching: Object groups are classified into one of multiple categories, representing different types of matches. If the task is to classify the objects to be matched into one of multiple categories (e.g., different types of matches), you can use Categorical Cross-Entropy Loss.

[0127] Complex matching (such as graph matching, sequence matching): It may be necessary to design more complex output structures and loss functions, such as structural similarity loss, graph matching loss, or sequence alignment loss.

[0128] During the training process, the model's robustness to noise can be improved by adding noise to simulate random interference in the real world. For example, generating random noise with a normal distribution and adding it to the input is equivalent to enhancing the generalization of the model by performing small perturbations such as random rotation and translation on the matching object; or adding Dropout between neural network layers or inputs, setting an appropriate dropout rate, to simulate the situation in actual engineering where frames are lost and data timestamps may not match, and improve the generalization ability of the model.

[0129] It should be noted that after continuous training of the deep learning model, it can be implemented using only the deep learning method to reduce the complexity, or a combination of the two methods can be used, because the accuracy of predicting the matching weight of one of the methods is improved, thus also improving the overall target matching accuracy.

[0130] In some embodiments of the present application, after step S520 trains the deep learning model based on the training samples to optimize the parameters of the weight extraction network and the target matching network, steps S610 and S620 are also included:

[0131] Step S610, when a plurality of sensor data collected for the N+1th time is obtained, if a matching condition is satisfied between the Nth first matching result and the Nth second matching result, the plurality of sensor data collected for the N+1th time is input into the deep learning model to obtain the N+1th second matching result output by the target matching network, wherein the Nth first matching result is a first result determined based on the plurality of sensor data collected for the N+1th time, and the N second matching results are second matching results output by the target matching network based on the plurality of sensor data collected for the Nth time;

[0132] Step S620: determine the N+1 second matching results as the N+1 first matching results.

[0133] In this embodiment, if the Nth first matching result (i.e., the matching result output based on two methods) and the Nth second matching result (the matching result output based only on the deep learning model) meet the matching condition, for example, the matching condition may be that the fusion weight between the two is less than a threshold, that is, both indicate the target object group, then when the deviation of the two values ​​(i.e., the fusion weight) is less than a threshold of 0.01, it proves that the deep learning model has achieved the same target matching effect as the combination of the two methods.

[0134] At N+1 times, the deep learning model can be directly used to predict the matching weight, and the matching weight is used to output the target matching result. This embodiment can be separated from the data support provided by the strategy based on logical matching, reducing the complexity of calculation.

[0135] It should be noted that a combination of the two methods can also be used, because the accuracy of predicting the matching weight of one of the methods is improved, thus also improving the accuracy of the overall target matching.

[0136] In some embodiments of the present application, a target matching device for multi-sensor data is shown, and the device includes:

[0137] The data acquisition unit 1100 is used to acquire multiple sensor data, where different sensor data are collected by different types of sensors;

[0138] The first matching unit 1200 is used to input the multiple sensor data into a deep learning model, and the deep learning model outputs a matching weight of at least one object group, wherein the at least one object group is associated with the multiple sensor data, each of the object groups includes two objects to be matched that are respectively detected by any two of the multiple sensor data, at least one of the objects to be matched is different between different object groups, and each matching weight is used to characterize the similarity between the two objects to be matched in the object group to which it corresponds;

[0139] The second matching unit 1300 is used to determine the relationship value of the at least one object group through a strategy based on logic matching, wherein each relationship value is used to measure the association relationship between two objects to be matched in the corresponding object group in terms of physical attributes;

[0140] The target matching unit 1400 is used to perform target matching on the objects to be matched that are respectively detected by the multiple sensor data according to the matching weight and the relationship value of the at least one object group, and obtain a matching result, wherein the matching result is used to indicate a target object group in the at least one object group, and the target object group is the object group in which two of the objects to be matched match each other.

[0141] It should be noted that the multi-sensor target matching device provided in this embodiment and the above-mentioned multi-sensor target matching method are based on the same inventive concept, so the relevant content of the above-mentioned multi-sensor target matching method is also applicable to the content of the multi-sensor target matching device, and therefore, it will not be repeated here.

[0142] An embodiment of the present application also discloses a vehicle, including a vehicle controller and a memory for communicating with the vehicle controller; the memory stores instructions that can be executed by the vehicle controller, and the instructions are executed by the vehicle controller so that the vehicle controller executes the target matching method of multi-sensor data as described above.

[0143] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned target matching method of multi-sensor data is implemented.

[0144] The memory is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device.

[0145] In some embodiments, the memory may include a memory remotely located relative to the processor, and the remote memory may be connected to the processor via a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0147] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0148] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0150] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0151] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0152] In the several embodiments provided in the present application, it should be understood that the disclosed controller and method can be implemented in other ways. For example, the controller embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0153] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0155] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0156] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A target matching method for multi-sensor data, characterized in that: The method comprises: Acquire multiple sensor data, where different sensor data are collected by different types of sensors; Inputting the plurality of sensor data into a deep learning model, and outputting a matching weight of at least one object group by the deep learning model, wherein the at least one object group is associated with the plurality of sensor data, each of the object groups includes two objects to be matched that are respectively detected by any two of the plurality of sensor data, at least one of the objects to be matched is different between different object groups, and each matching weight is used to characterize the degree of similarity between the two objects to be matched in the object group to which it corresponds; Determine the relationship value of the at least one object group through a strategy based on logic matching, each of the relationship values ​​being used to measure the association relationship between the two to-be-matched objects in the corresponding object group in terms of physical attributes; According to the matching weight and the relationship value of the at least one object group, target matching is performed on the objects to be matched respectively detected by the multiple sensor data to obtain a matching result, wherein the matching result is used to indicate a target object group in the at least one object group, and the target object group is the object group in which two of the objects to be matched match each other.

2. The target matching method of multi-sensor data according to claim 1, characterized in that: The step of performing target matching on the objects to be matched respectively detected by the plurality of sensor data according to the matching weight and the relationship value of the at least one object group to obtain a matching result includes: Obtaining a fusion weight corresponding to the object group according to the matching weight and the relationship value of each object group; Based on the fusion weight of the at least one object group, a target object group in which two of the to-be-matched objects in the at least one object group match each other is determined as a first matching result.

3. The target matching method of multi-sensor data according to claim 2, characterized in that: The relationship value is a numerical deviation, and the numerical deviation is used to measure the difference in physical properties between two matching objects in the object group corresponding to the relationship value. The obtaining, according to the matching weight and the relationship value of each object group, a fusion weight corresponding to the object group includes: Acquire a first weight corresponding to the matching weight of each of the object groups and a second weight corresponding to the numerical deviation of each of the object groups; The sum of the first weight product and the second weight product is determined as the fusion weight corresponding to the object group, the first weight product is the product of the matching weight and the first weight, the second weight product is the product of the second weight and a target ratio, the numerator of the target ratio is 1, the denominator of the target ratio includes the numerical deviation, and the sum of the second weight and the first weight is 1.

4. The target matching method of multi-sensor data according to claim 3, characterized in that: The numerical deviation includes N sub-deviations, the physical attribute includes N sub-physical attributes corresponding to the N sub-deviations one by one, each sub-deviation is used to measure the difference between two matching objects in the object group in the corresponding sub-physical attribute, and N is a positive integer greater than 1; Before determining the sum of the first weight product and the second weight product as the fusion weight corresponding to the object group, the method further includes: Obtaining a sub-weight of each of the N sub-physical attributes; Performing weight fusion on the sub-weight of each of the N sub-physical attributes and the sub-deviation corresponding to the sub-physical attribute to obtain a fused sub-weight; Divide 1 by the fusion sub-weight to obtain the target ratio.

5. The target matching method of multi-sensor data according to claim 4, characterized in that: The N sub-physical attributes include at least two of distance, speed, area, length, angle and identity.

6. The target matching method of multi-sensor data according to claim 2, characterized in that: The deep learning model includes a weight extraction network and a target matching network, and the target matching network is used to determine a second matching result according to the matching weight; The step of inputting the plurality of sensor data into a deep learning model, and outputting a matching weight of at least one object group by the deep learning model, comprises: Inputting the plurality of sensor data into a deep learning model to obtain a matching weight of at least one object group output by the weight extraction network; After performing target matching on the objects to be matched respectively detected by the plurality of sensor data according to the matching weight and the relationship value of the at least one object group and obtaining a matching result, the method further includes: generating a training sample based on the plurality of sensor data and the matching result; The deep learning model is trained based on the training samples to optimize parameters of the weight extraction network and the target matching network.

7. The target matching method of multi-sensor data according to claim 6, characterized in that: After the deep learning model is trained based on the training samples to optimize the parameters of the weight extraction network and the target matching network, the method further includes: When a plurality of sensor data collected for the N+1th time is obtained, if a matching condition is satisfied between the Nth first matching result and the Nth second matching result, the plurality of sensor data collected for the N+1th time is input into the deep learning model to obtain the N+1th second matching result output by the target matching network, wherein the Nth first matching result is a first result determined based on the plurality of sensor data collected for the N+1th time, and the N second matching results are second matching results output by the target matching network based on the plurality of sensor data collected for the Nth time; The N+1 second matching results are determined as the N+1 first matching results.

8. A target matching device for multi-sensor data, characterized in that: The device comprises: A data acquisition unit, used to acquire a plurality of sensor data, wherein the different sensor data are acquired by different types of sensors; A first matching unit, configured to input the plurality of sensor data into a deep learning model, and the deep learning model outputs a matching weight of at least one object group, wherein the at least one object group is associated with the plurality of sensor data, each of the object groups includes two objects to be matched that are respectively detected by any two of the plurality of sensor data, at least one of the objects to be matched is different between different object groups, and each matching weight is used to characterize a degree of similarity between two objects to be matched in the object group to which it corresponds; A second matching unit, configured to determine a relationship value of the at least one object group by a strategy based on logic matching, wherein each relationship value is used to measure an association relationship between two objects to be matched in the corresponding object group in terms of physical attributes; A target matching unit is used to perform target matching on the objects to be matched respectively detected by the multiple sensor data according to the matching weight and the relationship value of the at least one object group, and obtain a matching result, wherein the matching result is used to indicate a target object group in the at least one object group, and the target object group is the object group in which two of the objects to be matched match each other.

9. A vehicle, characterized in that: It includes a vehicle controller and a memory for communicating with the vehicle controller; the memory stores instructions that can be executed by the vehicle controller, and the instructions are executed by the vehicle controller to enable the vehicle controller to perform the target matching method of multi-sensor data as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the target matching method for multi-sensor data according to any one of claims 1 to 7 is implemented.

Images