Fusion target determination method and device and electronic equipment

Abstract

The embodiment of the invention provides a fusion target determination method and device and electronic equipment, and the method is applied to the field of intelligent driving, and is used for determining a fusion target under the condition that the number of a plurality of historical fusion targets determined in the previous frame is greater than or equal to the target number. A plurality of candidate targets and a plurality of reserved targets are determined from a plurality of historical fused targets and a plurality of sensor targets identified by a current frame, the historical fused targets being targets concerned by at least one sensor of the target vehicle in a previous frame. And determining the priority of each candidate target based on the target types of the plurality of candidate targets. Multiple fusion targets of the current frame are determined based on the priority of each candidate target, the multiple candidate targets and the multiple reserved targets, the number of the multiple fusion targets is smaller than or equal to the number of the targets, consumption of operation resources and storage resources of the target fusion system is reduced, the stability of the target fusion system is improved, and the user experience is improved. And the safety of the target vehicle is improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving, and more particularly, to a method, apparatus, and electronic device for determining a fused target in the field of intelligent driving. Background Art

[0002] With the development of intelligent driving technology, more and more vehicles will use intelligent driving technology to assist vehicle driving. Intelligent driving technology relies on environmental information collected by multiple sensors on the vehicle. By identifying the environmental information collected by multiple sensors, targets that may affect the normal driving of the vehicle can be obtained, that is, the targets of concern.

[0003] In the related art, over time, the number of targets that the sensors are concerned about may increase, resulting in an increasing consumption of computing resources and storage resources. However, the computing resources and storage resources of vehicles are often limited, which may lead to the problem of being unable to process environmental information in a timely manner and pose a safety hazard.

[0004] How to reduce the consumption of computing resources and storage resources during intelligent driving while ensuring vehicle safety is a research hotspot. Summary of the Invention

[0005] The present application provides a method, apparatus, and electronic device for determining a fused target. This method can reduce the consumption of computing resources and storage resources during intelligent driving while ensuring vehicle safety.

[0006] In a first aspect, a method for determining a fused target is provided. The method includes:

[0007] When the number of multiple historical fused targets determined in the previous frame is greater than or equal to the target number, determine multiple candidate targets and multiple retained targets from the multiple historical fused targets and multiple sensor targets recognized in the current frame, where the historical fused target is a target that was concerned by at least one sensor of the target vehicle in the previous frame;

[0008] Based on the target types of the multiple candidate targets, determine the priority of each candidate target, where the target type is used to indicate whether the candidate target is a historical fused target or a sensor target;

[0009] Based on the priority of each candidate target, the multiple candidate targets, and the multiple retained targets, determine multiple fused targets in the current frame, where the number of the multiple fused targets is less than or equal to the target number.

[0010] In a second aspect, a device for determining a fused target is provided. The device includes:

[0011] A first target determination module, configured to determine a plurality of candidate targets and a plurality of reserved targets from the plurality of historical fusion targets and the plurality of sensor targets recognized in the current frame when the number of the plurality of historical fusion targets determined in the previous frame is greater than or equal to the target number, where the historical fusion targets are targets that are concerned by at least one sensor of the target vehicle in the previous frame;

[0012] A priority determination module, configured to determine the priority of each of the candidate targets based on the target type of the plurality of candidate targets, where the target type is used to indicate whether the candidate target is a historical fusion target or a sensor target;

[0013] A second target determination module, configured to determine a plurality of fusion targets in the current frame based on the priority of each of the candidate targets, the plurality of candidate targets, and the plurality of reserved targets, where the number of the plurality of fusion targets is less than or equal to the target number.

[0014] In a possible implementation manner, the first target determination module is configured to compare the plurality of historical fusion targets and the plurality of sensor targets; determine the different plurality of targets among the plurality of historical fusion targets and the plurality of sensor targets as the plurality of candidate targets; and determine the same plurality of targets among the plurality of historical fusion targets and the plurality of sensor targets as the plurality of reserved targets.

[0015] In a possible implementation manner, the first target determination module is configured to determine the plurality of targets that exist in the plurality of historical fusion targets but do not exist in the plurality of sensor targets, and the plurality of targets that exist in the plurality of sensor targets but do not exist in the plurality of historical fusion targets, as the plurality of candidate targets.

[0016] In a possible implementation manner, the device further includes a deletion module, configured to determine the number of fusion existence cycles of the plurality of historical fusion targets, where the number of fusion existence cycles is the number of cycles determined as fusion targets; and delete the historical fusion targets among the plurality of historical fusion targets whose number of fusion existence cycles is greater than or equal to the cycle number threshold.

[0017] In a possible implementation, the priority determination module is configured to, for a first candidate target among the multiple candidate targets whose target type is a historical fusion target, obtain the fusion parameters, motion parameters, position parameters, and measurement parameters of the first candidate target; determine the priority of the first candidate target based on the fusion parameters, motion parameters, position parameters, and measurement parameters of the first candidate target; for a second candidate target among the multiple candidate targets whose target type is a sensor target, obtain the motion parameters, position parameters, and measurement parameters of the second candidate target; and determine the priority of the second candidate target based on the motion parameters, position parameters, and measurement parameters of the second candidate target.

[0018] In a possible implementation, the fusion parameters of the first candidate target include a first fusion type and a second fusion type, the motion parameters of the first candidate target include a motion state, a motion direction relative to the target vehicle, and an estimated collision time with the target vehicle, the position parameters of the first candidate target include a lateral distance from the target vehicle and a longitudinal distance from the target vehicle, and the measurement parameters of the first candidate target include a confidence level, a fusion existence period number, and a measurement state;

[0019] Wherein, the first fusion type is used to represent the number of sensors that detected the first candidate target, the second fusion type is used to represent whether there is a target sensor among the sensors that detected the first candidate target, the fusion existence period number is the number of periods in which the first candidate target is determined to be a fusion target, and the measurement state includes a newborn state, a processing state, and a prediction state.

[0020] In a possible implementation, the motion parameters of the second candidate target include a motion state, a motion direction relative to the target vehicle, and an estimated collision time with the target vehicle, the position parameters of the second candidate target include a lateral distance from the target vehicle and a longitudinal distance from the target vehicle, and the measurement parameters of the second candidate target include a confidence level, a sensor existence period number, and a sensor importance level;

[0021] Wherein, the sensor existence period number is the number of periods in which the second candidate target is determined to be a sensor target.

[0022] In a possible implementation, the priority determination module is configured to perform any one of the following:

[0023] Perform weighted fusion on the fusion parameters, motion parameters, position parameters, and measurement parameters of the first candidate target to obtain the priority of the first candidate target;

[0024] Input the fusion parameters, motion parameters, position parameters, and measurement parameters of the first candidate target into the first priority determination model; through the first priority determination model, extract features from the fusion parameters, motion parameters, position parameters, and measurement parameters of the first candidate target to obtain the first target feature of the first candidate target; through the first priority determination model, map the first target feature to obtain the priority of the first candidate target.

[0025] The priority determination module is used to perform any one of the following:

[0026] Perform weighted fusion on the motion parameters, position parameters, and measurement parameters of the second candidate target to obtain the priority of the second candidate target;

[0027] Input the motion parameters, position parameters, and measurement parameters of the second candidate target into the second priority determination model; through the second priority determination model, extract features from the motion parameters, position parameters, and measurement parameters of the second candidate target to obtain the second target feature of the second candidate target; through the second priority determination model, map the second target feature to obtain the priority of the second candidate target.

[0028] In a possible implementation, the second target determination module is used to determine a plurality of reference targets from the plurality of candidate targets based on the priorities of each candidate target. The plurality of reference targets are candidate targets whose priorities meet the priority conditions, and the number of the plurality of reference targets is the difference between the number of targets and the number of the plurality of reserved targets; determine the plurality of reference targets and the plurality of reserved targets as the plurality of fusion targets in the current frame.

[0029] In a possible implementation, the second target determination module is used to sort the plurality of candidate targets in descending order of priority; determine the first N candidate targets after sorting as the plurality of reference targets, where N is a positive integer and is the difference between the number of targets and the number of the plurality of reserved targets.

[0030] In a possible implementation, the device further includes: a filtering module, which is used to determine a plurality of initial sensor targets recognized by a plurality of sensors of the target vehicle in the current frame; merge the duplicate targets among the plurality of initial sensor targets to obtain the plurality of sensor targets, and the duplicate targets refer to the initial sensor targets recognized by at least two sensors in the current frame.

[0031] In a possible implementation, the device further includes a target number determination module, which is used to perform any one of the following:

[0032] Obtain the maximum historical fusion target quantity of multiple reference vehicles, where the multiple reference vehicles and the target vehicle belong to the same vehicle model; determine the target quantity based on the historical fusion target quantity;

[0033] Based on the computing resources and storage resources of the target vehicle, determine the target quantity.

[0034] In a third aspect, an electronic device is provided, including a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, so that the electronic device executes the method in the first aspect above.

[0035] In a fourth aspect, a computer program product is provided, which includes: computer program code, when the computer program code runs on a computer, the computer executes the method in the first aspect above.

[0036] In a fifth aspect, a computer-readable storage medium is provided, which stores computer program code, and when the computer program code runs on a computer, the computer executes the method in the first aspect above.

[0037] Through the technical solution provided by the embodiments of the present application, when the quantity of multiple historical fusion targets determined in the previous frame is greater than or equal to the target quantity, multiple candidate targets and multiple reserved targets are determined from the multiple historical fusion targets and multiple sensor targets recognized in the current frame, and the historical fusion target is a target that is concerned by at least one sensor of the target vehicle in the previous frame. Based on the target types of the multiple candidate targets, determine the priorities of the respective candidate targets. Based on the priorities of the respective candidate targets, the multiple candidate targets, and the multiple reserved targets, determine the multiple fusion targets in the current frame, and the quantity of the multiple fusion targets is less than or equal to the target quantity. The target quantity can be regarded as the upper limit of the quantity of the fusion targets, thereby reducing the consumption of the computing resources and storage resources of the target fusion system, improving the stability of the target fusion system, and thus improving the safety of the target vehicle. Description of the Drawings

[0038] Figure 1 is a schematic diagram of the implementation environment of a method for determining fusion targets provided by an embodiment of the present application;

[0039] Figure 2 is a flowchart of a method for determining fusion targets provided by an embodiment of the present application;

[0040] Figure 3 is a flowchart of a method for determining fusion targets provided by an embodiment of the present application;

[0041] Figure 4It is a schematic structural diagram of a fusion target determination device provided by an embodiment of the present application;

[0042] Figure 5 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0043] Next, the technical solutions in the present application will be clearly and elaborately described with reference to the accompanying drawings. Among them, in the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B may mean A or B. The "and / or" in the text is only a description of the association relationship between associated objects, indicating that there can be three relationships. For example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality" means two or more than two.

[0044] Hereinafter, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features.

[0045] Intelligent driving: Essentially, intelligent driving involves cognitive engineering of attention attraction and distraction, mainly including three links: network navigation, autonomous driving, and manual intervention. The prerequisite for intelligent driving is that the vehicle we select meets the dynamic requirements of driving, and the sensors on the vehicle can obtain relevant visual and auditory signals and information, and control the corresponding follow-up system through cognitive computing.

[0046] LiDAR: It is a radar system that detects the position, speed and other characteristic quantities of a target by emitting laser beams. Its working principle is to emit a detection signal (laser beam) to the target, and then compare the received signal (target echo) reflected from the target with the emitted signal. After appropriate processing, relevant information about the target can be obtained, such as target distance, azimuth, altitude, speed, attitude, and even shape parameters, so as to detect, track and identify targets such as airplanes and missiles.

[0047] Millimeter-wave radar: A radar that operates in the millimeter-wave band (Millimeter Wave). Generally, millimeter waves refer to the frequency range of 30 - 300 GHz (wavelength of 1 - 10 mm). The wavelength of millimeter waves is between that of microwaves and centimeter waves. Therefore, millimeter-wave radar has some advantages of both microwave radar and optoelectronic radar. Compared with centimeter-wave seekers, millimeter-wave seekers are characterized by small size, light weight, and high spatial resolution. Compared with optical seekers such as infrared, laser, and television, millimeter-wave seekers have strong ability to penetrate fog, smoke, and dust, and have the characteristics of all-weather (except heavy rain) and all-day operation. In addition, the anti-interference and anti-stealth capabilities of millimeter-wave seekers are also superior to other microwave seekers. Millimeter-wave radar can distinguish and identify very small targets and can also identify multiple targets simultaneously.

[0048] Target fusion system: The target fusion system is part of the intelligent driving system. Its main function is to process the information collected by multiple vehicle-mounted sensing devices (sensors), fuse the data detected with the same attributes, and obtain a method with better detection performance than a single sensor. This fusion method can enhance the system function or increase the system safety characteristics. For example, millimeter-wave radar and lidar work simultaneously to detect the object target in front and fuse the object target. The output of the fused object target not only improves the measurement accuracy but also enables the intelligent driving system to still operate in the case of the failure of a certain sensor, meeting the functional safety requirement of fail-operational.

[0049] In related technologies, due to the increasing number of fused sensors and the increasing number of targets output by each sensor, the number of fused targets output by fusion will also increase. The gradually increasing number of fused targets leads to continuous adaptation of the backend applications, with poor generality; and the relatively large number of targets causes a gradual increase in the consumption of computing power and memory resources of the fusion and subsequent application modules (such as target selection), and an increase in the load of the communication link, which is not suitable for the applications of subsequent small-computing-power platforms and small-memory platforms. Therefore, it is necessary to sort the priorities of the fused targets and fix the number of fused target outputs, so that the backend applications will be insensitive to sensor changes, and the computing power consumption and memory consumption of the overall link framework will be greatly reduced, and the development workload of each module will also be greatly reduced.

[0050] After introducing the nouns involved in the embodiments of the present application, the implementation environment of the embodiments of the present application will be introduced below. See Figure 1 , the implementation environment of the method for determining fused targets provided by the embodiments of the present application includes the target fusion system 101 of the vehicle 100 and multiple sensors 104 of the vehicle 100.

[0051] The target fusion system 101 belongs to the intelligent driving system of the vehicle 100. The target fusion system 101 is communicatively connected to multiple sensors 104 of the vehicle 100. The target fusion system 101 can obtain the information collected by the multiple sensors 104 and process the obtained information to obtain the targets around the vehicle 100. Additionally, since the vehicle 100 is equipped with multiple sensors 104, the target fusion system can also fuse the same target identified by different sensors 104, thereby improving the accuracy of identifying the target. The target fusion system 101 includes a processor and a memory. The processor is used for information processing and target fusion, and the memory is used for storing the information collected by the multiple sensors 104 and the results of information processing and target fusion.

[0052] The multiple sensors 104 are used for collecting the environmental information around the vehicle 100. The multiple sensors 104 can include sensors of the same type installed at different positions or different types of sensors. In some embodiments, the multiple sensors 104 include cameras, lidars, and millimeter-wave radars.

[0053] After introducing the implementation environment of the embodiments of the present application, the application scenarios of the method for determining fused targets provided by the embodiments of the present application will be described below in combination with the above implementation environment. In the following description process, the target fusion system is the target fusion system 101 in the above implementation environment, and the multiple sensors are the multiple sensors 104 in the above implementation environment.

[0054] The technical solution provided by the embodiment of the present application can be applied to vehicles using intelligent driving technology, especially vehicles including the above-mentioned target fusion system and multiple sensors. The target fusion system processes the information collected by multiple sensors in units of frames. Here, the process from the start to the end of a round of information processing by the target fusion system is called a frame. By adopting the technical solution provided by the embodiment of the present application, when the number of multiple historical fusion targets determined in the previous frame is greater than or equal to the target number, the target fusion system determines multiple candidate targets and multiple reserved targets from the multiple historical fusion targets and multiple sensor targets identified by the multiple sensors in the current frame. The historical fusion target is a target that is concerned by at least one sensor of the target vehicle. The multiple candidate targets are historical fusion targets or sensor targets waiting to be determined whether to be used as fusion targets in the current frame. The multiple reserved targets are historical fusion targets or sensor targets that are used as fusion targets in the current frame. The target fusion system determines the priority of each candidate target based on the target type of the multiple candidate targets. The target type includes historical fusion targets and sensor targets. The priority is used to represent the priority degree of being determined as the fusion target of the current frame. The target fusion system determines multiple fusion targets of the current frame based on the priority of each candidate target, the multiple candidate targets, and the multiple reserved targets. The number of the multiple fusion targets is less than or equal to the target number. Thus, it is ensured that the number of continuously processed fusion targets by the target fusion system is less than or equal to the target number, reducing the consumption of computing resources and storage resources of the target fusion system, improving the stability of the target fusion system, and thus improving the safety of the target vehicle.

[0055] After introducing the implementation environment and application scenarios of the present application, the method for determining fusion targets provided by the embodiments of the present application will be described below. See Figure 2 , taking the target fusion system of the target vehicle as the execution subject as an example, the method includes the following steps.

[0056] 201. When the number of multiple historical fusion targets determined in the previous frame is greater than or equal to the target number, the target fusion system determines multiple candidate targets and multiple reserved targets from the multiple historical fusion targets and multiple sensor targets identified in the current frame. The historical fusion target is a target that was concerned by at least one sensor of the target vehicle in the previous frame.

[0057] Among them, the process from the start to the end of a round of information processing by the target fusion system is called a frame, and the multiple historical fusion targets are the fusion targets determined by the target fusion system in the previous frame. The targets being monitored by at least one sensor will be continuously tracked and processed by the sensor, and the multiple historical fusion targets are non-repeating targets. The targets being monitored by at least one sensor refer to the targets with a relatively high degree of importance to the sensor, or to the targets with a relatively high degree of importance to the intelligent driving process, or to the targets with a relatively high degree of importance to the driving of the target vehicle at the current position. The target quantity is used to limit the number of fusion targets to ensure that the number of fusion targets remains within a reasonable range, so as to keep the consumption of computing resources and storage resources during the intelligent driving process within a reasonable range. The judgment criterion for whether the number of fusion targets is reasonable is based on the target quantity, and this target quantity is set by technicians according to the actual situation or determined according to the available computing resources and storage resources of the target fusion system. The embodiments of the present application do not make any limitations on this. In addition, since the difference in the amount of computing resources and storage resources consumed by processing a single fusion target is relatively small, the number of fusion targets being within a reasonable range (below the target quantity) also means that the consumption of computing resources and storage resources during the intelligent driving process is within a reasonable range. The target vehicle is a vehicle equipped with this target fusion system and multiple sensors. The sensor target refers to the target recognized by at least one sensor, and the multiple sensor targets are non-repeating targets. The multiple candidate targets include historical fusion targets and sensor targets whose status as fusion targets for the current frame needs to be determined, and the multiple reserved targets include historical fusion targets and sensor targets that are determined as fusion targets for the current frame.

[0058] 202. The target fusion system determines the priority of each candidate target based on the target type of the multiple candidate targets, and the target type is used to indicate whether the candidate target is a historical fusion target or a sensor target.

[0059] Among them, the priority of a candidate target is used to indicate the degree of priority of being determined as a fusion target for the current frame. The higher the priority of a candidate target, the greater the possibility that the candidate target will be determined as a fusion target for the current frame; the lower the priority of a candidate target, the smaller the possibility that the candidate target will be determined as a fusion target for the current frame.

[0060] 203. The target fusion system determines multiple fusion targets for the current frame based on the priority of each candidate target, the multiple candidate targets, and the multiple reserved targets, and the number of the multiple fusion targets is less than or equal to the target quantity.

[0061] Among them, the multiple fusion targets for the current frame are also the targets monitored by at least one sensor of the target vehicle in the current frame.

[0062] Through the technical solution provided by the embodiments of the present application, when the number of multiple historical fusion targets determined in the previous frame is greater than or equal to the target number, multiple candidate targets and multiple retained targets are determined from the multiple historical fusion targets and the multiple sensor targets recognized in the current frame, where the historical fusion target is a target that is concerned by at least one sensor of the target vehicle in the previous frame. Based on the target types of the multiple candidate targets, the priorities of the respective candidate targets are determined. Based on the priorities of the respective candidate targets, the multiple candidate targets, and the multiple retained targets, multiple fusion targets in the current frame are determined, and the number of the multiple fusion targets is less than or equal to the target number, thereby reducing the consumption of computing resources and storage resources of the target fusion system, improving the stability of the target fusion system, and thus improving the safety of the target vehicle.

[0063] It should be noted that the above steps 201-203 are a simple introduction to the method for determining fusion targets provided by the embodiments of the present application. Below, some examples will be combined to more clearly illustrate the method for determining fusion targets provided by the embodiments of the present application. Refer to Figure 3 , taking the target fusion system with the target vehicle as the execution subject as an example, the method includes the following steps.

[0064] 301. When the number of multiple historical fusion targets determined in the previous frame is greater than or equal to the target number, the target fusion system determines the number of cycles of existence of the fusion of the multiple historical fusion targets, where the number of cycles of existence of the fusion is the number of cycles determined as fusion targets, and the historical fusion target is a target that is concerned by at least one sensor of the target vehicle in the previous frame.

[0065] Among them, the process from the start to the end of a round of information processing by the target fusion system is called a frame, and the multiple historical fusion targets are the fusion targets determined by the target fusion system in the previous frame. The targets being monitored by at least one sensor will be continuously tracked and processed by the sensor, and the multiple historical fusion targets are non-repeating targets. The targets being monitored by at least one sensor refer to the targets with a relatively high degree of importance to the sensor, or to the intelligent driving process, or to the driving of the target vehicle at the current position. The target quantity is used to limit the number of fusion targets to ensure that the number of fusion targets remains within a reasonable range, so as to keep the consumption of computing resources and storage resources during the intelligent driving process within a reasonable range. The criterion for determining whether the number of fusion targets is reasonable is based on the target quantity, which is set by technicians according to the actual situation or determined according to the available computing resources and storage resources of the target fusion system. This application embodiment does not make a limitation on this. Additionally, since the resource consumption of processing a single fusion target has a relatively small difference in quantity, the number of fusion targets being within a reasonable range (below the target quantity) also means that the consumption of computing resources and storage resources during the intelligent driving process is within a reasonable range. The target vehicle is a vehicle equipped with the target fusion system and multiple sensors. In some embodiments, the multiple sensors include cameras, lidar, and millimeter-wave radars, and the number of cameras, lidar, and millimeter-wave radars can all be one or more.

[0066] In some embodiments, the multiple historical fusion targets are stored in a fusion target list, and the target fusion list is used to store the determined fusion targets. The fusion target list stores the fusion targets in a way of storing target attributes, and the target attributes include the target identifier of the fusion target (used to distinguish different fusion targets) and the target parameters of the fusion target (parameters related to the fusion target collected by at least one sensor, such as parameters including the speed and position of the fusion target). In this case, the target fusion system can add the determined fusion targets to the fusion target list for subsequent tracking and processing. The fusion target list is updated by the target fusion system, and the target fusion system can either add fusion targets to the fusion target list or delete the fusion targets in the fusion target list.

[0067] In a possible implementation manner, the target fusion system determines the number of multiple historical fusion targets determined in the previous frame. When the number of the multiple historical fusion targets is greater than or equal to the target quantity, the target fusion system determines the number of fusion existence cycles of each historical fusion target. When the number of the multiple historical fusion targets is less than the target quantity, the target fusion system determines the multiple historical fusion targets as the fusion targets of the current frame and adds fusion targets according to a preset algorithm.

[0068] Taking the example that the multiple historical fusion targets are stored in the fusion target list, the target fusion system determines the number of historical fusion targets stored in the fusion target list. When the number of historical fusion targets stored in the fusion target list is greater than or equal to the target number, the target fusion system obtains the fusion existence cycle numbers of the respective historical fusion targets from the fusion target list. Among them, for any historical fusion target in the fusion target list, after the historical fusion target is added to the fusion target list, every time the historical fusion target is determined as a fusion target in one frame, the target fusion system increments the fusion existence cycle number of the historical fusion target by one, and the fusion existence cycle number of each historical fusion target is stored in the fusion target list.

[0069] The above is described by taking the example that the fusion existence cycle numbers are stored in the fusion target list. In other possible implementation manners, the fusion existence cycle numbers of the respective historical fusion targets are stored in the fusion log of the target fusion system. In this case, the target fusion system can obtain the fusion existence cycles of the respective historical fusion targets from the fusion log.

[0070] For a clearer description of the embodiments of the present application, the method for determining the target number is described below.

[0071] In a possible implementation manner, the target fusion system obtains the maximum number of historical fusion targets of multiple reference vehicles, and the multiple reference vehicles belong to the same vehicle model as the target vehicle. The target fusion system determines the target number based on the number of historical fusion targets.

[0072] Among them, the maximum number of historical fusion targets refers to the maximum number of fusion targets determined by the reference vehicle without setting an upper limit on the number of fusion targets. The fact that the multiple reference vehicles belong to the same vehicle model as the target vehicle also means that the multiple reference vehicles have the same hardware configuration as the target vehicle.

[0073] In this implementation manner, using the maximum number of historical fusion targets of multiple reference vehicles of the same vehicle model as the target vehicle to determine the target number of the target vehicle can determine an appropriate target number on the premise of ensuring safety. The target number is related to the vehicle model, with high accuracy and strong adaptability.

[0074] For example, the target fusion system obtains the maximum historical fusion target quantity of multiple reference vehicles from the cloud platform. The multiple reference vehicles belong to the same vehicle model as the target vehicle. The target fusion system determines the average quantity of the maximum historical fusion target quantity of the multiple reference vehicles as the target quantity. Alternatively, the target fusion system arranges the maximum historical fusion target quantity of the multiple reference vehicles in ascending order, and determines the maximum historical fusion target quantity at a preset percentile as the target quantity. Among them, the preset percentile is set by the technician according to the actual situation, such as set to the 85th percentile or the 90th percentile. The embodiments of the present application do not limit this. Alternatively, the target fusion system determines the maximum historical fusion target quantity with the most occurrences among the maximum historical fusion target quantities of the multiple reference vehicles as the target quantity.

[0075] In a possible implementation manner, the target fusion system determines the target quantity based on the computing resources and storage resources of the target vehicle.

[0076] Among them, the computing resources of the target vehicle refer to the computing resources that can be invoked during target fusion, which are used for the computing ability of the target vehicle, or in other words, used to reflect the amount of computation that the target vehicle can perform per unit time. The storage resources refer to the storage resources that can be occupied during target fusion. For example, the storage resources refer to the storage space.

[0077] In this implementation manner, the target quantity can be determined according to the actual situation of the target vehicle, and the target quantity is more matched with the target vehicle.

[0078] For example, the target fusion system determines a first quantity corresponding to the computing resources of the target vehicle and a second quantity corresponding to the storage resources. Among them, the first quantity corresponding to the computing resources refers to the maximum quantity of fusion targets that the computing resources can process, and the second quantity corresponding to the storage resources refers to the maximum quantity of fusion targets that the storage resources can store. The target fusion system determines the smaller quantity among the first quantity and the second quantity as the target quantity.

[0079] Of course, in addition to the above implementation manners, the technician can also set the target quantity according to experience. For example, during normal driving, there are not so many targets threatening the host vehicle, and 50 targets can meet the judgment of the environmental situation, so the target quantity can be set to 50.

[0080] It should be noted that the above takes the target fusion system to determine the target quantity as an example for illustration. In other possible implementation manners, the target quantity can also be determined by other devices, such as determined by the technician's terminal or the cloud platform. The embodiments of the present application do not limit this.

[0081] 302. The target fusion system deletes the historical fusion targets among the multiple historical fusion targets whose fusion existence cycle number is greater than or equal to the cycle number threshold.

[0082] Among them, the cycle number threshold is set by the technical personnel according to the actual situation. For example, it is set to 5, and the embodiments of the present application do not limit this.

[0083] Through steps 301 and 302, when the number of multiple historical fusion targets is greater than or equal to the target number, the historical fusion targets among the multiple historical fusion targets whose fusion existence cycle number is greater than or equal to the cycle number threshold can be deleted to delete the stale historical fusion targets and retain the relatively new historical fusion targets.

[0084] It should be noted that the above steps 301 and 302 are optional steps. The target fusion system can either execute the above steps 301 and 302, or not execute the above steps 301 and 302 when the number of multiple historical fusion targets determined in the previous frame is greater than or equal to the target number. The embodiments of the present application do not limit this.

[0085] 303. The target fusion system determines multiple initial sensor targets recognized by multiple sensors of the target vehicle in the current frame.

[0086] Among them, an initial sensor target refers to a target recognized by one sensor of the target vehicle.

[0087] In a possible implementation manner, the target fusion system obtains multiple initial sensor targets recognized by the multiple sensors in the current frame from the target recognition system.

[0088] Among them, the target recognition system is used to obtain the environmental information collected by the multiple sensors and perform target recognition based on the environmental information to obtain the multiple initial sensor targets. In some embodiments, the multiple sensors include a camera, a lidar, and a millimeter-wave radar. The target recognition system can recognize the images collected by the camera to obtain the initial sensor targets corresponding to the camera; the target recognition system can also recognize the point clouds collected by the lidar and the millimeter-wave radar to obtain the initial sensor targets corresponding to the lidar and the millimeter-wave radar.

[0089] In this implementation manner, the multiple initial sensor targets recognized by the multiple sensors in the current frame can be directly obtained through the target recognition system, and the acquisition efficiency of the initial sensor targets is relatively high.

[0090] In a possible implementation, the target fusion system obtains the environmental information collected by the multiple sensors in the current frame. The target fusion system performs target recognition based on the environmental information collected by the multiple sensors in the current frame to obtain the multiple initial sensor targets. One initial sensor target corresponds to one of the multiple sensors, and one sensor can correspond to multiple initial sensor targets.

[0091] In this implementation, the target recognition of the environmental information collected by the multiple sensors is performed by the target fusion system, which can eliminate the need for an additional target recognition system and reduce the cost of the vehicle.

[0092] For example, the target fusion system obtains the environmental information collected by the multiple sensors in the current frame. The target fusion system inputs the environmental information collected by each sensor in the current frame into the target recognition model, extracts the features of the environmental information through the target recognition model to obtain the environmental features of the environmental information. Among them, the target recognition model is trained based on the sample environmental information and the labeled sensor targets corresponding to the sample environmental information, and has the ability to determine sensor targets based on the environmental information. The target fusion system maps the environmental features of the environmental information to obtain multiple initial sensor targets corresponding to the environmental information. Among them, the environmental information collected by one sensor in the current frame corresponds to at least one initial sensor target.

[0093] 304. The target fusion system merges the duplicate targets among the multiple initial sensor targets to obtain the multiple sensor targets. The duplicate target refers to the initial sensor target recognized by at least two sensors in the current frame.

[0094] Among them, the multiple sensor targets are obtained by merging the multiple initial sensor targets. The merging process can merge the same target recognized by different sensors into the same target. The sensor target refers to the target recognized by at least one sensor, and the multiple sensor targets are non-duplicate targets.

[0095] In a possible implementation, the target fusion system obtains the target parameters of the multiple initial sensor targets. The target parameters include motion parameters, position parameters, and target attributes. The target fusion system determines the duplicate targets among the multiple initial sensor targets based on the target parameters of the multiple initial sensor targets. The target fusion system merges the duplicate targets among the multiple initial sensor targets to obtain the multiple sensor targets.

[0096] Among them, the motion parameters are used to reflect the relative motion situation with the target vehicle. For example, the motion parameters include lateral speed, longitudinal speed, lateral acceleration, longitudinal acceleration, etc.; the position parameters are used to reflect the relative position relationship with the target vehicle. For example, the position parameters include lateral distance and longitudinal distance, etc.; the target attributes are used to reflect the characteristics of the target. For example, the target attributes include target length, target width, target shape, etc. It should be noted that in the process of merging at least two initially identified sensor targets into one sensor target, the target parameters collected by at least two sensors are all assigned to this sensor target. For example, the camera identifies the initial sensor target A, and the lidar identifies the initial sensor target B. When the initial sensor target A and the initial sensor target B are the same target, the target fusion system can merge the initial sensor target A and the initial sensor target B into the sensor target M, and assign the target parameters of the initial sensor target A collected by the camera and the target parameters of the initial sensor target B collected by the lidar to the sensor target M.

[0097] In this implementation manner, the target parameters of multiple initially identified sensor targets are obtained, and the duplicate targets are determined from the multiple initially identified sensor targets based on the target parameters. By merging the duplicate targets among the multiple initial sensors, multiple sensor targets can be obtained for subsequent use, with relatively high efficiency.

[0098] For example, the target fusion system obtains the target parameters of the multiple initially identified sensor targets, and the target parameters include motion parameters, position parameters, and target attributes. The target fusion system compares the target parameters of the multiple initially identified sensor targets. The target fusion system determines at least two initially identified sensor targets with a similarity between the target parameters greater than or equal to the similarity threshold as duplicate targets, where the similarity threshold is set by those skilled in the art according to the actual situation, and this application embodiment does not limit it. The target fusion system merges the duplicate targets among the multiple initial sensors to obtain the multiple sensor targets.

[0099] For example, the target fusion system obtains the target parameters of the multiple initially identified sensor targets, and the target parameters include motion parameters, position parameters, and target attributes. The target fusion system compares the motion parameters, position parameters, and target attributes of the initially identified sensor targets identified by different sensors among the multiple initially identified sensor targets. The target fusion system determines at least two initially identified sensor targets with a similarity between the motion parameters, position parameters, and target attributes greater than or equal to the similarity threshold as duplicate targets, and the at least two initially identified sensor targets are the initially identified sensor targets identified by different sensors. The target fusion system merges the duplicate targets among the multiple initial sensors to obtain the multiple sensor targets.

[0100] It should be noted that the above steps 303 and 304 are optional steps. The target fusion system can either execute the above steps 303 and 304, or not execute the above steps 303 and 304 when the number of multiple historical fusion targets determined in the previous frame is greater than or equal to the number of targets. The embodiments of the present application do not limit this.

[0101] 305. The target fusion system determines multiple candidate targets and multiple retained targets from the multiple historical fusion targets and the multiple sensor targets recognized in the current frame.

[0102] Among them, the multiple candidate targets include historical fusion targets and sensor targets to be determined whether to be the fusion targets of the current frame, and the multiple retained targets include historical fusion targets and sensor targets determined to be the fusion targets of the current frame.

[0103] In a possible implementation manner, the target fusion system compares the multiple historical fusion targets and the multiple sensor targets. The target fusion system determines different multiple targets among the multiple historical fusion targets and the multiple sensor targets as the multiple candidate targets. The target fusion system determines the same multiple targets among the multiple historical fusion targets and the multiple sensor targets as the multiple retained targets.

[0104] In this implementation manner, by comparing the multiple historical fusion targets and the multiple sensor targets, candidate targets and retained targets can be screened out, and the determination efficiency of candidate targets and retained targets is relatively high.

[0105] To more clearly illustrate the above implementation manner, the above implementation manner will be described in several parts below.

[0106] The first part: The target fusion system compares the multiple historical fusion targets and the multiple sensor targets.

[0107] Among them, comparing the multiple historical fusion targets and the multiple sensor targets is to determine the same targets and different targets among the multiple historical fusion targets and the multiple sensor targets.

[0108] In a possible implementation manner, the target fusion system compares the target identifiers of the multiple historical fusion targets and the target identifiers of the multiple sensor targets, where one target identifier is used to uniquely identify one target.

[0109] In this implementation manner, by comparing the target identifiers, the comparison of historical fusion targets and sensor targets can be realized, and the efficiency is relatively high.

[0110] Taking the example that the multiple historical fusion targets are stored in a fusion target list and the multiple sensor targets are stored in a sensor target list, the target identification of the multiple historical fusion targets is stored in the fusion target list, and the target identification of the multiple sensor targets is stored in the sensor target list. The target fusion system compares the target identifications stored in the fusion target list and the sensor target list.

[0111] In a possible implementation manner, the target fusion system compares the target parameters of the multiple historical fusion targets and the target parameters of the multiple sensor targets.

[0112] In this implementation manner, by comparing the target parameters, the comparison of the historical fusion targets and the sensor targets can be achieved, and the accuracy is relatively high.

[0113] Taking the example that the multiple historical fusion targets are stored in a fusion target list and the multiple sensor targets are stored in a sensor target list, the target parameters of the multiple historical fusion targets are stored in the fusion target list, and the target parameters of the multiple sensor targets are stored in the sensor target list. The target fusion system compares the target parameters stored in the fusion target list and the sensor target list.

[0114] Second part: The target fusion system determines the multiple different targets among the multiple historical fusion targets and the multiple sensor targets as the multiple candidate targets.

[0115] In a possible implementation manner, the target fusion system determines the multiple targets that exist in the multiple historical fusion targets but do not exist in the multiple sensor targets, and the multiple targets that exist in the multiple sensor targets but do not exist in the multiple historical fusion targets, as the multiple candidate targets.

[0116] Taking the example that the multiple historical fusion targets are stored in a fusion target list and the multiple sensor targets are stored in a sensor target list, the target fusion system determines the multiple targets that exist in the fusion target list but do not exist in the sensor target list, and the multiple targets that exist in the sensor target list but do not exist in the fusion target list, as the multiple candidate targets.

[0117] Third part: The target fusion system determines the multiple identical targets among the multiple historical fusion targets and the multiple sensor targets as the multiple retained targets.

[0118] In a possible implementation manner, the target fusion system determines the multiple targets that exist both in the multiple historical fusion targets and in the multiple sensor targets, as the multiple retained targets.

[0119] Taking the example that the multiple historical fusion targets are stored in the fusion target list and the multiple sensor targets are stored in the sensor target list, the target fusion system determines the multiple targets that exist both in the fusion target list and in the sensor target list as the multiple reserved targets.

[0120] 306. The target fusion system determines the priority of each candidate target based on the target type of the multiple candidate targets, where the target type is used to indicate whether the candidate target is a historical fusion target or a sensor target.

[0121] Among them, the priority of a candidate target is used to indicate the priority degree of being determined as the fusion target of the current frame. The higher the priority of a candidate target, the greater the possibility that the candidate target is determined as the fusion target of the current frame. The lower the priority of a candidate target, the smaller the possibility that the candidate target is determined as the fusion target of the current frame.

[0122] In a possible implementation manner, for a first candidate target whose target type among the multiple candidate targets is a historical fusion target, the target fusion system acquires the fusion parameter, motion parameter, position parameter, and measurement parameter of the first candidate target. The target fusion system determines the priority of the first candidate target based on the fusion parameter, motion parameter, position parameter, and measurement parameter of the first candidate target. For a second candidate target whose target type among the multiple candidate targets is a sensor target, the target fusion system acquires the motion parameter, position parameter, and measurement parameter of the second candidate target. The target fusion system determines the priority of the second candidate target based on the motion parameter, position parameter, and measurement parameter of the second candidate target.

[0123] Among them, the fusion parameter of the first candidate target is used to reflect the fusion type of the first candidate target. The motion parameter of the first candidate target is used to reflect the relative motion state between the first candidate target and the target vehicle. The position parameter of the first candidate target is used to reflect the relative position relationship between the first candidate target and the target vehicle. The measurement parameter of the first candidate target is an auxiliary parameter generated when measuring the first candidate target. Correspondingly, the motion parameter of the second candidate target is used to reflect the relative motion state between the second candidate target and the target vehicle. The position parameter of the second candidate target is used to reflect the relative position relationship between the second candidate target and the target vehicle. The measurement parameter of the second candidate target is an auxiliary parameter generated when measuring the second candidate target.

[0124] In this implementation manner, parameters are acquired according to the type of the candidate target, and the priority is determined according to the acquired parameters, and the accuracy of the determined priority is relatively high.

[0125] To illustrate the above embodiments more clearly, the above embodiments will be described in four parts below.

[0126] Part 1: For the first candidate target among the multiple candidate targets whose target type is a historical fusion target, the target fusion system acquires the fusion parameters, motion parameters, position parameters, and measurement parameters of the first candidate target.

[0127] In some embodiments, the fusion parameters of the first candidate target include a first fusion type and a second fusion type, the motion parameters of the first candidate target include a motion state, a motion direction relative to the target vehicle, and an estimated collision time with the target vehicle, the position parameters of the first candidate target include a lateral distance from the target vehicle and a longitudinal distance from the target vehicle, and the measurement parameters of the first candidate target include a confidence level, a fusion existence period number, and a measurement state. Among them, the first fusion type is used to represent the number of sensors that detect the first candidate target, the second fusion type is used to represent whether there is a target sensor among the sensors that detect the first candidate target, the fusion existence period number is the number of periods in which the first candidate target is determined to be a fusion target, and the measurement state includes a newborn state, a processing state, and a prediction state. The target sensor is a sensor with a relatively high degree of importance, and the target sensor is set by a technician according to the actual situation, and the embodiments of the present application do not limit this.

[0128] The above concepts will be further explained through several examples below.

[0129] In the case where the first candidate target is detected by three sensors (such as a camera, a lidar, and a millimeter-wave radar), then the first fusion type of the first candidate target is the first type, and the first type can be represented by a first numerical value. In the case where the first candidate target is detected by only one sensor (such as a camera), then the first fusion type of the first candidate target is the second type, and correspondingly, the second type can be represented by a second numerical value, and the second numerical value is different from the first numerical value and the first numerical value is greater than the second numerical value.

[0130] In the case where the first candidate target is detected by the target sensor (such as a lidar), then the second fusion type of the first candidate target is the third type, and the third type can be represented by a third numerical value. In the case where the first candidate target is not detected by the target sensor, then the second fusion type of the first candidate target is the fourth type, and the third type can be represented by a fourth numerical value, and the fourth numerical value is different from the third numerical value and the third numerical value is greater than the fourth numerical value.

[0131] The motion state of the first candidate target includes motion and stillness. The fifth value can be used to represent motion, and the sixth value can be used to represent stillness. The estimated time to collision between the first candidate target and the target vehicle refers to TTC (Time To Contact).

[0132] The confidence level of the first candidate target is determined based on the confidence level of the sensor that detected the first candidate target. For example, if two first candidate targets are both generated by millimeter-wave radars, but the confidence level of millimeter-wave radar 1 is 20% and the confidence level of millimeter-wave radar 2 is 90%, then the confidence levels of the first candidate targets formed by them are different, and the confidence level of the first candidate target detected by millimeter-wave radar 2 is higher.

[0133] Alternatively, the confidence level of the first candidate target is determined based on the number of sensors that detected the first candidate target. For example, if a first candidate target is detected by both a millimeter-wave radar and a camera, the confidence level of this first candidate target is higher than that of a first candidate target detected only by a millimeter-wave radar.

[0134] Second, based on the fusion parameters, motion parameters, position parameters, and measurement parameters of the first candidate target, the target fusion system determines the priority of the first candidate target.

[0135] In a possible implementation, the target fusion system performs weighted fusion on the fusion parameters, motion parameters, position parameters, and measurement parameters of the first candidate target to obtain the priority of the first candidate target.

[0136] Among them, the weights of the fusion parameters, motion parameters, position parameters, and measurement parameters are set by technicians according to the actual situation, and the embodiments of the present application do not limit this.

[0137] In this implementation, the priority of the first candidate target can be directly obtained by using the method of weighted summation, and the efficiency of determining the priority of the first candidate target is relatively high.

[0138] For example, the target fusion system performs weighted fusion on the first fusion type, second fusion type, motion state, motion direction relative to the target vehicle, estimated time to collision with the target vehicle, lateral distance from the target vehicle, longitudinal distance from the target vehicle, confidence level, number of fusion existence cycles, and measurement state of the first candidate target to obtain the priority of the first candidate target.

[0139] For example, the priority of the first candidate target = the first fusion type of the first candidate target * weight 1 + the second fusion type * weight 2 + the motion state * weight 3 + the motion direction relative to the target vehicle * weight 4 + the estimated collision time with the target vehicle * weight 5 + the lateral distance from the target vehicle * weight 6 + the longitudinal distance from the target vehicle * weight 7 + the confidence level * weight 8 + the number of fusion existence cycles * weight 9 + the measurement state * weight 10.

[0140] In a possible implementation, the target fusion system inputs the fusion parameters, motion parameters, position parameters, and measurement parameters of the first candidate target into the first priority determination model. The target fusion system extracts features from the fusion parameters, motion parameters, position parameters, and measurement parameters of the first candidate target through the first priority determination model to obtain the first target feature of the first candidate target. The target fusion system maps the first target feature through the first priority determination model to obtain the priority of the first candidate target.

[0141] Among them, the first priority determination model is trained based on sample fusion parameters, sample motion parameters, sample position parameters, sample measurement parameters, and labeled priorities, and has the ability to determine priorities based on fusion parameters, motion parameters, position parameters, and measurement parameters.

[0142] In this implementation, the first priority determination model is used to determine the priority of the first candidate target, and the accuracy of the priority of the first candidate target is relatively high.

[0143] For example, the target fusion system concatenates the fusion parameters, motion parameters, position parameters, and measurement parameters of the first candidate target into the first priority determination parameter. The target fusion system inputs the first priority determination parameter into the first priority determination model. The target fusion system performs at least one fully connected layer or at least one convolution on the first priority determination parameter through the first priority determination model to obtain the first target feature of the first candidate target. The target fusion system performs a fully connected layer and normalization on the first target feature through the first priority determination model to obtain the priority of the first candidate target.

[0144] Part Three: For the second candidate target whose target type among the multiple candidate targets is a sensor target, the target fusion system acquires the motion parameters, position parameters, and measurement parameters of the second candidate target.

[0145] In some embodiments, the motion parameters of the second candidate target include the motion state, the motion direction relative to the target vehicle, and the estimated time to collision with the target vehicle. The position parameters of the second candidate target include the lateral distance from the target vehicle and the longitudinal distance from the target vehicle. The measurement parameters of the second candidate target include the confidence level, the number of sensor presence cycles, and the sensor importance level. Among them, the number of sensor presence cycles is the number of cycles in which the second candidate target is determined as a sensor target. The sensor importance level is set by those skilled in the art according to the actual situation, and the sensor importance level is also referred to as the sensor confidence level, which is not limited in the embodiments of the present application.

[0146] It should be noted that the concepts with the same name in the above description have the same meaning as those in the first part above, and will not be elaborated here.

[0147] Part Four: The target fusion system determines the priority of the second candidate target based on the motion parameters, position parameters, and measurement parameters of the second candidate target.

[0148] In a possible implementation manner, the target fusion system performs weighted fusion on the motion parameters, position parameters, and measurement parameters of the second candidate target to obtain the priority of the second candidate target.

[0149] In this implementation manner, the priority of the second candidate target can be obtained directly by using the method of weighted summation, and the efficiency of determining the priority of the second candidate target is relatively high.

[0150] For example, the target fusion system performs weighted fusion on the motion state of the second candidate target, the motion direction relative to the target vehicle, the estimated time to collision with the target vehicle, the lateral distance from the target vehicle, the longitudinal distance from the target vehicle, the confidence level, the number of sensor presence cycles, and the sensor importance level to obtain the priority of the second candidate target.

[0151] For example, the priority of the second candidate target = motion state * weight A + motion direction relative to the target vehicle * weight B + estimated time to collision with the target vehicle * weight C + lateral distance from the target vehicle * weight D + longitudinal distance from the target vehicle * weight E + confidence level * weight F + number of sensor presence cycles * weight G + sensor importance level * weight H.

[0152] In a possible implementation, the target fusion system inputs the motion parameters, position parameters, and measurement parameters of the second candidate target into a second priority determination model. The target fusion system extracts features from the motion parameters, position parameters, and measurement parameters of the second candidate target through the second priority determination model to obtain the second target feature of the second candidate target. The target fusion system maps the second target feature through the second priority determination model to obtain the priority of the second candidate target.

[0153] Among them, the second priority determination model is trained based on sample motion parameters, sample position parameters, sample measurement parameters, and labeled priorities, and has the ability to determine priorities based on motion parameters, position parameters, and measurement parameters.

[0154] In this implementation, the second priority determination model is used to determine the priority of the second candidate target, and the accuracy of the priority of the second candidate target is relatively high.

[0155] For example, the target fusion system concatenates the motion parameters, position parameters, and measurement parameters of the second candidate target into second priority determination parameters. The target fusion system inputs the second priority determination parameters into the second priority determination model. The target fusion system performs at least one fully connected layer or at least one convolution on the second priority determination parameters through the second priority determination model to obtain the second target feature of the second candidate target. The target fusion system performs a fully connected layer and normalization on the second target feature through the second priority determination model to obtain the priority of the second candidate target.

[0156] 307. The target fusion system determines multiple fusion targets for the current frame based on the priorities of each candidate target, the multiple candidate targets, and the multiple reserved targets. The number of the multiple fusion targets is less than or equal to the number of targets.

[0157] Among them, the multiple fusion targets for the current frame are also the targets that at least one sensor of the target vehicle focuses on in the current frame.

[0158] In a possible implementation, the target fusion system determines multiple reference targets from the multiple candidate targets based on the priorities of each candidate target. The multiple reference targets are candidate targets whose priorities meet the priority conditions, and the number of the multiple reference targets is the difference between the number of targets and the number of the multiple reserved targets. The target fusion system determines the multiple reference targets and the multiple reserved targets as the multiple fusion targets for the current frame.

[0159] In this embodiment, multiple reference targets can be determined from multiple candidate targets based on the priorities of the respective candidate targets, and the multiple reference targets and the multiple reserved targets can be determined as the multiple fusion targets of the current frame, so the efficiency of determining the fusion targets is relatively high.

[0160] Next, the method for determining multiple reference targets in the above embodiment will be described. In a possible embodiment, the target fusion system sorts the multiple candidate targets in descending order of priority. The target fusion system determines the first N candidate targets in the sorted order as the multiple reference targets, where N is a positive integer and is the difference between the number of targets and the number of the multiple reserved targets.

[0161] In this embodiment, candidate targets with higher priorities can be retained, and candidate targets with lower priorities can be deleted, so as to ensure that the number of fusion targets in the current frame does not exceed the number of targets.

[0162] All the above optional technical solutions can be combined arbitrarily to form optional embodiments of the present application, which will not be elaborated one by one here.

[0163] Through the technical solution provided by the embodiment of the present application, when the number of multiple historical fusion targets determined in the previous frame is greater than or equal to the number of targets, multiple candidate targets and multiple reserved targets are determined from the multiple historical fusion targets and the multiple sensor targets recognized in the current frame, and the historical fusion target is a target that was concerned by at least one sensor of the target vehicle in the previous frame. Based on the target types of the multiple candidate targets, the priorities of the respective candidate targets are determined. Based on the priorities of the respective candidate targets, the multiple candidate targets, and the multiple reserved targets, the multiple fusion targets of the current frame are determined, and the number of the multiple fusion targets is less than or equal to the number of targets, thereby reducing the consumption of computing resources and storage resources of the target fusion system, improving the stability of the target fusion system, and thus improving the safety of the target vehicle.

[0164] Figure 4 is a schematic structural diagram of a device for determining fusion targets provided by an embodiment of the present application. Refer to Figure 4 , the device includes: a first target determination module 401, a priority determination module 402, and a second target determination module 403.

[0165] The first target determination module 401 is configured to, when the number of multiple historical fusion targets determined in the previous frame is greater than or equal to the number of targets, determine multiple candidate targets and multiple reserved targets from the multiple historical fusion targets and the multiple sensor targets recognized in the current frame, and the historical fusion target is a target that was concerned by at least one sensor of the target vehicle in the previous frame.

[0166] A priority determination module 402 is configured to determine the priority of each of the multiple candidate targets based on the target type of the multiple candidate targets, where the target type is used to indicate whether the candidate target is a historical fusion target or a sensor target.

[0167] A second target determination module 403 is configured to determine multiple fusion targets of the current frame based on the priorities of each of the multiple candidate targets, the multiple candidate targets, and the multiple retained targets, where the number of the multiple fusion targets is less than or equal to the number of targets.

[0168] In a possible implementation manner, the first target determination module 401 is configured to compare the multiple historical fusion targets and the multiple sensor targets. Determine the multiple different targets among the multiple historical fusion targets and the multiple sensor targets as the multiple candidate targets. Determine the multiple identical targets among the multiple historical fusion targets and the multiple sensor targets as the multiple retained targets.

[0169] In a possible implementation manner, the first target determination module 401 is configured to determine, as the multiple candidate targets, the multiple targets that exist in the multiple historical fusion targets but do not exist in the multiple sensor targets, and the multiple targets that exist in the multiple sensor targets but do not exist in the multiple historical fusion targets.

[0170] In a possible implementation manner, the apparatus further includes a deletion module, configured to determine the number of fusion existence cycles of the multiple historical fusion targets, where the number of fusion existence cycles is the number of cycles determined as fusion targets. Delete the historical fusion targets among the multiple historical fusion targets whose number of fusion existence cycles is greater than or equal to the cycle number threshold.

[0171] In a possible implementation manner, for a first candidate target whose target type among the multiple candidate targets is a historical fusion target, the priority determination module 402 is configured to obtain the fusion parameter, motion parameter, position parameter, and measurement parameter of the first candidate target. Determine the priority of the first candidate target based on the fusion parameter, motion parameter, position parameter, and measurement parameter of the first candidate target. For a second candidate target whose target type among the multiple candidate targets is a sensor target, obtain the motion parameter, position parameter, and measurement parameter of the second candidate target. Determine the priority of the second candidate target based on the motion parameter, position parameter, and measurement parameter of the second candidate target.

[0172] In a possible implementation, the fusion parameters of the first candidate target include a first fusion type and a second fusion type, the motion parameters of the first candidate target include a motion state, a motion direction relative to the target vehicle, and an estimated time to collision with the target vehicle, the position parameters of the first candidate target include a lateral distance from the target vehicle and a longitudinal distance from the target vehicle, and the measurement parameters of the first candidate target include a confidence level, a number of fusion existence cycles, and a measurement state.

[0173] Among them, the first fusion type is used to represent the number of sensors that detected the first candidate target, the second fusion type is used to represent whether there is a target sensor among the sensors that detected the first candidate target, the number of fusion existence cycles is the number of cycles in which the first candidate target is determined to be a fusion target, and the measurement state includes a newborn state, a processing state, and a prediction state.

[0174] In a possible implementation, the motion parameters of the second candidate target include a motion state, a motion direction relative to the target vehicle, and an estimated time to collision with the target vehicle, the position parameters of the second candidate target include a lateral distance from the target vehicle and a longitudinal distance from the target vehicle, and the measurement parameters of the second candidate target include a confidence level, a number of sensor existence cycles, and a sensor importance level.

[0175] Among them, the number of sensor existence cycles is the number of cycles in which the second candidate target is determined to be a sensor target.

[0176] In a possible implementation, the priority determination module 402 is configured to perform any one of the following:

[0177] Perform weighted fusion on the fusion parameters, motion parameters, position parameters, and measurement parameters of the first candidate target to obtain the priority of the first candidate target.

[0178] Input the fusion parameters, motion parameters, position parameters, and measurement parameters of the first candidate target into a first priority determination model. Through the first priority determination model, perform feature extraction on the fusion parameters, motion parameters, position parameters, and measurement parameters of the first candidate target to obtain a first target feature of the first candidate target. Through the first priority determination model, map the first target feature to obtain the priority of the first candidate target.

[0179] The priority determination module 402 is configured to perform any one of the following:

[0180] Perform weighted fusion on the motion parameters, position parameters, and measurement parameters of the second candidate target to obtain the priority of the second candidate target.

[0181] Input the motion parameters, position parameters, and measurement parameters of the second candidate target into the second priority determination model. Through the second priority determination model, extract features from the motion parameters, position parameters, and measurement parameters of the second candidate target to obtain the second target feature of the second candidate target. Through the second priority determination model, map the second target feature to obtain the priority of the second candidate target.

[0182] In a possible implementation manner, the second target determination module 403 is configured to determine a plurality of reference targets from the plurality of candidate targets based on the priorities of the respective candidate targets. The plurality of reference targets are candidate targets whose priorities meet the priority conditions, and the number of the plurality of reference targets is the difference between the number of targets and the number of the plurality of reserved targets. Determine the plurality of reference targets and the plurality of reserved targets as the plurality of fusion targets of the current frame.

[0183] In a possible implementation manner, the second target determination module 403 is configured to sort the plurality of candidate targets in descending order of priority. Determine the first N candidate targets after sorting as the plurality of reference targets, where N is a positive integer and is the difference between the number of targets and the number of the plurality of reserved targets.

[0184] In a possible implementation manner, the device further includes a filtering module configured to determine a plurality of initial sensor targets recognized by a plurality of sensors of the target vehicle in the current frame. Merge the duplicate targets among the plurality of initial sensor targets to obtain the plurality of sensor targets, where the duplicate targets refer to the initial sensor targets recognized by at least two sensors in the current frame.

[0185] In a possible implementation manner, the device further includes a target number determination module configured to perform any one of the following:

[0186] Obtain the maximum historical fusion target number of a plurality of reference vehicles, where the plurality of reference vehicles belong to the same vehicle model as the target vehicle. Determine the target number based on the historical fusion target number.

[0187] Determine the target number based on the computing resources and storage resources of the target vehicle.

[0188] It should be noted that when determining the fusion target, the fusion target determination device provided in the above embodiments only uses the division of the above functional modules as an example. In actual applications, the above functions can be allocated to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the fusion target determination device provided in the above embodiments and the fusion target determination method embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments and will not be repeated here.

[0189] According to the technical solution provided by the embodiments of the present application, when the number of multiple historical fusion targets determined in the previous frame is greater than or equal to the target number, multiple candidate targets and multiple reserved targets are determined from the multiple historical fusion targets and the multiple sensor targets recognized in the current frame, where the historical fusion target is a target that was concerned by at least one sensor of the target vehicle in the previous frame. Based on the target types of the multiple candidate targets, the priorities of the respective candidate targets are determined. Based on the priorities of the respective candidate targets, the multiple candidate targets, and the multiple reserved targets, multiple fusion targets for the current frame are determined, and the number of the multiple fusion targets is less than or equal to the target number, thereby reducing the consumption of computing resources and storage resources of the target fusion system, improving the stability of the target fusion system, and thus improving the safety of the target vehicle.

[0190] Embodiments of the present application also provide an electronic device. Figure 5 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

[0191] Generally, the electronic device 500 includes: one or more processors 501 and one or more memories 502.

[0192] The processor 501 may include one or more processing cores, such as a 4-core processor, a 5-core processor, etc. The processor 501 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 501 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 501 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 501 may also include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0193] The memory 502 may include one or more computer-readable storage media, which may be non-transitory. The memory 502 may also include high-speed random access memory, as well as non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage media in the memory 502 is used to store at least one computer program, and the at least one computer program is used to be executed by the processor 501 to implement the method for determining the fusion target provided in the method embodiments of the present application.

[0194] Those skilled in the art can understand that Figure 5 the structure shown in does not constitute a limitation on the electronic device 500, and it may include more or fewer components than shown in the figure, or combine certain components, or adopt different component arrangements.

[0195] In addition, the device provided in the embodiments of the present application may specifically be a chip, a component or a module. The chip may include a processor and a memory connected thereto; wherein, the memory is used to store instructions, and when the processor calls and executes the instructions, the chip may execute the method for determining a fusion target provided in the above embodiments.

[0196] This embodiment also provides a computer-readable storage medium, in which computer program code is stored. When the computer program code runs on a computer, the computer is caused to execute the above-related method steps to implement the method for determining a fusion target provided in the above embodiments.

[0197] This embodiment also provides a computer program product. When the computer program product runs on a computer, the computer is caused to execute the above-related steps to implement the method for determining a fusion target provided in the above embodiments.

[0198] Among them, the device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be elaborated here.

[0199] Through the description of the above embodiments, those skilled in the art can understand that for the convenience and brevity of description, only the above division of each functional module is used as an example. In actual applications, the above functions may be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0200] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical or other forms.

[0201] The above content is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed in the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for determining a fusion target, characterized in that, The method includes: When the number of multiple historical fusion targets determined in the previous frame is greater than or equal to the number of targets, determining multiple candidate targets and multiple reserved targets from the multiple historical fusion targets and multiple sensor targets recognized in the current frame, where the historical fusion targets are targets that were concerned by at least one sensor of the target vehicle in the previous frame; Based on the target types of the multiple candidate targets, determining the priority of each of the candidate targets, where the target type is used to indicate whether the candidate target is a historical fusion target or a sensor target; Based on the priority of each of the candidate targets, the multiple candidate targets and the multiple reserved targets, determining multiple fusion targets in the current frame, where the number of the multiple fusion targets is less than or equal to the number of targets.

2. The method according to claim 1, wherein The determining multiple candidate targets and multiple reserved targets from the multiple historical fusion targets and multiple sensor targets recognized in the current frame includes: Comparing the multiple historical fusion targets and the multiple sensor targets; Determining multiple different targets among the multiple historical fusion targets and the multiple sensor targets as the multiple candidate targets; Determining multiple identical targets among the multiple historical fusion targets and the multiple sensor targets as the multiple reserved targets.

3. The method according to claim 1, wherein Before the determining multiple candidate targets and multiple reserved targets from the multiple historical fusion targets and multiple sensor targets recognized in the current frame, the method further includes: Determining the number of fusion existence cycles of the multiple historical fusion targets, where the number of fusion existence cycles is the number of cycles determined as fusion targets; Deleting the historical fusion targets among the multiple historical fusion targets whose number of fusion existence cycles is greater than or equal to the cycle number threshold.

4. The method according to claim 1, characterized in that, The determining the priority of each of the candidate targets based on the target types of the multiple candidate targets includes: For a first candidate target among the multiple candidate targets whose target type is a historical fusion target, obtaining the fusion parameters, motion parameters, position parameters and measurement parameters of the first candidate target; based on the fusion parameters, motion parameters, position parameters and measurement parameters of the first candidate target, determining the priority of the first candidate target; For a second candidate target among the multiple candidate targets whose target type is a sensor target, obtaining the motion parameters, position parameters and measurement parameters of the second candidate target; based on the motion parameters, position parameters and measurement parameters of the second candidate target, determining the priority of the second candidate target.

5. The method according to claim 4, wherein The fusion parameters of the first candidate target include a first fusion type and a second fusion type, the motion parameters of the first candidate target include a motion state, a motion direction relative to the target vehicle, and an estimated collision time with the target vehicle, the position parameters of the first candidate target include a lateral distance from the target vehicle and a longitudinal distance from the target vehicle, and the measurement parameters of the first candidate target include a confidence level, a number of fusion existence cycles, and a measurement state; Wherein, the first fusion type is used to represent the number of sensors that detect the first candidate target, the second fusion type is used to represent whether there is a target sensor among the sensors that detect the first candidate target, the fusion existence period number is the number of periods in which the first candidate target is determined to be a fusion target, and the measurement state includes a newborn state, a processing state, and a prediction state.

6. The method according to claim 1, wherein Determining multiple fusion targets of the current frame based on the priorities of the candidate targets, the multiple candidate targets, and the multiple reserved targets includes: Determining multiple reference targets from the multiple candidate targets based on the priorities of the candidate targets, where the multiple reference targets are candidate targets whose priorities meet the priority conditions, and the number of the multiple reference targets is the difference between the target number and the number of the multiple reserved targets; Determining the multiple reference targets and the multiple reserved targets as the multiple fusion targets of the current frame.

7. The method according to claim 1, characterized in that When the number of multiple historical fusion targets determined in the previous frame is greater than or equal to the target number, before determining multiple candidate targets and multiple reserved targets from the multiple historical fusion targets and multiple sensor targets recognized in the current frame, the method further includes: Determining multiple initial sensor targets recognized by multiple sensors of the target vehicle in the current frame; Combining duplicate targets among the multiple initial sensor targets to obtain the multiple sensor targets, where the duplicate targets are initial sensor targets recognized by at least two sensors in the current frame.

8. The method according to claim 1, wherein The method for determining the target number includes any one of the following: Obtaining the maximum historical fusion target number of multiple reference vehicles, where the multiple reference vehicles belong to the same vehicle type as the target vehicle; Determining the target number based on the historical fusion target number; Determining the target number based on the computing resources and storage resources of the target vehicle.

9. An apparatus for determining a fusion target, characterized in that The device includes: A first target determination module, configured to determine multiple candidate targets and multiple reserved targets from the multiple historical fusion targets and multiple sensor targets recognized in the current frame when the number of multiple historical fusion targets determined in the previous frame is greater than or equal to the target number, where the historical fusion targets are targets concerned by at least one sensor of the target vehicle in the previous frame; A priority determination module, configured to determine the priority of each candidate target based on the target type of the multiple candidate targets, where the target type is used to represent whether the candidate target is a historical fusion target or a sensor target; A second target determination module, configured to determine multiple fusion targets of the current frame based on the priorities of the candidate targets, the multiple candidate targets, and the multiple reserved targets, where the number of the multiple fusion targets is less than or equal to the target number.

10. An electronic device, characterized in that, The electronic device includes: A memory, configured to store executable program code; A processor, configured to call and run the executable program code from the memory, so that the electronic device executes the method according to any one of claims 1 to 8.

Images