Multi-camera fusion trajectory processing method and device, electronic equipment and storage medium
By introducing an energy mechanism into the multi-camera fusion system, the energy value of the target camera is established and adjusted, and the multi-camera fusion trajectory is optimized. This solves the problems of trajectory splitting and identification code jumping in multi-camera fusion tracking, and achieves stable multi-camera fusion results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHIDAO NETWORK TECH (BEIJING) CO LTD
- Filing Date
- 2022-11-23
- Publication Date
- 2026-04-17
AI Technical Summary
The lack of effective methods for evaluating and adjusting the performance of multi-camera fusion tracking in existing technologies leads to poor trajectory processing results in multi-camera fusion tracking.
An energy mechanism is used to establish the target camera sub-energy value at the moment of successful matching, and the total energy value of the detected target is calculated based on the sub-energy values at multiple successful matching moments. The trajectory and camera of the multi-camera fusion are adjusted by the change of the total energy value to optimize the tracking process.
It achieves stability and uniqueness of multi-camera fusion trajectory, reduces splitting of fusion results, velocity jumps and universal unique identifier jumps, and improves the effect of multi-camera fusion tracking.
Smart Images
Figure CN115859214B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a multi-camera fusion trajectory processing method, apparatus, electronic device, and storage medium. Background Technology
[0002] In roadside scenarios, multiple cameras mounted on multiple roadside poles are typically used to track vehicles entering the target road segment or intersection. During the tracking process, the fusion results of multiple cameras are used as the final tracking result.
[0003] In related technologies, there is a lack of methods for evaluating or adjusting the tracking effect of multi-camera fusion tracking, which affects the trajectory processing effect of multi-camera fusion tracking. Summary of the Invention
[0004] This application provides a multi-camera fusion trajectory processing method, apparatus, electronic device, and storage medium to optimize the processing effect of roadside multi-camera tracking fusion through an energy mechanism.
[0005] The embodiments of this application adopt the following technical solutions:
[0006] In a first aspect, embodiments of this application provide a trajectory processing method for multi-camera fusion, wherein the method includes:
[0007] Based on the energy mechanism, a sub-energy value of the target camera is established at the moment of successful matching, wherein the moment of successful matching refers to the moment when the existing trajectory after multi-camera fusion is successfully matched with the newly detected target of the target camera.
[0008] The total energy value of the detected target is obtained based on the sub-energy values of the same / different target cameras at multiple successful matching times.
[0009] During the tracking and detection of the target, the trajectory and / or camera of the multi-camera fusion are adjusted according to the change in the total energy value of the target.
[0010] Secondly, embodiments of this application also provide a trajectory processing apparatus for multi-camera fusion, wherein the apparatus includes:
[0011] The sub-energy establishment module is used to establish the sub-energy value of the target camera at the moment of successful matching based on the energy mechanism, wherein the moment of successful matching refers to the moment when the existing trajectory after multi-camera fusion is successfully matched with the newly detected target of the target camera.
[0012] The total energy establishment module is used to obtain the total energy value of the detected target based on the sub-energy values of the same / different target cameras at multiple successful matching times;
[0013] The processing module is used to adjust the trajectory and / or camera of the multi-camera fusion based on the change in the total energy value of the detected target during the tracking and detection process.
[0014] Thirdly, embodiments of this application also provide an electronic device, including: a processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the above-described method.
[0015] Fourthly, embodiments of this application also provide a computer-readable storage medium that stores one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform the above-described method.
[0016] The at least one technical solution adopted in this application embodiment can achieve the following beneficial effects: by establishing the energy value of the target camera, the total energy of the detected target can be obtained during the tracking and detection process, and the trajectory and / or camera of the multi-camera fusion can be adjusted according to the change of the total energy value. By managing the tracking trajectory through the energy value, the trajectory fusion process is optimized in the tracking processing. Attached Figure Description
[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0018] Figure 1 This is a flowchart illustrating the trajectory processing method for multi-camera fusion in an embodiment of this application.
[0019] Figure 2 This is a schematic diagram of the jump threshold in the trajectory processing method of multi-camera fusion in the embodiments of this application;
[0020] Figure 3 This is a schematic diagram of the initial and transition states in the trajectory processing method of multi-camera fusion in the embodiments of this application;
[0021] Figure 4 This is a flowchart illustrating the trajectory processing method for multi-camera fusion in a preferred embodiment of this application.
[0022] Figure 5 This is a schematic diagram of the trajectory processing device for multi-camera fusion in the embodiments of this application;
[0023] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0025] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.
[0026] This application provides a trajectory processing method based on multi-camera fusion, such as... Figure 1 The diagram shows a schematic flowchart of a trajectory processing method for multi-camera fusion in an embodiment of this application. The method includes at least the following steps S110 to S130:
[0027] Step S110: Based on the energy mechanism, establish the sub-energy value of the target camera at the moment of successful matching, wherein the moment of successful matching refers to the moment when the existing trajectory after multi-camera fusion is successfully matched with the newly detected target of the target camera.
[0028] For each frame of a target camera that successfully matches the existing trajectory of the detected target, it is necessary to calculate the sub-energy value of the target camera at this time, and use this sub-energy value as the energy increment of the detected target.
[0029] Based on the actual business needs of roadside scenarios, a mechanism needs to be established to optimize the trajectory of multi-camera fusion, thereby solving problems such as poor perception of one or more cameras in the common field of view during multi-camera fusion, which leads to the overall fusion result splitting, speed jumps, and jumps in the universally unique identification code.
[0030] The energy value of the target camera is established based on the energy mechanism, and the following judgment process is required when determining the target camera:
[0031] Does the existing trajectory fused from the multi-camera fusion match the new detection target using a corresponding camera? If so, the camera corresponding to the successful match will be used as the target camera. After the existing trajectory fused from the multi-camera fusion successfully matches the new detection target, an energy mechanism is introduced. Preferably, the energy value corresponding to this energy mechanism increases by default, and the energy value decays over natural time by default.
[0032] It is understandable that no new targets were detected for the existing trajectory after multi-camera fusion, i.e., the historical trajectory after multi-camera fusion.
[0033] It's important to note that fusion result splitting refers to the splitting of fusion results that occurs during multi-camera detection and localization. For example, what originally belonged to the same vehicle may split into two different vehicles. Speed jump refers to an abnormal speed exceeding the previously tracked speed for the same vehicle during multi-pole tracking. A universally unique identifier (UUID) is a unique identifier for the same vehicle. UUID jump refers to an anomaly occurring when the unique identifier for the same vehicle jumps across poles.
[0034] It is understandable that the above-mentioned energy increase or energy decrease methods can be determined according to the decrease rate and whether there is a new detection target that matches the existing trajectory, or other feasible methods.
[0035] Step S120: Obtain the total energy value of the detected target based on the sub-energy values of the same / different target cameras at the multiple successful matching times.
[0036] The sum of the energy of a target camera at multiple successful matching moments refers to the sum of the energy values of each frame when the target camera successfully matches an existing trajectory.
[0037] The total energy of the detected target is obtained by summing the sub-energy values of multiple target cameras.
[0038] Here, "detection target" refers to the same target object across different / multiple cameras. In other words, through an energy mechanism, the total energy of the detection target is established, and the sub-energy of each target camera is used as the additional energy of the detection target.
[0039] It is understandable that during the initialization phase of tracking and detecting the target, an initial energy value is assigned to the target, and the addition of each target camera thereafter will be used as a new sub-energy value.
[0040] For example, the initial energy value a0 is accumulated based on the sub-energies a1, a2, a3, and a4 of multiple target cameras to obtain the current total energy value A1{a0, a1, a2, a3, a4} of the detected target. Furthermore, as more target cameras are added, the sub-energies are increased to the current total energy value A1 of the detected target.
[0041] Step S130: During the tracking and detection of the target, adjust the trajectory and / or camera of the multi-camera fusion according to the change in the total energy value of the target.
[0042] The changes in energy values include the energy value changes of each target camera and the total energy value of all target cameras (the sum of the energy values of each target camera). Tracking and detection of the target (vehicle) will terminate when the detected target (vehicle) leaves the shared field of view of the cameras.
[0043] It is understood that the total energy value of the target camera refers to the same detection target, specifically, the same detection target being a vehicle passing through multiple poles. During this process, the camera and / or the fused trajectory of multiple cameras are adjusted.
[0044] During the tracking and detection of a target using multi-camera fusion, the trajectory of the multi-camera fusion is adjusted based on the change in the total energy value of the detected target (the default energy value is highest when it is newly added and decays naturally over time). Preferably, adjusting the trajectory of the multi-camera fusion includes, but is not limited to, deleting the trajectory and adjusting the deletion time of the trajectory.
[0045] Furthermore, the cameras in the multi-camera fusion process can be adjusted based on changes in the total energy value of the detected target. Preferably, adjusting the cameras in the multi-camera fusion process includes, but is not limited to, adjusting the camera to be the current master camera in the fusion.
[0046] Building upon current multi-camera fusion tracking, an energy mechanism is introduced to determine the detection status of the target by detecting increases or decreases in the total energy value of the target. Furthermore, the main camera in the current trajectory of the multi-camera fusion is changed or maintained based on the total energy value. Thus, this method optimizes the fused tracking trajectory, achieving a stable and unique fusion result when a vehicle passes through the perception area formed by multiple cameras in a target road segment or intersection.
[0047] Furthermore, since the energy mechanism can control the timing of trajectory deletion by changing the natural decay rate (i.e., the proportion by which the energy value decreases over time), the predicted velocity of the detected target obtained from existing trajectories and historical motion states can be used to predict the location, thereby reducing the impact of missed detections and minimizing the influence on subsequent observation matching.
[0048] In one embodiment of this application, adjusting the trajectory and / or camera of multi-camera fusion according to the change in energy value during the tracking and detection of the target includes: after the existing trajectory fused by the multi-camera fusion is successfully matched with the new detection target, updating the existing trajectory and updating the increment of the total energy value corresponding to the detection target to obtain the increased total energy value; determining whether the increased energy value of a certain target camera is greater than a jump threshold; if the energy value of a certain target camera is greater than the jump threshold, then the main camera is switched to the target camera corresponding to the current energy value.
[0049] In the embodiments of this application, the detection status of the currently tracked target is determined by detecting the increase or decrease of the total energy value of the target. When the energy of a target camera exceeds the jump threshold, the main camera jumps.
[0050] Once the target camera is determined, the following operations can be performed on the camera during the target tracking and detection process:
[0051] After the existing trajectory fused by the multi-camera system successfully matches the newly detected target, the existing trajectory is updated along with the corresponding increment of the total energy value, resulting in an increased total energy value. If the sub-energy value of any camera in the trajectory is greater than the transition threshold, then that camera is considered capable of taking over the role of the main trajectory camera at subsequent moments, thus enabling the transition to the main trajectory camera. Specifically... Figure 2 As shown in the figure, the jump threshold, minimum energy, and energy value at which false detections may occur are illustrated. The horizontal axis represents the tracking time of the target, and the vertical axis represents the total energy value. The total energy value of the detected target decreases from high to low depending on the tracking time.
[0052] For further information, please refer to [link / reference]. Figure 3 This represents different states, including the initial state (single pole) and the transition state (switching between the main pole and the fusion pole, with no time limit). That is, if it is determined whether the increased energy value is greater than the transition threshold, the main camera switches to the camera corresponding to that energy value. It is important to note that the main camera maintaining the current trajectory is changed based on the total energy value of the multiple cameras.
[0053] Preferably, the system determines whether to use the current trajectory's main camera as the main camera for multi-camera fusion trajectory processing based on the energy value's change status. In actual use, the trajectory deletion time is adaptively adjusted according to the observation duration of the detected target. This ensures a stable and unique fusion result for a single vehicle passing through a perception area composed of multiple cameras in a real-time road segment / intersection, while avoiding jumps in position, speed, and universally unique identification codes.
[0054] In one embodiment of this application, adjusting the trajectory and / or camera of multi-camera fusion according to the change of the energy value during the tracking and detection of the target includes: establishing a sub-energy value of the target camera and simultaneously establishing an energy attenuation parameter corresponding to the detected target, wherein the increased total energy value is naturally attenuated based on the energy attenuation parameter to obtain the attenuated total energy value; determining whether the attenuated total energy value is lower than a presence threshold; if it is lower than the presence threshold, deleting the trajectory information collected by the camera corresponding to the sub-energy value in the attenuated total energy value.
[0055] like Figure 2 As shown, the "existence threshold" is the energy value between the jump threshold and the minimum energy.
[0056] In the embodiments of this application, the detection status of the current target is determined by the increase or decrease of the energy value. Once the target camera is determined, the following operations can be performed on the trajectory during the target tracking and detection process:
[0057] By establishing the sub-energy value of the target camera and the corresponding energy attenuation parameter of the detected target simultaneously, it can be understood that at the end of each fusion, a natural energy attenuation according to a preset attenuation rate can be set. Then, it is determined whether the total energy value after attenuation is lower than a presence threshold; if it is lower than the presence threshold, the trajectory information acquired by the camera corresponding to the total energy value after attenuation is deleted. The established energy mechanism can control the trajectory deletion time by changing the natural attenuation rate. When the energy value of the detected target's trajectory attenuates to below the presence threshold, the trajectory is directly deleted.
[0058] It's important to note that the increased total energy value is obtained by natural attenuation based on the aforementioned energy decay parameters. This can be understood as an increase in energy value for each newly added target camera. Therefore, the consideration here is always the increased energy value, which undergoes natural attenuation according to a specific energy decay parameter.
[0059] In one embodiment of this application, the method further includes: matching the existing trajectory with the updated perception data of each camera using a universally unique identifier to obtain a first matching result for cameras whose universally unique identifiers match; maintaining an association matrix between the perception data of the remaining cameras and the existing trajectory for cameras whose universally unique identifiers do not match; obtaining a second matching result between the existing trajectory and the perception data of each camera whose universally unique identifier does not match based on the global matching result of the association matrix; merging the matching results and sorting and matching them according to the universally unique identifier and the perception association distance; updating the existing trajectory based on the matching result, wherein the existing trajectory includes at least one of the following parameter information: detected target position, detected target velocity, and detected target heading angle.
[0060] For an existing trajectory, the matching result with the perception data of each camera is calculated. The matching result is the camera whose UUID matches the detected target. Excluding cameras with matching UUIDs, an association matrix equal to the number of cameras is maintained. For all association matrices, a global optimal match is performed to obtain the matching result between the current trajectory and the perception results of each camera. Then, for cameras whose UUIDs do not match according to the global matching results of the association matrices, the matching results are merged and sorted according to UUID and association distance. The existing trajectory is then updated again based on the matching results, and this process is repeated. Since UUID is a specific manifestation of a universally unique identifier, it indicates that for the same real object (detected target), the universally unique identifier given by the same camera may be the same, while the universally unique identifiers of different cameras will definitely be different. Therefore, fusion and deduplication are necessary.
[0061] Furthermore, when calculating the first matching result, the existing trajectory is matched with the updated perception data of each camera using a universally unique identifier to obtain the first matching result of the camera whose universally unique identifier matches; then, for the camera whose universally unique identifier does not match, the association matrix between the perception data of the remaining camera and the existing trajectory is maintained.
[0062] Then, based on the global matching result of the correlation matrix, a second matching result is obtained for the perception data of cameras whose existing trajectories do not match each of the universally unique identifiers.
[0063] Based on the maintained correlation matrix, the matching results are merged and sorted according to the universally unique identifier and the perceptual correlation distance. Finally, the detected target position, detected target velocity, and detected target heading angle in the existing trajectory are updated based on the matching results. At this time, if a new target is detected, the energy value of the corresponding target camera increases.
[0064] In one embodiment of this application, calculating the perception association distance between the existing trajectory and the perception data of each camera includes: if the existing trajectory successfully matches the observation value of the newly detected target, then the newly detected target and the target tracked in the existing trajectory are the same object; if the existing trajectory does not match the observation value of the newly detected target, then the currently maintained trajectory information has failed to find a current perception result corresponding to the newly detected target; if the observation value of the newly detected target does not match the multi-camera perception result, then the corresponding trajectory information has not been found in the current perception result.
[0065] While maintaining the correlation matrix, the perceptual correlation distance between the existing trajectory and the perceptual data of each camera also needs to be calculated as follows:
[0066] First, if the existing trajectory successfully matches the observation value of the newly detected target, then the newly detected target and the target tracked in the existing trajectory are the same object. Specifically, if the trajectory and observation successfully match, it means that the currently maintained trajectory and the latest acquired perception result have successfully corresponded, indicating that the macroscopic distance between the two, such as position, velocity, and type, is small, and they are considered to belong to the same real object.
[0067] It can be understood that the observed values here are used as the location estimation values for the detected target, and are input into the Kalman filter to estimate the location of the detected target.
[0068] Secondly, if the existing trajectory does not match the observation value of the new detection target, then the currently maintained trajectory information has failed to find a current perception result corresponding to the new detection target.
[0069] Specifically, if no matching trajectory is found, it means that the currently maintained trajectory has not found a corresponding latest perception result, which may be due to missed detection or the object moving out of the perception range.
[0070] Finally, if the observed value of the newly detected target does not match the multi-camera perception result, then no corresponding trajectory information can be found in the current perception result. Specifically, a mismatched perception result means that the latest (current) perception result fails to capture the trajectory corresponding to the threshold, possibly because it is a new, real object that has just entered the perception range, or because the current perception has a false detection, such as a split detection result.
[0071] In one embodiment of this application, the method further includes: if the existing trajectory does not match the observation value of the newly detected target, predicting the location information of the newly detected target based on historical trajectory information; if the observation value of the newly detected target does not match the multi-camera sensing result, creating new trajectory information; and predicting the location information of the detected target at the current moment based on the existing trajectory.
[0072] According to the matching method described above, if the existing trajectory does not match the observation value of the newly detected target, the location information of the new target is predicted based on historical trajectory information. For example, the observation value can be input into a Kalman filter for prediction. Since the existing trajectory cannot be matched, the location information of the new target is predicted based on historical trajectory information.
[0073] If the observed value of the new detected target does not match the multi-camera sensing result, new trajectory information is created, that is, new trajectory information is created for the new detected target. Since the observed value of the new detected target does not match the multi-camera sensing result, trajectory information for the new detected target is created.
[0074] In one embodiment of this application, establishing the sub-energy value of the target camera and the total energy attenuation parameter corresponding to the detected target, wherein the target camera includes the camera corresponding to the successful matching of the existing trajectory after multi-camera fusion with the new detected target, and the new detected target is the first detected target to enter the target camera, includes: when the existing trajectory after multi-camera fusion successfully matches the new detected target, increasing the energy value of the corresponding newly added camera.
[0075]
[0076] The energy represents the correlation distance between the existing trajectory and the newly detected target.
[0077] The energy of the association distance between the existing trajectory and the new detected target includes, but is not limited to, the matching cost calculated from one or more of the following factors: spatial distance, pixel distance, intersection-over-union ratio, detection box shape, and motion state. It should be noted that the embodiments of this application do not specifically limit which parameter(s) are used; one or more parameters can be used to characterize the energy of the association distance between the existing trajectory and the new detected target. Alternatively, one or more parameters can be used according to a preset weight / proportion to characterize the energy of the association distance between the existing trajectory and the new detected target.
[0078] Set the attenuation rate in the energy attenuation parameter to α, e i =(1-α)e i , i = 1, 2, ..., n.
[0079] Similarly, it can be understood that the natural decay rate α can also decay in various forms such as exponential, logarithmic, power function, and reciprocal. In the embodiments of this application, no specific limitation is made. Those skilled in the art can configure or select according to the actual situation.
[0080] In one embodiment of this application, before establishing the energy value and energy attenuation parameter of the target camera based on the energy mechanism, the method further includes: when receiving the perception results of the multiple cameras, creating new trajectory information and initial energy value for each perception result; when a new perception result is input, predicting the position and velocity of the target based on the existing trajectory according to a preset interval time; matching according to the universally unique identifier of the perception result, and storing the successfully matched perception result in the trajectory as the observation value to be fused, wherein the universally unique identifier is the same for the same real object and the same camera, and the universally unique identifier is different for different cameras.
[0081] The identification of universally unique identifiers enables this method to generate globally unique universally unique identifiers.
[0082] During initialization, such as when the first sensing information is input (not when a new detection target appears), a new trajectory is created for each sensing result. At this time, an initial energy value is attached. That is, when the sensing results from the multiple cameras are received, new trajectory information and an initial energy value are created for each sensing result.
[0083] In subsequent tracking and detection, each time sensing information is input, the existing trajectory is first analyzed using a Kalman filter and a motion model based on the time difference to predict the target position and velocity. That is, matching is performed based on the universally unique identifier of the sensing results, and the successfully matched sensing results are stored in the trajectory as observations to be fused.
[0084] Furthermore, the universally unique identifier in the camera's perception results can be used for matching, and the matched perception results can be stored in the trajectory as observations to be fused.
[0085] After the above initialization steps, it is also necessary to calculate the correlation distance between the trajectory and the perception results of each camera. Then, the obtained matching results are merged and sorted according to trajectory ID and perception correlation distance.
[0086] Furthermore, the trajectory is updated based on the matching results, increasing the energy. After the energy increase, the main camera is adjusted based on the energy value during tracking and detection. Also, due to the natural decay of the energy value, trajectories below the energy threshold can be deleted.
[0087] Preferably, for trajectories that fail to match, the location of the new target is predicted.
[0088] Preferably, for perception results that fail to match, a new trajectory is created for the new detection target.
[0089] To better understand the implementation methods in this application, such as Figure 4 The diagram shown is a flowchart of a trajectory processing method for multi-camera fusion in a preferred embodiment of this application, which specifically includes the following implementation steps:
[0090] Step S410: Predict the current position of the existing trajectory. This step is repeated cyclically after creating a new trajectory based on the sensing results of unsuccessful matches.
[0091] Step S420: Perform UUID matching between the existing trajectory and the latest perception data from each camera.
[0092] Step S430: Except for cameras whose UUIDs are successfully matched, maintain the correlation matrix between the sensor observation results of all cameras and the currently existing trajectories.
[0093] Step S440: Perform global optimal matching on all correlation matrices to obtain the matching results between the current trajectory and the perception results of each camera.
[0094] Step S450: Merge the obtained matching results and sort and match them according to trajectory ID and perception association distance.
[0095] Specifically, the existing trajectory is matched with the updated perception data of each camera using a universally unique identifier to obtain the first matching result of the camera with the universally unique identifier.
[0096] Step S460: Update the existing trajectory based on the final matching result, including updates to position, velocity, heading angle, and corresponding increase in energy value.
[0097] For cameras whose universally unique identifiers do not match, maintain the association matrix between the perception data of the remaining cameras and the existing trajectory; based on the global matching result of the association matrix, obtain a second matching result between the existing trajectory and the perception data of each camera whose universally unique identifier does not match; merge the matching results and sort and match them according to the universally unique identifier and the perception association distance; update the existing trajectory according to the matching result, wherein the existing trajectory includes at least one of the following parameter information: detected target position, detected target velocity, and detected target heading angle.
[0098] Step S470: Determine whether the updated energy value is higher than the jump threshold, and change the trajectory main camera accordingly.
[0099] After the existing trajectory fused by the multi-camera system is successfully matched with the new detected target, the existing trajectory is updated and the corresponding increment of the total energy value is updated to obtain the increased total energy value.
[0100] Determine whether the increased total energy value is greater than the jump threshold;
[0101] If the value is greater than that, the main camera will switch to the camera corresponding to that sub-energy value.
[0102] In step S480, the total energy value naturally decays.
[0103] Step S490: Determine whether the total energy value after decay is below the existence threshold, and delete the trajectory.
[0104] While establishing the sub-energy value of the target camera, a corresponding energy attenuation parameter is also established, wherein the increased total energy value is obtained by natural attenuation based on the energy attenuation parameter.
[0105] Determine whether the total energy value after attenuation is lower than the existence threshold;
[0106] If the value is lower, the trajectory information captured by the camera corresponding to the attenuated total energy value will be deleted.
[0107] Step S4100: For unsuccessfully matched trajectories, make predictions based on historical information.
[0108] Step S4110: For the perception results that did not match successfully, create a new trajectory for tracking.
[0109] If the existing trajectory does not match the observation value of the new detection target, the location information of the new detection target is predicted based on the historical trajectory information;
[0110] If the observed value of the newly detected target does not match the multi-camera sensing results, then new trajectory information is created; and
[0111] Based on the existing trajectory, predict the location information of the target at the current moment.
[0112] The above method enables adaptive adjustment of trajectory deletion time based on the observed duration, and allows for lifecycle management of trajectories based on energy values without the need to introduce additional parameters.
[0113] This application embodiment also provides a trajectory processing device 500 for multi-camera fusion, such as... Figure 5 The diagram shows a schematic representation of a trajectory processing device for multi-camera fusion in an embodiment of this application. The trajectory processing device 500 for multi-camera fusion includes at least: a sub-energy establishment module 510, a total energy establishment module 520, and a processing module 530, wherein:
[0114] In one embodiment of this application, the sub-energy establishment module 510 is specifically used to: establish the sub-energy value of the target camera at the moment of successful matching based on the energy mechanism, wherein the moment of successful matching refers to the moment when the existing trajectory after multi-camera fusion is successfully matched with the new detection target of the target camera.
[0115] For each frame of a target camera that successfully matches the existing trajectory of the detected target, it is necessary to calculate the sub-energy value of the target camera at this time, and use this sub-energy value as the energy increment of the detected target.
[0116] Based on the actual business needs of roadside scenarios, a mechanism needs to be established to optimize the trajectory of multi-camera fusion, thereby solving problems such as poor perception of one or more cameras in the common field of view during multi-camera fusion, which leads to the overall fusion result splitting, speed jumps, and jumps in the universally unique identification code.
[0117] The energy value of the target camera is established based on the energy mechanism, and the following judgment process is required when determining the target camera:
[0118] Does the existing trajectory from the multi-camera fusion match the new detection target using a corresponding camera? If so, the camera that matches successfully will be used as the target camera. After the existing trajectory from the multi-camera fusion successfully matches the new detection target, an energy mechanism is introduced. Preferably, the energy value corresponding to this energy mechanism increases by default, and the energy value decays over time by default.
[0119] It is understandable that no new targets were detected for the existing trajectory after multi-camera fusion, i.e., the historical trajectory after multi-camera fusion.
[0120] It's important to note that fusion result splitting refers to the splitting of fusion results that occurs during multi-camera detection and localization. For example, what originally belonged to the same vehicle may split into two different vehicles. Speed jump refers to an abnormal speed exceeding the previously tracked speed for the same vehicle during multi-pole tracking. Unique identifier jump refers to an anomaly occurring when the unique identifier for the same vehicle crosses a pole.
[0121] It is understandable that the above-mentioned energy increase or energy decrease methods can be determined according to the decrease rate and whether there is a new detection target that matches the existing trajectory, or other feasible methods.
[0122] In one embodiment of this application, the total energy establishment module 520 is specifically used to: obtain the total energy value of the detected target based on the sub-energy values of the plurality of target cameras.
[0123] The sum of the energy of a target camera at multiple successful matching moments refers to the sum of the energy values of each frame when the target camera successfully matches an existing trajectory.
[0124] The total energy of the detected target is obtained by summing the sub-energy values of multiple target cameras.
[0125] Here, "detection target" refers to the same target object across different / multiple cameras. In other words, through an energy mechanism, the total energy of the detection target is established, and the sub-energy of each target camera is used as the additional energy of the detection target.
[0126] It is understandable that during the initialization phase of tracking and detecting the target, an initial energy value is assigned to the target, and the addition of each target camera thereafter will be used as a new sub-energy value.
[0127] For example, the initial energy value a0 is accumulated based on the sub-energies a1, a2, a3, and a4 of multiple target cameras to obtain the current total energy value A1{a0, a1, a2, a3, a4} of the detected target. Furthermore, as more target cameras are added, the sub-energies are increased to the current total energy value A1 of the detected target.
[0128] In one embodiment of this application, the processing module 530 is specifically used to: adjust the trajectory and / or camera of multi-camera fusion according to the change in the total energy value of the detected target during the tracking and detection of the target.
[0129] The changes in energy values include the energy value changes of each target camera and the total energy value of all target cameras (the sum of the energy values of each target camera). Tracking and detection of the target (vehicle) will terminate when the detected target (vehicle) leaves the shared field of view of the cameras.
[0130] It is understood that the total energy value of the target camera refers to the same detection target, specifically, the same detection target being a vehicle passing through multiple poles. During this process, the camera and / or the fused trajectory of multiple cameras are adjusted.
[0131] During the tracking and detection of a target using multi-camera fusion, the trajectory of the multi-camera fusion is adjusted based on the change in the total energy value of the detected target (the default energy value is highest when it is newly added and decays naturally over time). Preferably, adjusting the trajectory of the multi-camera fusion includes, but is not limited to, deleting the trajectory and adjusting the deletion time of the trajectory.
[0132] Furthermore, the cameras in the multi-camera fusion process can be adjusted based on changes in the total energy value of the detected target. Preferably, adjusting the cameras in the multi-camera fusion process includes, but is not limited to, adjusting the camera to be the current master camera in the fusion.
[0133] Building upon current multi-camera fusion tracking, an energy mechanism is introduced to determine the detection status of the target by detecting increases or decreases in the total energy value of the target. Furthermore, the main camera in the current trajectory of the multi-camera fusion is changed or maintained based on the total energy value. Thus, this method optimizes the fused tracking trajectory, achieving a stable and unique fusion result when a vehicle passes through the perception area formed by multiple cameras in a target road segment or intersection.
[0134] Furthermore, since the energy mechanism can control the timing of trajectory deletion by changing the natural decay rate (i.e., the proportion by which the energy value decreases over time), the predicted velocity of the detected target obtained from existing trajectories and historical motion states can be used to predict the location, thereby reducing the impact of missed detections and minimizing the influence on subsequent observation matching.
[0135] It is understood that the above-mentioned multi-camera fusion trajectory processing device can implement each step of the multi-camera fusion trajectory processing method provided in the foregoing embodiments. The relevant explanations of the multi-camera fusion trajectory processing method are applicable to the multi-camera fusion trajectory processing device, and will not be repeated here.
[0136] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 6 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.
[0137] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.
[0138] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.
[0139] The processor reads the corresponding computer program from non-volatile memory into main memory and then executes it, forming a multi-camera fusion trajectory processing device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:
[0140] Based on the energy mechanism, a sub-energy value of the target camera is established at the moment of successful matching, wherein the moment of successful matching refers to the moment when the existing trajectory after multi-camera fusion is successfully matched with the newly detected target of the target camera.
[0141] The total energy value of the detected target is obtained based on the sub-energy values of the same / different target cameras at multiple successful matching times.
[0142] During the tracking and detection of the target, the trajectory and / or camera of the multi-camera fusion are adjusted according to the change in the total energy value of the target.
[0143] The above is as stated in this application. Figure 1The method executed by the multi-camera fusion trajectory processing device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.
[0144] The electronic device can also perform Figure 1 A method for executing a trajectory processing device for multi-camera fusion, and implementation of the trajectory processing device for multi-camera fusion in... Figure 1 The functions of the embodiments shown are not described in detail here.
[0145] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform... Figure 1 The method executed by the multi-camera fusion trajectory processing device in the illustrated embodiment is specifically used to perform:
[0146] Based on the energy mechanism, a sub-energy value of the target camera is established at the moment of successful matching, wherein the moment of successful matching refers to the moment when the existing trajectory after multi-camera fusion is successfully matched with the newly detected target of the target camera.
[0147] The total energy value of the detected target is obtained based on the sub-energy values of the same / different target cameras at multiple successful matching times.
[0148] During the tracking and detection of the target, the trajectory and / or camera of the multi-camera fusion are adjusted according to the change in the total energy value of the target.
[0149] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0150] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0151] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0152] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0153] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0154] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0155] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0156] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0157] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0158] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.
Claims
1. A trajectory processing method for multi-camera fusion, wherein, The method includes: Based on the energy mechanism, a sub-energy value of the target camera is established at the moment of successful matching, wherein the moment of successful matching refers to the moment when the existing trajectory after multi-camera fusion is successfully matched with the newly detected target of the target camera. The total energy value of the detected target is obtained based on the sub-energy values of the same / different target cameras at multiple successful matching times. During the tracking and detection of the target, the trajectory and / or camera of the multi-camera fusion are adjusted according to the change in the total energy value of the target. The step of adjusting the trajectory and / or camera of the multi-camera fusion system based on the change in the total energy value of the detected target during the tracking and detection process includes: After the existing trajectory fused by the multi-camera system is successfully matched with the new target detected by the target camera, the existing trajectory is updated and the increment of the total energy value corresponding to the detected target is updated to obtain the increased total energy value. Determine whether the energy value of a target camera after the increase is greater than the jump threshold; If the energy value of a target camera is greater than the jump threshold, the main camera switches to the target camera corresponding to the current energy value.
2. The method of claim 1, wherein, The step of adjusting the trajectory and / or camera of the multi-camera fusion system based on the change in the total energy value of the detected target during the tracking and detection process includes: While establishing the sub-energy value of the target camera, an energy attenuation parameter corresponding to the detected target is also established, wherein the increased total energy value is naturally attenuated based on the energy attenuation parameter to obtain the attenuated total energy value; Determine whether the total energy value of the detected target after attenuation is lower than the presence threshold; If the value is below the existence threshold, the trajectory information of the detected target will be deleted.
3. The method of claim 1, wherein, The method further includes: The existing trajectory is matched with the perception data of a certain frame of each target camera after the update using a universally unique identifier to obtain the first matching result of the camera with the universally unique identifier. For cameras whose universal unique identifiers do not match, maintain the correlation matrix between the perception data of the remaining cameras and the existing trajectories; Based on the global matching result of the correlation matrix, a second matching result is obtained for the perception data of cameras whose existing trajectories do not match each of the universally unique identifiers; The matching results are merged and sorted according to the universal unique identifier and the perceptual association distance. The existing trajectory is updated based on the matching result, wherein the existing trajectory includes at least one of the following parameter information: the detected target position, the detected target velocity, and the detected target heading angle.
4. The method of claim 3, wherein, Calculating the perceptual association distance between the existing trajectory and the perceptual data of each camera includes: If the existing trajectory successfully matches the observation value of the newly detected target, then the newly detected target and the target tracked in the existing trajectory are the same object; If the existing trajectory does not match the observation value of the new detection target, then the currently maintained trajectory information has failed to find a current perception result corresponding to the new detection target; If the observed value of the newly detected target does not match the multi-camera perception result, then no corresponding trajectory information can be found in the current perception result.
5. The method of claim 3, wherein, The method further includes: If the existing trajectory does not match the observation value of the new detection target, the location information of the new detection target is predicted based on the historical trajectory information. If the observed value of the newly detected target does not match the multi-camera sensing result, then new trajectory information is created; and Based on the existing trajectory, predict the location information of the target at the current moment.
6. The method of claim 1, wherein, The sub-energy value of the target camera at the moment of successful matching is established based on the energy mechanism. The moment of successful matching refers to the moment when the existing trajectory after multi-camera fusion successfully matches the newly detected target of the target camera, including: When the existing trajectory fused from the multi-camera system successfully matches the newly detected target, the energy value of the corresponding newly added camera is increased.
7. The energy representing the correlation distance between the existing trajectory and the newly detected target; setting an attenuation rate in the energy attenuation parameter to , 。 8. The method of claim 1, wherein, The method of establishing the sub-energy value of the target camera at the moment of successful matching based on the energy mechanism, wherein the moment of successful matching refers to the time before the existing trajectory after multi-camera fusion successfully matches the newly detected target of the target camera, also includes: When the perception results from the multiple cameras are received, new trajectory information and an initial energy value are created for each perception result; When new sensing results are input, the position and velocity of the tracking target are predicted based on the existing trajectory according to a preset time interval. Matching is performed based on the universally unique identifier of the perception results, and the successfully matched perception results are stored in the trajectory as observations to be fused.
9. A multi-camera fusion trajectory processing apparatus, wherein, The device includes: The sub-energy establishment module is used to establish the sub-energy value of the target camera at the moment of successful matching based on the energy mechanism, wherein the moment of successful matching refers to the moment when the existing trajectory after multi-camera fusion is successfully matched with the newly detected target of the target camera. The total energy establishment module is used to obtain the total energy value of the detected target based on the sub-energy values of the same / different target cameras at multiple successful matching times; The processing module is used to adjust the trajectory and / or camera of the multi-camera fusion according to the change in the total energy value of the detected target during the tracking and detection process; The step of adjusting the trajectory and / or camera of the multi-camera fusion system based on the change in the total energy value of the detected target during the tracking and detection process includes: After the existing trajectory fused by the multi-camera system is successfully matched with the new target detected by the target camera, the existing trajectory is updated and the increment of the total energy value corresponding to the detected target is updated to obtain the increased total energy value. Determine whether the energy value of a target camera after the increase is greater than the jump threshold; If the energy value of a target camera is greater than the jump threshold, the main camera switches to the target camera corresponding to the current energy value.
10. An electronic device, comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the method of any one of claims 1 to 7.
11. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of applications, cause the electronic device to perform the method of any one of claims 1 to 7.