Target fusion methods, apparatus and electronic devices, and computer-readable storage media

Abstract

This application discloses a target fusion method, apparatus, electronic device, and computer-readable storage medium. The method, executed by the cloud, includes: acquiring target detection results reported by the roadside camera, the target detection results including the absolute position of the target detection box and the target's heading angle; determining the target's orientation type relative to the roadside camera based on the orientation of the roadside camera and the target's heading angle; determining the absolute position of the target center based on the absolute position of the target detection box and the target's orientation type relative to the roadside camera; and determining the target fusion result based on the absolute position of the target center. This application, based on the 2D target detection results from the roadside camera, further combines data such as the orientation of the roadside camera to determine the target's orientation type relative to the roadside camera and estimates the absolute position of the target center. This allows for the fusion of target detection results from different roadside cameras, significantly improving the accuracy of target fusion across multiple roadside cameras.

Description

Technical Field

[0001] This application relates to the field of target fusion technology, and in particular to a target fusion method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Currently, in the field of vehicle-road cooperation, multiple road poles are generally installed at intersections and road sections. Each road pole is equipped with roadside cameras facing the direction of travel in the same lane and the direction of travel in the opposite lane, which are used to collect road images with different field of view and different travel directions, thereby realizing road monitoring of the entire road section.

[0003] In intersections or road sections between poles, there are overlapping fields of view from multiple roadside cameras. Based on 2D object detection algorithms, each roadside camera in the overlapping area will detect a 2D bounding box for the same target. Multiple roadside cameras will generate multiple 2D bounding boxes. Because 2D object detection algorithms are used, some roadside cameras may detect the front of a vehicle, while others may detect the rear. After projecting the bottom center or center of the bounding box onto the world coordinate system, the same target will be projected with the target detection results from multiple roadside cameras. Therefore, the cloud needs to fuse the target detection results from multiple roadside cameras to ensure that only one location result is output for each target.

[0004] However, due to errors in 2D target detection results and calibration errors, the target detection results between multiple roadside cameras often deviate significantly. Furthermore, an excessively large fusion range may result in two different targets being merged into one, thereby reducing the accuracy of the fusion results. Summary of the Invention

[0005] This application provides a target fusion method, apparatus, electronic device, and computer-readable storage medium to improve the accuracy of target fusion.

[0006] The embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, embodiments of this application provide a target fusion method, the method being executed in the cloud, wherein the method includes:

[0008] Obtain the target detection results reported by the roadside, the target detection results including the absolute position of the target detection box and the heading angle of the target;

[0009] Based on the orientation of the roadside camera corresponding to the road end and the heading angle of the target, determine the orientation type of the target relative to the roadside camera;

[0010] The absolute position of the target center is determined based on the absolute position of the target detection box and the orientation type of the target relative to the roadside camera;

[0011] The target fusion result is determined based on the absolute position of the target center.

[0012] Optionally, determining the orientation type of the target relative to the roadside camera based on the orientation of the roadside camera corresponding to the road end and the heading angle of the target includes:

[0013] Determine the relative angular deviation between the orientation of the roadside camera and the heading angle of the target;

[0014] If the relative angle deviation between the orientation of the roadside camera and the heading angle of the target meets the first angle deviation requirement, then the orientation type of the target relative to the roadside camera is determined to be the target facing the roadside camera.

[0015] If the relative angle deviation between the orientation of the roadside camera and the heading angle of the target meets the second angle deviation requirement, then the orientation type of the target relative to the roadside camera is determined to be the target facing away from the roadside camera.

[0016] Otherwise, the orientation type of the target relative to the roadside camera is determined to be the target side facing the roadside camera.

[0017] Optionally, the absolute position of the target detection box is the absolute position of the center of its bottom edge. Determining the absolute position of the target center based on the absolute position of the target detection box and the orientation type of the target relative to the roadside camera includes:

[0018] Based on the orientation type of the target relative to the roadside camera and the target type, a preset offset strategy is determined corresponding to the absolute position of the bottom center of the target detection box. The preset offset strategy includes the offset direction and the offset distance.

[0019] Based on the preset offset strategy corresponding to the absolute position of the bottom center of the target detection box, the absolute position of the bottom center of the target detection box is offset to obtain the absolute position of the target center.

[0020] Optionally, the orientation type of the target relative to the roadside camera includes the target facing the roadside camera from the front, the target facing the roadside camera from the back, and the target facing the roadside camera from the side. The preset offset strategy for determining the absolute position of the bottom center of the target detection box based on the orientation type of the target relative to the roadside camera and the target type includes:

[0021] If the orientation of the target relative to the roadside camera is either the front of the target facing the roadside camera or the back of the target facing the roadside camera, then based on the target type, a first preset offset strategy corresponding to the absolute position of the bottom center of the target detection box is determined.

[0022] If the orientation type of the target relative to the roadside camera is that the side of the target faces the roadside camera, then according to the target type, a second preset offset strategy corresponding to the absolute position of the bottom center of the target detection box is determined.

[0023] Optionally, determining the target fusion result based on the absolute position of the target center includes:

[0024] Based on the absolute position of each target center, determine the target area corresponding to the absolute position of each target center;

[0025] Determine the intersection-union ratio of the target regions corresponding to the absolute positions of any two target centers;

[0026] The target fusion result is determined based on the intersection-union ratio of the target regions corresponding to the absolute positions of any two target centers.

[0027] Optionally, determining the target fusion result based on the intersection-union ratio of the target regions corresponding to the absolute positions of any two target centers includes:

[0028] Compare the intersection-union ratio of the target regions corresponding to the absolute positions of any two target centers with a preset intersection-union ratio threshold.

[0029] If the intersection-union ratio (IU) of the target regions corresponding to the absolute positions of any two target centers is greater than the preset IU threshold, then the targets corresponding to the absolute positions of any two target centers are determined to be the same target.

[0030] Otherwise, the targets corresponding to the absolute positions of any two target centers are determined to be different targets.

[0031] Secondly, embodiments of this application also provide a target fusion apparatus, the apparatus being applied in the cloud, wherein the apparatus includes:

[0032] The acquisition unit is used to acquire the target detection results reported by the roadside, the target detection results including the absolute position of the target detection box and the heading angle of the target;

[0033] The first determining unit is used to determine the orientation type of the target relative to the roadside camera based on the orientation of the roadside camera corresponding to the road end and the heading angle of the target.

[0034] The second determining unit is used to determine the absolute position of the target center based on the absolute position of the target detection box and the orientation type of the target relative to the roadside camera;

[0035] The third determining unit is used to determine the target fusion result based on the absolute position of the target center.

[0036] Thirdly, embodiments of this application also provide a target fusion system, the system comprising a cloud and a road end, wherein the cloud is used to execute any of the methods described above, and the road end is used to execute:

[0037] Obtain multi-frame target detection results corresponding to the roadside camera;

[0038] The target tracking results of the roadside camera are obtained by using a preset tracking and matching algorithm to track and match the target detection results of multiple frames.

[0039] The heading angle of the target is determined based on the target tracking results from the roadside camera.

[0040] Optionally, the road end is also used to perform:

[0041] Obtain the transformation relationship between the roadside camera coordinate system and the world coordinate system;

[0042] Based on the transformation relationship between the roadside camera coordinate system and the world coordinate system, the absolute position of the target detection box is transformed to the world coordinate system to obtain the position of the target in the world coordinate system.

[0043] Fourthly, embodiments of this application also provide an electronic device, including:

[0044] Processor; and

[0045] A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform any of the methods described above.

[0046] Fifthly, embodiments of this application also provide a computer-readable storage medium that stores one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform any of the methods described above.

[0047] The above-mentioned at least one technical solution adopted in the embodiments of this application can achieve the following beneficial effects: The target fusion method of the embodiments of this application is executed by the cloud. First, the target detection results reported by the roadside are obtained. The target detection results include the absolute position of the target detection box and the heading angle of the target. Based on the orientation of the roadside camera corresponding to the roadside and the heading angle of the target, the orientation type of the target relative to the roadside camera is determined. Based on the absolute position of the target detection box and the orientation type of the target relative to the roadside camera, the absolute position of the target center is determined. Based on the absolute position of the target center, the target fusion result is determined. The target fusion method of the embodiments of this application, based on the 2D target detection results of the roadside, further combines the orientation data of the roadside camera to determine the orientation type of the target relative to the roadside camera and estimates the center position of the target. In this way, the target detection results of different roadside cameras are fused, which greatly improves the accuracy of target fusion of multiple roadside cameras. Attached Figure Description

[0048] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0049] Figure 1 This is a flowchart illustrating a target fusion method in an embodiment of this application;

[0050] Figure 2 This is a schematic diagram of the structure of a target fusion device according to an embodiment of this application;

[0051] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0054] This application provides a target fusion method, which is executed by the cloud, such as... Figure 1 The diagram shows a flowchart of a target fusion method according to an embodiment of this application. The method includes at least the following steps S110 to S140:

[0055] Step S110: Obtain the target detection results reported by the roadside. The target detection results include the absolute position of the target detection box and the heading angle of the target.

[0056] The target fusion method in this application embodiment is executed by the cloud. Based on the communication connection between the cloud and the roadside, the target detection results reported by each roadside can be obtained first. The target detection results mainly include the absolute position of the target detection box and the corresponding heading angle of the target. The absolute position of the target detection box refers to the position of the target detection box in the world coordinate system, i.e., the WGS-84 (World Geodetic System 1984) geodetic coordinate system. Of course, it can also include the unique identifier of the roadside camera corresponding to the roadside, so as to facilitate the cloud to distinguish the detection results of different roadsides and the relationship between different roadsides. The types of detected targets can include, for example, vehicles, pedestrians and other targets on the road.

[0057] Step S120: Determine the orientation type of the target relative to the roadside camera based on the orientation of the roadside camera corresponding to the road end and the heading angle of the target.

[0058] The heading angle of a target reflects its orientation in the world coordinate system, while the orientation of the roadside camera reflects its shooting angle or shooting direction in the world coordinate system. Therefore, by comparing the heading angle of the target with the orientation of the roadside camera, the orientation of the target relative to the roadside camera can be determined, that is, the orientation of the target as seen from the perspective of the roadside camera. This orientation can include, for example, the target facing the roadside camera from the front, the target facing the roadside camera from the back, and the target facing the roadside camera from the side.

[0059] Step S130: Determine the absolute position of the target center based on the absolute position of the target detection box and the orientation type of the target relative to the roadside camera.

[0060] Since the same target may be oriented differently relative to different roadside cameras, different roadside cameras will see targets facing different directions. For example, some roadside cameras only see the front of the car, some only see the rear of the car, and some only see the side of the car. Therefore, after projecting the 2D target detection boxes corresponding to these different roadside cameras onto the world coordinate system, the absolute position of the target in the world coordinate system will also be different. For example, some are the absolute positions corresponding to the front of the car, some are the absolute positions corresponding to the rear of the car, and some are the absolute positions corresponding to the side of the car.

[0061] Based on this, the embodiments of this application need to combine the absolute position of the target detection box and the orientation type of the target relative to the roadside camera to estimate the absolute position of the center of the entire target, so as to provide an accurate fusion basis for subsequent target fusion of multiple roadside cameras.

[0062] Step S140: Determine the target fusion result based on the absolute position of the target center.

[0063] For the same target, the absolute position of the target center estimated based on the target detection results reported by each roadside camera may not completely correspond to the same position, i.e. there is a certain degree of deviation. Therefore, the embodiments of this application can adopt a certain fusion strategy to fuse the absolute positions of the target centers corresponding to the views of multiple roadside cameras, thereby obtaining the final target fusion result and ensuring that the same target corresponds to only one final position.

[0064] The target fusion method in this application embodiment is based on the 2D target detection results from the roadside camera. It further combines data such as the orientation of the roadside camera to determine the target's orientation type relative to the roadside camera and estimates the target's center position. This allows for the fusion of target detection results from different roadside cameras, significantly improving the accuracy of target fusion across multiple roadside cameras. Furthermore, this application embodiment does not require additional detection of specific categories such as vehicle front, rear, or body. The effect of 3D target detection can be achieved by appropriately processing the existing 2D target detection results, resulting in low implementation cost.

[0065] In some embodiments of this application, determining the orientation type of the target relative to the roadside camera based on the orientation of the roadside camera corresponding to the road end and the heading angle of the target includes: determining the relative angular deviation between the orientation of the roadside camera and the heading angle of the target; if the relative angular deviation between the orientation of the roadside camera and the heading angle of the target meets a first angular deviation requirement, then the orientation type of the target relative to the roadside camera is determined to be the target facing the roadside camera directly; if the relative angular deviation between the orientation of the roadside camera and the heading angle of the target meets a second angular deviation requirement, then the orientation type of the target relative to the roadside camera is determined to be the target facing away from the roadside camera; otherwise, the orientation type of the target relative to the roadside camera is determined to be the target facing the roadside camera from the side.

[0066] When determining the orientation of a target relative to a roadside camera, the orientation of the roadside camera can be compared with the target's heading angle to determine the relative angular deviation. If the relative angular deviation between the roadside camera's orientation angle and the target's heading angle meets the first angular deviation requirement, for example, a relative angular deviation close to 180 degrees, it means that from the roadside camera's perspective, the front of the target is facing the roadside camera. If the relative angular deviation between the roadside camera's orientation angle and the target's heading angle meets the second angular deviation requirement, for example, a relative angular deviation close to 0 degrees, it means that from the roadside camera's perspective, the back of the target is facing the roadside camera.

[0067] If the relative angle deviation between the roadside camera's orientation angle and the target's heading angle does not meet either the first angle deviation requirement or the second angle deviation requirement, it means that from the roadside camera's perspective, the target's side is facing the roadside camera.

[0068] In some embodiments of this application, the absolute position of the target detection box is the absolute position of the center of its bottom edge. Determining the absolute position of the target center based on the absolute position of the target detection box and the orientation type of the target relative to the roadside camera includes: determining a preset offset strategy corresponding to the absolute position of the center of the bottom edge of the target detection box based on the orientation type of the target relative to the roadside camera and the target type, wherein the preset offset strategy includes an offset direction and an offset distance; and performing offset processing on the absolute position of the center of the bottom edge of the target detection box according to the preset offset strategy corresponding to the absolute position of the center of the bottom edge of the target detection box to obtain the absolute position of the target center.

[0069] The 2D target detection algorithm on the roadside can directly obtain the position of the bottom center of the target detection box in the roadside camera coordinate system. When it is projected into the world coordinate system, since the same target has different orientations relative to different roadside camera views, multiple different target detection boxes corresponding to the same target will have multiple absolute positions after being projected into the world coordinate system. Moreover, the differences between multiple absolute positions are also large. Therefore, a certain strategy is needed to compensate for this difference.

[0070] The embodiments of this application predefine a preset offset strategy. The core of the preset offset strategy is to compensate the absolute position of the target detection box corresponding to different roadside camera views to the actual center position of the target. For example, it may include compensation for the offset direction and offset distance of the absolute position of the target detection box. In this way, the influence of the deviation between the absolute positions of multiple target detection boxes corresponding to the same target on target fusion can be greatly reduced, thereby improving the accuracy of target fusion.

[0071] Since the orientation of the target relative to the roadside camera varies (e.g., the target is facing the front, back, or side of the roadside camera), and the specific target type varies (e.g., vehicle or pedestrian), the setting of the preset offset strategy will be affected. Therefore, in this embodiment, the preset offset strategy corresponding to the absolute position of the target detection box can be determined first based on the orientation of the target relative to the roadside camera and the specific target type.

[0072] In some embodiments of this application, the orientation type of the target relative to the roadside camera includes the target facing the roadside camera head-on, the target facing the roadside camera from the back, and the target facing the roadside camera from the side. The step of determining the preset offset strategy corresponding to the absolute position of the bottom center of the target detection frame based on the orientation type of the target relative to the roadside camera and the target type includes: if the orientation type of the target relative to the roadside camera is the target facing the roadside camera head-on or the target facing the roadside camera from the back, then a first preset offset strategy corresponding to the absolute position of the bottom center of the target detection frame is determined based on the target type; if the orientation type of the target relative to the roadside camera is the target facing the roadside camera from the side, then a second preset offset strategy corresponding to the absolute position of the bottom center of the target detection frame is determined based on the target type.

[0073] Taking a vehicle target as an example, if the vehicle's orientation relative to the roadside camera is either front-end or rear-end, it means that from the roadside camera's perspective, the absolute position of the bottom center of the target detection box differs from the target's center by approximately half a vehicle length. If the vehicle's orientation relative to the roadside camera is side-end, it means that from the roadside camera's perspective, the absolute position of the bottom center of the target detection box differs from the target's center by approximately half a vehicle width. Therefore, it can be seen that the deviation between the absolute position of the bottom center of the target detection box and the absolute position of the target's center varies depending on the target's orientation relative to the roadside camera. Thus, the influence of the target's orientation relative to the roadside camera can be considered when setting the preset offset strategy.

[0074] Of course, different target types will also affect the specific settings of the above-mentioned preset offset strategies. For example, there is a significant difference between the size of a vehicle and the size of a pedestrian, and there is also a significant difference between the size of a sedan and the size of a bus. Therefore, the absolute position of the bottom edge center of the corresponding target detection box will also be different from the absolute position of the target center. Thus, different offset strategies can be adopted in combination with the specific target type.

[0075] The preset offset strategy in this application mainly includes setting the offset distance and offset direction. For setting the offset distance, the influence of the target's orientation relative to the roadside camera and the specific target type must be considered simultaneously. Of course, those skilled in the art can flexibly adjust the specific setting of the offset distance according to actual needs, and no specific limitation is made here.

[0076] Regarding the setting of the offset direction, from the perspective of the roadside camera, regardless of the orientation type of the target relative to the roadside camera, or the specific target type, the absolute position of the bottom center of the target detection box is always closer to the roadside camera than the absolute position of the target center. Therefore, the orientation type of the target relative to the roadside camera and the specific target type do not have a significant impact on the setting of the offset direction. The offset direction can be uniformly set to move towards the orientation direction of the roadside camera, that is, move towards the absolute position closer to the target center.

[0077] To make it easier to understand, let's take a further example. For instance, for a bus target facing the roadside camera, the absolute position of its bottom center can be moved 1.5 meters (about half the length of the bus) in the direction the roadside camera is facing. For a bus target facing the side of the roadside camera, the absolute position of its bottom center can be moved 1.3 meters (about half the width of the bus) in the direction the roadside camera is facing, and so on, so as to estimate the absolute position of the target's center in the world coordinate system.

[0078] In some embodiments of this application, determining the target fusion result based on the absolute position of the target center includes: determining the target region corresponding to the absolute position of each target center based on the absolute position of each target center; determining the intersection-union ratio of the target regions corresponding to the absolute positions of any two target centers; and determining the target fusion result based on the intersection-union ratio of the target regions corresponding to the absolute positions of any two target centers.

[0079] For the same target, the absolute position of the target center estimated based on the target detection results reported by each road end may not completely correspond to the same position, i.e., there is a certain degree of deviation. Based on this, the embodiments of this application can first use the absolute position of each target center as a benchmark, and assign the target the corresponding length and width according to the length and width dimensions corresponding to the target type, thereby obtaining the target area corresponding to the absolute position of each target center.

[0080] Next, the intersection-union ratio (CIU) of the target regions corresponding to the absolute positions of any two target centers is calculated. The size of the CIU reflects the size of the overlap between the two target regions. If they are the same target, the overlap should be large enough; conversely, if they are different targets, the overlap should be small enough. Based on this, targets corresponding to the absolute positions of different target centers can be merged to obtain the final target fusion result. The final target fusion result includes the final position of each merged target. If the same target corresponds to multiple target centers with absolute positions, the average of the absolute positions of these multiple target centers can be used as the final position of the target in the world coordinate system.

[0081] In some embodiments of this application, determining the target fusion result based on the intersection-union ratio (IU) of the target regions corresponding to the absolute positions of any two target centers includes: comparing the IU of the target regions corresponding to the absolute positions of any two target centers with a preset IU threshold; if the IU of the target regions corresponding to the absolute positions of any two target centers is greater than the preset IU threshold, then the targets corresponding to the absolute positions of any two target centers are determined to be the same target; otherwise, the targets corresponding to the absolute positions of any two target centers are determined to be different targets.

[0082] When determining the target fusion result based on the intersection-union ratio (CIRR) of the target regions corresponding to the absolute positions of any two target centers, a pre-set CIRR threshold can be used for measurement. If the CIRR of the target regions corresponding to the absolute positions of the two target centers is greater than a certain CIRR threshold, the targets corresponding to the absolute positions of the two target centers can be considered to be the same target; otherwise, they are different targets. The specific threshold value can be flexibly set by those skilled in the art according to actual needs, and is not specifically limited here.

[0083] This application embodiment also provides a target fusion device 200, which is applied in the cloud, such as... Figure 2 The diagram shows a schematic representation of a target fusion device according to an embodiment of this application. The device 200 includes: an acquisition unit 210, a first determination unit 220, a second determination unit 230, and a third determination unit 240, wherein:

[0084] The acquisition unit 210 is used to acquire the target detection results reported by the roadside, the target detection results including the absolute position of the target detection box and the heading angle of the target;

[0085] The first determining unit 220 is used to determine the orientation type of the target relative to the roadside camera based on the orientation of the roadside camera corresponding to the road end and the heading angle of the target.

[0086] The second determining unit 230 is used to determine the absolute position of the target center based on the absolute position of the target detection box and the orientation type of the target relative to the roadside camera.

[0087] The third determining unit 240 is used to determine the target fusion result based on the absolute position of the target center.

[0088] In some embodiments of this application, the first determining unit 220 is specifically used to: determine the relative angle deviation between the orientation of the roadside camera and the heading angle of the target; if the relative angle deviation between the orientation of the roadside camera and the heading angle of the target meets a first angle deviation requirement, then determine that the orientation type of the target relative to the roadside camera is the target facing the roadside camera; if the relative angle deviation between the orientation of the roadside camera and the heading angle of the target meets a second angle deviation requirement, then determine that the orientation type of the target relative to the roadside camera is the target facing away from the roadside camera; otherwise, determine that the orientation type of the target relative to the roadside camera is the target facing the roadside camera from the side.

[0089] In some embodiments of this application, the absolute position of the target detection box is the absolute position of the center of the bottom edge of the target detection box. The second determining unit 230 is specifically used to: determine a preset offset strategy corresponding to the absolute position of the center of the bottom edge of the target detection box according to the orientation type of the target relative to the roadside camera and the target type, wherein the preset offset strategy includes an offset direction and an offset distance; and perform offset processing on the absolute position of the center of the bottom edge of the target detection box according to the preset offset strategy corresponding to the absolute position of the center of the bottom edge of the target detection box to obtain the absolute position of the target center.

[0090] In some embodiments of this application, the orientation type of the target relative to the roadside camera includes the target facing the roadside camera from the front, the target facing the roadside camera from the back, and the target facing the roadside camera from the side. The second determining unit 230 is specifically used to: if the orientation type of the target relative to the roadside camera is the target facing the roadside camera from the front or the target facing the roadside camera from the back, then determine a first preset offset strategy corresponding to the absolute position of the bottom edge center of the target detection frame according to the target type; if the orientation type of the target relative to the roadside camera is the target facing the roadside camera from the side, then determine a second preset offset strategy corresponding to the absolute position of the bottom edge center of the target detection frame according to the target type.

[0091] In some embodiments of this application, the third determining unit 240 is specifically used to: determine the target region corresponding to the absolute position of each target center based on the absolute position of each target center; determine the intersection-union ratio of the target regions corresponding to the absolute positions of any two target centers; and determine the target fusion result based on the intersection-union ratio of the target regions corresponding to the absolute positions of any two target centers.

[0092] In some embodiments of this application, the third determining unit 240 is specifically used to: compare the intersection-union ratio (IUR) of the target regions corresponding to the absolute positions of any two target centers with a preset IUR threshold; if the IUR of the target regions corresponding to the absolute positions of any two target centers is greater than the preset IUR threshold, then determine that the targets corresponding to the absolute positions of any two target centers are the same target; otherwise, determine that the targets corresponding to the absolute positions of any two target centers are different targets.

[0093] It is understood that the above-mentioned target fusion device can realize each step of the target fusion method in the foregoing embodiments. The relevant explanations of the target fusion method are applicable to the target fusion device and will not be repeated here.

[0094] This application embodiment also provides a target fusion system, the system including a cloud and a roadside, wherein the cloud is used to execute the aforementioned method, and the roadside is used to execute: acquiring multi-frame target detection results corresponding to a roadside camera; using a preset tracking matching algorithm to track and match the multi-frame target detection results to obtain the target tracking result of the roadside camera; and determining the heading angle of the target based on the target tracking result of the roadside camera.

[0095] When performing target detection at the roadside, the roadside camera can first acquire the road image captured by the current roadside camera, and then use a certain 2D target detection algorithm to perform 2D target detection on the road image. For example, the 2D target detection algorithm can be trained using the existing YOLO series network. Of course, those skilled in the art can flexibly choose which algorithm to use for 2D target detection in combination with existing technology, and no specific limitation is made here.

[0096] After obtaining the target detection result corresponding to the current roadside camera, the heading angle of the target can be further determined based on the target detection results of multiple consecutive frames of the current roadside camera. Using certain target tracking and matching algorithms, such as optical flow tracking algorithm and particle filtering algorithm, the target detection results of multiple consecutive frames can be tracked and matched, so as to obtain the position of the same target in multiple consecutive frames of images. Then, based on the transformation relationship between the roadside camera and the world coordinate system, the absolute position of the same target in the world coordinate system of multiple consecutive frames can be obtained.

[0097] Although calculating the heading angle only requires the absolute position of the target in two frames of data, considering that the motion change of the target between adjacent frames may be small, making it impossible to accurately calculate the heading angle, this embodiment of the application can select the target detection result that is a certain number of frames away from the current frame from the target detection results of historical multi-frame targets to calculate the heading angle of the target in the current frame. Of course, those skilled in the art can determine the specific calculation method in conjunction with existing technology, and it will not be elaborated here.

[0098] For example, if the current frame is the 10th frame, the target position detected in the 2nd frame can be selected from the historical multi-frame target detection results to calculate the target heading angle corresponding to the 10th frame. If the current frame is the 11th frame, the target position detected in the 3rd frame can be selected from the historical multi-frame target detection results to calculate the target heading angle corresponding to the 11th frame. And so on, that is, the target detection results are selected at certain frame intervals to calculate the heading angle to ensure the accuracy of the heading angle calculation.

[0099] In some embodiments of this application, the roadside device is further configured to perform: obtaining the transformation relationship between the roadside camera coordinate system and the world coordinate system; and transforming the absolute position of the target detection box to the world coordinate system according to the transformation relationship between the roadside camera coordinate system and the world coordinate system, thereby obtaining the position of the target in the world coordinate system.

[0100] Since the target detection box obtained directly from the roadside camera is located in the roadside camera coordinate system, and in order to ensure the accuracy of fusion, the target detection results from the roadside camera are fused in the world coordinate system in the cloud, the roadside camera can first transform the position of the target detection box in the roadside camera coordinate system to the world coordinate system based on the pre-calibrated transformation relationship between the roadside camera coordinate system and the world coordinate system, so as to obtain the absolute position of the target detection box in the world coordinate system. Finally, the target's heading angle and the unique identifier of the roadside camera are sent to the cloud for processing.

[0101] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 3 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0102] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0103] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0104] The processor reads the corresponding computer program from non-volatile memory into main memory and then executes it, forming the target fusion device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0105] Obtain the target detection results reported by the roadside, the target detection results including the absolute position of the target detection box and the heading angle of the target;

[0106] Based on the orientation of the roadside camera corresponding to the road end and the heading angle of the target, determine the orientation type of the target relative to the roadside camera;

[0107] The absolute position of the target center is determined based on the absolute position of the target detection box and the orientation type of the target relative to the roadside camera;

[0108] The target fusion result is determined based on the absolute position of the target center.

[0109] The above is as stated in this application. Figure 1The method executed by the target fusion device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0110] The electronic device can also perform Figure 1 The method for executing the target fusion device, and realizing the target fusion device in Figure 1 The functions of the embodiments shown are not described in detail here.

[0111] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform... Figure 1 The method executed by the target fusion device in the illustrated embodiment is specifically used to perform:

[0112] Obtain the target detection results reported by the roadside, the target detection results including the absolute position of the target detection box and the heading angle of the target;

[0113] Based on the orientation of the roadside camera corresponding to the road end and the heading angle of the target, determine the orientation type of the target relative to the roadside camera;

[0114] The absolute position of the target center is determined based on the absolute position of the target detection box and the orientation type of the target relative to the roadside camera;

[0115] The target fusion result is determined based on the absolute position of the target center.

[0116] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0117] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0118] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0119] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0120] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0121] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0122] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0123] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0124] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A target fusion method, wherein the method is executed in the cloud, The method includes: Obtain the target detection results reported by the roadside, the target detection results including the absolute position of the target detection box and the heading angle of the target; Based on the orientation of the roadside camera corresponding to the road end and the heading angle of the target, determine the orientation type of the target relative to the roadside camera; The absolute position of the target center is determined based on the absolute position of the target detection box and the orientation type of the target relative to the roadside camera; The target fusion result is determined based on the absolute position of the target center; The absolute position of the target detection box is the absolute position of the center of its bottom edge. Determining the absolute position of the target center based on the absolute position of the target detection box and the orientation type of the target relative to the roadside camera includes: Based on the orientation type of the target relative to the roadside camera and the target type, a preset offset strategy is determined corresponding to the absolute position of the bottom center of the target detection box. The preset offset strategy includes the offset direction and the offset distance. Based on the preset offset strategy corresponding to the absolute position of the bottom edge center of the target detection box, the absolute position of the bottom edge center of the target detection box is offset to obtain the absolute position of the target center. The orientation type of the target relative to the roadside camera includes the target facing the roadside camera from the front, the target facing the roadside camera from the back, and the target facing the roadside camera from the side. The preset offset strategy for determining the absolute position of the bottom center of the target detection box based on the orientation type of the target relative to the roadside camera and the target type includes: If the orientation of the target relative to the roadside camera is either the front of the target facing the roadside camera or the back of the target facing the roadside camera, then based on the target type, a first preset offset strategy corresponding to the absolute position of the bottom center of the target detection box is determined. If the orientation type of the target relative to the roadside camera is that the side of the target faces the roadside camera, then according to the target type, a second preset offset strategy corresponding to the absolute position of the bottom center of the target detection box is determined.

2. The method as described in claim 1, wherein, Determining the orientation type of the target relative to the roadside camera based on the orientation of the roadside camera corresponding to the road end and the heading angle of the target includes: Determine the relative angular deviation between the orientation of the roadside camera and the heading angle of the target; If the relative angle deviation between the orientation of the roadside camera and the heading angle of the target meets the first angle deviation requirement, then the orientation type of the target relative to the roadside camera is determined to be the target facing the roadside camera. If the relative angle deviation between the orientation of the roadside camera and the heading angle of the target meets the second angle deviation requirement, then the orientation type of the target relative to the roadside camera is determined to be the target facing away from the roadside camera. Otherwise, the orientation type of the target relative to the roadside camera is determined to be the target side facing the roadside camera.

3. The method as described in claim 1, wherein, Determining the target fusion result based on the absolute position of the target center includes: Based on the absolute position of each target center, determine the target area corresponding to the absolute position of each target center; Determine the intersection-union ratio of the target regions corresponding to the absolute positions of any two target centers; The target fusion result is determined based on the intersection-union ratio of the target regions corresponding to the absolute positions of any two target centers.

4. The method as described in claim 3, wherein, Determining the target fusion result based on the intersection-union ratio of the target regions corresponding to the absolute positions of any two target centers includes: Compare the intersection-union ratio of the target regions corresponding to the absolute positions of any two target centers with a preset intersection-union ratio threshold. If the intersection-union ratio (IU) of the target regions corresponding to the absolute positions of any two target centers is greater than the preset IU threshold, then the targets corresponding to the absolute positions of any two target centers are determined to be the same target. Otherwise, the targets corresponding to the absolute positions of any two target centers are determined to be different targets.

5. A target fusion device, wherein the device is applied in the cloud, The device includes: The acquisition unit is used to acquire the target detection results reported by the roadside, the target detection results including the absolute position of the target detection box and the heading angle of the target; The first determining unit is used to determine the orientation type of the target relative to the roadside camera based on the orientation of the roadside camera corresponding to the road end and the heading angle of the target. The second determining unit is used to determine the absolute position of the target center based on the absolute position of the target detection box and the orientation type of the target relative to the roadside camera; The third determining unit is used to determine the target fusion result based on the absolute position of the target center; The absolute position of the target detection box is the absolute position of the center of the bottom edge of the target detection box, and the second determining unit is specifically used for: Based on the orientation type of the target relative to the roadside camera and the target type, a preset offset strategy is determined corresponding to the absolute position of the bottom center of the target detection box. The preset offset strategy includes the offset direction and the offset distance. Based on the preset offset strategy corresponding to the absolute position of the bottom edge center of the target detection box, the absolute position of the bottom edge center of the target detection box is offset to obtain the absolute position of the target center. The orientation type of the target relative to the roadside camera includes the target facing the roadside camera directly, the target facing away from the roadside camera, and the target facing the side of the roadside camera. The second determining unit is specifically used for: If the orientation of the target relative to the roadside camera is either the front of the target facing the roadside camera or the back of the target facing the roadside camera, then based on the target type, a first preset offset strategy corresponding to the absolute position of the bottom center of the target detection box is determined. If the orientation type of the target relative to the roadside camera is that the side of the target faces the roadside camera, then according to the target type, a second preset offset strategy corresponding to the absolute position of the bottom center of the target detection box is determined.

6. A target fusion system, the system comprising a cloud and a roadside, wherein, The cloud platform is used to execute the method according to any one of claims 1 to 4, and the routing terminal is used to execute: Obtain multi-frame target detection results corresponding to the roadside camera; The target tracking results of the roadside camera are obtained by using a preset tracking and matching algorithm to track and match the target detection results of multiple frames. The heading angle of the target is determined based on the target tracking results from the roadside camera.

7. The system of claim 6, wherein, The road end is also used to perform: Obtain the transformation relationship between the roadside camera coordinate system and the world coordinate system; Based on the transformation relationship between the roadside camera coordinate system and the world coordinate system, the absolute position of the target detection box is transformed to the world coordinate system to obtain the position of the target in the world coordinate system.

8. An electronic device, comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the method of any one of claims 1 to 4.

9. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of applications, cause the electronic device to perform the method of any one of claims 1 to 4.

Images