Target fusion method, apparatus, electronic device and storage medium

Abstract

This disclosure provides a target fusion method, apparatus, electronic device, and storage medium. The target fusion method of this disclosure includes: acquiring the positioning information of a first target using a vehicle's millimeter-wave radar, and acquiring the positioning information of a second target using a vehicle's vision sensor; determining the pose of the second target based on its positioning information, and determining a matching region for the second target based on its pose and positioning information; calculating the similarity between the second target and a first target located within its matching region based on the positioning information of the first target, and determining a first target that matches the second target based on the similarity; correcting the pose of the second target based on the positioning information of the first target that matches the second target; determining the observation information of a third target based on the corrected pose of the second target and the positioning information of the first target that matches it, wherein the third target is a fusion target of the second target and the first target that matches it; and filtering the observation information of the third target to obtain the fusion information of the third target. This disclosure can effectively reduce mismatches and missed matches between millimeter-wave radar targets and vision sensor targets.

Description

Technical Field

[0001] This disclosure relates to a target fusion method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, in scenarios such as autonomous driving and assisted driving, accurate target detection is mainly achieved through the fusion of onboard visual sensors and radar sensors.

[0003] In related technologies, the target fusion methods of visual sensors and radar sensors mainly include the following approaches: 1) Distance matching in the world coordinate system according to the nearest neighbor principle, as disclosed in Chinese patents with publication numbers CN 115546597, CN115082712, and CN115184917; 2) IOU matching in the image coordinate system by projecting the radar target onto the image, as disclosed in Chinese patents with publication numbers CN114724110 and CN115097441; 3. Other methods, such as velocity similarity disclosed in Chinese patent CN115372958 and trajectory similarity disclosed in Chinese patent CN 115423845.

[0004] Because millimeter-wave radar outputs target information as a sparse point cloud, each point represents a target, and each point can only describe a certain part of the actual target. For example, for large targets such as vehicles, the vehicle's location points in the sparse point cloud output by millimeter-wave radar can only describe the location of a certain part of the vehicle, which could be a part of the rear, body, or front. Furthermore, the vehicle's location points cannot accurately describe their specific location within the vehicle, which part of the vehicle they belong to, nor can they describe the vehicle's pose, size, or other information. Therefore, the location point information output by millimeter-wave radar is insufficient to completely describe the information of the entire target.

[0005] Figure 1 An example scene diagram showing the detection results of millimeter-wave radar is provided. Figure 1 In the image, the quadrilateral detection box is a visual representation of the target detection results from the vehicle-mounted camera, while the set of white dots that traverse the image and pass through the rear of the vehicle is a visual representation of the target detection results from the millimeter-wave radar. Figure 1 In traditional radar systems, for vehicles within the detection frame, the point cloud output by millimeter-wave radar contains several points on the vehicle. These points can only represent a portion of the vehicle and cannot accurately describe which part of the vehicle they represent, their location within the vehicle, or information such as the vehicle's dimensions and heading angle. Related matching methods do not consider these uncertainties and limitations of millimeter-wave radar positioning points, easily leading to mismatches and missed matches between targets, resulting in increased errors in target positioning and poor positioning accuracy.

[0006] During target localization, only by knowing the location of the target's location point can the target location point be transformed to a fixed position in the vehicle coordinate system (e.g., the center or rear center of the target vehicle). Otherwise, when the target location point changes significantly (e.g., the location point was at the front of the vehicle at one moment and at the rear of the vehicle at the next moment), the aforementioned uncertainties and deficiencies of the millimeter-wave radar location point will further increase the error of the target localization result. Summary of the Invention

[0007] To address at least one of the aforementioned technical problems, this disclosure provides a target fusion method, apparatus, electronic device, and storage medium.

[0008] According to a first aspect of this disclosure, a target fusion method is provided, comprising:

[0009] The vehicle's millimeter-wave radar acquires the location information of the first target, and the vehicle's vision sensor acquires the location information of the second target.

[0010] The pose of the second target is determined based on the positioning information of the second target, and the matching area of ​​the second target is determined based on the pose and positioning information of the second target.

[0011] Calculate the similarity between the second target and the first target located in its matching area based on the location information of the first target, and determine the first target matched by the second target based on the similarity.

[0012] The pose of the second target is corrected based on the positioning information of the first target matched with the second target;

[0013] The observation information of the third target is determined based on the corrected pose of the second target and the positioning information of the first target it matches. The third target is the fusion target of the second target and the first target it matches.

[0014] The observation information of the third target is filtered to obtain the fused information of the third target.

[0015] In some embodiments of this disclosure, the positioning information of the second target includes the vertex coordinates of the second target in the image coordinate system of the visual sensor; determining the pose of the second target based on the positioning information of the second target includes: determining the coordinates of the ground vertex of the second target in the vehicle coordinate system based on the vertex coordinates of the second target in the image coordinate system of the visual sensor; determining the pose of the second target based on the coordinates of the ground vertex of the second target in the vehicle coordinate system, wherein the pose of the second target includes the size of the second target, heading information and vertex coordinates in the vehicle coordinate system.

[0016] In some embodiments of this disclosure, the matching region is a fan-shaped region determined based on the vertex coordinates of the second target.

[0017] In some embodiments of this disclosure, determining the matching region of the second target based on the pose and positioning information of the second target includes: determining the polar coordinates of the vertex of the second target based on the vertex coordinates of the second target in the vehicle coordinate system; determining the size of the matching region of the second target based on the polar coordinates of the vertex of the second target; and determining the size and weight of each sub-region in the matching region of the second target based on the polar coordinates of the vertex of the second target and the size of the matching region.

[0018] In some embodiments of this disclosure, the size parameters of the matching region of the second target include a maximum radius, a minimum radius, a minimum angle, and a maximum angle. The maximum radius is determined based on the maximum polar radius in the polar coordinates of the second target vertex, the minimum radius is determined based on the minimum polar radius in the polar coordinates of the second target vertex, the maximum angle is determined based on the maximum polar angle in the polar coordinates of the second target vertex, and the minimum angle is determined based on the minimum polar angle in the polar coordinates of the second target vertex.

[0019] In some embodiments of this disclosure, the sub-region includes a nearest sub-region, an intermediate sub-region, and a farthest sub-region arranged from the nearest to the origin of the vehicle coordinate system. The weight of the nearest sub-region is greater than the weight of the intermediate sub-region, and the weight of the intermediate sub-region is greater than the weight of the farthest sub-region.

[0020] In some embodiments of this disclosure, the positioning information of the first target includes the radial distance of the first target and the angle between it and the X-axis of the vehicle coordinate system; calculating the similarity between the second target and the first target located in its matching area based on the positioning information of the first target includes: determining whether the first target falls into the matching area of ​​the second target based on the radial distance of the first target, the angle between it and the X-axis of the vehicle coordinate system, and the size of the matching area of ​​the second target; when the first target falls into the matching area of ​​the second target, determining the weight of the sub-region to which the first target belongs, calculating the minimum distance between the first target and the second target, and calculating the similarity between the first target and the second target based on the weight of the sub-region to which the first target belongs, the minimum distance between the first target and the second target, and the effective matching distance threshold of the second target.

[0021] In some embodiments of this disclosure, the similarity between the first target and the second target is calculated using the following formula:

[0022]

[0023] Among them, S rv W represents the similarity between the first objective and the second objective. s D represents the weight of the sub-region to which the first target belongs. min D represents the minimum distance between the first and second targets. max This represents the effective matching distance threshold for the second target.

[0024] In some embodiments of this disclosure, the step of correcting the pose of the second target based on the positioning information of the first target matched by the second target includes: when the minimum distance between the first target matched by the second target and the second target is greater than a predetermined first threshold, translating the entire second target along the vertical direction of the side closest to the first target in the OXY plane of the vehicle coordinate system toward the first target by a predetermined distance, thereby obtaining the corrected pose of the second target.

[0025] In some embodiments of this disclosure, the predetermined distance is determined based on the minimum distance between the first target and the second target matched by the second target.

[0026] In some embodiments of this disclosure, the pose of the second target is corrected based on the positioning information of the first target matched by the second target, including: calculating the minimum distance between the first target matched by the second target and the second target based on the positioning information of the first target matched by the second target and the vertex coordinates of the second target in the vehicle coordinate system; when the minimum distance is greater than a predetermined first threshold, determining the translation distance of the second target based on the minimum distance, and calculating the corrected coordinates of each vertex of the second target based on the translation distance of the second target so that the corrected second target is closer to the first target.

[0027] A second aspect of this disclosure provides a target fusion apparatus, comprising:

[0028] The information acquisition unit is used to acquire the positioning information of the first target through the vehicle's millimeter-wave radar and the positioning information of the second target through the vehicle's vision sensor.

[0029] The pose determination unit is used to determine the pose of the second target based on the positioning information of the second target.

[0030] The matching region determination unit is used to determine the matching region of the second target based on the pose and positioning information of the second target;

[0031] The target matching unit is used to calculate the similarity between the second target and the first target located in its matching area based on the positioning information of the first target, and to determine the first target matched by the second target based on the similarity.

[0032] The correction unit is used to correct the pose of the second target based on the positioning information of the first target matched by the second target.

[0033] The information fusion unit is used to determine the observation information of the third target based on the corrected pose of the second target and the positioning information of the first target it matches. The third target is the fusion target of the second target and the first target it matches.

[0034] The filtering unit is used to filter the observation information of the third target to obtain the fused information of the third target.

[0035] A third aspect of this disclosure provides an electronic device, comprising: a memory storing execution instructions; and a processor that executes the execution instructions stored in the memory to cause the processor to perform the target fusion method described above.

[0036] A fourth aspect of this disclosure provides a readable storage medium storing executable instructions that, when executed by a processor, are used to implement the aforementioned target fusion method.

[0037] This embodiment calculates the target's pose state using the detection results of a visual sensor, and dynamically calculates the effective matching range between the visual target and the millimeter-wave radar target using the target's pose state. Target matching is then performed within the effective matching range. This fully utilizes the characteristics of both the millimeter-wave radar sensor and the visual sensor, effectively reducing mismatches and missed matches between targets caused by uncertainties or insufficient information in the millimeter-wave radar's positioning points. Furthermore, during the target fusion stage, the pose of the visual sensor target can be corrected based on the millimeter-wave radar's detection results, further improving the accuracy of the target fusion results. Attached Figure Description

[0038] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0039] Figure 1 This is a visualization example of the target detection results from millimeter-wave radar.

[0040] Figure 2 This is a flowchart illustrating a target fusion method according to one embodiment of the present disclosure.

[0041] Figure 3 This is a visualization example of the target detection results in the visual coordinate system.

[0042] Figure 4 This is a visual schematic diagram of the 3D detection frame of the second target in the XY plane of the vehicle coordinate system according to one embodiment of the present disclosure.

[0043] Figure 5 This is a visual schematic diagram of the matching region and its sub-regions of the second target in the XY plane of the vehicle coordinate system according to one embodiment of this disclosure.

[0044] Figure 6This is a visual schematic diagram of the second target pose correction process in the vehicle coordinate system XY plane according to one embodiment of this disclosure.

[0045] Figure 7 This is a schematic block diagram of the structure of a target fusion device that employs a hardware implementation of a processing system according to one embodiment of the present disclosure. Detailed Implementation

[0046] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0047] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0048] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0049] The use of crosshairs and / or shading in the accompanying drawings is generally used to clarify the boundaries between adjacent components. Thus, unless otherwise stated, the presence or absence of crosshairs or shading does not convey or indicate any preference or requirement for the specific material, material properties, dimensions, proportions, commonalities between the illustrated components, or any other characteristics, properties, etc., of the components. Furthermore, in the accompanying drawings, the dimensions and relative dimensions of components may be exaggerated for clarity and / or descriptive purposes. When exemplary embodiments can be implemented differently, a specific process sequence may be performed in a different order than that described. For example, two consecutively described processes may be performed substantially simultaneously or in the reverse order of their description. Furthermore, the same reference numerals denote the same components.

[0050] When a component is referred to as being "on" or "above" another component, "connected to," or "joined to" another component, the component may be directly on, directly connected to, or directly joined to the other component, or there may be intermediate components. However, when a component is referred to as being "directly on" another component, "directly connected to," or "directly joined to" another component, there are no intermediate components. Therefore, the term "connection" can refer to a physical connection, an electrical connection, etc., and may or may not have intermediate components.

[0051] The terminology used herein is for the purpose of describing particular embodiments and is not intended to be limiting. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0052] As described in the background section, existing target fusion methods for millimeter-wave radar and visual sensors have the following shortcomings: 1. They fail to address the target positioning jump problem caused by the uncertainty of the positioning point when millimeter-wave radar locates the target, which can easily lead to target matching failure or reduced accuracy of the fusion result; 2. They fail to fully utilize the effective target information of the visual sensor to correct the fusion result.

[0053] In view of this, embodiments of this disclosure provide a target fusion method, apparatus, electronic device, and storage medium, which fully utilize the characteristics of millimeter-wave radar sensors and vision sensors. The vision sensor calculates the target's pose state, and the effective matching range between the visual target and the millimeter-wave radar target is dynamically calculated using the target's pose state. Target matching is performed within the effective matching range. That is, the possible range of the millimeter-wave radar target is determined by the target's pose state, and target matching is performed within this range. This effectively reduces mismatches and missed matches between targets caused by the uncertainty or insufficient information of the millimeter-wave radar positioning point. Simultaneously, during the target fusion stage, the pose of the target from the vision sensor can be corrected based on the detection results of the millimeter-wave radar, further improving the accuracy of the target fusion results.

[0054] Terminology Explanation:

[0055] The first objective is to identify targets in the vehicle camera's target detection results. These targets can be, but are not limited to, people, animals, and road signs such as lane lines, utility poles, and buildings.

[0056] Second target: The target in the target detection results of millimeter-wave radar can be, but is not limited to, people, animals, and road signs such as lane lines, utility poles, and buildings.

[0057] The third target: The target obtained by fusing the target detection results of millimeter-wave radar with the target detection results of vehicle-mounted camera can be, but is not limited to, people, animals, and road signs such as lane lines, utility poles, and buildings.

[0058] Figure 2 A flowchart illustrating a target fusion method according to some embodiments of this disclosure is shown.

[0059] like Figure 2 As shown, the target fusion method of this disclosure embodiment may include:

[0060] Step S202: Obtain the positioning information of the first target by using the millimeter-wave radar mounted on the vehicle, and obtain the positioning information of the second target by using the vision sensor mounted on the vehicle.

[0061] Step S204: Determine the pose of the second target based on the positioning information of the second target, and determine the matching area of ​​the second target based on the pose of the second target and its positioning information.

[0062] Step S206: Calculate the similarity between the second target and the first target located in its matching area based on the positioning information of the first target, and determine the first target matched by the second target based on the similarity.

[0063] Step S208: Correct the pose of the second target based on the positioning information of the first target matched with the second target;

[0064] Step S210: Determine the observation information of the third target based on the corrected pose of the second target and the positioning information of the matched first target. The third target is the fusion target of the second target and the matched first target.

[0065] Step S212: Filter the observation information of the third target to obtain the fused information of the third target. The fused information of the third target is the target fusion result of the millimeter-wave radar and the visual sensor, which can be used to indicate the position, velocity and direction of the third target (i.e. the fused target).

[0066] In step S202, the positioning information of the first target around the vehicle detected by the millimeter-wave radar, that is, the target detection result of the millimeter-wave radar, can be expressed as [ρ, θ, v], where ρ represents the radial distance of the first target, θ represents the angle between the first target and the X-axis of the vehicle coordinate system, and v represents the radial velocity of the first target.

[0067] In step S202, the vision sensor acquires images containing the scene around the vehicle, and the location information of the second target can be obtained by performing target detection on these images. Figure 3 The detection results of the second target, obtained through a visual sensor, are shown. Figure 3The 3D bounding box in the image is a visual representation of the second target. The detection result of the second target, i.e., its localization information, can include the vertex coordinates of the 3D bounding box in the image coordinate system of the vision sensor. Figure 3 Information such as page 1 to page 6, target category, etc.

[0068] In step S202, the location information of the first target and the location information of the second target are location information obtained at the same time or within a predetermined time difference. That is, the location information of the first target and the location information of the second target are target location information of the vehicles at the same time and in the same scenario.

[0069] In step S204, determining the pose of the second target based on its positioning information may include the following steps a1 to a2:

[0070] Step a1: Determine the coordinates of the grounding vertex of the second target in the vehicle coordinate system based on the vertex coordinates of the second target in the image coordinate system of the vision sensor;

[0071] Specifically, the coordinates of the grounding vertex of the second target in the vehicle coordinate system can be determined by coordinate system transformation based on the intrinsic and extrinsic parameters of the vision sensor.

[0072] For example, based on the imaging principle of pinhole cameras and the assumption of flat ground, the grounding point in the vehicle coordinate system satisfies Z=0. The coordinates of the grounding vertices p1(u1,v1), p2(u2,v2), and p3(u3,v3) of the second target in the vehicle coordinate system can be calculated according to the following formula (1): P1(X1,Y1,0), P2(X2,Y2,0), and P3(X3,Y3,0).

[0073]

[0074] Where M1 represents the intrinsic parameter matrix of the vision sensor, M2 represents the extrinsic parameter matrix of the vision sensor, s is the scaling factor, (u,v) represents the coordinates of the point in the visual coordinate system and the image coordinate system, and (X,Y,Z) represents the coordinates of the point in the vehicle coordinate system.

[0075] Step a2: Determine the pose of the second target based on the coordinates of its grounding vertices in the vehicle coordinate system. The pose of the second target may include, but is not limited to, the target's size, heading information, and vertex coordinates in the vehicle coordinate system. The vertex coordinates in the vehicle coordinate system include, but are not limited to, the coordinates of the four grounding vertices in the vehicle coordinate system.

[0076] Specifically, the coordinates of the other grounding vertices of the second target in the vehicle coordinate system, the length of the second target, the width of the second target, and the heading angle of the second target can be calculated according to the following formulas (2) to (4) based on the coordinates of the three grounding vertices of the second target in the vehicle coordinate system.

[0077]

[0078]

[0079]

[0080] Where W represents the width of the second target, L represents the length of the second target, P1(X1,Y1,0), P2(X2,Y2,0), and P3(X3,Y3,0) are the three grounding vertices of the second target, and yaw represents the heading angle of the second target.

[0081] Figure 4 The diagram shows the detection frame of the second target in the XY plane of the vehicle coordinate system. P1 to P4 are the four vertices of the second target, and yaw represents the heading angle of the second target.

[0082] Since images captured by vision sensors typically contain two faces of an object, in most cases there will be three joint vertices in the 3D detection box.

[0083] For cases where there are fewer than three grounding points, for example, in special cases where the image acquired by the vision sensor contains only one face of an object and the second target has only two grounding vertices, the size of the second target can be set to the default size corresponding to its category. Based on the coordinates of the two grounding vertices in the vehicle coordinate system and the size of the second target, the coordinates of the other two vertices of the second target in the vehicle coordinate system can be calculated using the aforementioned formulas (2) to (4).

[0084] In step S204, determining the matching region of the second target based on its pose and positioning information may include the following steps b1 to b3:

[0085] Step b1: Determine the polar coordinates of the second target's vertex based on the vertex coordinates of the second target in the vehicle coordinate system;

[0086] Specifically, based on the coordinates P of each vertex of the second target in the vehicle coordinate system i (X i ,Y i (i=1,2,3,4), using the transformation rules from Cartesian coordinates to polar coordinates, i.e., the following equations (6)~(7), calculate the polar coordinates P' of these points. i (R i ,θ i ),

[0087]

[0088]

[0089] The origin of the polar coordinate system can be the same as the origin of the vehicle coordinate system.

[0090] Step b2: Determine the size of the matching region of the second target based on the polar coordinates of the second target vertex;

[0091] Since millimeter-wave radar detects targets within a fan-shaped area based on its measurement principle, this embodiment defines the matching region of the second target as a fan-ring region. The size of this fan-ring region can be characterized by four parameters, namely, the maximum radius R. max , minimum radius R min Minimum angle θ min and maximum angle θ max That is, the matching region of a certain second target can be represented as {R}. min ,R max ,θ min ,θ max}

[0092] Specifically, the maximum radius of the second target matching region can be determined based on the maximum polar radius in the polar coordinates of the second target vertex, the minimum radius of the second target matching region can be determined based on the minimum polar radius in the polar coordinates of the second target vertex, the maximum angle of the second target matching region can be determined based on the maximum polar angle in the polar coordinates of the second target vertex, and the minimum angle of the second target matching region can be determined based on the minimum polar angle in the polar coordinates of the second target vertex.

[0093] In some implementations, the size of the matching region of the second target can be determined by the following formulas (8) to (11):

[0094]

[0095]

[0096]

[0097]

[0098] in, This represents the minimum polar radius of the four vertices of the second objective. This represents the maximum value of the polar radius of the four vertices of the second target. This represents the minimum polar angle of the four vertices of the second target. This represents the maximum value of the polar angles at the four vertices of the second target. δ1, δ2, ε1, and ε2 are adjustable measurement noise parameters, which can be set to fixed values ​​or dynamic values ​​that change with the size of the matching region as needed. For example, they can be configured as follows:

[0099] Figure 5 A visual diagram of the matching region of a certain second target P1P2P3P4 is shown. Figure 5 In the example, the fan-ring region A1A2A3A4 is the matching region of the second target P1P2P3P4, and the maximum radius R of the fan-ring region A1A2A3A4 is... max , minimum radius R min Minimum angle θ min and maximum angle θ max It can be calculated from the polar coordinates of the four vertices of the second target according to equations (8) to (11).

[0100] Step b3: Determine the size and weight of each sub-region in the matching region of the second target based on the polar coordinates of the vertex of the second target.

[0101] Considering that targets detected by millimeter-wave radar are more likely to appear in areas relatively close to the vehicle, the matching area for the second target can be divided into three sub-regions, arranged from farthest to farthest from the vehicle's coordinate system origin: the closest sub-region, the middle sub-region, and the farthest sub-region. Different matching weights are assigned to these sub-regions. For example, the weight of the closest sub-region is greater than that of the middle sub-region, and the weight of the middle sub-region is greater than that of the farthest sub-region. In this way, the accuracy of target matching can be further improved by dividing the target into sub-regions and assigning different weights.

[0102] When the matching region of the second target is defined as a fan-ring region, its three sub-regions are three consecutive fan-ring regions within the matching region. Specifically, the maximum and minimum angles of the three sub-regions are the same as those of the matching region. The maximum radius of the nearest sub-region can be the second minimum value of the polar radius of the four vertices of the second target, and its minimum radius is equal to the minimum radius of the matching region. The maximum radius of the middle sub-region can be the second maximum value of the polar radius of the four vertices of the second target, and its minimum radius is the same as the maximum radius of the nearest sub-region. The maximum radius of the farthest sub-region is equal to the maximum radius of the matching region, and its minimum radius is the same as the maximum radius of the middle sub-region.

[0103] like Figure 5 As shown, the matching region A1A2A3A4 of a certain second target P1P2P3P4 is divided into three sub-regions from near to far, as shown in the figure: the nearest sub-region S1 (A1A2B2B1), the middle sub-region S2 (B1B2B3B4), and the farthest sub-region S3 (B4B3A3A4). The matching weights corresponding to these three sub-regions can be calculated by the following formulas (12) to (14):

[0104]

[0105]

[0106]

[0107] Among them, W S1 R represents the weight of the nearest subregion S1. max To match the maximum radius of region A1A2A3A4, R min The minimum radius of the matching region A1A2A3A4 is given. OB1 represents the radial distance from the origin o to point B1. R3 represents the second minimum value of the polar radius of the four vertices of the second target P1P2P3P4, which is the polar radius of vertex P3. B1B4 represents the polar radius of the middle sub-region S2. B4A4 represents the polar radius of the farthest sub-region S3. R1 represents the second maximum value of the polar radius of the four vertices of the second target P1P2P3P4, which is the polar radius of vertex P1. max() represents taking the larger value, and min() represents taking the smaller value.

[0108] In step S206, calculating the similarity between the second target and the first target located in its matching region based on the positioning information of the first target may include the following steps c1 to c2:

[0109] Step c1: Determine whether the first target falls into the second target matching area based on the radial distance of the first target, its angle with the X-axis of the vehicle coordinate system, and the size of the second target matching area;

[0110] For example, assuming the location information of a first target is [ρ, θ, v], we can first determine whether the first target is located within the matching region of the second target. If R min ≤ρ≤R max ,θ min ≤θ≤θ max If the first target is within the matching area of ​​the second target, then it means that the first target is within the matching area of ​​the second target; otherwise, it means that the first target is not within the matching area of ​​the second target.

[0111] Step c1: When the first target falls into the matching region of the second target, determine the weight of the sub-region to which the first target belongs, calculate the minimum distance between the first target and the second target, and calculate the similarity between the first target and the second target based on the weight of the sub-region to which the first target belongs, the minimum distance between the first target and the second target, and the effective matching distance threshold of the second target.

[0112] For the first target [ρ, θ, v], the radial distance ρ of the first target is compared with the maximum and minimum radii of each sub-region. If the radial distance ρ of the first target is greater than or equal to the minimum radius of a certain sub-region and less than the maximum radius of that sub-region, then the first target can be considered to belong to that sub-region. After determining the sub-region to which the first target belongs, the matching weight Ws of that sub-region is obtained.

[0113] Specifically, the distances between the first target and each edge of the second target can be calculated based on the principle of calculating the distance from a point to a line, and the minimum value among these distances can be taken as the minimum distance Dmin between the first target and the second target.

[0114] In some implementations, the similarity between the first target and the second target can be calculated using the following formula (15):

[0115]

[0116] Among them, S rv W represents the similarity between the first objective and the second objective. s D represents the weight of the sub-region to which the first target belongs. min D represents the minimum distance between the first and second targets. max This represents the effective matching distance threshold for the second target.

[0117] Among them, D max Different thresholds can be set based on the category of the second objective. Alternatively, D max It can be obtained through the following equation (16), that is, D max You can choose the larger value between the width and length of the second target.

[0118] D max =max(W,L) (16)

[0119] Where W represents the width of the second target and L represents the length of the second target.

[0120] In step S206, after determining the similarity between each first target falling into each second target matching region and the target, a target matching matrix can be established based on the similarity. The optimal matching result can be obtained by, for example, the KM algorithm (Hungarian algorithm). The optimal matching result contains one or more first targets matched by each second target.

[0121] After a first target detected by millimeter-wave radar is matched with a second target detected by a visual sensor, that is, when it is determined that a first target and a second target belong to the same target, the accuracy of the millimeter-wave radar detection result is usually higher than that of the visual sensor detection result. Therefore, in this embodiment of the present disclosure, after target matching in step S206, the target detection result of the millimeter-wave radar is used to correct the pose of the target in the visual sensor, thereby improving the accuracy of the target fusion result and avoiding inaccurate fusion results due to errors in the target detection result of the visual sensor.

[0122] In step S208, when the minimum distance between the first target and the second target matched by the second target is greater than a predetermined first threshold, the second target can be moved along the edge closest to its matched first target (i.e., the edge corresponding to the minimum distance D in the second target) in the OXY plane of the vehicle coordinate system (i.e., the plane parallel to the ground in the vehicle coordinate system). min The second target's pose is obtained by translating the edge of the second target (the first target) perpendicularly to the first target by a predetermined distance. The predetermined distance can be determined based on the minimum distance between the first target and the second target that are matched with the second target.

[0123] In some implementations, step S208, correcting the pose of the second target based on the positioning information of the first target matched with the second target, may include the following steps d1 to d2:

[0124] Step d1: Based on the location information of the first target matched by the second target and the vertex coordinates of the second target in the vehicle coordinate system, calculate the minimum distance D between the first target and the second target matched by the second target. min ;

[0125] Here, the method for calculating the minimum distance is as described in section D above. min The calculation method is the same, so I will not repeat it here.

[0126] Step d2: When the minimum distance is greater than a predetermined first threshold, determine the translation distance of the second target based on the minimum distance, and calculate the corrected coordinates of each vertex of the second target based on the translation distance of the second target so that the corrected second target is closer to the first target.

[0127] Specifically, the minimum distance D calculated in step d1 min and the predetermined first threshold D th Comparison, if D min <=D th If D, then no correction is needed for the pose of the second target; if D min >D th Then the pose of the second target needs to be corrected.

[0128] Specifically, the process of correcting the pose of the second target may include: first calculating the translation distance of the second target using the following formulas (17) to (18), and then determining the corrected coordinates of each vertex of the second target using the following formula (19), so that the corrected second target is closer to the first target.

[0129]

[0130]

[0131] P′ i (X′ i ,Y′ i ) = P i (X i +X tran ,Y i +Y tran (i = 1, 2, 3, 4) (19)

[0132] Among them, X tran The X-axis represents the translation distance in the vehicle coordinate system, and the Y-axis represents the translation distance. tran X′ represents the Y-axis translation distance in the vehicle coordinate system. i This represents the corrected X-axis coordinate and Y′ of the i-th vertex of the second target in the vehicle coordinate system. i This represents the corrected Y-axis coordinate of the i-th vertex of the second target in the vehicle coordinate system, X. i This represents the uncorrected X-axis coordinate of the i-th vertex of the second target in the vehicle coordinate system, and the Y-axis coordinate is... i This represents the corrected Y-axis coordinate of the i-th vertex of the second target in the vehicle coordinate system.

[0133] The first threshold Dth can be manually set according to actual working conditions without specific constraints. For example, it can be configured as Dth = 0.3 meters. In specific applications, the value of the first threshold can also be flexibly and dynamically updated according to the specific scenario.

[0134] Figure 6 An example diagram of the second target pose correction process is shown. Figure 6 In the diagram, quadrilateral P1P2P3P4 represents the position of the second target before correction, and point P... rad This indicates that the first target matches the second target. Figure 6 In the example, point P rad Minimum distance D to the second target min Greater than D th Point P rad If the edge P1P2 is closest to the second target, then the entire quadrilateral P1P2P3P4 is oriented towards point P along a direction perpendicular to edge P1P2. radThe quadrilateral P1'P2'P3'P4' obtained by translating a certain distance is the corrected position of the second target.

[0135] In step S210, the corrected pose of the second target and the location information [ρ, θ, v] of the matched first target can be used, for example... Figure 6 The quadrilateral P1'P2'P3'P4' in the vehicle coordinate system determines the observation information matrix [X,Y,yaw,v] of the third target, where X and Y represent the X-axis coordinates and Y-axis coordinates of the center point of quadrilateral P1'P2'P3'P4' in the vehicle coordinate system, respectively, yaw represents the heading angle of the third target, and v represents the velocity of the third target. This third target is the fusion target of the second target and its matched first target.

[0136] In step S212, the observation information matrix of the third target determined in step S210 can be passed into a lossless Kalman filter (UKF) for filtering to obtain the fused information of the third target. The fused information of the third target can be represented as [X',Y',yaw',v'], which can be used as the positioning information of the third target, that is, the target fusion result of millimeter-wave radar and visual sensor, and can be used to indicate the position, direction and velocity of the third target.

[0137] The target fusion method described in this embodiment first calculates the pose state of each visual sensor target, then dynamically calculates a matching region with matching weights based on the pose of the visual target, calculates the similarity between targets based on information such as position, velocity, and orientation, and completes target matching based on the similarity, thereby reducing false matching and missed matching between targets.

[0138] Furthermore, the target fusion method of this disclosure also corrects the target pose of the visual sensor using target information from the millimeter-wave radar, and uses the corrected target pose from the visual sensor and the target information from the millimeter-wave radar to determine the observation information of the fused target. Finally, the observation information is used as a measurement value and input into a lossless Kalman filter to obtain the fusion result. This reduces data fluctuations caused by jumps in the target positioning point of the millimeter-wave radar and improves the accuracy of the fusion result. Therefore, this disclosure improves the accuracy of the target fusion result by fully utilizing the operating characteristics of the millimeter-wave radar sensor and the visual sensor.

[0139] Figure 7 This is a schematic block diagram of the structure of a target fusion device that employs a hardware implementation of a processing system according to one embodiment of the present disclosure.

[0140] The apparatus may include corresponding modules that perform one or more steps in the flowchart above. Therefore, each or more steps in the flowchart above can be performed by a corresponding module, and the apparatus may include one or more of these modules. A module may be one or more hardware modules specifically configured to perform a corresponding step, or implemented by a processor configured to perform a corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented through some combination thereof.

[0141] This hardware architecture can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 800 connects various circuits, including one or more processors 900, memory 1000, and / or hardware modules. Bus 800 can also connect various other circuits 1100, such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0142] Bus 800 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, this diagram uses only one connection line, but this does not imply that there is only one bus or one type of bus.

[0143] Any process or method description in the flowcharts or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain. The processor performs the various methods and processes described above. For example, the method embodiments of this disclosure may be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some embodiments, part or all of the software program may be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform one of the methods described above by any other suitable means (e.g., by means of firmware).

[0144] The logic and / or steps represented in the flowchart or otherwise described herein may be specifically implemented in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0145] For the purposes of this specification, a "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM). Furthermore, a readable storage medium can even be paper or other suitable media on which a program can be printed, since a program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in memory.

[0146] It should be understood that various parts of this disclosure can be implemented in hardware, software, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0147] Those skilled in the art will understand that all or part of the steps of the methods described above can be implemented by a program instructing related hardware. The program can be stored in a readable storage medium, and when executed, the program includes one or a combination of the steps of the method implementation.

[0148] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a single processing module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. The storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0149] Figure 7 This is a schematic diagram of a target fusion apparatus according to one embodiment of the present disclosure. Figure 7 As shown, the target fusion apparatus 700 according to an embodiment of this disclosure may include:

[0150] The information acquisition unit 702 is used to acquire the positioning information of the first target through the vehicle's millimeter-wave radar and to acquire the positioning information of the second target through the vehicle's vision sensor.

[0151] The pose determination unit 704 is used to determine the pose of the second target based on the positioning information of the second target.

[0152] The matching region determination unit 706 is used to determine the matching region of the second target based on the pose and positioning information of the second target;

[0153] The target matching unit 708 is used to calculate the similarity between the second target and the first target located in its matching area based on the positioning information of the first target, and to determine the first target matched by the second target based on the similarity.

[0154] The correction unit 710 is used to correct the pose of the second target based on the positioning information of the first target matched by the second target;

[0155] The information fusion unit 712 is used to determine the observation information of the third target based on the corrected pose of the second target and the positioning information of the matched first target. The third target is the fusion target of the second target and the matched first target.

[0156] The filtering unit 714 is used to filter the observation information of the third target to obtain the fused information of the third target.

[0157] Other technical details of the target fusion device 700 in this embodiment can be found in the preceding method section, and will not be repeated here.

[0158] This disclosure also provides an electronic device, including: a memory storing execution instructions; and a processor or other hardware module executing the execution instructions stored in the memory, causing the processor or other hardware module to perform the target fusion method described above.

[0159] This disclosure also provides a readable storage medium storing executable instructions, which, when executed by a processor, are used to implement the aforementioned target fusion method.

[0160] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0161] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0162] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A target fusion method, characterized by, The method comprises: acquiring positioning information of a first target through a millimeter wave radar of a vehicle, and acquiring positioning information of a second target through a vision sensor of the vehicle; determining a pose of the second target according to the positioning information of the second target, and determining a matching region of the second target according to the pose of the second target and the positioning information of the second target; calculating a similarity between the second target and a first target located in the matching region of the second target according to the positioning information of the first target, and determining the first target matched by the second target according to the similarity; correcting the pose of the second target according to the positioning information of the first target matched by the second target; determining observation information of a third target according to the corrected pose of the second target and the positioning information of the first target matched by the second target, the third target being a fusion target of the second target and the first target matched by the second target; filtering the observation information of the third target to obtain fusion information of the third target.

2. The target fusion method according to claim 1, wherein the positioning information of the second target comprises vertex coordinates of the second target in an image coordinate system of the vision sensor; the pose of the second target is determined according to the positioning information of the second target, comprising: determining coordinates of a ground vertex of the second target in a vehicle coordinate system according to the vertex coordinates of the second target in the image coordinate system of the vision sensor; and determining the pose of the second target according to the coordinates of the ground vertex of the second target in the vehicle coordinate system, the pose of the second target comprising a size, a heading information and vertex coordinates of the second target in the vehicle coordinate system. The matching region is a fan ring region determined according to the vertex coordinates of the second target. The matching region of the second target is determined according to the pose of the second target and the positioning information of the second target, comprising: determining vertex polar coordinates of the second target according to the vertex coordinates of the second target in the vehicle coordinate system; determining a size of the matching region of the second target according to the vertex polar coordinates of the second target; 3. The target fusion method of claim 2, wherein, determining sizes of sub-regions in the matching region of the second target and weights of the sub-regions according to the vertex polar coordinates of the second target and the size of the matching region of the second target.

4. The target fusion method according to claim 2 or 3, characterized in that, The size parameters of the matching region of the second target comprise a maximum radius, a minimum radius, a maximum angle and a minimum angle, the maximum radius being determined according to a maximum polar radius in the vertex polar coordinates of the second target, the minimum radius being determined according to a minimum polar radius in the vertex polar coordinates of the second target, the maximum angle being determined according to a maximum polar angle in the vertex polar coordinates of the second target, and the minimum angle being determined according to a minimum polar angle in the vertex polar coordinates of the second target. The sub-regions comprise a nearest sub-region, an intermediate sub-region and a farthest sub-region arranged from near to far from the origin of the vehicle coordinate system, the weight of the nearest sub-region being greater than the weight of the intermediate sub-region, and the weight of the intermediate sub-region being greater than the weight of the farthest sub-region.

7. The target fusion method according to claim 4, wherein the positioning information of the first target comprises a radial distance of the first target and an included angle between the first target and an X axis of the vehicle coordinate system; and the similarity between the second target and the first target located in the matching region of the second target is calculated according to the positioning information of the first target, comprising: ​ 5. The target fusion method of claim 4, wherein, ​ 6. The target fusion method of claim 4, wherein, ​ ​ ​ ​ The first target is determined to fall into the second target matching region according to a radial distance of the first target, an angle between the first target and an X-axis of a vehicle coordinate system, and a size of the second target matching region; When the first target falls into the matching region of the second target, a weight of a sub-region to which the first target belongs is determined, a minimum distance between the first target and the second target is calculated, and a similarity between the first target and the second target is calculated according to the weight of the sub-region to which the first target belongs, the minimum distance between the first target and the second target, and an effective matching distance threshold of the second target.

8. The target fusion method of claim 7, wherein, The similarity between the first target and the second target is calculated according to the following formula: wherein, represents a similarity of the first target to the second target, represents a weight of a sub-region to which the first target belongs, represents a minimum distance of the first target to the second target, represents an effective matching distance threshold of the second target.

9. The target fusion method of claim 1, wherein, The pose of the second target is corrected according to the positioning information of the first target matched with the second target, and the pose of the second target after correction is determined according to the following formula: When the minimum distance between the first target matched with the second target and the second target is greater than a predetermined first threshold, the second target is translated in a vertical direction of an edge closest to the first target in an OXY plane of the vehicle coordinate system by a predetermined distance, so as to obtain the pose of the second target after correction.

10. The target fusion method of claim 9, wherein, The predetermined distance is determined according to the minimum distance between the first target matched with the second target and the second target.

11. The target fusion method of claim 1 or 9, wherein, The pose of the second target is corrected according to the positioning information of the first target matched with the second target, and the pose of the second target after correction is determined according to the following formula: The minimum distance between the first target matched with the second target and the second target is calculated according to the positioning information of the first target matched with the second target and vertex coordinates of the second target in the vehicle coordinate system; When the minimum distance is greater than a predetermined first threshold, a translation distance of the second target is determined according to the minimum distance, and the corrected coordinates of each vertex of the second target are calculated according to the translation distance of the second target, so that the second target after correction is closer to the first target.

12. A target fusion device, characterized by The method comprises the following steps: An information acquisition unit is configured to acquire the positioning information of the first target through a millimeter wave radar of the vehicle and acquire the positioning information of the second target through a visual sensor of the vehicle; A pose determination unit is configured to determine the pose of the second target according to the positioning information of the second target; A matching region determination unit is configured to determine the matching region of the second target according to the pose and the positioning information of the second target; A target matching unit is configured to calculate the similarity between the second target and the first target located in the matching region of the second target according to the positioning information of the first target, and determine the first target matched with the second target according to the similarity; A correction unit is configured to correct the pose of the second target according to the positioning information of the first target matched with the second target; An information fusion unit is configured to determine the observation information of a third target according to the pose of the second target after correction and the positioning information of the first target matched with the second target, the third target being a fusion target of the second target and the first target matched with the second target; A filtering unit is configured to filter the observation information of the third target to obtain the fusion information of the third target.

13. An electronic device, comprising: The method comprises the following steps: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the target fusion method in any one of claims 1 to 11. ​ 14. A readable storage medium, characterized by, The readable storage medium stores execution instructions, and the execution instructions are used for realizing the target fusion method in any one of claims 1 to 11 when executed by the processor.

Images