Multi-sensor target matching method, device and equipment, moving tool and medium

Abstract

The invention relates to a multi-sensor target matching method, device and equipment, a moving tool and a medium, and the method comprises the steps: obtaining a to-be-matched millimeter wave target and a to-be-matched visual target; constructing a first cost matrix of the millimeter wave target and the visual target on an image coordinate system, and a second cost matrix and a third cost matrix on a preset coordinate system; obtaining a target cost matrix according to the first cost matrix, the second cost matrix and the third cost matrix; and taking the target cost matrix as input of a Hungary algorithm to obtain a matching result of the millimeter wave target and the visual target. A plurality of cost matrixes of a millimeter wave target and a visual target are constructed, then the most accurate target cost matrix is obtained according to the plurality of cost matrixes, and finally the target cost matrix is used as the input of a Hungary algorithm to obtain a matching result so as to reduce the matching error of the millimeter wave target and the visual image target.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a multi-sensor target matching method, device, equipment, mobile tool and medium. Background Art

[0002] In recent years, autonomous driving cars have been a hot research direction. At present, vehicle control is relatively mature. Relying solely on the information of a single sensor cannot meet the needs of autonomous driving, and the accuracy of the information from a single sensor cannot be judged. Therefore, multi-sensors have become the mainstream choice for autonomous driving. However, how to correctly integrate the information between multiple sensors has also become the focus of everyone's attention.

[0003] In the prior art, if the millimeter wave target is mapped to the visual image target for matching, the accuracy is low. If the visual image target is mapped to the vehicle coordinate system and matched with the millimeter wave target, it will be difficult to match once the vehicle is bumped and shaken. Therefore, there is a large error in the matching of the millimeter wave target and the visual image target.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The present application provides a multi-sensor target matching method, device, equipment, mobile tool and medium to solve the technical problem of large errors in matching the above-mentioned millimeter wave targets and visual image targets.

[0006] According to one aspect of an embodiment of the present application, the present application provides a multi-sensor target matching method, including: obtaining a millimeter wave target and a visual target to be matched; constructing a first cost matrix of the millimeter wave target and the visual target in an image coordinate system, a second cost matrix and a third cost matrix in a preset coordinate system, wherein the preset coordinate system is constructed with the center of a moving tool as the origin and the front of the moving tool as the horizontal coordinate; obtaining a target cost matrix according to the first cost matrix, the second cost matrix and the third cost matrix; using the target cost matrix as the input of a Hungarian algorithm to obtain a matching result of the millimeter wave target and the visual target.

[0007] Optionally, constructing a first cost matrix of the millimeter wave target and the visual target in the image coordinate system specifically includes: mapping the millimeter wave target to the image coordinate system to obtain a first coordinate; correcting the first coordinate according to the visual target to obtain a second coordinate; and constructing the first cost matrix using the second coordinate and the third coordinate of the visual target in the image coordinate system.

[0008] Optionally, a second cost matrix of the millimeter wave target and the visual target in the preset coordinate system is constructed, specifically including: mapping the visual target to the preset coordinate system to obtain a fourth coordinate; calculating the first position distance, speed difference and heading deviation between the visual target and the millimeter wave target based on the fourth coordinate and the fifth coordinate of the millimeter wave target in the preset coordinate system; and constructing a second cost matrix based on the first position distance, speed difference and heading deviation.

[0009] Optionally, constructing a third cost matrix of the millimeter wave target and the visual target in the preset coordinate system specifically includes: obtaining the historical trajectory of the millimeter wave target and the visual target in the preset coordinate system; determining the sixth coordinate of the preset frame of the millimeter wave target on the historical trajectory, and determining the seventh coordinate of the preset frame of the visual target on the historical trajectory, wherein the preset frame is a preset number of frames closest to the current moment; calculating the second position distance between the millimeter wave target and the visual target in each frame according to the sixth coordinate and the seventh coordinate, and using multiple second position distances to construct the third cost matrix.

[0010] Optionally, a target cost matrix is ​​obtained according to the first cost matrix, the second cost matrix and the third cost matrix, specifically including: normalizing the first cost matrix, the second cost matrix and the third cost matrix to obtain a fourth cost matrix, a fifth cost matrix and a sixth cost matrix, respectively; assigning weight values ​​to the fourth cost matrix, the fifth cost matrix and the sixth cost matrix, respectively; adding the fourth cost matrix, the fifth cost matrix and the sixth cost matrix to which the weight values ​​are assigned to obtain the target cost matrix.

[0011] Optionally, after obtaining the matching result of the millimeter wave target and the visual target, the method also includes: if the matching result indicates that the millimeter wave target and the visual target match, binding the first identifier of the millimeter wave target with the second identifier of the visual target, and marking the millimeter wave target and the visual target as a strong matching state; calculating a third position distance between the millimeter wave target and the visual target, and comparing the third position distance with a preset distance; adjusting the matching coefficient of the millimeter wave target and the visual target according to the comparison result; and unbinding the millimeter wave target from the visual target when the matching coefficient is adjusted to a preset value.

[0012] According to another aspect of an embodiment of the present application, the present application provides a multi-sensor target matching device, including: an acquisition module, used to acquire a millimeter-wave target and a visual target to be matched; a construction module, used to construct a first cost matrix of the millimeter-wave target and the visual target in an image coordinate system, a second cost matrix and a third cost matrix in a preset coordinate system, wherein the preset coordinate system is constructed with the center of a moving tool as the origin and the front of the moving tool as the horizontal coordinate; a processing module, used to obtain a target cost matrix according to the first cost matrix, the second cost matrix and the third cost matrix; a matching module, used to use the target cost matrix as the input of a Hungarian algorithm to obtain a matching result of the millimeter-wave target and the visual target.

[0013] According to another aspect of an embodiment of the present application, the present application provides an electronic device, including a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, the memory and the processor communicate through the communication bus and the communication interface, and the steps of the above method are implemented when the processor executes the computer program.

[0014] According to another aspect of an embodiment of the present application, the present application provides a mobile tool, including the above-mentioned electronic device.

[0015] According to another aspect of an embodiment of the present application, the present application also provides a computer-readable medium having a non-volatile program code executable by a processor, and the program code enables the processor to execute the above method.

[0016] The above technical solution provided by the embodiment of the present application has the following advantages compared with the related art:

[0017] The present application uses a multi-sensor target matching method, including: obtaining the millimeter wave target and the visual target to be matched; constructing the first cost matrix of the millimeter wave target and the visual target in the image coordinate system, the second cost matrix and the third cost matrix in the preset coordinate system, wherein the preset coordinate system is constructed with the center of the mobile tool as the origin and the front of the mobile tool as the horizontal coordinate; obtaining the target cost matrix according to the first cost matrix, the second cost matrix and the third cost matrix; using the target cost matrix as the input of the Hungarian algorithm to obtain the matching result of the millimeter wave target and the visual target. By constructing multiple cost matrices of the millimeter wave target and the visual target, and then obtaining the most accurate target cost matrix according to the multiple cost matrices, and finally using the target cost matrix as the input of the Hungarian algorithm, the matching result is obtained to reduce the error of matching the millimeter wave target and the visual image target. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 A flowchart of an optional multi-sensor target matching method provided according to an embodiment of the present application;

[0021] Figure 2 A block diagram of an optional multi-sensor target matching device provided according to an embodiment of the present application;

[0022] Figure 3 A schematic diagram of an optional electronic device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0024] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0025] With the rapid development of autonomous driving technology, relying solely on the information from a single sensor cannot meet the needs of autonomous driving, and the accuracy of the information from a single sensor cannot be determined, so multi-sensors have become the mainstream choice for autonomous driving. However, how to correctly integrate the information between multiple sensors has also become the most critical technology.

[0026] Cameras and millimeter-wave radars are two common sensors. Visual target detection can accurately detect target categories, sizes, and other information, but relying on single vision cannot provide accurate speed and distance information; millimeter-wave radar can accurately measure the speed and position of the target, but cannot know the target category. Therefore, visual millimeter-wave fusion technology can complement each other and obtain richer target attributes. Therefore, how to make visual targets and millimeter-wave targets match more accurately is the key point and difficulty in the field of autonomous driving.

[0027] Existing visual millimeter wave fusion technology is generally fused in the same coordinate system. For example, in the image coordinate system, the millimeter wave point cloud is mapped to the image coordinate system through matrix transformation. The position information of the visual target and the millimeter wave target on the image is used for matching and fusion. Another solution is to map the target on the image to the vehicle coordinate system through the mapping matrix between the image coordinate system and the vehicle coordinate system in the vehicle coordinate system. In the vehicle coordinate system, the relative position relationship between the visual target and the millimeter wave target is calculated for matching and fusion.

[0028] The scheme of mapping the millimeter wave point cloud to the image for matching and fusion has the following problems: first, there is no concept of the target's true movement speed and direction in the image coordinate system, and the matching and fusion can only rely on the image pixel coordinates, which loses the two important features of the target's speed and direction, reducing the matching accuracy. Second, in the image coordinate system, for targets that are occluded in front and behind, the distance in the image is very close, but in the vehicle coordinate system, the two targets are far apart, so it is easy to cause mismatching.

[0029] The disadvantages of this solution of mapping the target on the image to the vehicle coordinate system and then matching it with the millimeter wave target are: first, the target on the image is converted to the vehicle coordinate system through the extrinsic matrix, which relies on the ground plane assumption and the accuracy of the extrinsic matrix. This causes large errors when the target on the image is mapped to the vehicle coordinate system, especially when the vehicle is bumpy and shaking, making it difficult to match the millimeter wave target.

[0030] Therefore, in the prior art, there is still a large error in the matching of millimeter wave targets and visual image targets.

[0031] In order to solve the problems mentioned in the background technology, according to one aspect of the embodiments of the present application, an embodiment of a multi-sensor target matching method is provided, such as Figure 1 As shown, including:

[0032] Step 101, obtaining a millimeter wave target and a visual target to be matched;

[0033] Step 103, constructing a first cost matrix of the millimeter wave target and the visual target in the image coordinate system, a second cost matrix and a third cost matrix in the preset coordinate system, wherein the preset coordinate system is constructed with the center of the mobile tool as the origin and the front of the mobile tool as the horizontal coordinate;

[0034] Step 105, obtaining a target cost matrix according to the first cost matrix, the second cost matrix and the third cost matrix;

[0035] Step 107, using the target cost matrix as the input of the Hungarian algorithm to obtain the matching result between the millimeter wave target and the visual target.

[0036] The millimeter wave target is collected by the millimeter wave radar, the visual target is collected by the camera collection device, and then the millimeter wave target and the visual target are fused and matched using the method provided in the present application.

[0037] Specifically, the millimeter wave radar and the camera acquisition equipment are pre-calibrated with a preset coordinate system, and the preset coordinate system is the vehicle coordinate system, which is obtained by constructing a coordinate system with the center of the mobile tool as the origin and the front of the mobile tool as the horizontal coordinate.

[0038] The cost matrix provided in this application can represent the matching relationship between the millimeter wave target and the visual target, wherein the first cost matrix represents the first matching relationship in the image coordinate system, the second cost matrix represents the second matching relationship in the preset coordinate system, and the third cost matrix represents the third matching relationship on the historical trajectory in the preset coordinate system. The three matching relationships are weighted to obtain the final target matching relationship (i.e., the target cost matrix), and then the fused target is output according to the Hungarian matching algorithm, and the fused target is judged as a strong matching target.

[0039] As an optional embodiment, constructing a first cost matrix of the millimeter wave target and the visual target in the image coordinate system specifically includes: mapping the millimeter wave target to the image coordinate system to obtain a first coordinate; correcting the first coordinate according to the visual target to obtain a second coordinate; and using the second coordinate and the third coordinate of the visual target in the image coordinate system to construct the first cost matrix.

[0040] A single-frame millimeter-wave target is mapped to the image coordinate system to obtain a first coordinate. Due to the error in external parameter calibration, errors may occur in the points of the millimeter-wave target in the image coordinate system. Therefore, the first coordinate needs to be corrected according to the visual target, specifically including: calculating the RT transformation matrix from the millimeter-wave target to the visual target in the image coordinate system through a matching algorithm (for example, an ICP point cloud matching algorithm), and then correcting the millimeter-wave target in the image coordinate system through the RT transformation matrix to obtain a second coordinate.

[0041] The first cost matrix is ​​constructed by the second coordinate of the millimeter wave target and the third coordinate of the visual target in the image coordinate system. The specific formula is:

[0042]

[0043] Among them, vis.x represents the horizontal coordinate of the visual target, vis.y represents the vertical coordinate of the visual target, radar.x represents the horizontal coordinate of the millimeter wave target, and radar.y represents the horizontal coordinate of the millimeter wave target.

[0044] As an optional embodiment, a second cost matrix of the millimeter wave target and the visual target in a preset coordinate system is constructed, specifically including: mapping the visual target to the preset coordinate system to obtain a fourth coordinate; calculating the first position distance, speed difference and heading deviation between the visual target and the millimeter wave target based on the fourth coordinate and the fifth coordinate of the millimeter wave target in the preset coordinate system; and constructing a second cost matrix based on the first position distance, speed difference and heading deviation.

[0045] The single-frame visual target is mapped to the preset coordinate system through the coordinate system transformation matrix.

[0046] Specifically, according to the fourth coordinate and the fifth coordinate of the millimeter wave target in the preset coordinate system, the first position distance between the visual target and the millimeter wave target is calculated according to formula (2):

[0047]

[0048] Among them, dist_cost is the first position distance between the visual target and the millimeter wave target, vis.vcs_x is the horizontal coordinate of the visual target in the preset coordinate system, vis.vcs_y is the vertical coordinate of the visual target in the preset coordinate system, radar.vcs_x is the horizontal coordinate of the millimeter wave target in the preset coordinate system, and radar.vcs_y is the horizontal coordinate of the millimeter wave target in the preset coordinate system.

[0049] Specifically, the lateral speed and longitudinal speed of the millimeter wave target at the fifth coordinate are detected, and the lateral speed and longitudinal speed of the visual target at the fourth coordinate are detected, and the speed difference between the visual target and the millimeter wave target is calculated according to formula (3):

[0050]

[0051] Among them, v_cost represents the speed difference, radar.vcs_vx represents the lateral speed of the millimeter wave target at the fifth coordinate, radar.vcs_vy represents the lateral speed of the millimeter wave target at the fifth coordinate, vis.vcs_vx represents the lateral speed of the visual target at the fourth coordinate, and vis.vcs_vy represents the longitudinal speed of the visual target at the fourth coordinate.

[0052] Specifically, the heading angle when collecting the visual target and the heading angle when collecting the millimeter wave target are obtained, and the heading deviation between the visual target and the millimeter wave target is calculated according to formula (4):

[0053] h_cost=vis.heading-radar.heading formula (4)

[0054] Among them, h_cost represents the heading deviation, vis.heading represents the heading angle when collecting visual targets, and radar.heading represents the heading angle when collecting millimeter wave targets.

[0055] Specifically, a second cost matrix is ​​constructed according to the first position distance, speed difference and heading deviation, including: assigning weight coefficients to the first position distance, speed difference and heading deviation, and then adding the first position distance, speed difference and heading deviation including the weight coefficient to obtain a third cost matrix.

[0056] The specific values ​​of the weight coefficients assigned to the first position distance, speed difference and heading deviation in this application are not limited and can be set according to the accuracy of each characteristic value of the sensor target in the actual project. Features with high accuracy can be assigned higher weight coefficients, and features with low accuracy can be assigned lower weight coefficients.

[0057] For example, if the weight coefficient assigned to the first position distance is 1, the weight coefficient assigned to the speed difference is 5, and the weight coefficient assigned to the heading deviation is 0.5, then the third cost matrix can be obtained by formula (5):

[0058] cost2=dist_cost+v_cost*5+h_cost*0.1 Formula (5)

[0059] As an optional embodiment, constructing a third cost matrix of the millimeter wave target and the visual target in a preset coordinate system specifically includes: obtaining the historical trajectory of the millimeter wave target and the visual target in the preset coordinate system; determining the sixth coordinate of the preset frame of the millimeter wave target on the historical trajectory, and determining the seventh coordinate of the preset frame of the visual target on the historical trajectory, wherein the preset frame is a preset number of frames closest to the current moment; calculating the second position distance between the millimeter wave target and the visual target in each frame according to the sixth coordinate and the seventh coordinate, and using multiple second position distances to construct the third cost matrix.

[0060] For example, the preset number is not limited. If the preset number is 30, then the preset frames are the 30 frames closest to the current frame on the historical trajectory, and the second position distance between the millimeter wave target and the visual target on the historical trajectory is calculated for each frame.

[0061] The calculation formula is:

[0062]

[0063] Among them, vis.vcs_ix represents the horizontal coordinate of the visual target in the i-th frame, vis.vcs_iy represents the vertical and horizontal coordinates of the visual target in the i-th frame, vis.vcs_iy represents the horizontal coordinate of the millimeter wave target in the i-th frame, radar.vcs_iy represents the vertical coordinate of the millimeter wave target in the i-th frame, and n represents the preset number.

[0064] As an optional embodiment, a target cost matrix is ​​obtained according to the first cost matrix, the second cost matrix and the third cost matrix, specifically including: normalizing the first cost matrix, the second cost matrix and the third cost matrix to obtain a fourth cost matrix, a fifth cost matrix and a sixth cost matrix respectively; assigning weight values ​​to the fourth cost matrix, the fifth cost matrix and the sixth cost matrix respectively; adding the fourth cost matrix, the fifth cost matrix and the sixth cost matrix after the weight values ​​are assigned to obtain the target cost matrix.

[0065] Specifically, the three cost matrices are normalized to obtain cost matrices at the same scale. The first cost matrix is ​​normalized to obtain a fourth cost matrix, the second cost matrix is ​​normalized to obtain a fifth cost matrix, and the third cost matrix is ​​normalized to obtain a sixth cost matrix.

[0066] Different weight values ​​are assigned to each normalized cost matrix. For example, if the image matching error is large, a lower weight coefficient (e.g., 0.1) is assigned to the first cost matrix, if the matching error in the preset coordinate system is low, a higher weight coefficient (e.g., 0.2) is assigned to the second cost matrix, and if the historical trajectory matching error is the lowest, the third cost matrix is ​​assigned the largest weight coefficient (e.g., 0.7).

[0067] As an optional embodiment, after obtaining the matching result of the millimeter wave target and the visual target, the method also includes: if the matching result indicates that the millimeter wave target and the visual target match, binding the first identifier of the millimeter wave target with the second identifier of the visual target, and marking the millimeter wave target and the visual target as a strong matching state; calculating a third position distance between the millimeter wave target and the visual target, and comparing the third position distance with a preset distance; adjusting the matching coefficient of the millimeter wave target and the visual target according to the comparison result; and unbinding the millimeter wave target from the visual target when the matching coefficient is adjusted to a preset value.

[0068] The preset value in this application can be set to 0.

[0069] The target cost matrix is ​​used as the input of the Hungarian algorithm, and an association matrix representing the matching results can be output. The association matrix is ​​used to determine whether the millimeter wave target and the visual target match. If they match, the millimeter wave target and the visual target are marked as a strong match state, and the first identifier of the millimeter wave target is bound to the second identifier of the visual target. In subsequent processing, only the identifier needs to be matched, and there is no need to rebuild the cost matrix for matching.

[0070] The targets of the two sources in the strong matching state need to be verified at regular intervals. The specific verification method is: calculate the third position distance between the visual target and the millimeter wave target, and compare the third position distance with the preset distance; if the third position distance is less than the preset distance, increase the matching coefficient; if the third position distance is greater than or equal to the preset distance, reduce the matching coefficient.

[0071] When the distance of the third position is greater than or equal to the preset distance, the visual target and the millimeter wave target are considered to be in a pseudo-matching state, and the distance error between the visual target and the millimeter wave target is calculated, and then the distance error is multiplied by the current matching coefficient to reduce the matching coefficient. When the matching coefficient is about to be 0, it means that the visual target and the millimeter wave target do not match, so the binding of the visual target and the millimeter wave target is released, and the visual target and the millimeter wave target are added to the matching pool to restart the matching.

[0072] By verifying the bound source target, it can be ensured that the binding will not be directly untied when interference occurs between the visual target and the millimeter wave target. In addition, if an incorrect binding occurs, it can be quickly untied.

[0073] This application does not limit the preset distance, which can be set according to actual conditions.

[0074] The present application uses a multi-sensor target matching method, including: obtaining the millimeter wave target and the visual target to be matched; constructing the first cost matrix of the millimeter wave target and the visual target in the image coordinate system, the second cost matrix and the third cost matrix in the preset coordinate system, wherein the preset coordinate system is constructed with the center of the mobile tool as the origin and the front of the mobile tool as the horizontal coordinate; obtaining the target cost matrix according to the first cost matrix, the second cost matrix and the third cost matrix; using the target cost matrix as the input of the Hungarian algorithm to obtain the matching result of the millimeter wave target and the visual target. By constructing multiple cost matrices of the millimeter wave target and the visual target, and then obtaining the most accurate target cost matrix according to the multiple cost matrices, and finally using the target cost matrix as the input of the Hungarian algorithm, the matching result is obtained to reduce the error of matching the millimeter wave target and the visual image target.

[0075] According to another aspect of the embodiment of the present application, the present application provides a multi-sensor target matching device, such as Figure 2 As shown, including:

[0076] An acquisition module 202 is used to acquire a millimeter wave target and a visual target to be matched;

[0077] A construction module 204 is used to construct a first cost matrix of the millimeter wave target and the visual target in the image coordinate system, a second cost matrix and a third cost matrix in the preset coordinate system, wherein the preset coordinate system is constructed with the center of the moving tool as the origin and the front of the moving tool as the horizontal coordinate;

[0078] A processing module 206, configured to obtain a target cost matrix according to the first cost matrix, the second cost matrix and the third cost matrix;

[0079] The matching module 208 is used to use the target cost matrix as the input of the Hungarian algorithm to obtain the matching result between the millimeter wave target and the visual target.

[0080] It should be noted that the acquisition module 202 in this embodiment can be used to execute step 101 in the embodiment of the present application, the construction module 204 in this embodiment can be used to execute step 103 in the embodiment of the present application, the processing module 206 in this embodiment can be used to execute step 105 in the embodiment of the present application, and the matching module 208 in this embodiment can be used to execute step 107 in the embodiment of the present application.

[0081] Optionally, the construction module 204 is also used to map the millimeter wave target to the image coordinate system to obtain a first coordinate; correct the first coordinate according to the visual target to obtain a second coordinate; and construct a first cost matrix using the second coordinate and the third coordinate of the visual target in the image coordinate system.

[0082] Optionally, the construction module 204 is also used to map the visual target to a preset coordinate system to obtain a fourth coordinate; calculate the first position distance, speed difference and heading deviation between the visual target and the millimeter wave target based on the fourth coordinate and the fifth coordinate of the millimeter wave target in the preset coordinate system; and construct a second cost matrix based on the first position distance, speed difference and heading deviation.

[0083] Optionally, the construction module 204 is also used to obtain the historical trajectory of the millimeter wave target and the visual target in a preset coordinate system; determine the sixth coordinate of the preset frame of the millimeter wave target on the historical trajectory, and determine the seventh coordinate of the preset frame of the visual target on the historical trajectory, wherein the preset frame is a preset number of frames closest to the current moment; calculate the second position distance between the millimeter wave target and the visual target in each frame according to the sixth coordinate and the seventh coordinate, and use multiple second position distances to construct a third cost matrix.

[0084] Optionally, the processing module 206 is also used to normalize the first cost matrix, the second cost matrix and the third cost matrix to obtain a fourth cost matrix, a fifth cost matrix and a sixth cost matrix, respectively; assign weight values ​​to the fourth cost matrix, the fifth cost matrix and the sixth cost matrix, respectively; and add the fourth cost matrix, the fifth cost matrix and the sixth cost matrix after the weight values ​​are assigned to obtain a target cost matrix.

[0085] Optionally, the device also includes an adjustment module, which is used to, after obtaining the matching result of the millimeter wave target and the visual target, if the matching result indicates that the millimeter wave target and the visual target match, bind the first identifier of the millimeter wave target with the second identifier of the visual target, and mark the millimeter wave target and the visual target as a strong matching state; calculate the third position distance between the millimeter wave target and the visual target, and compare the third position distance with a preset distance; adjust the matching coefficient of the millimeter wave target and the visual target according to the comparison result; and unbind the millimeter wave target from the visual target when the matching coefficient is adjusted to a preset value.

[0086] It should be noted here that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments.

[0087] According to another aspect of the embodiments of the present application, the present application provides an electronic device, such as Figure 3 As shown, it includes a memory 301, a processor 303, a communication interface 305 and a communication bus 307. The memory 301 stores a computer program that can be run on the processor 303. The memory 301 and the processor 303 communicate through the communication interface 305 and the communication bus 307. When the processor 303 executes the computer program, the steps of the above method are implemented.

[0088] The memory and processor in the above electronic device communicate via a communication bus and a communication interface. The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc.

[0089] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0090] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0091] According to another aspect of the embodiments of the present application, a mobile tool is provided, including the above-mentioned electronic device.

[0092] The mobile tool of the present application can be a vehicle or robot with an autonomous driving function, such as a vacuum truck, a floor scrubber, a sprinkler truck, a sewage treatment truck, a sanitation truck, a lawn mower, a golf cart, an entertainment vehicle, an amusement park autonomous driving device, a balance car, a sweeping robot, a food delivery robot, a shopping guide robot, etc.

[0093] In some possible implementations, the electronic device provided in the present application is an autonomous driving domain controller, and the mobile tool provided in the present application is equipped with an autonomous driving domain controller and various sensors for sensing environmental information and the mobile tool's own information (including but not limited to lidar, camera, millimeter wave radar, ultrasonic radar, inertial measurement unit, wheel speed meter, etc.). When the mobile tool is in working state, the autonomous driving domain controller is communicated with the various sensors loaded on the mobile tool, and path planning is performed for the mobile tool based on the environmental information collected by these sensors and the mobile tool's own information, so as to achieve the purpose of controlling the mobile tool to drive automatically.

[0094] According to another aspect of the embodiments of the present application, a computer-readable medium having a non-volatile program code executable by a processor is provided.

[0095] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0096] When the embodiments of the present application are specifically implemented, reference may be made to the above-mentioned embodiments, which have corresponding technical effects.

[0097] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0098] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0099] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0100] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0101] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0102] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0103] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0104] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk. It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0105] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A multi-sensor target matching method, characterized in that: include: Obtain the millimeter wave target and visual target to be matched; Constructing a first cost matrix of the millimeter wave target and the visual target in the image coordinate system, a second cost matrix and a third cost matrix in the preset coordinate system, wherein the preset coordinate system is constructed with the center of the mobile tool as the origin and the front of the mobile tool as the horizontal coordinate; Obtain a target cost matrix according to the first cost matrix, the second cost matrix and the third cost matrix; The target cost matrix is ​​used as an input of the Hungarian algorithm to obtain a matching result between the millimeter wave target and the visual target.

2. The method according to claim 1, characterized in that The constructing a first cost matrix of the millimeter wave target and the visual target in the image coordinate system specifically includes: Mapping the millimeter wave target to the image coordinate system to obtain a first coordinate; Correcting the first coordinate according to the visual target to obtain a second coordinate; The first cost matrix is ​​constructed using the second coordinates and the third coordinates of the visual target in the image coordinate system.

3. The method according to claim 1, characterized in that The constructing a second cost matrix of the millimeter wave target and the visual target in a preset coordinate system specifically includes: Mapping the visual target to the preset coordinate system to obtain a fourth coordinate; Calculating a first position distance, a speed difference, and a heading deviation between the visual target and the millimeter wave target according to the fourth coordinate and a fifth coordinate of the millimeter wave target in the preset coordinate system; The second cost matrix is ​​constructed according to the first position distance, the speed difference and the heading deviation.

4. The method according to claim 3, characterized in that The constructing a third cost matrix of the millimeter wave target and the visual target in a preset coordinate system specifically includes: Acquire historical trajectories of the millimeter wave target and the visual target in the preset coordinate system; Determine a sixth coordinate of a preset frame of the millimeter wave target on the historical trajectory, and determine a seventh coordinate of the preset frame of the visual target on the historical trajectory, wherein the preset frame is a preset number of frames closest to the current moment; The second position distance between the millimeter wave target and the visual target in each frame is calculated according to the sixth coordinate and the seventh coordinate, and the third cost matrix is ​​constructed using a plurality of the second position distances.

5. The method according to claim 1, characterized in that The obtaining of a target cost matrix according to the first cost matrix, the second cost matrix and the third cost matrix specifically includes: Normalizing the first cost matrix, the second cost matrix, and the third cost matrix to obtain a fourth cost matrix, a fifth cost matrix, and a sixth cost matrix, respectively; Assigning weight values ​​to the fourth cost matrix, the fifth cost matrix and the sixth cost matrix respectively; The fourth cost matrix, the fifth cost matrix, and the sixth cost matrix after being assigned the weight values ​​are added together to obtain the target cost matrix.

6. The method according to claim 1, characterized in that After obtaining the matching result between the millimeter wave target and the visual target, the method further includes: If the matching result indicates that the millimeter wave target matches the visual target, binding the first identifier of the millimeter wave target with the second identifier of the visual target, and marking the millimeter wave target and the visual target as a strong match state; Calculating a third position distance between the millimeter wave target and the visual target, and comparing the third position distance with a preset distance; Adjusting the matching coefficient between the millimeter wave target and the visual target according to the comparison result; When the matching coefficient is adjusted to a preset value, the millimeter wave target is unbound from the visual target.

7. A multi-sensor target matching device, characterized in that: include: An acquisition module, used to acquire the millimeter wave target and the visual target to be matched; A construction module, used to construct a first cost matrix of the millimeter wave target and the visual target in the image coordinate system, a second cost matrix and a third cost matrix in the preset coordinate system, wherein the preset coordinate system is constructed with the center of the moving tool as the origin and the front of the moving tool as the horizontal coordinate; A processing module, configured to obtain a target cost matrix according to the first cost matrix, the second cost matrix and the third cost matrix; The matching module is used to use the target cost matrix as an input of the Hungarian algorithm to obtain a matching result between the millimeter wave target and the visual target.

8. An electronic device, comprising a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, and the memory and the processor communicate through the communication bus and the communication interface, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A mobile tool, characterized in that: Comprising the electronic device as claimed in claim 8.

10. A computer readable medium having a non-volatile program code executable by a processor, characterized in that: The program code enables the processor to execute the method according to any one of claims 1 to 6.

Images