Target matching method and device, electronic equipment and computer readable storage medium

Abstract

This application provides a target matching method, apparatus, electronic device, and computer-readable storage medium. The method includes: acquiring data collected by a first sensor and data collected by a second sensor; determining multiple first targets based on the data collected by the first sensor, and determining multiple second targets based on the data collected by the second sensor; performing global estimation processing based on a first spatial deviation between the first targets and the second targets to obtain a global transformation matrix; performing spatial alignment processing on the data collected by the first sensor and the data collected by the second sensor based on the global transformation matrix; and performing matching processing on the spatially aligned data collected by the first sensor and the data collected by the second sensor to obtain a matching target pair, identifying the two targets in the matching target pair as the same target; wherein, the matching target pair includes one first target and one second target. This application improves the accuracy of target matching.

Description

Technical Field

[0001] This application relates to intelligent sensing technology, and more particularly to a target matching method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Intelligent sensing refers to the process of collecting signals from the real world through cameras, microphones, or other sensors, mapping them to the digital world using technologies such as image recognition and speech recognition, and then further elevating this digital information to a cognitive level, such as memory, understanding, planning, and decision-making. Target recognition is a major branch of intelligent sensing, primarily studying how to identify specific targets, such as pedestrians and vehicles, from sensor data.

[0003] Given the limited scope of target recognition results from a single sensor, current methods increasingly employ sensor fusion, which integrates the target recognition results from multiple sensors to determine which targets are accurately identified. Current solutions typically align data from multiple sensors in both time and space for target matching. However, extensive research has revealed that time alignment is prone to errors, and these errors introduce additional spatial errors, resulting in low accuracy and a high false-match rate in target matching. Summary of the Invention

[0004] This application provides a target matching method, apparatus, electronic device, and computer-readable storage medium that can effectively fuse data from multiple sensors, thereby improving the accuracy of target matching.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a target matching method, characterized in that it includes:

[0007] Acquire data collected by the first sensor and data collected by the second sensor;

[0008] Multiple first targets are determined based on the data collected by the first sensor, and multiple second targets are determined based on the data collected by the second sensor;

[0009] A global estimation process is performed based on the first spatial deviation between the first target and the second target to obtain the global transformation matrix;

[0010] Spatial alignment processing is performed on the data collected by the first sensor and the data collected by the second sensor according to the global transformation matrix;

[0011] The data collected by the first sensor and the data collected by the second sensor after spatial alignment are matched to obtain a matching target pair, and the two targets in the matching target pair are identified as the same target; wherein, the matching target pair includes a first target and a second target.

[0012] This application provides a target matching device, including:

[0013] The acquisition module is used to acquire data collected by the first sensor and data collected by the second sensor;

[0014] The target determination module is used to determine multiple first targets based on the data collected by the first sensor, and to determine multiple second targets based on the data collected by the second sensor;

[0015] The optimization estimation module is used to perform global estimation based on the first spatial deviation between the first target and the second target to obtain the global transformation matrix.

[0016] The spatial alignment module is used to perform spatial alignment processing on the data collected by the first sensor and the data collected by the second sensor according to the global transformation matrix;

[0017] The matching module is used to perform matching processing on the data collected by the first sensor and the data collected by the second sensor after spatial alignment processing to obtain a matching target pair, and to determine the two targets in the matching target pair as the same target; wherein, the matching target pair includes a first target and a second target.

[0018] This application provides an electronic device, including:

[0019] Memory, used to store executable instructions;

[0020] The processor, when executing executable instructions stored in the memory, implements the target matching method provided in the embodiments of this application.

[0021] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the target matching method provided in this application.

[0022] This application provides a computer program product including executable instructions for inducing a processor to execute and implement the target matching method provided in this application.

[0023] The embodiments of this application have the following beneficial effects:

[0024] In this embodiment, when acquiring data collected by the first sensor and data collected by the second sensor, multiple first targets are determined based on the data collected by the first sensor, and multiple second targets are determined based on the data collected by the second sensor. A global transformation matrix is ​​calculated based on the first spatial deviation between the first targets and the second targets, and then spatial alignment processing is performed. This enables effective fusion of data collected by the two sensors, reduces the occurrence of missed and incorrect matches, and improves the accuracy of target matching. Since the entire process utilizes spatially related information, it avoids time alignment errors and additional spatial errors introduced by time alignment errors. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the target matching system provided in an embodiment of this application;

[0027] Figure 2 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0028] Figure 3A This is a flowchart illustrating a target matching method provided in an embodiment of this application;

[0029] Figure 3B This is another flowchart illustrating the target matching method provided in the embodiments of this application;

[0030] Figure 3C This is another flowchart illustrating the target matching method provided in the embodiments of this application;

[0031] Figure 4 This is a flowchart illustrating a target matching method for a vehicle driving environment provided in an embodiment of this application;

[0032] Figure 5 This is a schematic flowchart illustrating the process of solving the global transformation matrix provided in an embodiment of this application;

[0033] Figure 6 This is a schematic diagram of a radar target and a visual target provided in an embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] In the following description, references to "some embodiments" describe a subset of all possible embodiments; however, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict. In the following description, the term "a plurality of" means at least two.

[0036] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0038] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0039] 1) Target: refers to the specific type of object that needs to be identified. This application does not limit the specific type, but can be determined according to the actual scenario. For example, in a traffic scenario, the target that needs to be identified may include people, vehicles, road traffic signs, etc.

[0040] 2) Data collected by the sensors: This application does not limit the type of sensor, such as a camera, millimeter-wave radar, or lidar. Correspondingly, the data collected by the camera can be image data, and the data collected by the millimeter-wave radar can be point cloud data. For the data collected by each sensor, multiple targets can be identified through target recognition, and the specific target recognition algorithm is not limited.

[0041] 3) Spatial attributes: These are a series of parameters that describe the target based on the space corresponding to the data collected by the sensor. For example, spatial attributes may include at least one of the following parameters: horizontal coordinate, vertical coordinate, velocity, and angle.

[0042] 4) Spatial deviation: refers to the deviation in at least some parameters of a spatial attribute. Depending on the parameters of interest, different types of spatial deviations can be formed. In the embodiments of this application, a first spatial deviation, a second spatial deviation, and a third spatial deviation are involved. The first spatial deviation, the second spatial deviation, and the third spatial deviation can be the same type of spatial deviation or different types of spatial deviations.

[0043] 5) Transformation matrix: The spatial deviation between two targets is represented by a matrix. Based on this matrix, a linear transformation of the targets in space can be achieved to eliminate the spatial deviation. Such a matrix is ​​called a transformation matrix.

[0044] Multi-sensor fusion technology is a key technology in the field of intelligent sensing. In related solutions, time alignment (time synchronization) and motion estimation are typically used to overcome the problem of inconsistent observation times from different sensors. Furthermore, target matching is usually performed based on individual targets. Through extensive research, the inventors have discovered that the solutions provided by these technologies have at least the following problems: 1) Errors often occur during time alignment, and these errors also introduce additional spatial errors, resulting in low accuracy and a high false-match rate in the final target matching; 2) When matching is based on a single target, false matching and missed matching are prone to occur.

[0045] In view of this, embodiments of this application provide a target matching method, apparatus, electronic device, and computer-readable storage medium, which can improve the accuracy of target matching and thus accurately identify targets in real-world scenarios. The following describes exemplary applications of the electronic device provided in embodiments of this application. The electronic device provided in embodiments of this application can be implemented as various types of terminal devices or as a server.

[0046] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the target matching system 100 provided in the embodiment of this application. The terminal device 400 is connected to the server 200 through the network 300, wherein the network 300 can be a wide area network or a local area network, or a combination of the two.

[0047] In some embodiments, taking the electronic device as a terminal device as an example, the target matching method provided in this application embodiment can be implemented by the terminal device. For example, the terminal device 400 acquires data collected by a first sensor and data collected by a second sensor, wherein the first sensor and the second sensor can collect data for the same real environment; determines multiple first targets based on the data collected by the first sensor, and determines multiple second targets based on the data collected by the second sensor; performs global estimation processing based on the first spatial deviation between the first target and the second target to obtain a global transformation matrix; performs spatial alignment processing on the data collected by the first sensor and the data collected by the second sensor based on the global transformation matrix; performs matching processing on the spatially aligned data collected by the first sensor and the data collected by the second sensor to obtain a matching target pair, and determines the two targets in the matching target pair as the same target; wherein the matching target pair includes a first target and a second target.

[0048] Figure 1 The example illustrates the deployment of terminal device 400 in a vehicle. In this scenario, both the first and second sensors are also deployed in the vehicle and connected to terminal device 400 via wired or wireless means. For instance, the first sensor could be an onboard millimeter-wave radar, and the second sensor could be an onboard camera. The real-world environment refers to the vehicle's driving environment. Terminal device 400 can perform corresponding operations based on the obtained matching results (i.e., matching target pairs). For example, it can plan an autonomous driving scheme and perform autonomous driving based on the matching results; or, for another example, it can display the matching results on the vehicle's display screen so that the driver can understand the surrounding situation.

[0049] In some embodiments, taking a server as an example, the target matching method provided in this application can be implemented by a server. For example, the server 200 receives data collected by a first sensor and data collected by a second sensor sent by the terminal device 400; determines multiple first targets based on the data collected by the first sensor, and determines multiple second targets based on the data collected by the second sensor; performs global estimation processing based on the first spatial deviation between the first target and the second target to obtain a global transformation matrix; performs spatial alignment processing on the data collected by the first sensor and the data collected by the second sensor based on the global transformation matrix; performs matching processing on the spatially aligned data collected by the first sensor and the data collected by the second sensor to obtain a matching target pair, and determines the two targets in the matching target pair as the same target; wherein, the matching target pair includes a first target and a second target.

[0050] Server 200 can generate corresponding instructions (such as autonomous driving instructions or display instructions) based on the matching results and send them to terminal device 400 for execution. Of course, server 200 can also directly send the matching results to terminal device 400 so that terminal device 400 can perform corresponding operations based on the matching results.

[0051] In some embodiments, the terminal device 400 or server 200 can implement the target matching method provided in this application embodiment by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run; it can also be a small program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a small program that can be embedded in any APP, wherein the small program can be controlled by the user to run or close. In short, the above-mentioned computer program can be any form of application, module or plugin.

[0052] In some embodiments, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The cloud service can be a target matching service, which can be invoked by terminal device 400. Terminal device 400 can be an in-vehicle device, smartphone, tablet computer, laptop computer, desktop computer, smart TV, smartwatch, etc., but is not limited to these. Terminal devices and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0053] Taking the example of a terminal device provided in this application embodiment, it can be understood that in the case where the electronic device is a server, Figure 2 Some parts of the structure shown (such as the user interface, presentation module, and input processing module) can be omitted. See also Figure 2 , Figure 2 This is a schematic diagram of the structure of the terminal device 400 provided in the embodiments of this application. Figure 2The terminal device 400 shown includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the terminal device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.

[0054] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0055] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0056] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0057] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0058] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0059] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0060] The network communication module 452 is used to reach other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0061] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with user interface 430;

[0062] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0063] In some embodiments, the target matching device provided in this application can be implemented in software. Figure 2 A target matching device 455 stored in memory 450 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: an acquisition module 4551, a target determination module 4552, an optimization estimation module 4553, a spatial alignment module 4554, and a matching module 4555. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0064] The target matching method provided in this application will be described in conjunction with exemplary applications and implementations of the electronic devices provided in the embodiments of this application.

[0065] See Figure 3A , Figure 3A This is a flowchart illustrating a target matching method provided in an embodiment of this application, which will be combined with... Figure 3A The steps shown are explained.

[0066] In step 101, the data collected by the first sensor and the data collected by the second sensor are acquired.

[0067] This application embodiment fuses data collected by two sensors, named a first sensor and a second sensor. The first and second sensors can be different types of sensors, such as a radar for the first sensor and a camera for the second sensor; alternatively, they can be the same type of sensor, such as both being radars. First, the data collected by the first sensor and the data collected by the second sensor are acquired. For example, data obtained from data collection by both sensors on the same real-world environment is acquired. The real-world environment refers to the environment in the real world where target identification is required. This environment can be static or dynamic, such as a dynamic vehicle driving environment. In this case, the sensor can be an onboard sensor.

[0068] In step 102, multiple first targets are determined based on the data collected by the first sensor, and multiple second targets are determined based on the data collected by the second sensor.

[0069] Here, the data collected by the first sensor is processed for target recognition to obtain multiple first targets, and the data collected by the second sensor is processed for target recognition to obtain multiple second targets. The method of target recognition processing is not limited; for example, a Convolutional Neural Network (CNN) algorithm can be used. For the identified targets, spatial attributes are used to describe them. These spatial attributes include at least one of the following: horizontal coordinate, vertical coordinate, angle, and velocity. The parameters required for these spatial attributes can be set according to actual needs.

[0070] It is worth noting that, in order to ensure the effectiveness of target matching, the time when the first sensor collects data should be as close as possible to the time when the second sensor collects data. For example, it can be set that both the first and second sensors collect data periodically, and each time the latest data collected by the first sensor and the latest data collected by the second sensor are used for target matching.

[0071] In some embodiments, after step 102, the method further includes: performing the following processing on the data collected by any one of the sensors: identifying targets with speeds greater than a speed threshold as targets for global estimation processing; wherein the data collected by any one of the sensors is the data collected by the first sensor or the data collected by the second sensor.

[0072] In dynamic real-world environments, some types of sensors (such as radar) may not be able to accurately detect low-speed targets. Therefore, the velocities of all targets in the sensor's data can be compared with a velocity threshold. Targets with velocities exceeding the threshold are identified as dynamic targets for global estimation. This approach, using dynamic targets for global estimation, improves the accuracy of the estimation.

[0073] It is worth noting that dynamic targets are only used for global estimation. In the subsequent spatial alignment process, all targets (including dynamic and non-dynamic targets) in the data collected by the sensor need to be spatially aligned to avoid omissions.

[0074] In step 103, a global estimation process is performed based on the first spatial deviation between the first target and the second target to obtain the global transformation matrix.

[0075] In this embodiment, a first spatial deviation between a first target and a second target is calculated, and a corresponding transformation matrix is ​​determined based on the first spatial deviation. The transformation matrix is ​​used to perform spatial transformation on either the first target or the second target to eliminate the first spatial deviation. It is worth noting that the first spatial deviation refers to the deviation calculated for at least some parameters in the spatial attributes.

[0076] Understandably, since there are multiple first targets and multiple second targets, multiple transformation matrices can be calculated, for example, by enumeration. Then, a global transformation matrix is ​​selected from these multiple transformation matrices through global estimation. This global transformation matrix is ​​applied to the data collected by either the first or second sensor to achieve global spatial alignment. Global estimation refers to estimating the effectiveness of the transformation matrix using specific metrics to select the best-performing transformation matrix as the global transformation matrix. These metrics, such as the matching degree metric, will be explained later.

[0077] In some embodiments, the real environment is the vehicle driving environment; the first spatial deviation is determined in the following ways: when the vehicle is in a non-steering state, the positional deviation between the first target and the second target is determined as the first spatial deviation; when the vehicle is in a steering state, both the positional deviation and the angular deviation between the first target and the second target are determined as the first spatial deviation.

[0078] Since the data collected by the first and second sensors are from the real-world environment, the parameters of interest when calculating spatial deviation can be selected based on the actual conditions of that environment. For example, in a vehicle driving environment, the following two situations exist:

[0079] 1) When the vehicle is in a non-steering state, the angular deviation (or rotation angle) between the first and second targets is small and within an acceptable range, therefore it does not need to be considered. Here, a global estimation process is performed based on the positional deviation between the first and second targets to obtain the global transformation matrix. That is, in this case, the first spatial deviation used for global estimation refers to the positional deviation, which can include at least one of the lateral distance (horizontal coordinate deviation) and the longitudinal distance (vertical coordinate deviation).

[0080] 2) When the vehicle is turning, the angular deviation between the first and second targets is large. Therefore, both the positional and angular deviations need to be used for global estimation to obtain the global transformation matrix. That is, in this case, the first spatial deviation used for global estimation includes both positional and angular deviations.

[0081] This application does not limit the method of determining the state of the vehicle. For example, the yaw rate collected by a special sensor (such as a Yaw-G sensor) installed on the vehicle can be obtained. When the yaw rate is greater than the angular velocity threshold, the vehicle is determined to be in a steering state; when the yaw rate is less than or equal to the angular velocity threshold, the vehicle is determined to be in a non-steering state.

[0082] Using the above method, when the vehicle is in a non-steering state, the angular deviation has a relatively small impact on spatial alignment, so the global transformation matrix is ​​determined based on the positional deviation. When the vehicle is steering, the angular deviation has a significant impact on spatial alignment, so the global transformation matrix is ​​determined based on both the positional and angular deviations. In this way, the accuracy of the determined global transformation matrix and the spatial alignment can be guaranteed for different vehicle states.

[0083] In step 104, spatial alignment processing is performed on the data collected by the first sensor and the data collected by the second sensor according to the global transformation matrix.

[0084] Here, spatial alignment refers to applying the global transformation matrix to the data collected by the first sensor or the data collected by the second sensor to achieve global alignment. The meaning of "global" is all targets in the data collected by a certain sensor.

[0085] In some embodiments, the above-mentioned spatial alignment processing of the data collected by the first sensor and the data collected by the second sensor according to the global transformation matrix can be achieved by performing any of the following processes: performing spatial transformation processing on multiple first targets according to the global transformation matrix; performing spatial transformation processing on multiple second targets according to the global transformation matrix.

[0086] For example, a spatial transformation can be performed on all first targets in the data collected by the first sensor using a global transformation matrix, thereby aligning them with all second targets in the data collected by the second sensor; similarly, a spatial transformation can be performed on all second targets in the data collected by the second sensor using a global transformation matrix, thereby aligning them with all first targets in the data collected by the first sensor. It is worth noting that spatial transformation refers to updating at least some parameters of the target's spatial attributes (parameters involved in the first spatial deviation) using a transformation matrix. This improves the flexibility of spatial alignment.

[0087] In step 105, the data collected by the first sensor and the data collected by the second sensor after spatial alignment are matched to obtain a matching target pair, and the two targets in the matching target pair are identified as the same target; wherein, the matching target pair includes a first target and a second target.

[0088] Spatial alignment processing can reduce the overall spatial deviation between the first target in the data collected by the first sensor and the second target in the data collected by the second sensor. Then, the data collected by the first sensor and the data collected by the second sensor after spatial alignment processing can be matched to obtain a matching target pair. The accuracy of the matching target pair determined in this way is relatively high. The matching processing actually refers to the matching between the first target and the second target. The matching target pair includes a first target and a second target.

[0089] The embodiments of this application do not limit the matching process. For example, the first target in the data collected by the first sensor can be traversed, and the third spatial deviation between the traversed first target and all second targets in the data collected by the second sensor can be determined. Based on the third spatial deviation, a second target is selected from the data collected by the second sensor (for example, the second target with the smallest third spatial deviation) to form a matching target pair with the traversed first target. Alternatively, global matching can be performed based on preset spatial deviation constraints to achieve global optimal matching.

[0090] For the obtained matching target pair, the two targets in the matching target pair are identified as the same target, and at the same time, the same target is identified as a real target that actually exists in the real environment.

[0091] In some embodiments, the matching target pairs can be further applied. For example, in a real-world scenario where the vehicle is in motion, an autonomous driving plan can be developed based on the matching target pairs, or the matching target pairs can be displayed on the vehicle's screen to inform the driver about the surrounding environment. It is worth noting that in these applications, spatial attributes may be required. In such cases, the spatial attribute of any target in the matching target pair can be used as the spatial attribute to be employed; alternatively, the spatial attributes of the two targets in the matching target pair can be averaged to obtain the spatial attribute to be employed.

[0092] This application describes the process of fusing data collected by two sensors to achieve target matching. It is understood that this application can also support the fusion of data collected by a larger number of sensors.

[0093] like Figure 3A As shown, this embodiment of the application effectively fuses the data collected by the two sensors through spatial alignment, which can reduce the occurrence of missed matching and incorrect matching, improve the accuracy of target matching, and avoid time alignment errors and additional spatial errors introduced by time alignment errors.

[0094] In some embodiments, see Figure 3B , Figure 3B This is a flowchart illustrating a target matching method provided in an embodiment of this application. Figure 3A Step 103 shown can be implemented through steps 201 to 203, and will be explained in conjunction with each step.

[0095] In step 201, multiple transformation matrices are determined based on the first spatial deviation between the first target and the second target.

[0096] Here, for a first target and a second target, a transformation matrix is ​​determined based on the first spatial deviation between the two targets. It is understandable that, since there are multiple first targets and multiple second targets, multiple transformation matrices can be obtained.

[0097] In step 202, for the target transformation matrix among multiple transformation matrices, multiple matching target pairs that can be obtained when the target transformation matrix is ​​used as the global transformation matrix are determined, and the matching degree index of the target transformation matrix is ​​determined based on the multiple matching target pairs.

[0098] For each of the multiple transformation matrices, assume that multiple matching target pairs can be obtained when the transformation matrix is ​​used as the global transformation matrix, and determine the matching degree index of the transformation matrix based on the multiple matching target pairs obtained. The matching degree index is used to estimate the effect of the transformation matrix.

[0099] To facilitate understanding, we will use the target transformation matrix among multiple transformation matrices as an example. The target transformation matrix represents any one of the multiple transformation matrices. The process of "determining the multiple matching target pairs that can be obtained when the target transformation matrix is ​​used as the global transformation matrix" simulates the process of "performing spatial alignment processing on the data collected by the first sensor and the data collected by the second sensor according to the target transformation matrix; and performing matching processing on the spatially aligned data collected by the first sensor and the data collected by the second sensor to obtain matching target pairs." It is worth noting that this process assumes that the target transformation matrix is ​​used as the global transformation matrix and does not substantially modify the data collected by the first sensor and the data collected by the second sensor.

[0100] The embodiments of this application do not limit the matching degree index, which can be various data related to the matching target pair. For example, it can include at least one of the following: the number of matching target pairs, and the second spatial deviation between two targets in the matching target pair (if the number of matching target pairs includes multiple pairs, the average value of the second spatial deviations corresponding to the multiple matching target pairs can be calculated).

[0101] In some embodiments, the matching degree index includes the number of matching target pairs and the second spatial deviation between the two targets in the matching target pairs; after determining the matching degree index of the target transformation matrix based on multiple matching target pairs, the method further includes: updating the target transformation matrix based on the second spatial deviation in the matching degree index of the target transformation matrix, and redetermining the matching degree index based on the updated target transformation matrix, until the number of matching target pairs in the new matching degree index no longer increases.

[0102] For the target transformation matrix, after obtaining the matching degree index, the target transformation matrix can be further updated based on the matching degree index to improve its accuracy. First, the target transformation matrix is ​​updated based on the second spatial deviation in the matching degree index corresponding to the target transformation matrix. For example, the transformation matrix is ​​determined based on the second spatial deviation, and this transformation matrix is ​​superimposed on the target transformation matrix to obtain the updated target transformation matrix. Then, the matching degree index is re-determined based on the updated target transformation matrix. Compared to the matching degree index corresponding to the target transformation matrix before the update, if the number of matching target pairs in the new matching degree index increases, the updated target transformation matrix is ​​updated again based on the new matching degree index. This process is repeated until the number of matching target pairs in the latest matching degree index no longer increases. Through this method, the target transformation matrix can be adjusted to further improve its accuracy.

[0103] In some embodiments, the second spatial deviation is determined as follows: when the vehicle is in a non-steering state, the positional deviation between the two targets in the matching target pair is determined as the second spatial deviation; when the vehicle is in a steering state, the angular deviation between the two targets in the matching target pair is determined as the second spatial deviation.

[0104] Here, in a real-world vehicle driving environment, the parameters of interest for the second spatial deviation are determined based on whether the vehicle is turning. For example, when the vehicle is not turning, there is no need to focus on the angle deviation; instead, the positional deviation between the two targets in the matching target pair is determined as the second spatial deviation. When the vehicle is turning, the angle deviation between the two targets in the matching target pair is determined as the second spatial deviation. It is worth noting that since the second spatial deviation is used to update the transformation matrix (equivalent to fine-tuning), when the vehicle is turning, the angle deviation can be primarily focused on for angle fine-tuning. By determining which parameters to update based on the actual state of the vehicle, the update effect of the transformation matrix can be improved.

[0105] In step 203, the multiple transformation matrices are filtered according to the matching degree index corresponding to each of the multiple transformation matrices to obtain the global transformation matrix.

[0106] Step 202 yields a corresponding matching degree index for each transformation matrix. Then, based on the matching degree indices corresponding to the multiple transformation matrices, the matrices are filtered to obtain the global transformation matrix. For example, preset index conditions can be used to select the transformation matrix corresponding to the matching degree index that meets the conditions as the global transformation matrix. These conditions include: maximizing the number of matched target pairs; exceeding a threshold number of matched target pairs; the proportion of the first target involved in the matched target pair among all first targets in the data collected by the first sensor exceeding a preset ratio; and minimizing the second spatial deviation between the two targets in the matched target pair.

[0107] In some embodiments, the above-described determination of multiple transformation matrices based on the first spatial deviation between the first target and the second target can be achieved by: selecting multiple first targets; and determining multiple transformation matrices based on the first spatial deviation between the selected first targets and each second target.

[0108] The above-mentioned filtering process based on the matching degree index corresponding to multiple transformation matrices can be achieved in the following way to obtain the global transformation matrix: when there is a matching degree index that satisfies the first index condition, the transformation matrix corresponding to the matching degree index that satisfies the first index condition is determined as the global transformation matrix; when there is no matching degree index that satisfies the first index condition, the step of selecting multiple first targets is re-executed.

[0109] When calculating the transformation matrix, one approach is to enumerate all possible transformation matrices. However, this approach involves excessive computation and can lead to inefficient target matching.

[0110] Therefore, multiple first targets in the data collected by the first sensor can be selected. This selection can be random or strategic; for example, considering that closer proximity leads to more accurate data acquisition, the first target with the smallest ordinate (closest to the sensor) can be prioritized. Then, the first spatial deviation between the selected first target and each second target is calculated, and multiple transformation matrices are determined. That is, a transformation matrix can be determined for each second target. Then, a matching degree index is determined for each transformation matrix. For the matching degree indices of the transformation matrices corresponding to the multiple second targets, if there is a matching degree index that satisfies the first index condition, the transformation matrix corresponding to the matching degree index that satisfies the first index condition is determined as the global transformation matrix. If there is no matching degree index that satisfies the first index condition, the step of selecting multiple first targets in the data collected by the first sensor is repeated, i.e., the next loop is entered until a matching degree index that satisfies the first index condition appears. The reselected first target should be different from the previously selected first target; that is, the selection process refers to selection without replacement.

[0111] It is worth noting that the above example illustrates the selection process for the first target. In some embodiments, the selection process for the second target can also be performed, meaning that the first target and the second target can be interchanged.

[0112] In some embodiments, the optimal transformation matrix can be selected from the transformation matrices corresponding to multiple second targets based on the matching degree index, and then it can be determined whether the matching degree index corresponding to the optimal transformation matrix satisfies the first index condition. In this way, the processing efficiency can be improved and the matching degree index of each transformation matrix can be compared with the first index condition.

[0113] For example, when the matching index includes the number of matched target pairs and the second spatial deviation between the two targets in a matched target pair, the transformation matrix with the most matched target pairs can be selected from the transformation matrices corresponding to multiple second targets as the optimal transformation matrix. If there are multiple transformation matrices with the most matched target pairs, the transformation matrix with the smallest second spatial deviation is selected as the optimal transformation matrix.

[0114] In some embodiments, the number of selected first targets includes multiple targets; before step 202, the method further includes: for any transformation matrix corresponding to any selected first target, performing the following processing: determining the matching target pairs that can be obtained when any transformation matrix is ​​used as the global transformation matrix, including other selected first targets; when the number of matching target pairs including other selected first targets is zero, discarding any transformation matrix.

[0115] Here, in each loop, multiple first targets can be selected. For ease of explanation, we will use the case where the first targets selected in a certain loop include first target A and first target B. For first target A, based on the first spatial deviation between first target A and each second target, multiple transformation matrices are determined. For each transformation matrix (for ease of understanding, we will use transformation matrix A as the explanation), we determine the matching target pairs that can be obtained by using transformation matrix A as the global transformation matrix and that include first target B. This simulates the process of "performing spatial transformation on first target B according to transformation matrix A; matching multiple second targets and the spatially transformed first target B to obtain matching target pairs." This process is only hypothetical and will not substantially modify the data collected by the first sensor or the second sensor. When the number of matching target pairs including first target B is not zero, transformation matrix A is retained; when the number of matching target pairs including first target B is zero, it proves that the effect of transformation matrix A is poor, and therefore transformation matrix A is discarded.

[0116] The same principle applies to the first target B. Based on the first spatial deviation between the first target B and each second target, multiple transformation matrices need to be determined. For each transformation matrix (taking transformation matrix B as an example), determine the matching target pairs that can be obtained when transformation matrix B is used as the global transformation matrix, and that include the first target A. If the number of matching target pairs including the first target A is not zero, retain transformation matrix B; if the number of matching target pairs including the first target A is zero, it proves that the effect of transformation matrix B is poor, and therefore discard transformation matrix B.

[0117] By using the above method, the effectiveness of the transformation matrix can be evaluated against each other based on multiple selected primary targets, and unqualified transformation matrices can be discarded, which can effectively reduce the amount of subsequent calculations and improve the efficiency of target matching.

[0118] In some embodiments, after re-executing the step of selecting multiple first targets in the data collected by the first sensor, the method further includes: when the number of re-executings is greater than the number threshold and there is a matching degree index that satisfies the second index condition, determining the transformation matrix corresponding to the matching degree index that satisfies the second index condition as the global transformation matrix.

[0119] Here, when the number of re-executions (i.e., the number of loops) exceeds a threshold, if no matching index satisfying the first criterion has yet appeared, then among all historically observed matching indices, it is determined whether a matching index satisfying the second criterion exists. If it does, the transformation matrix corresponding to the matching index satisfying the second criterion is determined and used as the global transformation matrix; if it does not exist, the loop ends, and a target matching failure message is output. It is understandable that the constraint of the second criterion should be weaker than that of the first criterion. For example, if the first criterion is that the proportion of the matched target to the first target among all the first targets in the data collected by the first sensor exceeds a first preset proportion, then the second criterion could be that the proportion of the matched target to the first target among all the first targets in the data collected by the first sensor exceeds a second preset proportion, where the second preset proportion is less than the first preset proportion.

[0120] By using the above method, when a satisfactory global transformation matrix cannot be determined after multiple iterations, the loop can be terminated by adopting a fallback approach, which can avoid unnecessary waste of resources and improve overall processing efficiency.

[0121] like Figure 3B As shown, this embodiment evaluates the effect of the transformation matrix based on the matching degree index to filter out the global transformation matrix, which can ensure the accuracy of the obtained global transformation matrix and thus improve the accuracy of target matching.

[0122] In some embodiments, see Figure 3C , Figure 3C This is a flowchart illustrating a target matching method provided in an embodiment of this application. Figure 3A Step 105 shown can be implemented through steps 301 to 305, and will be explained in conjunction with each step.

[0123] In step 301, global matching processing is performed on the data collected by the first sensor and the data collected by the second sensor after spatial alignment processing, based on the spatial deviation constraint condition, to obtain at least one target set; the target set includes at least one first target and at least one second target, and the third spatial deviation between the first target and the second target in the target set satisfies the spatial deviation constraint condition.

[0124] In this embodiment, the data collected by the first sensor and the second sensor after spatial alignment processing are matched to obtain a matching target pair. This matching process can be performed on a single target basis; however, single-target matching is prone to mismatches and missed matches. Therefore, a global matching process can be performed based on spatial deviation constraints to obtain at least one target set. This target set includes at least one first target and at least one second target. The third spatial deviation between any first target and any second target in the target set satisfies the spatial deviation constraint. The third spatial deviation can be set according to actual conditions, for example, it can include lateral distance (horizontal coordinate deviation), longitudinal distance (vertical coordinate deviation), and velocity deviation.

[0125] For each target set obtained, one first target and one second target are selected from the target set to form a matching target pair. The selection method will vary depending on the number of first targets and second targets in the target set.

[0126] In step 302, when the target set includes a first target and a second target, the target set is determined as a matching target pair.

[0127] When the target set includes a first target and a second target, the target set already meets the requirements for a matching target pair, so the target set is directly used as the matching target pair.

[0128] In step 303, when the target set includes multiple first targets and one second target, a greedy matching process is performed based on the spatial attributes of the multiple first targets in the target set, and the resulting first target is combined with one second target in the target set to form a matching target pair.

[0129] When the target set includes multiple primary targets and one secondary target, it is necessary to filter the primary targets. For example, a greedy matching process can be performed based on the spatial attributes of the primary targets in the target set to obtain a primary target. The greedy matching process uses a greedy algorithm, which means that when solving a problem, it always makes the choice that seems best at the current moment, i.e., it seeks a locally optimal solution. For example, regarding the spatial attributes of the multiple primary targets in the target set, we can focus on certain parameters of the spatial attributes and take the primary target that achieves the maximum or minimum value for these parameters as the result of the greedy matching. Which parameters to focus on can be determined based on the actual situation.

[0130] For example, in a real-world environment where vehicles are in motion, considering the spatial attributes of multiple primary targets in a target set, the primary target with the smallest ordinate (closest to the vehicle) can be selected as the result of a greedy matching algorithm. The rationale is that targets identified as closer to the vehicle are more likely to be real-world targets.

[0131] It is worth noting that when the target set includes a first target and multiple second targets, the filtering method is the same, that is, a greedy matching process is performed based on the spatial attributes of multiple second targets in the target set, and the resulting second target is combined with a first target in the target set to form a matching target pair.

[0132] In step 304, when the target set includes multiple first targets and multiple second targets, the multiple first targets and multiple second targets in the target set are enumerated and combined to obtain multiple combination schemes. Based on the third space deviation of the target pairs in each combination scheme, a target combination scheme is selected from the multiple combination schemes, and the target pairs in the target combination scheme are determined as matching target pairs; wherein, each combination scheme includes multiple target pairs.

[0133] When the target set includes multiple first targets and multiple second targets, multiple combination schemes can be obtained by enumerating and combining the multiple first targets and multiple second targets in the target set. Each combination scheme includes multiple candidate target pairs, and each target pair includes one first target and one second target. For each combination scheme, based on the third space deviation of the target pairs in the combination scheme, a target combination scheme is selected from the multiple combination schemes, and the target pairs in the target combination scheme are determined as the matching target pairs.

[0134] It is worth noting that since the combined scheme includes multiple target pairs, the third space deviations corresponding to each of the multiple target pairs can be averaged to obtain the average third space deviation. Then, based on the average third space deviations corresponding to the multiple combined schemes, the target combined scheme can be selected from the multiple combined schemes.

[0135] It is worth noting that the parameters of interest in the third spatial deviation can be set according to the actual situation. For example, the third spatial deviation may include lateral distance, longitudinal distance, and velocity deviation. When the third spatial deviation involves multiple types of parameters, in order to facilitate subsequent screening, the multiple types of parameters can be fused (e.g., a weighted sum of lateral distance, longitudinal distance, and velocity deviation). During subsequent screening, the results obtained from the fusion process can be directly compared. For example, the combination scheme with the smallest fusion result can be taken as the target combination scheme.

[0136] In some embodiments, when the number of first targets and the number of second targets meet a preset quantity condition, all first targets and all second targets are enumerated and combined to obtain multiple combination schemes. Based on the third space deviation of the target pairs in each combination scheme, a target combination scheme is selected from the multiple combination schemes, and the target pairs in the target combination scheme are determined as matching target pairs. Here, when the number of first targets and the number of second targets are sufficiently small (meeting the preset quantity condition), the computational cost of the enumeration method is within an acceptable range, so the enumeration method can be used to achieve target matching. The preset quantity condition can be set according to the actual application scenario, for example, the number of first targets and the number of second targets are both less than or equal to 2. Through the above method, the flexibility of target matching can be improved.

[0137] In step 305, the two targets in the matching target pair are determined to be the same target.

[0138] like Figure 3C As shown, the embodiments of this application obtain matching target pairs through global matching, which can achieve the best results globally and minimize the occurrence of mismatches and missed matches. At the same time, targeted strategies for determining matching target pairs are provided for different situations of the target set.

[0139] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario. For ease of understanding, the description will be based on a real-world vehicle driving environment.

[0140] Multi-sensor fusion technology is a key technology in the field of intelligent vehicle perception. In traditional fusion methods, time synchronization and motion estimation are usually used to overcome the problem of inconsistent observation time of different sensors. However, time synchronization is prone to errors and will also introduce additional spatial errors. Meanwhile, motion estimation has high accuracy requirements and is also prone to errors. In addition, traditional fusion methods mainly consider the matching of individual targets, which can easily lead to missed matching and mismatch.

[0141] To address the aforementioned problems, the inventors, after long-term research, proposed a global graph matching method for dynamic targets. This method primarily improves the matching performance of dynamic targets in a vehicle driving environment by solving for the optimal global graph matching solution. Firstly, by directly solving the transformation matrix, it avoids dependence on time synchronization and the accuracy of vehicle motion estimation. Secondly, by aligning the matching errors of each target after transformation matrix alignment, it eliminates time alignment errors, meaning only observation errors need to be considered. Therefore, corresponding matching methods can be designed for different observation methods, further improving matching accuracy. Graph matching refers to treating the data collected by sensors as a graph and matching different graphs to achieve the fusion of data collected by different sensors.

[0142] To facilitate understanding, we will use an example where the first sensor is an onboard millimeter-wave radar and the second sensor is an onboard camera. Figure 4 The process of target matching is explained step by step.

[0143] Step 1) Dynamic radar target selection.

[0144] Here, considering that the detection effect of vehicle-mounted millimeter-wave radar on low-speed targets may be slightly weaker, dynamic radar targets R_d are selected from all radar targets R (corresponding to the first target mentioned above) based on the speed threshold. That is, if the speed of a radar target is greater than the speed threshold, the radar target is identified as a dynamic radar target and used to solve the global transformation matrix.

[0145] If the number of dynamic radar targets is greater than 2, and the number of visual targets C (corresponding to the second target mentioned above) identified by the vehicle-mounted camera is also greater than 2, then a steering decision is made. It is worth noting that the embodiments of this application do not limit the algorithm for identifying targets from the data collected by sensors.

[0146] In some embodiments, if the conditions that the number of dynamic radar targets is greater than 2 and the number of visual targets C is also greater than 2 are not met, then step 8) can be referred to to obtain matching target pairs by enumeration matching.

[0147] Step 2) Determine the vehicle's steering direction.

[0148] The system reads the yaw rate value collected by the vehicle's specialized sensors and determines whether the vehicle is turning based on the yaw rate value. For example, when the yaw rate value is greater than the angular velocity threshold, the vehicle is determined to be turning; when the yaw rate value is less than or equal to the angular velocity threshold, the vehicle is determined to be not turning.

[0149] Step 3) Solve the translation model.

[0150] When the vehicle is in a non-steering state, the global transformation matrix T is solved using a translation model based on the dynamic radar target R_d and the visual target C.

[0151] For example, the global transformation matrix can be set as [1,0,△x; 0,1,△y], where △x is the horizontal distance and △y is the vertical distance. The horizontal and vertical distances correspond to the positional deviations mentioned above.

[0152] Step 4) Solve the rotation model.

[0153] When the vehicle is turning, the global transformation matrix T is solved using a rotation model based on the dynamic radar target R_d and the visual target C.

[0154] For example, the global transformation matrix can be set as [cosθ, -sinθ, Δx; sinθ, cosθ, Δy], where Δx is the horizontal distance, Δy is the vertical distance, and θ is the rotation angle (corresponding to the angle deviation mentioned above).

[0155] Step 5) Global alignment.

[0156] The radar target R and the visual target C are globally aligned based on the solved global transformation matrix T (corresponding to the spatial alignment process mentioned above). For example, the radar target R can be spatially transformed using the global transformation matrix T, i.e., R' = T*R. Of course, the visual target C can also be spatially transformed. Here, we will illustrate the former case.

[0157] Then, a global matching process is performed on R' and C according to the matching constraints (corresponding to the spatial deviation constraints mentioned above) to obtain at least one target set. Each target set includes at least one radar target and at least one visual target, and the radar targets and visual targets in the target set satisfy the matching constraints. The matching constraints can constrain attributes such as lateral distance, longitudinal distance, and velocity deviation. For example, the matching constraints could be that the lateral distance is less than or equal to 3 meters, the longitudinal distance is less than or equal to 5 meters, and the velocity deviation is less than or equal to 5 meters per second.

[0158] Step 6) Perfect match judgment.

[0159] After performing global matching based on the matching constraints, if each target set obtained contains only one radar target and one visual target, it is considered a perfect match, that is, each target set is treated as a matching target pair, and the process ends.

[0160] Step 7) Single visual target matching judgment.

[0161] If the target set includes one visual target and multiple radar targets, a greedy matching method is used for further matching based on the distance to the vehicle. Greedy matching means prioritizing the matching of the target with the closest longitudinal distance to the vehicle. For example, a matching target pair is formed based on one visual target in the target set and the radar target with the smallest longitudinal coordinate in the target set.

[0162] Step 8) Enumerate the matching.

[0163] If the target set includes multiple visual targets and multiple radar targets, then the multiple visual targets and multiple radar targets in the target set are enumerated and combined to obtain multiple combination schemes. The combination scheme with the minimum total cost is the optimal combination scheme (target combination scheme). The total cost can take into account factors such as lateral distance, longitudinal distance and velocity deviation.

[0164] Next, we will combine Figure 5 and Figure 6 This describes the process of solving the global transformation matrix in the embodiments of this application.

[0165] Step 1) Select two dynamic radar targets. This selection can be random, or, to improve efficiency, priority can be given to selecting nearby (smaller ordinate) dynamic radar targets with a larger lateral distance. For ease of understanding, we will refer to the following steps for selecting... Figure 6 Let's take R2 and R3 as examples to illustrate.

[0166] Step 2) Determine the set of transformation matrices {T_2X} between the selected R2 and the visual target set {Cn}, and determine the set of transformation matrices {T_3X} between the selected R3 and the visual target {Cn}. To facilitate understanding, an example is given to illustrate the process of solving {T_3X}:

[0167] Step a) Traverse the set of visual targets {Cn} and solve the transformation matrix between R3 and each visual target (it is necessary to distinguish between translation model and rotation model) to obtain the initial set of transformation matrices {T_3X}.

[0168] For example, to calculate the transformation matrix T_34 between R3 and C4, the formula is as follows:

[0169] [R3X,R3Y,1] T =[1,0,△x;0,1,△y]*[C4X,C4Y,1] T

[0170] In the above formula, R3X is the x-coordinate of R3, R3Y is the y-coordinate of R3; C4X is the x-coordinate of C4, C4Y is the y-coordinate of C4; [1,0,△x; 0,1,△y] is the transformation matrix T_34 to be solved.

[0171] After obtaining the transformation matrix corresponding to R3, the effect of the transformation matrix corresponding to R3 can be judged based on R2. That is, R2 is spatially transformed according to the transformation matrix corresponding to R3, and it is determined whether there is a visual target that satisfies the matching constraint condition with R2 after spatial transformation. If there is, the corresponding transformation matrix is ​​retained; otherwise, the corresponding transformation matrix is ​​discarded to reduce the subsequent computation and improve efficiency.

[0172] For example, R2 can be spatially transformed using the transformation matrix T_34, i.e., R2' = T_34 * R2. Since no visual target satisfies the matching constraint for R2', T_34 is discarded. As another example, the transformation matrix T_33 between R3 and C3 is calculated. Similarly, R2 is processed using the transformation matrix T_33 to obtain R2'. Since R2' satisfies the matching constraint for C2, T_33 is retained.

[0173] Step b) For each transformation matrix in {T_3X}, perform spatial transformation processing on the dynamic radar target R_d according to the transformation matrix to obtain {R_d'}, and calculate the matching degree index M_3X between {R_d'} and {Cn}.

[0174] Taking the transformation matrix T_33 as an example, we can obtain the matching degree index M_33, which includes the number of matches N_33 and the average matching error W_33. The number of matches N_33 refers to the number of matched target pairs, and the average matching error W_33 is the average of the matching errors of all matched target pairs. The matching error refers to the error between the two targets in a matched target pair (corresponding to the second spatial deviation mentioned above). If the transformation matrix corresponds to a translation model, the matching error can include lateral and longitudinal distances; if the transformation matrix corresponds to a rotation model, the matching error can include the rotation angle.

[0175] Step c) Update the corresponding transformation matrix according to the matching degree index to obtain {T_3X'}.

[0176] For example, for the transformation matrix T_33, the average matching error W_33 obtained in step b) can be superimposed with T_33 to obtain T_33'.

[0177] Step d) Recalculate M_3X' based on {T_3X'} until the number of matches in M_3X' no longer increases.

[0178] For example, based on T_33', step b) is re-executed to obtain a new matching index M_33' (including a new number of matches N_33' and an average matching error W_33'). The above process is repeated until the number of matches no longer increases. Then, the transformation matrix corresponding to the maximum number of matches that has occurred is selected as the final transformation matrix in {T_3X}.

[0179] Similarly, the transformation matrix set {T_2X} corresponding to R2 can be solved by solving for {T_3X}.

[0180] Step 3) Select the optimal transformation matrix.

[0181] Here, the optimal transformation matrix is ​​selected from {T_2X} and {T_3X} based on the matching degree index. For example, the transformation matrix with the largest number of matches is set as the optimal transformation matrix. If there are multiple transformation matrices with the largest number of matches, the one with the smallest average matching error is selected as the optimal transformation matrix.

[0182] Step 4) Determine the ratio between the number of dynamic radar targets in the matching results (referring to all matched target pairs) and the total number of dynamic radar targets when the optimal transformation matrix is ​​used as the global transformation matrix. When this ratio exceeds the threshold K_1% (corresponding to the first index condition above), the optimal transformation matrix is ​​determined as the global transformation matrix, and the loop ends.

[0183] Step 5) If the proportion does not exceed the threshold K_1%, determine whether the number of loops exceeds N (corresponding to the number threshold mentioned above). If the number of loops does not exceed N, increment the number of loops by one and re-execute step 1), that is, reselect two new dynamic radar targets.

[0184] Step 6) When the number of iterations exceeds N and the largest historical proportion exceeds K_2% (corresponding to the second indicator condition above), the transformation matrix corresponding to the largest proportion is determined as the global transformation matrix, and the loop ends.

[0185] Step 7) When the number of iterations exceeds N and the highest historical percentage does not exceed K_2%, the matching fails and the loop ends.

[0186] It is worth noting that the spatial transformation and matching processes involved in steps 1) to 7) above are hypothetical processes and will not substantially modify the relevant data of visual targets or dynamic radar targets.

[0187] like Figure 6 As shown, if the traditional target matching method is followed, it is easy to obtain the matching results of R1-C3 and R3-C4. However, according to the target matching method provided in the embodiments of this application, the correct matching results of C1-R1, C2-R2, and C3-R3 can be obtained, and no mismatch will occur because the distance between R1-C3 and R3-C4 is close.

[0188] The embodiments of this application can achieve at least the following technical effects:

[0189] 1) To address the issue of inconsistent sampling times between camera and radar sensors, a global transformation matrix is ​​solved for dynamic targets, avoiding the need for high-precision time synchronization and motion compensation operations. This saves computational resources and improves processing efficiency.

[0190] 2) The global matching operation reduces the error caused by differences in sampling motion compensation, which is beneficial for the design of matching constraints;

[0191] 3) The embodiments of this application adopt the graph matching method to directly match all targets, which reduces the problems of mismatch and missed match that are easy to occur when solving the matching of a single target.

[0192] The following continues to describe an exemplary structure of the target matching device 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the target matching device 455 in the memory 450 may include: an acquisition module 4551, used to acquire data collected by a first sensor and data collected by a second sensor; a target determination module 4552, used to determine multiple first targets based on the data collected by the first sensor and multiple second targets based on the data collected by the second sensor; an optimization estimation module 4553, used to perform global estimation processing based on a first spatial deviation between the first target and the second target to obtain a global transformation matrix; a spatial alignment module 4554, used to perform spatial alignment processing on the data collected by the first sensor and the data collected by the second sensor based on the global transformation matrix; and a matching module 4555, used to perform matching processing on the spatially aligned data collected by the first sensor and the data collected by the second sensor to obtain a matching target pair, and to determine the two targets in the matching target pair as the same target; wherein, the matching target pair includes one first target and one second target.

[0193] In some embodiments, the optimization estimation module 4553 is further configured to: determine multiple transformation matrices based on the first spatial deviation between the first target and the second target; for the target transformation matrix among the multiple transformation matrices, determine multiple matching target pairs that can be obtained when the target transformation matrix is ​​used as the global transformation matrix, and determine the matching degree index of the target transformation matrix based on the multiple matching target pairs; and perform filtering processing on the multiple transformation matrices based on the matching degree index corresponding to the multiple transformation matrices respectively to obtain the global transformation matrix.

[0194] In some embodiments, the optimization estimation module 4553 is further configured to: perform selection processing on a plurality of first targets; determine a plurality of transformation matrices based on the first spatial deviation between the selected first targets and each second target; when there is a matching degree index that satisfies the first index condition, determine the transformation matrix corresponding to the matching degree index that satisfies the first index condition as the global transformation matrix; when there is no matching degree index that satisfies the first index condition, re-execute the step of selecting a plurality of first targets.

[0195] In some embodiments, the number of selected first targets includes multiple targets; the target matching device 455 further includes a discarding module, which performs the following processing for any transformation matrix corresponding to any selected first target: determining the matching target pairs that can be obtained when any transformation matrix is ​​used as the global transformation matrix and that include other selected first targets; when the number of matching target pairs including other selected first targets is zero, discarding any transformation matrix.

[0196] In some embodiments, the optimization estimation module 4553 is further configured to determine the transformation matrix corresponding to the matching degree index that satisfies the second index condition as the global transformation matrix when the number of re-executions is greater than the number threshold and there is a matching degree index that satisfies the second index condition.

[0197] In some embodiments, the matching degree index includes the number of matching target pairs and the second spatial deviation between the two targets in the matching target pair; the optimization estimation module 4553 is further configured to: update the target transformation matrix according to the second spatial deviation in the matching degree index of the target transformation matrix, and redetermine the matching degree index according to the updated target transformation matrix until the number of matching target pairs in the new matching degree index no longer increases.

[0198] In some embodiments, the second spatial deviation is determined as follows: when the vehicle is in a non-steering state, the positional deviation between the two targets in the matching target pair is determined as the second spatial deviation; when the vehicle is in a steering state, the angular deviation between the two targets in the matching target pair is determined as the second spatial deviation.

[0199] In some embodiments, the matching module 4555 is further configured to: perform global matching processing on the data collected by the first sensor and the data collected by the second sensor after spatial alignment processing, according to the spatial deviation constraint condition, to obtain at least one target set; the target set includes at least one first target and at least one second target, and the third spatial deviation between the first target and the second target in the target set satisfies the spatial deviation constraint condition; and select one first target and one second target from the target set to form a matching target pair.

[0200] In some embodiments, the matching module 4555 is further configured to: determine the target set as a matching target pair when the target set includes a first target and a second target; perform greedy matching processing based on the spatial attributes of the multiple first targets in the target set when the target set includes multiple first targets and a second target, and combine the obtained first target with a second target in the target set as a matching target pair when the target set includes multiple first targets and multiple second targets; perform enumeration and combination processing on the multiple first targets and multiple second targets in the target set to obtain multiple combination schemes when the target set includes multiple first targets and multiple second targets, filter out target combination schemes from the multiple combination schemes based on the third spatial deviation of the target pairs in each combination scheme, and determine the target pairs in the target combination schemes as matching target pairs; wherein each combination scheme includes multiple target pairs.

[0201] In some embodiments, the first spatial deviation is determined as follows: when the vehicle is in a non-steering state, the positional deviation between the first target and the second target is determined as the first spatial deviation; when the vehicle is in a steering state, both the positional deviation and the angular deviation between the first target and the second target are determined as the first spatial deviation.

[0202] In some embodiments, the spatial alignment module 4554 is further configured to perform any one of the following processes: perform spatial transformation processing on a plurality of first targets according to a global transformation matrix; perform spatial transformation processing on a plurality of second targets according to a global transformation matrix.

[0203] In some embodiments, the optimization estimation module 4553 is further configured to perform the following processing on the data collected by any one of the sensors: determine targets with speeds greater than a speed threshold as targets for global estimation processing; wherein the data collected by any one of the sensors is the data collected by the first sensor or the data collected by the second sensor.

[0204] This application provides a computer program product or computer program that includes executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, causing the electronic device to perform the target matching method described in this application.

[0205] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to perform the method provided in this application, for example... Figure 3A , Figure 3B and Figure 3C The target matching method is shown.

[0206] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0207] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0208] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0209] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0210] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A target matching method characterized by, include: Acquire data collected by the first sensor and data collected by the second sensor; Multiple first targets are determined based on the data collected by the first sensor, and multiple second targets are determined based on the data collected by the second sensor; A global transformation matrix is ​​obtained by performing a global estimation process based on the first spatial deviation between the first and second targets. Multiple transformation matrices are determined based on this first spatial deviation. For the target transformation matrix among these multiple transformation matrices, multiple matching target pairs are determined when using the target transformation matrix as the global transformation matrix. A matching degree index for the target transformation matrix is ​​then determined based on these multiple matching target pairs. The target transformation matrix can be any one of the multiple transformation matrices. Finally, the multiple transformation matrices are filtered based on their respective matching degree indices to obtain the global transformation matrix. Spatial alignment processing is performed on the data collected by the first sensor and the data collected by the second sensor according to the global transformation matrix; The data collected by the first sensor and the data collected by the second sensor after spatial alignment are matched to obtain a matching target pair, and the two targets in the matching target pair are identified as the same target; wherein, the matching target pair includes a first target and a second target.

2. The method according to claim 1, characterized in that, The step of determining multiple transformation matrices based on the first spatial deviation between the first target and the second target includes: The selection process is performed on the plurality of first targets; Based on the first spatial deviation between the selected first target and each second target, multiple transformation matrices are determined; The step of filtering the multiple transformation matrices according to the matching degree index corresponding to each of the multiple transformation matrices to obtain the global transformation matrix includes: When there exists a matching degree index that satisfies the first indicator condition, the transformation matrix corresponding to the matching degree index that satisfies the first indicator condition is determined as the global transformation matrix. If no matching index meets the first index condition, the step of selecting the multiple first targets is re-executed.

3. The method according to claim 2, characterized in that, The number of selected first targets includes multiple targets; before determining the multiple matching target pairs that can be obtained when using the target transformation matrix as the global transformation matrix for the multiple transformation matrices, the method further includes: For any transformation matrix corresponding to any selected first target, perform the following processing: Determine the matching target pairs that can be obtained when any one of the transformation matrices is used as the global transformation matrix, including other selected first targets; When the number of matching target pairs including other selected first targets is zero, any one of the transformation matrices is discarded.

4. The method according to claim 2, characterized in that, The method further includes: When the number of re-executions exceeds the threshold and there exists a matching degree index that satisfies the second indicator condition, the transformation matrix corresponding to the matching degree index that satisfies the second indicator condition is determined as the global transformation matrix.

5. The method according to claim 1, characterized in that, The matching degree index includes the number of matching target pairs and the second spatial deviation between the two targets in the matching target pair; After determining the matching degree index of the target transformation matrix based on the plurality of matching target pairs, the method further includes: The target transformation matrix is ​​updated based on the second spatial deviation in the matching degree index of the target transformation matrix, and the matching degree index is re-determined based on the updated target transformation matrix until the number of matching target pairs in the new matching degree index no longer increases.

6. The method according to claim 5, characterized in that, The second spatial deviation is determined in the following way: When the vehicle is in a non-steering state, the positional deviation between the two targets in the matching target pair is determined as the second spatial deviation; When the vehicle is turning, the angular deviation between the two targets in the matching target pair is determined as the second spatial deviation.

7. The method according to any one of claims 1 to 6, characterized in that, The first spatial deviation is determined in the following way: When the vehicle is in a non-steering state, the positional deviation between the first target and the second target is defined as the first spatial deviation; When the vehicle is turning, the positional and angular deviations between the first and second targets are both defined as the first spatial deviation.

8. The method according to any one of claims 1 to 6, characterized in that, The step of spatially aligning the data collected by the first sensor and the data collected by the second sensor according to the global transformation matrix includes: Perform any of the following processes: The plurality of first targets are subjected to spatial transformation processing based on the global transformation matrix; The multiple second targets are subjected to spatial transformation processing based on the global transformation matrix.

9. A target matching device, characterized in that, include: The acquisition module is used to acquire data collected by the first sensor and data collected by the second sensor; The target determination module is used to determine multiple first targets based on the data collected by the first sensor, and to determine multiple second targets based on the data collected by the second sensor; The optimization estimation module is used to perform global estimation processing based on the first spatial deviation between the first target and the second target to obtain a global transformation matrix: Multiple transformation matrices are determined based on the first spatial deviation between the first target and the second target; for the target transformation matrix among the multiple transformation matrices, multiple matching target pairs that can be obtained when the target transformation matrix is ​​used as the global transformation matrix are determined, and a matching degree index of the target transformation matrix is ​​determined based on the multiple matching target pairs, where the target transformation matrix is ​​any one of the multiple transformation matrices; the multiple transformation matrices are filtered based on the matching degree index corresponding to each of the multiple transformation matrices to obtain the global transformation matrix. The spatial alignment module is used to perform spatial alignment processing on the data collected by the first sensor and the data collected by the second sensor according to the global transformation matrix; The matching module is used to perform matching processing on the data collected by the first sensor and the data collected by the second sensor after spatial alignment processing to obtain a matching target pair, and to determine the two targets in the matching target pair as the same target; wherein, the matching target pair includes a first target and a second target.

10. An electronic device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the target matching method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, It stores executable instructions for implementing the target matching method according to any one of claims 1 to 8 when executed by a processor.

Images