Matching relationship determination method and device, storage medium and electronic device

Abstract

The invention discloses a matching relation determination method and device, a storage medium and an electronic device, and the method comprises the steps: determining first obstacle information corresponding to a plurality of pieces of image data, and filtering a plurality of first obstacles in the plurality of pieces of image data according to the first obstacle information, so as to determine an obstacle set, the plurality of image data are image data of a plurality of first obstacles acquired by a plurality of image acquisition devices of the target vehicle; predicting a plurality of motion tracks according to the plurality of image data to obtain a motion track set; and performing cascade matching on each obstacle in the obstacle set and each movement track in the movement track set to determine a matching relationship between the target obstacle and the target movement track. According to the technical scheme, the problem that in the related technology, due to the fact that a distortion area exists in a multi-look-around camera, large errors exist when the multi-look-around camera is used for obstacle detection, and the matching integrity of the obstacle and the track is low is solved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving, and more specifically, to a method and apparatus for determining matching relationships, a storage medium, and an electronic device. Background Technology

[0002] With the development of the automotive industry, the radius of human activity has expanded. However, this expansion also brings potential dangers such as traffic congestion reducing the driving experience and fatigue from prolonged driving increasing the risk of accidents. Simultaneously, with the accelerating pace of work and increasing pressure, providing a relaxed and enjoyable commute experience is a key factor considered by automakers when developing new models. Furthermore, as the level of automotive intelligence gradually improves, the implementation of autonomous driving systems is becoming less of a distant dream. If autonomous driving can be widely implemented in daily life, the human driving experience will be significantly enhanced, and the incidence of traffic accidents will be greatly reduced. Therefore, various autonomous driving functions are increasingly appearing in new models from various automakers.

[0003] Cameras are widely used in object detection and even perception modules in autonomous driving due to their low cost and rich extraction of texture and color information. In the field of autonomous driving, which mainly relies on vision sensors, multiple cameras are usually arranged around the vehicle to form a surround-view perception system to avoid blind spots, thereby acquiring information about all obstacles around the vehicle. For obstacles located at the field of view boundaries of each camera, the position detection of obstacles near the field of view boundaries often has a large error due to the distortion area of ​​the camera. When the same obstacle is observed by multiple cameras, the detected position of the multiple cameras often differs significantly due to differences in camera calibration and model detection. How to confirm whether the detection results are affected by distortion and reduce the impact of the distortion position on the system is one of the problems that surround-view camera fusion urgently needs to solve.

[0004] Currently, there is no effective solution to the problem that multi-view cameras have distortion areas, which can lead to large errors in obstacle detection and low accuracy in matching obstacles with trajectories.

[0005] Therefore, it is necessary to improve the relevant technology to overcome the aforementioned defects. Summary of the Invention

[0006] This application provides a method and apparatus for determining matching relationships, a storage medium, and an electronic device to at least solve the problem in the related art where, due to the distortion area of ​​multi-view cameras, there will be a large error when detecting obstacles through multi-view cameras, resulting in low completeness of obstacle-trajectory matching.

[0007] According to one aspect of the embodiments of this application, a method for determining a matching relationship is provided, comprising: determining first obstacle information corresponding to multiple image data respectively, and filtering multiple first obstacles in the multiple image data according to the first obstacle information to determine an obstacle set, wherein the multiple image data are image data of multiple first obstacles acquired by multiple image acquisition devices of a target vehicle; predicting multiple motion trajectories according to the multiple image data to obtain a motion trajectory set; and performing cascade matching between each obstacle in the obstacle set and each motion trajectory in the motion trajectory set to determine a matching relationship between a target obstacle and a target motion trajectory.

[0008] According to another aspect of the embodiments of this application, a matching relationship determination apparatus is also provided, comprising: a determination module, configured to determine first obstacle information corresponding to multiple image data respectively, and filter multiple first obstacles in the multiple image data according to the first obstacle information to determine an obstacle set, wherein the multiple image data are image data of multiple first obstacles acquired by multiple image acquisition devices of a target vehicle; a prediction module, configured to predict multiple motion trajectories according to the multiple image data to obtain a motion trajectory set; and a matching module, configured to perform cascade matching between each obstacle in the obstacle set and each motion trajectory in the motion trajectory set to determine the matching relationship between a target obstacle and a target motion trajectory.

[0009] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the above-described method for determining the matching relationship when it is run.

[0010] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method for determining the matching relationship through the computer program.

[0011] This application embodiment determines the first obstacle information corresponding to multiple image data of multiple first obstacles collected by multiple image acquisition devices of a target vehicle, and filters the multiple first obstacles in the multiple image data according to the first obstacle information to determine an obstacle set. Multiple motion trajectories are predicted based on the multiple image data to obtain a motion trajectory set. The obstacle set and the motion trajectory set are then cascaded and matched to determine the matching relationship between the target obstacle and the target motion trajectory. In other words, this application embodiment obtains an obstacle set by filtering multiple first obstacles collected by image acquisition devices, and obtains a motion trajectory set by predicting the motion trajectories of the multiple first obstacles. The obstacle set and the motion trajectory set are then cascaded and matched to determine the matching relationship between the target obstacle and the target motion trajectory. This solves the problem in related technologies where the detection of obstacles using multi-view cameras results in large errors due to distortion areas, leading to low completeness of obstacle-trajectory matching. This effectively filters abnormal obstacles and ensures completeness of obstacle-trajectory matching. Attached Figure Description

[0012] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and, together with the description thereof, serve to explain this application and do not constitute an undue limitation thereof. In the drawings:

[0013] Figure 1 This is a hardware structure block diagram of a computer device for determining a matching relationship according to an embodiment of this application;

[0014] Figure 2 This is a flowchart of a method for determining matching relationships according to an embodiment of this application;

[0015] Figure 3 This is a schematic diagram of a projection frame verification according to an optional embodiment of this application;

[0016] Figure 4 This is a structural block diagram of a projection-based multi-view camera fusion processing device according to an optional embodiment of this application;

[0017] Figure 5 This is a flowchart of a projection-based multi-view camera fusion processing method according to an optional embodiment of this application;

[0018] Figure 6 This is a flowchart of cascading matching according to an optional embodiment of this application;

[0019] Figure 7 This is a structural block diagram of a matching relationship determination device according to an embodiment of this application. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0021] It should be noted that the terms and terms such as "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] The methods and embodiments provided in this application can be executed in a computer device or a similar computing device. Taking running on a computer device as an example, Figure 1 This is a hardware structure block diagram of a computer device for a matching relationship determination method according to an embodiment of this application. For example... Figure 1 As shown, a computer device may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor unit (MPU) or a programmable logic device (PLD)) and a memory 104 for storing data are also shown. In one exemplary embodiment, the computer device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer device described above. For example, the computer device may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 Equivalent functions or ratios shown Figure 1 The functions shown have more different configurations.

[0023] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the matching relationship determination method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, which is equivalent to implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0024] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer equipment. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0025] This embodiment provides a method for determining matching relationships. Figure 2 This is a flowchart of a method for determining a matching relationship according to an embodiment of this application, which is run in the aforementioned computer device. The process includes the following steps:

[0026] Step S202: Determine the first obstacle information corresponding to the multiple image data respectively, and filter the multiple first obstacles in the multiple image data according to the first obstacle information to determine the obstacle set, wherein the multiple image data are image data of multiple first obstacles collected by multiple image acquisition devices of the target vehicle;

[0027] The aforementioned obstacles can be vehicles, pedestrians, etc.

[0028] Step S204: Predict multiple motion trajectories based on the multiple image data to obtain a set of motion trajectories;

[0029] Step S206: Cascade matching of each obstacle in the obstacle set and each motion trajectory in the motion trajectory set to determine the matching relationship between the target obstacle and the target motion trajectory.

[0030] Through the above steps, the first obstacle information corresponding to multiple image data of multiple first obstacles collected by multiple image acquisition devices of the target vehicle is determined. The multiple first obstacles in the multiple image data are then filtered based on the first obstacle information to determine an obstacle set. Multiple motion trajectories are predicted based on the multiple image data to obtain a motion trajectory set. The obstacle set and the motion trajectory set are then cascaded and matched to determine the matching relationship between the target obstacle and the target motion trajectory. In other words, this embodiment of the application obtains an obstacle set by filtering multiple first obstacles collected by the image acquisition device, and obtains a motion trajectory set by predicting the motion trajectories of the multiple first obstacles. The obstacle set and the motion trajectory set are then cascaded and matched to determine the matching relationship between the target obstacle and the target motion trajectory. This solves the problem in related technologies where the detection of obstacles using multi-view cameras results in large errors due to distortion areas, leading to low completeness of obstacle-trajectory matching. This effectively filters abnormal obstacles and ensures completeness of obstacle-trajectory matching.

[0031] Optionally, step S202 above, which filters multiple first obstacles in the multiple image data based on the first obstacle information, includes: determining the number of detection boxes corresponding to each first obstacle based on the first obstacle information; determining a first size relationship between the number of detection boxes and a preset detection box number threshold; determining the confidence level of each first detection box when the first size relationship indicates that the number of detection boxes is greater than the preset detection box number threshold; determining the first detection box with the highest confidence level among the multiple first detection boxes, and determining the first detection box with the highest confidence level as the target detection box of each first obstacle; and filtering the multiple first obstacles based on the target detection boxes of each first obstacle.

[0032] The plurality of first obstacles are filtered by at least one of the following methods: determining a second detection box corresponding to each first obstacle; determining a first intersection-union ratio (IUU) of the target detection box and the second detection box for each first obstacle; filtering the first obstacles according to the first IUU, wherein the target detection box and the second detection box are detection boxes of different types; determining a first distance between the target detection box and the edge region of the corresponding image data; and filtering the plurality of first obstacles according to the first distance.

[0033] Since a single obstacle may be detected by multiple image acquisition devices, it is necessary to filter the detected first obstacles. Specifically, the number of detection boxes for each first obstacle is determined. For example, if it is determined that an obstacle has been captured by an image acquisition device, the first detection box corresponding to the obstacle can be determined to include the detection boxes of the obstacle captured by the image acquisition device.

[0034] The aforementioned preset threshold for the number of detection boxes can be 1. When the number of detection boxes is greater than 1, it can be determined that the same obstacle is captured by multiple image acquisition devices at the same time. At this time, the confidence level of each first detection box can be determined, and then the detection box with the highest confidence level can be determined. The first detection box with the highest confidence level is then determined as the target detection box of each obstacle.

[0035] The first detection box and the target detection box mentioned above can be 2D detection boxes.

[0036] Therefore, multiple first obstacles can be filtered out in the following two ways, specifically:

[0037] (1) A second detection box can be determined for each first obstacle, and then a first cross-union ratio (CUI) between the second detection box and the target detection box can be determined. First detection boxes with a CUI less than the CUI threshold are deleted. The second detection box and the first detection box are of different types. For example, the first detection box can be a 2D detection box, and the second detection box can be a 3D detection box.

[0038] (2) The first distance between the target detection box of each first obstacle and the edge region of the image data corresponding to each first obstacle can be determined, and the first detection box with a first distance less than the distance threshold can be deleted.

[0039] Optionally, step S204 above, which predicts multiple motion trajectories based on the multiple image data, includes: determining a first time difference between consecutive time frames corresponding to each first obstacle based on the multiple image data; determining a state transition matrix for each first obstacle based on the first time difference; predicting a state quantity of each first obstacle in the current time frame based on the state transition matrix; determining an error covariance matrix for each first obstacle in the current time frame; and predicting the motion trajectory corresponding to each first obstacle based on the state quantity and the error covariance matrix.

[0040] It is understandable that the state variables of each first obstacle in the previous frame, as well as the first time difference between consecutive time frames, can be determined; thus, the state transition matrix and the error covariance matrix of each first obstacle can be determined. The trajectory of each first obstacle can then be determined using the error covariance matrix and the state transition matrix.

[0041] Optionally, step S206 above, which involves cascading matching of each obstacle in the obstacle set and each motion trajectory in the motion trajectory set, includes: determining multiple second motion trajectories for first-level matching in the motion trajectory set, and performing first-level matching of the second motion trajectories with each obstacle in the obstacle set to obtain a first-level matching result; determining second obstacles for second-level matching in the obstacle set based on the first-level matching result, and determining third motion trajectories for second-level matching in the motion trajectory set based on the first-level matching result, and performing second-level matching of the second obstacles and the third motion trajectory to obtain a second-level matching result; determining third obstacles for third-level matching in the obstacle set based on the first-level matching result and the second-level matching result, and determining fourth motion trajectories for third-level matching in the motion trajectory set based on the first-level matching result and the second-level matching result, and performing third-level matching of the third obstacles and the fourth motion trajectory to obtain a third-level matching result.

[0042] Understandably, cascading matching involves three steps: first-level matching, second-level matching, and third-level matching.

[0043] It is necessary to determine the motion trajectory and obstacles for first-level matching, second-level matching, and third-level matching respectively, and then perform first-level matching, second-level matching, and third-level matching.

[0044] (1) First-level matching:

[0045] Determine multiple second motion trajectories for primary matching: obtain the previous frame motion trajectory of each motion trajectory; determine whether the previous frame motion trajectory has been updated; determine the motion trajectory whose previous frame motion trajectory has been updated as the multiple second motion trajectories.

[0046] The specific steps of the first-level matching are as follows: First, determine the third detection box for each second motion trajectory and the first detection box for each obstacle in the obstacle set; second, establish a first cost matrix based on the second intersection-union ratio (IUU) of each third detection box and the first detection box of each obstacle, wherein each first element in the first cost matrix indicates each second IUU; third, perform distance verification on the obstacle and motion trajectory corresponding to the first element to determine the second distance between the obstacle and motion trajectory corresponding to the first element; fourth, adjust the first cost matrix based on the second distance to obtain an adjusted first cost matrix; fifth, perform Hungarian matching on the adjusted first cost matrix to determine the target obstacle and target motion trajectory that are successfully matched in the first-level matching.

[0047] It is understandable that we can determine the second motion trajectory by checking whether the motion trajectory in the previous frame has been updated. If it is determined that the motion trajectory in the previous frame has been updated, then the motion trajectory can be identified as the second motion trajectory.

[0048] The obstacles for first-level matching are all obstacles in the obstacle set.

[0049] The second intersection-over-union (IoU) ratio between the obstacle and the trajectory can be determined. This second IoU ratio is then negative and used as the first cost matrix. The first cost matrix is ​​then adjusted as follows: if all elements in a row or column of the first cost matrix are greater than the IoU matching threshold, that row or column is deleted. Furthermore, elements in the first cost matrix that exceed the IoU matching threshold are set to the IoU matching threshold to prevent the Hungarian algorithm from entering local optima. (The adjustment of the second and third cost matrices uses the same processing method, which will not be elaborated here). Finally, Hungarian matching is performed on the processed first cost matrix to determine the matching relationship between the target obstacle and the target trajectory.

[0050] (2) Secondary matching:

[0051] Determine the motion trajectory and obstacles for secondary matching: Based on the results of the primary matching, determine the target obstacles and target motion trajectories that have been successfully matched; determine the other obstacles in the obstacle set besides the target obstacles that were successfully matched in the primary matching as the second obstacles, and determine the other motion trajectories in the motion trajectory set besides the target motion trajectories that were successfully matched in the primary matching as the third motion trajectories.

[0052] Specific steps: Establish a second cost matrix based on the third distance between the second obstacle and the third motion trajectory, wherein each second element in the second cost matrix is ​​used to indicate the third distance; determine a first element threshold for each second element based on the second obstacle and the third motion trajectory; adjust the second cost matrix based on multiple first element thresholds to obtain an adjusted second cost matrix; perform Hungarian matching on the adjusted second cost matrix to determine the target obstacle and target motion trajectory that are successfully matched in the second-level matching.

[0053] Understandably, obstacles in the obstacle set other than the target obstacle that was successfully matched in the first-level matching participate in the second-level matching; and motion trajectories in the motion trajectory set other than the target motion trajectory that was successfully matched in the first-level matching participate in the second-level matching.

[0054] A second cost matrix is ​​determined based on the distance between the second obstacle and the third trajectory, and then adjusted. A Hungarian matching process is then performed on the adjusted second cost matrix to determine the matching relationship between the target obstacle and the target trajectory.

[0055] (3) Three-level matching:

[0056] The motion trajectory and obstacles for three-level matching are determined: based on the first-level matching results, the target obstacles and target motion trajectories that are successfully matched are determined; and based on the second-level matching results, the target obstacles and target motion trajectories that are successfully matched are determined; the obstacles in the obstacle set other than the target obstacles that are successfully matched in the first-level matching and the target obstacles that are successfully matched in the second-level matching are determined as the third obstacles, and the motion trajectories in the motion trajectory set other than the target motion trajectories that are successfully matched in the first-level matching and the target motion trajectories that are successfully matched in the second-level matching are determined as the fourth motion trajectories.

[0057] Specific steps: Determine the first projection frame of the third obstacle in the bird's-eye view and the second projection frame of the fourth motion trajectory in the bird's-eye view; establish a third cost matrix based on the third intersection-union ratio (IUU) of the first and second projection frames, wherein each third element in the third cost matrix is ​​used to indicate each third IUU; determine the second element threshold of each third element based on the third obstacle and the fourth motion trajectory; process the third cost matrix based on multiple second element thresholds to obtain a processed third cost matrix; perform Hungarian matching on the processed third cost matrix to determine the target obstacle and target motion trajectory that are successfully matched in the three-level matching.

[0058] It is understandable that obstacles in the obstacle set other than those successfully matched in the first-level matching and the second-level matching participate in the third-level matching; and motion trajectories in the motion trajectory set other than those successfully matched in the first-level matching and the second-level matching participate in the third-level matching.

[0059] The third cost matrix is ​​determined based on the first projection frame of the fourth obstacle in the bird's-eye view and the second projection frame of the fourth motion trajectory in the bird's-eye view, and then adjusted. Hungarian matching is then performed on the adjusted third cost matrix to determine the matching relationship between the target obstacle and the target motion trajectory.

[0060] After performing a three-level cascaded matching of obstacles and trajectories, there are still unmatched obstacles and unmatched motion trajectories. The unmatched fifth obstacle and unmatched sixth motion trajectory can be processed as follows: A fourth obstacle that failed to match is identified in the obstacle set; a fifth motion trajectory corresponding to the fourth obstacle is established and input into the motion trajectory set; a sixth motion trajectory that failed to match is identified in the motion trajectory set; a first matching threshold for the sixth motion trajectory to undergo cascaded matching is determined based on the first trajectory state of the sixth motion trajectory, wherein the first trajectory state includes: a pending state and a stable state; it is determined whether the sixth motion trajectory successfully matches within the first matching threshold; if it is determined that the sixth motion trajectory fails to match within the first matching threshold, it is determined that the sixth motion trajectory has no matching relationship with any obstacle in the obstacle set.

[0061] In this embodiment, a fifth motion trajectory can be established for the fifth obstacle that failed to match, and the unmatched motion trajectories and the fifth motion trajectory in the motion trajectory set are collectively referred to as the sixth motion trajectory. This allows for the determination of whether the sixth motion trajectory needs to be deleted. Specifically:

[0062] The first motion state of the sixth motion trajectory is determined. For example, if the first motion state is determined to be an undetermined state, the first matching threshold can be 1, meaning that if the sixth motion trajectory fails to match once, it is deleted. If the first motion state is determined to be a stable state, the first matching threshold can be 3, meaning that if the sixth motion trajectory fails to match three times, it is deleted.

[0063] Optionally, after filtering multiple first obstacles in the multiple image data according to the first obstacle information in step S202 above, the method further includes: determining the number of first image acquisition devices that detect a fifth obstacle in the obstacle set, wherein the multiple image acquisition devices include: the first image acquisition devices; determining a second size relationship between the number and a preset number threshold; when the second size relationship indicates that the number is greater than the preset number threshold, determining the angle of deviation of the fifth obstacle from the camera optical axis in each first image acquisition device; determining the angle difference between the fifth obstacle and the optical axis angle of each first image acquisition device according to the angle of deviation from the camera optical axis; and determining the position information of the fifth obstacle according to the angle difference.

[0064] The determination of the position information of the fifth obstacle based on the angle difference includes: determining a third magnitude relationship between the angle difference and a preset angle difference; when the third magnitude relationship indicates that the angle difference is greater than the preset angle difference, determining a minimum deviation angle from the camera optical axis among multiple deviation angles from the camera optical axis, and determining a second image acquisition device corresponding to the minimum deviation angle from the camera optical axis, and determining the position information of the fifth obstacle detected by the second image acquisition device as the target position information of the fifth obstacle, wherein the multiple first image acquisition devices include: the second image acquisition device; when the third magnitude relationship indicates that the angle difference is less than or equal to the preset angle difference, determining a target detection box of the fifth obstacle; and determining the position information of the fifth obstacle based on the target detection box of the fifth obstacle and the deviation angle from the camera optical axis.

[0065] Understandably, after filtering, the location information of the filtered obstacles can be determined. Specifically:

[0066] The number of image acquisition devices that simultaneously detect an obstacle in the obstacle set is determined. If there are two or more first image acquisition devices, each obstacle detected by the first image acquisition device can be transformed into the target vehicle's coordinate system for distance and bird's-eye view bounding box matching. Then, the angle of deviation of an obstacle from the camera's optical axis under each first image acquisition device is calculated, along with the angle difference between the optical axis angles of each first image acquisition device. The unique location of the obstacle is determined based on the angle difference. For example, if the angle difference is greater than 10°, the obstacle location with the smaller angle deviation from the camera's optical axis is directly selected as the unique location of the obstacle; when the angle difference is less than or equal to 10°, a voting process is performed combining the pixel distance of the 2D detection box from the image edge and the confidence level. That is, the 2D detection box farther from the image edge receives one vote, the location corresponding to the 2D detection box with high confidence receives one vote, and the obstacle location with the smaller angle deviation from the optical axis receives one vote. The location with the highest number of votes is the location of the successfully matched obstacle.

[0067] Optionally, after performing cascade matching between each obstacle in the obstacle set and each motion trajectory in the motion trajectory set in step S206 above, it is also necessary to update the target motion trajectory, including: updating the motion state of the target motion trajectory; and / or updating the trajectory type of the target motion trajectory; and / or updating the trajectory speed of the target motion trajectory; and / or updating the trajectory length and trajectory width of the target motion trajectory; and / or updating the trajectory speed angle of the target motion trajectory; and / or updating the trajectory position of the target motion trajectory.

[0068] (1) Determine the first number of matching of the target motion trajectory, and determine the second motion state of the target motion trajectory based on the first number of matching; if the first number of matching is greater than the second number of matching threshold, update the second motion state of the target motion trajectory to a stable state; if the first number of matching is less than or equal to the second number of matching threshold, update the second motion state of the target motion trajectory to a pending state.

[0069] Understandably, the state of the target trajectory is determined based on the number of first matches. When the number of first matches is less than 2, the state of the target trajectory is set to pending; when the number of first matches exceeds 2, the state of the target trajectory is set to stable.

[0070] (2) Determine multiple sixth obstacles matched by the target motion trajectory based on a preset time frame, and vote on each first obstacle type according to multiple first obstacle types corresponding to each sixth obstacle; determine the first obstacle type with the highest number of votes as the target obstacle type; determine whether the target obstacle type belongs to the second obstacle type; if the target obstacle type belongs to the second obstacle type, update the trajectory type corresponding to the target motion trajectory according to the target obstacle type; if the target obstacle type does not belong to the second obstacle type, determine the second number of times the target motion trajectory has been continuously matched with the target obstacle type; update the trajectory type corresponding to the target motion trajectory according to the second number of matches.

[0071] Understandably, for each target motion trajectory, a voting system is used to determine the type of each trajectory. The type of obstacle that appears most frequently in the two consecutive matches from the creation of each target motion trajectory is determined as the trajectory type.

[0072] For determining the type of non-vehicle targets (the type of vehicle targets is the second obstacle type in this application): the type recognition judgment can be further divided into two functions. Function 1: Target type change output: The target needs to be detected as a pedestrian for 8 consecutive frames before the pedestrian category can be output. The target type will only be adjusted downstream after three consecutive frames of cyclist detection. Function 2: Target type jump: If the target's type is inconsistent with the current type output in even one frame, it is judged as a flash, and the type output remains unchanged until the target disappears or a type change is detected.

[0073] (3) Determine the target position information of multiple sixth obstacles matched based on a preset time frame for the target motion trajectory; determine the position sequence of each sixth obstacle based on the target position information of each sixth obstacle, and determine the time difference sequence based on the second time difference of the preset time frame; update the trajectory speed of the target motion trajectory based on the position sequence and the time difference sequence.

[0074] Understandably, for each target motion trajectory, the velocity is fitted using the historical frame position information matched to that trajectory, employing the least squares method as the fitting algorithm. Specifically:

[0075] Record the sequence of N frames of obstacle positions matched to the motion trajectory of each target [(x1, y1), (x2, y2), (x3, y3), ..., (x...]. N y N [], with N frames of obstacles and their corresponding time difference sequences [dt1, dt2, dt3, ..., dt] N [,] where each element represents the time difference from the current moment to the starting moment. The velocity v in the x-direction is calculated using this method. x For example, assuming the obstacle moves at a constant speed, according to the equation of motion x0 + v x dt i =x i Write the position sequence and time difference sequence in matrix form:

[0076]

[0077] Then, by using the least squares method, its velocity v in the x-direction can be obtained. x , where x0 is the trajectory position corresponding to the initial time obtained from the fitting. The velocity v in the y-direction. y The solution is the same as above. Finally, the fitted velocity (v) is... x v y ( ) represents the velocity of the trajectory.

[0078] For non-vehicle targets, such as pedestrians, their position sequence is [(x1, y1), (x2, y2), (x3, y3), ..., (x...]. N y N [dt1, dt2, dt3, ..., dt], and the corresponding time difference sequence [dt1, dt2, dt3, ..., dt] N The fitted curve is called the moving curve, and the velocity is obtained. The points that are 20% of the distance from the velocity curve are removed, and a new velocity curve is fitted. The velocity (v) is then calculated. x v y The velocity output is used as the trajectory velocity.

[0079] (4) Determine multiple sixth obstacles matched based on a preset time frame for the target motion trajectory; determine the first length and first width of the detection box corresponding to each sixth obstacle; establish a length and width list based on the first length and the first width; determine the average length and average width in the length and width list; update the trajectory length and trajectory width of the target motion trajectory based on the average length and the average width.

[0080] Specifically, it is determined whether the first length of each sixth obstacle is greater than a preset length threshold, and whether the first width corresponding to each sixth obstacle is greater than a preset width threshold; if it is determined that the first length is greater than the preset length threshold and / or the first width is greater than the preset width threshold, the first length and / or the first width of each sixth obstacle are updated to the preset length threshold and / or the preset width threshold, and the updated first length and updated first width corresponding to each sixth obstacle are input into the length-width list; if it is determined that the first length is less than or equal to the preset length threshold, and the first width is less than or equal to the preset width threshold, the first length and the first width of each sixth obstacle are input into the length-width list.

[0081] Understandably, for each target motion trajectory, outlier filtering and updates are performed on the length and width (i.e., the first length and first width of this application) of each target motion trajectory. Specifically:

[0082] For each target motion trajectory, the length and width of N frames of obstacles are recorded in a length-width list. During recording, if the length and width of an obstacle significantly differ from its default length and width (i.e., the preset width threshold of this application), the default length and width are selected and recorded in the length-width list. The average length and width in the length-width list are calculated. If the length or width of an obstacle in a frame exceeds 20% of the average, the obstacle's length or width in that frame is considered abnormal, and it is removed from the obstacle list. After deleting abnormal lengths or widths, the average length and width in the length-width list are calculated again to obtain the trajectory length and width. This updates the trajectory length and width of the target motion trajectory.

[0083] For non-vehicle targets, such as pedestrians, the system determines whether the type is consistent with previous observations. If there is a very short jump, the previous average length and width values ​​will be maintained. If a category change is detected, the data pool for calculating length and width will be cleared, and the target information will be re-added. During recording, if the length and width of an obstacle differ significantly from its default type, the default length and width will be selected and recorded in the length and width list. The average length and width values ​​in the length and width list will be calculated. If the length or width of an obstacle in a frame exceeds 20% of the average, the obstacle length or width in that frame is considered abnormal, and the abnormal length or width will be removed from the obstacle list. After deleting abnormal lengths or widths, the average length and width values ​​in the length and width list will be calculated again as the length and width of the trajectory output.

[0084] (5) Determine the target motion trajectory based on multiple sixth obstacles matched by a preset time frame; determine the velocity angle corresponding to the target motion trajectory and the obstacle angle of each sixth obstacle; update the trajectory velocity angle corresponding to the target motion trajectory according to the fourth size relationship between the velocity angle and the obstacle angle.

[0085] Understandably, for each target's motion trajectory, the angle is updated for each trajectory. The filtering of the trajectory angle consists of two steps: velocity-angle verification and angle jump strategy.

[0086] For non-vehicle targets, such as pedestrians, the orientation angle is output using the orientation angle output by the model when the model is accurate. When the model output becomes unstable or fluctuates, the orientation angle is obtained using component velocity, and relevant information is output.

[0087] 1) When the absolute speed of the trajectory is greater than 60km / h, the angle jump strategy is triggered. If the difference between the obstacle's angle and the speed angle is greater than 10°, the speed angle is used as the trajectory angle.

[0088] 2) When the ratio of the difference between the obstacle angle and the trajectory angle to the time interval between two frames is greater than 10° per second, the danger angular velocity count of the trajectory is incremented by 1, but the trajectory angle remains unchanged. If the danger angular velocity count of the trajectory is incremented by 1 three times consecutively, the obstacle angle is assigned to the trajectory angle, and the danger angular velocity count is reset to zero.

[0089] This allows for the updating of the trajectory velocity and angle corresponding to the target's motion trajectory.

[0090] (6) Determine the Kalman gain of the target motion trajectory; update the trajectory position of the target motion trajectory according to the Kalman gain.

[0091] Understandably, for each target's motion trajectory, the position of each trajectory is updated using a Kalman filter. Specifically, the Kalman gain K is calculated. k+1 ,

[0092] Where H is the observation matrix, H T Let H be the transpose of H, and R be the noise matrix of the obstacle observations. In an optional embodiment of this application,

[0093] Understandably, obstacle positions need to be verified before an update. This can be done using 3D projection frames and 2D detection frames of the obstacles. Figure 3 This is a schematic diagram of a projection frame verification according to an optional embodiment of this application, such as... Figure 3 As shown:

[0094] When an obstacle is detected by the model, a 2D bounding box surrounds it. Simultaneously, the model provides the obstacle's center point position and dimensions (length, width, and height) in the vehicle's coordinate system. The obstacle's center point position and dimensions constitute its 3D information, which can be projected onto the image domain using the camera's intrinsic and extrinsic parameters to form a 3D projection box. When the model detects the obstacle's position accurately, the intersection-over-union (IoU) ratio of its 3D projection box and 2D bounding box is typically greater than a constant. In this embodiment, we set this constant to 0.4 as a threshold for verifying the model's position detection accuracy. If it is lower than this threshold, the obstacle's position is considered abnormal, and the corresponding element value in the observation noise matrix R needs to be increased during the update. In this embodiment, if the IoU ratio of the 3D projection box and 2D bounding box is less than the threshold, the element in the observation noise matrix R can be set to 10000.

[0095] Furthermore, combining Kalman gain K k+1 The observed value Z of the obstacle k+1 Calculate the posterior position of the trajectory:

[0096] Finally, the error covariance matrix of the trajectory is updated:

[0097] This completes the update of the trajectory position of the target's motion trajectory.

[0098] Obviously, the embodiments described above are only some embodiments of this application, and not all embodiments. To better understand the method for determining the matching relationship described above, the process is explained below with reference to embodiments, but this is not intended to limit the technical solutions of the embodiments of this application. Specifically:

[0099] An optional embodiment of this application provides a projection-based multi-view camera fusion processing method and apparatus, specifically:

[0100] Figure 4 This is a structural block diagram of a projection-based multi-view camera fusion processing device according to an optional embodiment of this application, such as... Figure 4 As shown, the device includes:

[0101] The preprocessing module is used for multi-view obstacle filtering and spatial alignment;

[0102] The prediction module is used to predict multiple motion trajectories based on multiple image data to obtain a set of motion trajectories;

[0103] The matching module is used for multi-level matching of trajectories and obstacles, including but not limited to: distance matching, image detection box matching, bird's-eye view projection box matching, and three-dimensional (Three Dimensions, abbreviated as 3D) projection box matching.

[0104] The update module is configured to combine the intersection-over-union (IOU) ratio of the obstacle 3D projection frame and the 2D detection frame to set the observation noise, and use Kalman filtering to update the position, fit the velocity, and maintain the angle of the trajectory.

[0105] Figure 5 This is a flowchart of a projection-based multi-view camera fusion processing method according to an optional embodiment of this application, applied to... Figure 4 The apparatus shown includes:

[0106] Step S501: Filter and spatially synchronize the obstacles observed by the multi-view camera.

[0107] Specifically:

[0108] Step 501-1: For obstacles detected by a single camera, perform multi-detection filtering. If multiple cameras have two or more 2D detection boxes for the same obstacle, and the sizes of the 2D detection boxes are different, there will be some error in the position of the obstacle detected by multiple cameras. In some cases, this error will affect the stability of the subsequent filtering algorithm. Therefore, outlier filtering is required before filtering. This includes: performing non-maximum suppression based on the confidence level of the 2D detection box (i.e., the first detection box in this application), that is, selecting the detection box with the highest confidence level as the detection box (target detection box) of the obstacle. Specifically, this includes: judging based on the intersection-union ratio (IUGR) of the 2D and 3D detection boxes (i.e., the second detection box in this application) projected in the image domain. For example: if the obstacle is within 20 meters and the IUGR of the 2D and 3D detection boxes projected in the image domain is less than 0.5, then the outlier obstacle is deleted; if the obstacle is located more than 25 meters away, then the threshold can be set to 0.4. The threshold parameters in the optional embodiments of this application are for reference only. In actual applications, the threshold can be appropriately adjusted according to the changes of the actual vehicle sensors and models. In addition, obstacles within the distortion zone at the edge of the camera lens can be filtered out. Specifically, this includes filtering based on whether the pixel distance between the obstacle's 2D detection box and the image edge is less than a threshold. If it is less than the threshold, the obstacle is deleted.

[0109] Step 501-2: Spatial synchronization of the positions of obstacles detected in the overlapping observation area of ​​the multi-camera surround view is performed. This mainly addresses situations where the same obstacle is observed by more than two cameras. Specifically, the obstacles observed by the two cameras are transformed into the vehicle's coordinate system, and distance and projection frame matching are performed in the bird's-eye view. Due to differences in camera lenses and calibration, it is necessary to determine the unique position of each successfully matched obstacle in the vehicle's coordinate system. This involves calculating the magnitude of the obstacle's deviation from the camera's optical axis under each camera, and calculating the difference between the two obstacles' deviations from their respective camera's optical axis. The unique location of the obstacle is determined based on the angle difference. If the angle difference is greater than 10°, the location of the obstacle with the smallest angle away from the camera's optical axis is directly selected as the unique location of the obstacle. If the angle difference is less than or equal to 10°, a vote is taken by combining the pixel distance of the 2D detection box from the image edge and the confidence level. The 2D detection box that is far from the image edge gets one vote, the location corresponding to the 2D box with high confidence gets one vote, and the location of the obstacle with the smallest angle away from the optical axis gets one vote. Finally, the location with the most votes is the location of the successfully matched obstacle.

[0110] Step S502: Predict the state variables of the multi-view camera trajectory.

[0111] Step S502-1: Assume the vehicle's motion is uniform and the angular rotation is uniform angular velocity. Based on the state variables [x, y, vx, vy, yaw, yaw_rate] of the previous frame and the time difference Δt between the two frames, predict the state variables of the trajectory in the current frame.

[0112] The state transition formula for the motion model is:

[0113]

[0114] Where, k≥1, Let be the state transition matrix.

[0115] Step S502-2: Update the prior error covariance matrix

[0116] Where, P k Let Q be the error covariance matrix of the previous frame, and let Q be the process noise matrix.

[0117] In this embodiment:

[0118]

[0119] This allows us to determine the trajectory of the current frame.

[0120] Step S503, cascade matching.

[0121] The filtered obstacles in step S501 and the trajectory of the current frame obtained in step S502 are cascaded and matched, and the trajectory is created, updated, or deleted based on the matching results. Figure 6 This is a flowchart of cascading matching according to an optional embodiment of this application, such as... Figure 6 As shown, cascaded matching is divided into three levels: the first level is matching 2D detection boxes in the image domain, the second level is matching using Euclidean distance, and the third level is matching projected boxes under the bird's-eye view. Before matching, all obstacles are stored in an obstacle list, and all trajectories are stored in a trajectory list.

[0122] Step S503-1: The first-level matching uses the 2D detection box in the image domain as a feature for matching.

[0123] Step S503-1.1: Determine whether the trajectory participates in the current 2D detection box. If the trajectory was not updated in the previous frame, then the trajectory does not participate in the matching of the dimension of the 2D detection box in the current frame image domain; if the trajectory was updated in the previous frame, then it can participate in the matching of the dimension of the current 2D detection box.

[0124] Step S503-1.2: Calculate the intersection-union ratio (IUR) of the 2D bounding boxes of all obstacles and the 2D bounding boxes of the objects with updated trajectories from the previous frame. Negate the IUR and use it as the cost matrix for the Hungarian algorithm.

[0125] Step S503-1.3: Perform distance verification on the obstacles and trajectories corresponding to the elements in the cost matrix. If the distance between the obstacle and the trajectory exceeds the distance matching threshold, then regardless of the value of the element in the cost matrix, it is set to the intersection-union ratio matching threshold.

[0126] Step S503-1.4: Filter the cost matrix. If all elements in a row or column of the cost matrix exceed the intersection-union (IU) matching threshold, then delete that row or column, indicating that the trajectory corresponding to that row or the obstacle corresponding to that column will not participate in the matching.

[0127] Step S503-1.5: For elements in the cost matrix that exceed the crossover-union ratio (CUI) threshold, set the CUI threshold to prevent the Hungarian algorithm from entering a local optimum.

[0128] Step S503-1.6: Perform Hungarian matching using the optimized cost matrix. If an obstacle and a trajectory are successfully matched, remove the obstacle from the obstacle list, remove the trajectory from the trajectory list, and record the obstacle and the trajectory in the successful matching list.

[0129] Step S503-2: The second-level matching uses distance as a feature for matching.

[0130] Step S503-2.1: Calculate the distance between obstacles in the obstacle list and trajectories in the trajectory list as the cost matrix of the Hungarian algorithm.

[0131] Step S503-2.2: Perform type verification on obstacles and trajectories in the cost matrix. If the types of the trajectory and obstacle are consistent, the distance corresponding to the cost matrix remains unchanged. If the types of the trajectory and obstacle are inconsistent, continue to verify the similarity of other dimensions between the obstacle and the trajectory, such as the intersection-union ratio (IU) of the projected boxes in the bird's-eye view. If the IU is less than the IU threshold, it is considered that the trajectory and obstacle cannot match, and the elements in the cost matrix are set to the distance threshold.

[0132] Step S503-2.3: Set different thresholds for the elements in the cost matrix. If either the obstacle or the trajectory is a truck, then the distance threshold for that element should be set larger than the normal distance threshold.

[0133] Step S503-2.4: Filter the cost matrix. If all elements in a row or column of the cost matrix exceed the distance matching threshold, delete that row or column, indicating that the trajectory corresponding to that row or the obstacle corresponding to that column will not participate in the matching.

[0134] Step S503-2.5: For elements in the cost matrix that exceed the distance threshold, set the distance threshold to prevent the Hungarian algorithm from entering a local optimum.

[0135] Step S503-2.6: Perform Hungarian matching using the optimized cost matrix. If an obstacle and a trajectory are successfully matched, remove the obstacle from the obstacle list, remove the trajectory from the trajectory list, and record the obstacle and the trajectory in the successful matching list.

[0136] Step S503-3: The third-level matching uses the projected bounding box under the bird's-eye view as a feature for matching.

[0137] Step S503-3.1: Calculate the intersection-union ratio of the projection frames of obstacles in the obstacle list and trajectories in the trajectory list in the bird's-eye view as the cost matrix of the Hungarian algorithm.

[0138] Step S503-3.2: Perform type verification on obstacles and trajectories in the cost matrix. If the types of the trajectory and obstacle are consistent, the cross-union ratio (CUP) of the cost matrix remains unchanged. If the types of the trajectory and obstacle are inconsistent, continue with the CUP of the projection boxes in the bird's-eye view. If the CUP is less than the CUP threshold, it is considered that the trajectory and obstacle cannot match, and the elements in the cost matrix are set to the CUP threshold.

[0139] Step S503-3.3: Filter the cost matrix. If all elements in a row or column of the cost matrix exceed the intersection-union (IU) matching threshold, delete that row or column, indicating that the trajectory corresponding to that row or the obstacle corresponding to that column will not participate in the matching.

[0140] Step S503-3.4: For elements in the cost matrix that exceed the crossover-union ratio (CUI) threshold, set the CUI threshold to prevent the Hungarian algorithm from entering a local optimum.

[0141] Step S503-3.5: Perform Hungarian matching using the optimized cost matrix. If an obstacle and a trajectory are successfully matched, remove the obstacle from the obstacle list, remove the trajectory from the trajectory list, and record the obstacle and the trajectory in the successful matching list.

[0142] Step S503-4: Manage the matching results. For successfully matched trajectories and obstacles, update the trajectory using the obstacle information and proceed to step 4; for unmatched trajectories, decrease their lifespan by 1. If the trajectory's state is pending, delete the trajectory as soon as its lifespan decreases by 1; if the trajectory's state is stable, delete the trajectory from the trajectory list after its lifespan decreases by 1 three times consecutively; for unmatched obstacles, create a new trajectory using its information and add it to the trajectory list.

[0143] In this embodiment, by fusing the results detected by multiple surround-view cameras, functions such as filtering abnormal detection values ​​of obstacles and tracking trajectories can be effectively achieved, thereby providing reliable environmental perception data for autonomous driving and ensuring the safety of vehicles during autonomous driving.

[0144] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0145] This embodiment also provides a matching relationship determination device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0146] Figure 7 This is a structural block diagram of a matching relationship determination device according to an embodiment of this application, the device comprising:

[0147] The determining module 72 is used to determine the first obstacle information corresponding to the multiple image data respectively, and to filter the multiple first obstacles in the multiple image data according to the first obstacle information to determine the obstacle set, wherein the multiple image data are image data of multiple first obstacles collected by multiple image acquisition devices of the target vehicle;

[0148] The prediction module 74 is used to predict multiple motion trajectories based on the multiple image data to obtain a set of motion trajectories;

[0149] The matching module 76 is used to perform cascade matching between each obstacle in the obstacle set and each motion trajectory in the motion trajectory set to determine the matching relationship between the target obstacle and the target motion trajectory.

[0150] The aforementioned device determines the first obstacle information corresponding to multiple image data of multiple first obstacles acquired by multiple image acquisition devices of the target vehicle, and filters the multiple first obstacles in the multiple image data according to the first obstacle information to determine an obstacle set. Multiple motion trajectories are predicted based on the multiple image data to obtain a motion trajectory set. The obstacle set and the motion trajectory set are then cascaded and matched to determine the matching relationship between the target obstacle and the target motion trajectory. In other words, this embodiment of the application obtains an obstacle set by filtering multiple first obstacles acquired by the image acquisition device, and obtains a motion trajectory set by predicting the motion trajectories of the multiple first obstacles. The obstacle set and the motion trajectory set are then cascaded and matched to determine the matching relationship between the target obstacle and the target motion trajectory. This solves the problem in related technologies where, due to distortion areas in multi-view cameras, there are large errors in obstacle detection, resulting in low completeness of obstacle-trajectory matching. This effectively filters abnormal obstacles and ensures completeness of obstacle-trajectory matching.

[0151] In an exemplary embodiment, the determining module 72 is further configured to: determine the number of detection boxes corresponding to each first obstacle based on the first obstacle information; determine a first size relationship between the number of detection boxes and a preset detection box number threshold; determine the confidence level of each first detection box when the first size relationship indicates that the number of detection boxes is greater than the preset detection box number threshold; determine the first detection box with the highest confidence level among the multiple first detection boxes, and determine the first detection box with the highest confidence level as the target detection box of each first obstacle; and filter the multiple first obstacles based on the target detection boxes of each first obstacle.

[0152] In an exemplary embodiment, the determining module 72 is further configured to filter the plurality of first obstacles by at least one of the following methods: determining a second detection box corresponding to each first obstacle; determining a first intersection-union ratio (IUU) of the target detection box and the second detection box for each first obstacle; filtering the first obstacles according to the first IUU, wherein the target detection box and the second detection box are detection boxes of different types; determining a first distance between the target detection box and the edge region of the corresponding image data; and filtering the plurality of first obstacles according to the first distance.

[0153] In an exemplary embodiment, the prediction module 74 is further configured to: determine a first time difference between consecutive time frames corresponding to each first obstacle based on the plurality of image data; determine a state transition matrix for each first obstacle based on the first time difference; predict a state quantity of each first obstacle in the current time frame based on the state transition matrix; determine an error covariance matrix for each first obstacle in the current time frame; and predict a motion trajectory corresponding to each first obstacle based on the state quantity and the error covariance matrix.

[0154] In an exemplary embodiment, the matching module 76 is further configured to: determine a plurality of second motion trajectories for first-level matching in the set of motion trajectories; perform first-level matching between the second motion trajectories and each obstacle in the set of obstacles to obtain a first-level matching result; determine a second obstacle for second-level matching in the set of obstacles based on the first-level matching result; determine a third motion trajectory for second-level matching in the set of motion trajectories based on the first-level matching result; perform second-level matching between the second obstacle and the third motion trajectory to obtain a second-level matching result; determine a third obstacle for third-level matching in the set of obstacles based on the first-level matching result and the second-level matching result; determine a fourth motion trajectory for third-level matching in the set of motion trajectories based on the first-level matching result and the second-level matching result; and perform third-level matching between the third obstacle and the fourth motion trajectory to obtain a third-level matching result.

[0155] In an exemplary embodiment, the matching module 76 is further configured to obtain the previous frame motion trajectory of each motion trajectory; determine whether the previous frame motion trajectory has been updated; and determine the motion trajectory whose previous frame motion trajectory has been updated as the plurality of second motion trajectories.

[0156] In an exemplary embodiment, the matching module 76 is further configured to: determine a third detection box for each second motion trajectory and a first detection box for each obstacle in the obstacle set; establish a first cost matrix based on a second intersection-union ratio (IUU) of each third detection box and the first detection box of each obstacle, wherein each first element in the first cost matrix is ​​used to indicate each second IUU; perform distance verification on the obstacle and motion trajectory corresponding to the first element to determine a second distance between the obstacle and motion trajectory corresponding to the first element; adjust the first cost matrix based on the second distance to obtain an adjusted first cost matrix; and perform Hungarian matching on the adjusted first cost matrix to determine the target obstacle and target motion trajectory that are successfully matched in the first-level matching.

[0157] In an exemplary embodiment, the matching module 76 is further configured to determine the target obstacle and the target motion trajectory that were successfully matched based on the first-level matching result; to determine the other obstacles in the obstacle set besides the target obstacle that was successfully matched in the first-level matching as the second obstacle; and to determine the other motion trajectories in the motion trajectory set besides the target motion trajectory that was successfully matched in the first-level matching as the third motion trajectory.

[0158] In an exemplary embodiment, the matching module 76 is further configured to: establish a second cost matrix based on a third distance between the second obstacle and the third motion trajectory, wherein each second element in the second cost matrix is ​​used to indicate the third distance; determine a first element threshold for each second element based on the second obstacle and the third motion trajectory; adjust the second cost matrix based on a plurality of first element thresholds to obtain an adjusted second cost matrix; and perform Hungarian matching on the adjusted second cost matrix to determine the target obstacle and target motion trajectory that are successfully matched in the second-level matching.

[0159] In an exemplary embodiment, the matching module 76 is further configured to determine the successfully matched target obstacle and target motion trajectory based on the first-level matching result; and to determine the successfully matched target obstacle and target motion trajectory based on the second-level matching result; to determine the other obstacles in the obstacle set besides the target obstacles successfully matched in the first-level matching and the target obstacles successfully matched in the second-level matching as the third obstacle; and to determine the other motion trajectories in the motion trajectory set besides the target motion trajectories successfully matched in the first-level matching and the target motion trajectories successfully matched in the second-level matching as the fourth motion trajectory.

[0160] In an exemplary embodiment, the matching module 76 is further configured to: determine a first projection frame of the third obstacle in a bird's-eye view and a second projection frame of the fourth motion trajectory in the bird's-eye view; establish a third cost matrix based on a third intersection-union ratio (IUU) of the first and second projection frames, wherein each third element in the third cost matrix is ​​used to indicate each third IUU; determine a second element threshold for each third element based on the third obstacle and the fourth motion trajectory; process the third cost matrix based on multiple second element thresholds to obtain a processed third cost matrix; and perform Hungarian matching on the processed third cost matrix to determine the target obstacle and target motion trajectory that are successfully matched in the three-level matching.

[0161] In one exemplary embodiment, the apparatus further includes: an update module, configured to: determine a fourth obstacle that failed to match in the obstacle set; establish a fifth motion trajectory corresponding to the fourth obstacle and input the fifth motion trajectory into the motion trajectory set; and determine a sixth motion trajectory that failed to match in the motion trajectory set; determine a first matching count threshold for the sixth motion trajectory to be cascaded matched based on a first trajectory state of the sixth motion trajectory, wherein the first trajectory state includes: a pending state and a stable state; determine whether the sixth motion trajectory successfully matches within the first matching count threshold; and, if it is determined that the sixth motion trajectory failed to match within the first matching count threshold, determine that the sixth motion trajectory does not have a matching relationship with any obstacle in the obstacle set.

[0162] In an exemplary embodiment, the updating module is further configured to: determine a second size relationship between the quantity and a preset quantity threshold; determine the angle of deviation of the fifth obstacle from the camera optical axis in each first image acquisition device when the second size relationship indicates that the quantity is greater than the preset quantity threshold; determine the angle difference between the fifth obstacle and the optical axis of each first image acquisition device based on the angle of deviation from the camera optical axis; and determine the position information of the fifth obstacle based on the angle difference.

[0163] In an exemplary embodiment, the updating module is further configured to: determine a third magnitude relationship between the angle difference and a preset angle difference; when the third magnitude relationship indicates that the angle difference is greater than the preset angle difference, determine a minimum deviation angle from the camera optical axis among multiple deviation angles from the camera optical axis, and determine a second image acquisition device corresponding to the minimum deviation angle from the camera optical axis, and determine the position information of the fifth obstacle detected by the second image acquisition device as the target position information of the fifth obstacle, wherein the multiple first image acquisition devices include: the second image acquisition device; when the third magnitude relationship indicates that the angle difference is less than or equal to the preset angle difference, determine a target detection box of the fifth obstacle; and determine the position information of the fifth obstacle based on the target detection box of the fifth obstacle and the deviation angle from the camera optical axis.

[0164] In an exemplary embodiment, the updating module is further configured to update the motion state of the target motion trajectory; and / or update the trajectory type of the target motion trajectory; and / or update the trajectory speed of the target motion trajectory; and / or update the trajectory length and trajectory width of the target motion trajectory; and / or update the trajectory speed angle of the target motion trajectory; and / or update the trajectory position of the target motion trajectory.

[0165] In an exemplary embodiment, the updating module is further configured to determine a first number of matching of the target motion trajectory, and determine a second motion state of the target motion trajectory based on the first number of matching; if the first number of matching is greater than a second number of matching threshold, update the second motion state of the target motion trajectory to a stable state; if the first number of matching is less than or equal to the second number of matching threshold, update the second motion state of the target motion trajectory to a pending state.

[0166] In an exemplary embodiment, the updating module is further configured to: determine multiple sixth obstacles matched by the target motion trajectory based on a preset time frame; vote on each first obstacle type according to multiple first obstacle types corresponding to each sixth obstacle; determine the first obstacle type with the highest number of votes as the target obstacle type; determine whether the target obstacle type belongs to a second obstacle type; if the target obstacle type belongs to the second obstacle type, update the trajectory type corresponding to the target motion trajectory according to the target obstacle type; if the target obstacle type does not belong to the second obstacle type, determine a second number of consecutive matches of the target motion trajectory with the target obstacle type; and update the trajectory type corresponding to the target motion trajectory according to the second number of matches.

[0167] In an exemplary embodiment, the updating module is further configured to determine the target position information of a plurality of sixth obstacles matched by the target motion trajectory based on a preset time frame; determine the position sequence of each sixth obstacle according to the target position information of each sixth obstacle, and determine the time difference sequence according to the second time difference of the preset time frame; and update the trajectory velocity of the target motion trajectory according to the position sequence and the time difference sequence.

[0168] In an exemplary embodiment, the updating module is further configured to: determine a plurality of sixth obstacles matched by the target motion trajectory based on a preset time frame; determine a first length and a first width of the detection box corresponding to each sixth obstacle; establish a length and width list based on the first length and the first width; determine the average length and the average width in the length and width list; and update the trajectory length and trajectory width of the target motion trajectory based on the average length and the average width.

[0169] In an exemplary embodiment, the updating module is further configured to determine whether the first length of each sixth obstacle is greater than a preset length threshold, and whether the first width corresponding to each sixth obstacle is greater than a preset width threshold; if it is determined that the first length is greater than the preset length threshold and / or the first width is greater than the preset width threshold, the first length and / or the first width of each sixth obstacle is updated to the preset length threshold and / or the preset width threshold, and the updated first length and the updated first width corresponding to each sixth obstacle are input into the length-width list; if it is determined that the first length is less than or equal to the preset length threshold, and the first width is less than or equal to the preset width threshold, the first length and the first width of each sixth obstacle are input into the length-width list.

[0170] In an exemplary embodiment, the updating module is further configured to determine a plurality of sixth obstacles matched by the target motion trajectory based on a preset time frame; determine the velocity angle corresponding to the target motion trajectory and the obstacle angle of each sixth obstacle; and update the trajectory velocity angle corresponding to the target motion trajectory according to a fourth relationship between the velocity angle and the obstacle angle.

[0171] In an exemplary embodiment, the updating module is further configured to determine the Kalman gain of the target motion trajectory; and update the trajectory position of the target motion trajectory according to the Kalman gain.

[0172] Optionally, in this embodiment, the storage medium may be configured to store computer programs corresponding to the steps of the method for determining the matching relationship.

[0173] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0174] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0175] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0176] Optionally, in this embodiment, the processor can be configured to execute the steps in the method for determining the matching relationship via a computer program.

[0177] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0178] Embodiments of this application also provide a computer program product, including a computer program that is executed by a processor through the steps of any of the above method embodiments.

[0179] Optionally, in this embodiment, the computer program product described above can be executed by a processor using the steps in the method for determining the matching relationship.

[0180] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0181] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0182] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for determining a matching relationship, characterized in that, include: First obstacle information corresponding to multiple image data is determined, and multiple first obstacles in the multiple image data are filtered according to the first obstacle information to determine an obstacle set, wherein the multiple image data are image data of multiple first obstacles collected by multiple image acquisition devices of the target vehicle; Multiple motion trajectories are predicted based on the multiple image data to obtain a set of motion trajectories; each obstacle in the obstacle set and each motion trajectory in the motion trajectory set are cascaded and matched to determine the matching relationship between the target obstacle and the target motion trajectory.

2. The method for determining the matching relationship according to claim 1, characterized in that, Filtering multiple first obstacles in the multiple image data based on the first obstacle information includes: The number of detection boxes corresponding to the first detection box for each first obstacle is determined based on the first obstacle information; Determine a first size relationship between the number of detection frames and a preset threshold number of detection frames; When the first size relationship indicates that the number of detection boxes is greater than the preset detection box number threshold, the confidence level of each first detection box is determined; Among multiple first detection boxes, the first detection box with the highest confidence is determined, and the first detection box with the highest confidence is determined as the target detection box for each first obstacle; The plurality of first obstacles are filtered based on the target detection box of each first obstacle.

3. The method for determining the matching relationship according to claim 2, characterized in that, Filtering the plurality of first obstacles based on the target detection bounding box of each first obstacle includes: The plurality of first obstacles are filtered by at least one of the following methods: Determine a second detection box corresponding to each first obstacle; determine a first intersection-union ratio (IUU) of the target detection box and the second detection box for each first obstacle; filter the first obstacles according to the first IUU, wherein the target detection box and the second detection box are detection boxes of different types; Determine a first distance between the target detection box and the edge region of the corresponding image data; filter the plurality of first obstacles based on the first distance.

4. The method for determining the matching relationship according to claim 1, characterized in that, The process of cascading and matching each obstacle in the obstacle set with each trajectory in the trajectory set includes: Multiple second motion trajectories for primary matching are determined from the set of motion trajectories, and the second motion trajectories are matched with each obstacle in the set of obstacles to obtain a primary matching result; Based on the first-level matching result, a second obstacle for second-level matching is determined in the obstacle set, and a third motion trajectory for second-level matching is determined in the motion trajectory set based on the first-level matching result. The second obstacle and the third motion trajectory are then matched in a second-level manner to obtain a second-level matching result. Based on the first-level matching result and the second-level matching result, a third obstacle for third-level matching is determined in the obstacle set, and a fourth motion trajectory for third-level matching is determined in the motion trajectory set based on the first-level matching result and the second-level matching result. The third obstacle and the fourth motion trajectory are then matched at the third level to obtain the third-level matching result.

5. The method for determining the matching relationship according to claim 4, characterized in that, Multiple second motion trajectories for primary matching are determined from the set of motion trajectories, including: Obtain the motion trajectory of the previous frame for each of the motion trajectories; Determine whether the motion trajectory of the previous frame has been updated; The motion trajectory that has been updated in the previous frame is determined as the plurality of second motion trajectories.

6. The method for determining the matching relationship according to claim 4, characterized in that, Perform a first-level match between the second motion trajectory and each obstacle in the obstacle set, including: Determine the third detection box for each second motion trajectory and the first detection box for each obstacle in the obstacle set; A first cost matrix is ​​established based on the second intersection-union ratio of each third detection box and the first detection box of each obstacle, wherein each first element in the first cost matrix is ​​used to indicate each second intersection-union ratio; Distance verification is performed on the obstacles and movement trajectory corresponding to the first element to determine the second distance between the obstacles and movement trajectory corresponding to the first element; The first cost matrix is ​​adjusted according to the second distance to obtain the adjusted first cost matrix; The adjusted first cost matrix is ​​subjected to Hungarian matching to determine the target obstacles and target trajectories that are successfully matched in the first-level matching.

7. The method for determining the matching relationship according to claim 4, characterized in that, Based on the first-level matching result, a second obstacle for second-level matching is determined from the obstacle set, and a third motion trajectory for second-level matching is determined from the motion trajectory set based on the first-level matching result, including: Based on the first-level matching results, the successfully matched target obstacles and target movement trajectories are determined. The obstacles in the obstacle set other than the target obstacle that was successfully matched in the first-level matching are identified as the second obstacle, and the motion trajectories in the motion trajectory set other than the target motion trajectory that was successfully matched in the first-level matching are identified as the third motion trajectory.

8. The method for determining the matching relationship according to claim 4, characterized in that, Performing a secondary matching between the second obstacle and the third trajectory includes: A second cost matrix is ​​established based on a third distance between the second obstacle and the third trajectory, wherein each second element in the second cost matrix is ​​used to indicate the third distance; Determine the first element threshold of each second element based on the second obstacle and the third motion trajectory; The second cost matrix is ​​adjusted based on multiple first element thresholds to obtain the adjusted second cost matrix; Hungarian matching is performed on the adjusted second cost matrix to identify the target obstacles and target trajectories that were successfully matched in the second-level matching.

9. The method for determining the matching relationship according to claim 4, characterized in that, Based on the first-level matching result and the second-level matching result, a third obstacle for third-level matching is determined from the obstacle set, and a fourth motion trajectory for third-level matching is determined from the motion trajectory set based on the first-level matching result and the second-level matching result, including: Based on the first-level matching results, the successfully matched target obstacles and target motion trajectories are determined; and based on the second-level matching results, the successfully matched target obstacles and target motion trajectories are determined. The obstacles in the obstacle set other than the target obstacles that were successfully matched in the first-level matching and the target obstacles that were successfully matched in the second-level matching are identified as the third obstacles, and the motion trajectories in the motion trajectory set other than the target motion trajectories that were successfully matched in the first-level matching and the target motion trajectories that were successfully matched in the second-level matching are identified as the fourth motion trajectories.

10. The method for determining the matching relationship according to claim 4, characterized in that, Perform a three-level matching process between the third obstacle and the fourth motion trajectory, including: Determine the first projection frame of the third obstacle in the bird's-eye view and the second projection frame of the fourth motion trajectory in the bird's-eye view; A third cost matrix is ​​established based on the third intersection-union ratio of the first projection frame and the second projection frame, wherein each third element in the third cost matrix is ​​used to indicate each third intersection-union ratio; Determine the second element threshold of each third element based on the third obstacle and the fourth motion trajectory; The third cost matrix is ​​processed based on multiple second element thresholds to obtain the processed third cost matrix; Hungarian matching is performed on the processed third cost matrix to determine the target obstacles and target trajectories that are successfully matched in the third-level matching.

11. The method for determining the matching relationship according to claim 1, characterized in that, After cascading and matching each obstacle in the obstacle set with each motion trajectory in the motion trajectory set, the method further includes: Identify a fourth obstacle in the set of obstacles that failed to match successfully; Establish a fifth motion trajectory corresponding to the fourth obstacle, and input the fifth motion trajectory into the motion trajectory set; and In the set of motion trajectories, identify the sixth motion trajectory that failed to match. The first matching count threshold for the sixth motion trajectory to be cascaded is determined based on the first trajectory state of the sixth motion trajectory, wherein the first trajectory state includes: Undetermined state and stable state; Determine whether the sixth motion trajectory is successfully matched within the first matching count threshold; if it is determined that the sixth motion trajectory is not successfully matched within the first matching count threshold, determine that the sixth motion trajectory has no matching relationship with any obstacle in the obstacle set.

12. The method for determining the matching relationship according to claim 1, characterized in that, After filtering multiple first obstacles in the multiple image data based on the first obstacle information, the method further includes: Determine the number of first image acquisition devices that detect the fifth obstacle in the obstacle set, wherein the plurality of image acquisition devices includes: the first image acquisition device; determine a second size relationship between the number of the first image acquisition devices and a preset number threshold; When the second size relationship indicates that the number is greater than the preset number threshold, the angle of the fifth obstacle's deviation from the camera optical axis in each first image acquisition device is determined; The angle difference between the fifth obstacle and the optical axis of each first image acquisition device is determined based on the angle of deviation from the camera's optical axis. The position information of the fifth obstacle is determined based on the angle difference.

13. The method for determining the matching relationship according to claim 12, characterized in that, Determining the position information of the fifth obstacle based on the angle difference includes: Determine a third relationship between the angle difference and the preset angle difference; When the third size relationship indicates that the angle difference is greater than the preset angle difference, the minimum deviation angle from the camera optical axis is determined among multiple deviation angles from the camera optical axis, and the second image acquisition device corresponding to the minimum deviation angle from the camera optical axis is determined, and the position information of the fifth obstacle detected by the second image acquisition device is determined as the target position information of the fifth obstacle, wherein the multiple image acquisition devices include: the second image acquisition device; When the third size relationship indicates that the angle difference is less than or equal to the preset angle difference, the target detection box of the fifth obstacle is determined; The position information of the fifth obstacle is determined based on the target detection box of the fifth obstacle and the angle of deviation from the camera optical axis.

14. The method for determining the matching relationship according to claim 1, characterized in that, After cascading and matching each obstacle in the obstacle set with each motion trajectory in the motion trajectory set, the method further includes: Update the motion state of the target trajectory; and / or Update the trajectory type of the target motion trajectory; and / or Update the trajectory and velocity of the target motion trajectory; and / or Update the trajectory length and trajectory width of the target motion trajectory; and / or Update the trajectory, velocity, and angle of the target motion trajectory; and / or The trajectory position of the target motion trajectory is updated.

15. The method for determining the matching relationship according to claim 14, characterized in that, Updating the motion state of the target trajectory includes: Determine the first number of matchings for the target motion trajectory, and determine the second motion state of the target motion trajectory based on the first number of matchings; If the first number of matching is greater than the second number of matching threshold, the second motion state of the target motion trajectory is updated to a stable state; If the first number of matches is less than or equal to the second number of matches threshold, the second motion state of the target motion trajectory is updated to a pending state.

16. The method for determining the matching relationship according to claim 14, characterized in that, Updating the trajectory type of the target motion trajectory includes: The target motion trajectory is determined based on multiple sixth obstacles matched by a preset time frame, and each first obstacle type is voted on according to multiple first obstacle types corresponding to each sixth obstacle; The type of obstacle with the highest number of votes is identified as the target obstacle type; Determine whether the target obstacle type belongs to the second obstacle type; If it is determined that the target obstacle type belongs to the second obstacle type, the trajectory type corresponding to the target movement trajectory is updated according to the target obstacle type; if it is determined that the target obstacle type does not belong to the second obstacle type, the second number of consecutive matches of the target movement trajectory to the target obstacle type is determined. The trajectory type corresponding to the target motion trajectory is updated based on the second number of matching.

17. The method for determining the matching relationship according to claim 14, characterized in that, Updating the trajectory velocity of the target motion trajectory includes: The target's motion trajectory is determined based on the target position information of multiple sixth obstacles matched within a preset time frame; The position sequence of each sixth obstacle is determined based on the target position information of each sixth obstacle, and the time difference sequence is determined based on the second time difference of the preset time frame; The trajectory velocity of the target motion trajectory is updated based on the position sequence and the time difference sequence.

18. The method for determining the matching relationship according to claim 14, characterized in that, Updating the trajectory length and trajectory width of the target motion trajectory includes: The target motion trajectory is determined based on multiple sixth obstacles matched within a preset time frame; the first length and first width of the detection box corresponding to each sixth obstacle are determined; Create a length and width list based on the first length and the first width; Determine the average length and average width in the list of length and width; The trajectory length and trajectory width of the target motion trajectory are updated based on the average length and the average width.

19. The method for determining the matching relationship according to claim 18, characterized in that, A length and width list is created based on the first length and the first width, including: Determine whether the first length of each sixth obstacle is greater than a preset length threshold, and determine whether the first width corresponding to each sixth obstacle is greater than a preset width threshold; If it is determined that the first length is greater than the preset length threshold and / or the first width is greater than the preset width threshold, the first length and / or the first width of each sixth obstacle are updated to the preset length threshold and / or the preset width threshold, and the updated first length and the updated first width corresponding to each sixth obstacle are input into the length and width list; If it is determined that the first length is less than or equal to the preset length threshold and the first width is less than or equal to the preset width threshold, the first length and the first width of each sixth obstacle are input into the length and width list.

20. The method for determining the matching relationship according to claim 14, characterized in that, Updating the trajectory velocity and angle of the target motion trajectory includes: The target motion trajectory is determined based on multiple sixth obstacles matched within a preset time frame; the velocity angle corresponding to the target motion trajectory and the obstacle angle of each sixth obstacle are determined. The trajectory velocity angle corresponding to the target motion trajectory is updated based on the fourth relationship between the velocity angle and the obstacle angle.

21. A device for determining a matching relationship, characterized in that, include: The determining module is used to determine the first obstacle information corresponding to multiple image data respectively, and to filter the multiple first obstacles in the multiple image data according to the first obstacle information to determine the obstacle set, wherein the multiple image data are image data of multiple first obstacles collected by multiple image acquisition devices of the target vehicle; The prediction module is used to predict multiple motion trajectories based on the multiple image data to obtain a set of motion trajectories; The matching module is used to perform cascade matching between each obstacle in the obstacle set and each motion trajectory in the motion trajectory set to determine the matching relationship between the target obstacle and the target motion trajectory.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method described in any one of claims 1 to 20.

23. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 20 through the computer program.

Images