Multi-object association method, system, computer device and storage medium

Abstract

Embodiments of the present disclosure relate to the technical field of target tracking, and specifically relate to a multi-target association method, system, computer device and storage medium. The main steps of the foregoing method include: obtaining a plurality of candidate bounding boxes corresponding to a plurality of to-be-associated targets in current frame data, taking each to-be-associated target and its corresponding plurality of candidate bounding boxes as a sample, determining the sample type of each sample, extracting the appearance features of the candidate bounding boxes and the to-be-associated targets in difficult samples, obtaining association probability data based on a preset classification model according to the basic bounding box features of simple samples, the basic bounding box features and the appearance features of difficult samples, and determining the target bounding boxes having an association relationship with each to-be-associated target from the plurality of candidate bounding boxes according to the association probability data and a preset matching algorithm. The foregoing method can improve the accuracy of target object recognition.

Description

Technical Field

[0001] This disclosure relates to the field of target tracking technology, and in particular to a multi-target association method, system, computer device, and storage medium. Background Technology

[0002] In fields such as intelligent transportation, video surveillance, and autonomous driving, associating the same object across different frames of data, and linking the identity information of the tracked object (the target to be associated) across frames, concatenates discrete intra-frame detection results into a continuous target trajectory. This is the foundation for target tracking, target behavior analysis, and anomaly warning. For example, in traffic scenarios, associating vehicle trajectories identifies dangerous behaviors such as illegal lane changes and pedestrians crossing the road; in autonomous driving scenarios, associating obstacles dynamically assesses collision risks. Accurate multi-target association is a prerequisite for accurate multi-target tracking and a key prerequisite for ensuring the effective operation of various intelligent systems.

[0003] However, current multi-objective association methods mainly rely on single features for analysis and judgment, resulting in inaccurate association results. Summary of the Invention

[0004] Therefore, it is necessary to provide a multi-target association method, system, computer device, and storage medium to improve the accuracy of association, addressing at least one of the aforementioned technical problems.

[0005] In a first aspect, embodiments of this disclosure provide a multi-target association method, which may include the following steps: Obtain multiple candidate detection boxes corresponding to multiple targets to be associated in the current frame data; Each target to be associated and its corresponding multiple candidate detection boxes are treated as a sample, and the sample type of each sample is determined. Extract the appearance features of candidate bounding boxes and the target to be associated from hard samples; Based on the preset classification model, the association probability data is obtained according to the basic detection box features of simple samples, the basic detection box features and appearance features of difficult samples; Based on the association probability data and the preset matching algorithm, target detection boxes that are associated with each target to be associated are determined from multiple candidate detection boxes.

[0006] The target to be associated is a dynamic target in historical frame data. The historical frame data and the current frame data are road traffic data at different times, including image data and / or point cloud data. The sample types include simple samples indicating that there is a similarity score between any candidate detection box and the target to be associated that is greater than a preset similarity threshold, and difficult samples indicating that there is no similarity score between any candidate detection box and the target to be associated that is greater than a preset similarity threshold. The basic detection box features include at least attribute features, motion trajectory features, and spatiotemporal features.

[0007] In some embodiments, extracting the appearance features of candidate detection boxes and targets to be associated in difficult samples may include the following steps: extracting the appearance features of candidate detection boxes and targets to be associated in difficult samples based on a preset re-identification model.

[0008] The re-identification model runs in parallel with the target detection model used to identify and detect static targets in the current frame data.

[0009] In some embodiments, determining the sample type of each sample may include the following steps: The sample type is determined based on the similarity score between the target to be associated in the sample and the candidate detection box; If any candidate detection box in the sample has a similarity score greater than the preset similarity threshold with the target to be associated, the sample is a simple sample. A sample is considered a difficult sample if no candidate detection box in the sample has a similarity score greater than the preset similarity threshold with the target to be associated.

[0010] In some embodiments, determining the sample type of each sample may further include the following steps: The similarity score is determined based on the overlap score and velocity similarity score between the target to be associated and the candidate detection box in the sample. The sample type is determined based on the similarity score, the target to be associated, and the category information of the candidate detection box; If any candidate bounding box in the sample has a similarity score greater than the preset similarity threshold with the target to be associated, and the category information is consistent, the sample is a simple sample. A sample is considered a difficult sample if no candidate detection box in the sample has a similarity score greater than the preset similarity threshold with the target to be associated.

[0011] In some embodiments, the multi-target association method may further include the following steps: pairing multiple targets to be associated with multiple candidate detection boxes one by one to obtain multiple association groups, and obtaining the basic detection box features of the multiple association groups.

[0012] Based on a pre-defined classification model, association probability data is obtained according to the basic detection box features of simple samples, the basic detection box features of hard samples, and appearance features. This may include the following steps: The basic detection box features of the association groups corresponding to simple samples are fused to obtain the first feature vector; The basic detection box features and appearance features of the association group corresponding to the difficult sample are fused to obtain the second feature vector; By inputting the first and second feature vectors into the classification model, association probability data for multiple association groups are obtained.

[0013] In some embodiments, the classification model is a random forest model, and the matching algorithm is the Hungarian algorithm. Based on the association probability data and the preset matching algorithm, determining multiple target detection boxes that are associated with multiple targets to be associated from multiple candidate detection boxes may include the following steps: Construct an association cost matrix based on the association probability data; The optimal solution of the association cost matrix is ​​obtained by using the Hungarian algorithm, and preliminary matching results are obtained. Based on the preliminary matching results and the preset confidence threshold, target detection boxes that are the same as the target to be associated are determined from the candidate detection boxes corresponding to the preliminary matching results.

[0014] In some embodiments, obtaining multiple candidate detection boxes corresponding to multiple targets to be associated in the current frame data may include the following steps: Based on the location and trajectory information of multiple targets to be associated, candidate object regions are determined from the current frame data; Based on the preset detection model, multiple candidate detection boxes corresponding to multiple candidate objects are extracted from the candidate object region.

[0015] In a second aspect, embodiments of this disclosure provide a multi-target association system including sensors and a processor.

[0016] Sensors are used to acquire road traffic data and send it to a processor.

[0017] The processor is used to receive road traffic images and perform steps to implement the multi-target association method provided in any embodiment of the first aspect of this disclosure.

[0018] In a third aspect, embodiments of this disclosure provide a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the multi-target association method provided in any embodiment of the first aspect of this disclosure.

[0019] In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multi-target association method provided in any embodiment of the first aspect of the present disclosure.

[0020] The aforementioned multi-target association method, system, computer equipment, and storage medium determine the sample type based on the similarity score between the candidate detection box and the target to be associated, extract appearance features for difficult samples, determine the association probability based on the appearance features and the basic detection box features, determine the association probability based on the basic detection box for simple samples, and finally determine the target detection box that is associated with the target to be associated based on the association probability data and the preset matching algorithm.

[0021] Candidate bounding boxes and targets to be associated are categorized into simple and difficult samples based on similarity scores. For candidate bounding boxes and targets corresponding to simple samples, only basic bounding box features are extracted for association judgment, eliminating the need for appearance feature extraction. This significantly reduces unnecessary computation, saves system computing time and resources, and improves system real-time performance. For candidate detection boxes and targets to be associated with difficult samples, appearance features are extracted. The basic detection box features and appearance features are used together to make association judgments. Combining the two types of features can combine the advantages of each type of feature and make up for each other's shortcomings (the basic detection box features are easily affected by factors such as inaccurate positioning, position offset, and object occlusion, while appearance features are easily affected by factors such as lighting, lens blur, and objects with similar appearances). Through feature fusion, and using a classification model to obtain association probability data, it can adapt to various complex scenarios and improve the accuracy of association judgments for targets to be associated. Attached Figure Description

[0022] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a diagram illustrating the application environment of the multi-target association method in some embodiments; Figure 2 This is a flowchart illustrating the multi-target association method in some embodiments; Figure 3 This is a flowchart illustrating the steps of the multi-target association method in some embodiments; Figure 4 This is a flowchart illustrating the steps involved in determining a target detection box based on association probability in some embodiments; Figure 5This is a flowchart illustrating the steps involved in obtaining candidate detection boxes in some embodiments; Figure 6 Here are some block diagrams of the multi-objective association system in some embodiments; Figure 7 This is a diagram showing the internal structure of a computer device in some embodiments. Detailed Implementation

[0024] To make the technical solutions and advantages of this disclosure clearer, the embodiments and related technical content of this disclosure will be further described in detail below with reference to the accompanying drawings and text description. It should be understood that the embodiments described below are only used to explain the technical solutions of the embodiments of this disclosure and are not intended to limit more possible implementations of this disclosure.

[0025] In a first aspect, embodiments of this disclosure provide a multi-target association method. This method can be applied to, for example... Figure 1 In the application environment shown, server 101 can communicate with sensor 102 and terminal 103 via a network. Server 101 can be a standalone server or a server cluster consisting of multiple servers. Sensor 102 can be a sensor cluster installed on road infrastructure, autonomous vehicles, or non-autonomous vehicles. The sensor cluster can include multiple cameras and multiple LiDAR sensors; there are no specific restrictions on the type and number of sensors. Terminal 103 can be a management terminal controlled by staff in the automotive manufacturer's backend system. It can be used to display road traffic videos or images collected by the sensors, as well as trajectory information of targets to be associated. Terminal 103 includes, but is not limited to, various personal computers, laptops, smartphones, and tablets.

[0026] The server includes processors that run algorithms or models related to detection box extraction, appearance feature extraction, and detection box association recognition. There may be multiple processors, and the processors may be implemented in at least one of the following hardware forms: programmable logic array (PLA), field-programmable gate array (FPGA), digital signal processor (DSP), application-specific integrated circuit (ASIC), general-purpose processor, or other programmable logic devices.

[0027] Applying multi-objective association methods Figure 1 Taking server 101 as an example, in some embodiments, such as Figure 2 As shown, the multi-target association method includes steps S201 to S205 that can be executed by server 101. Each step is described in detail below.

[0028] Step S201: Obtain multiple candidate detection boxes corresponding to multiple targets to be associated in the current frame data.

[0029] The historical frame data and current frame data are road traffic data at different times. Specifically, the historical frame data can include the initial frame image of the road traffic video corresponding to the road traffic image and the historical frame point cloud data collected by radar. The historical frame data and current frame data can be two images from consecutive frames in the road traffic video. The current frame data and historical frame data can be images collected by camera sensors or point cloud data collected based on radar (ultrasonic radar, lidar, millimeter-wave radar).

[0030] The targets to be associated are dynamic targets in road traffic data, such as pedestrians and vehicles.

[0031] Specifically, obtaining multiple candidate bounding boxes corresponding to multiple targets to be associated in the current frame data can be achieved by extracting multiple candidate detection boxes from the current frame data based on the object information of the targets to be associated.

[0032] Object information can include the category, speed, location, and trajectory information of the target to be associated. Category information indicates the object type of the target to be associated, such as vehicles or pedestrians. Speed ​​information indicates the moving speed of the target to be associated, location information indicates the spatial location of the target to be associated, trajectory information indicates the motion trajectory of the target to be associated in historical frame data and subsequent frame images, and image frame information indicates the frame number of the target to be associated.

[0033] Specifically, based on the object information of the target to be associated, multiple candidate detection boxes are extracted from the current frame data. This can be done by identifying candidate objects that are likely to be the same object as the target to be associated from all objects included in the current frame data, and then extracting multiple candidate detection boxes corresponding to these candidate objects. In some other examples, it can also be done by determining the possible movement range of the target to be associated from the current frame data, identifying candidate regions, and then using objects within these candidate regions as candidate objects for detection box extraction. A target to be associated can have one or more corresponding candidate detection boxes.

[0034] Step S202: Treat each target to be associated and its corresponding multiple candidate detection boxes as a sample, and determine the sample type of each sample.

[0035] The sample types include simple samples, which indicate that there is a similarity score between any candidate detection box and the target to be associated that is greater than a preset similarity threshold, and difficult samples, which indicate that there is no similarity score between any candidate detection box and the target to be associated that is greater than a preset similarity threshold.

[0036] Specifically, step S202 may include the following steps: taking each target to be associated and its corresponding candidate detection as a sample, calculating the similarity score between the target to be associated and the candidate detection box in the sample, and determining the sample type of each sample based on the similarity score.

[0037] The similarity score can be determined based on the overlap score and / or velocity similarity.

[0038] When the similarity score is greater than the preset similarity threshold, it indicates that the candidate detection box is highly similar to the detection box of the target to be associated, and the probability that the two are the same object is high. The relationship can be determined by conventional features (such as position features, attribute features, and motion trajectory features). When the similarity score is less than or equal to the similarity threshold, the candidate detection box is less similar to the detection box of the target to be associated. They may be detection boxes of different objects, or they may be detection boxes of the same object, but there may be positional offset, object occlusion, etc., which is more difficult to identify than candidate detection boxes in simple samples.

[0039] Step S203: Extract the appearance features of candidate detection boxes and targets to be associated in difficult samples.

[0040] Among these, appearance features can be used to represent the consistency information of an object's identity. Specifically, appearance features can include basic appearance attribute features and semantic structured features. Basic appearance attribute features include color features, texture features, contour features, and other local detail features (such as pedestrian accessory image features, hairstyle detail image features, and license plate area features). Semantic structured features can include semantic attribute features (such as pedestrian gender, age, clothing style, vehicle type, brand, etc.) and posture and behavior features (such as walking posture, formal posture, orientation, etc.).

[0041] In some specific examples, the appearance features of the second type of candidate detection boxes can be extracted in parallel with other object detection models. These other object detection models can be models used to detect and identify static objects, or other detection models that are not related to the identification and detection of the object to be associated.

[0042] Step S204: Based on the preset classification model, obtain the association probability data according to the basic detection box features of simple samples, the basic detection box features of difficult samples, and the appearance features.

[0043] The basic detection box features include at least attribute features, motion trajectory features, and spatiotemporal features. Attribute features may include the type, source, size, and distance information of the object corresponding to the detection box from the acquisition device. Motion trajectory features may include trajectory type, speed, motion state (whether it is moving), observation state, observation source (e.g., radar or camera), and number of observations. Spatiotemporal features may include trajectory history features (used to record historical changes in the trajectory), overlap features, projection features, and spatiotemporal fusion features (e.g., trajectory center statistical features, trajectory velocity statistical features, and trajectory overlap statistical features).

[0044] Association probability data can be used to represent the association probability value between the target to be associated and the candidate detection box. The higher the association probability value, the higher the probability that the target to be associated and the candidate detection box are the same target.

[0045] Specifically, the basic detection box features of simple samples are input into the classification model, and the basic detection box features and appearance features of difficult samples are input into the classification model to obtain the association probability data.

[0046] Step S205: Based on the association probability data and the preset matching algorithm, determine the target detection boxes that are associated with each target to be associated from multiple candidate detection boxes.

[0047] The preset matching algorithm can be the Hungarian algorithm, which solves the association probability data using the Hungarian matching algorithm to determine the target detection boxes that are associated with each target to be associated from multiple candidate detection boxes. The target detection box and the target to be associated are the same object.

[0048] In the above multi-target association method, the sample type is determined based on the similarity score between the candidate detection box and the target to be associated. Appearance features are extracted for difficult samples. The association probability is determined based on the appearance features and the basic detection box features. For simple samples, the association probability is determined based on the basic detection box. Finally, the target detection box that is associated with the target to be associated is determined based on the association probability data and the preset matching algorithm.

[0049] Candidate detection boxes and targets to be associated are categorized into simple and difficult samples based on similarity scores. For candidate detection boxes and targets to be associated corresponding to simple samples, only basic detection box features are obtained for association judgment, eliminating the need for appearance feature extraction. This significantly reduces unnecessary computation, saves system computing time and resources, and improves system real-time performance.

[0050] For candidate detection boxes and targets to be associated with difficult samples, appearance features are extracted. The basic detection box features and appearance features are used to make association judgments. Combining the two types of features can combine the advantages of each type of feature and make up for each other's shortcomings (the basic detection box features are easily affected by factors such as inaccurate positioning, position offset, and object occlusion, while the appearance features are easily affected by factors such as lighting, lens blur, and objects with similar appearances). Through feature fusion and using a classification model to obtain association probability data, it can adapt to various complex scenarios and improve the accuracy of association judgments for targets to be associated.

[0051] In some embodiments, extracting the appearance features of candidate detection boxes and targets to be associated in difficult samples includes: extracting the appearance features of candidate detection boxes and targets to be associated in difficult samples based on a preset re-identification model.

[0052] The re-identification model runs in parallel with the target detection model used to identify and detect static targets in the current frame data.

[0053] Static targets can include transportation facilities, environmental elements, etc. The re-identification model is the REID (Re-identification) model, and the corresponding appearance features are REID features.

[0054] In some specific examples, the re-identification model can be pre-trained on a labeled training dataset. This model can be a convolutional neural network architecture or other network architectures that can be used for feature extraction.

[0055] As will be readily understood by those skilled in the art, target association methods can include a detection phase and a tracking phase. In related technologies, the detection phase runs a target detection model to identify and locate candidate detection boxes from road traffic data. In the tracking phase, association matching is performed, pairing candidate detection boxes with targets to be associated and determining the association relationship to achieve continuous tracking. That is, in related technologies, to extract appearance features, the target detection models for dynamic targets, static targets, etc., in the current frame data must be run before entering the tracking phase, where appearance features are extracted. The re-identification model used to extract appearance features and the target detection model in the detection phase run sequentially. However, in the embodiments of this application, the re-identification model used to extract appearance features and the target detection model for identifying and detecting static targets run in parallel. Running the re-identification model in parallel during the detection phase fully utilizes system resources, does not significantly increase overall time consumption, improves computational efficiency, and can meet the needs of application scenarios with high real-time requirements.

[0056] Furthermore, the method provided in this application embodiment, by first extracting appearance features and then forming teams for association judgment, avoids the situation where the features of the same target to be associated are extracted repeatedly during the association process of multiple targets to be associated, thus saving computing resources and improving computing efficiency.

[0057] In some embodiments, determining the sample type of each sample may include the following steps: The sample type is determined based on the similarity score between the target to be associated in the sample and the candidate detection box; If any candidate detection box in the sample has a similarity score greater than the preset similarity threshold with the target to be associated, the sample is a simple sample. A sample is considered a difficult sample if no candidate detection box in the sample has a similarity score greater than the preset similarity threshold with the target to be associated.

[0058] In some embodiments, determining the sample type of each sample may further include the following steps: The similarity score is determined based on the overlap score and velocity similarity score between the target to be associated and the candidate detection box in the sample. The sample type is determined based on the similarity score, the target to be associated, and the category information of the candidate detection box; If any candidate bounding box in the sample has a similarity score greater than the preset similarity threshold with the target to be associated, and the category information is consistent, the sample is a simple sample. A sample is considered a difficult sample if no candidate bounding box in the sample has a similarity score greater than the preset similarity threshold and the category information is consistent with the target to be associated.

[0059] The overlap score, calculated based on the intersection-union ratio (IU), represents the degree of overlap between the candidate detection box and the detection box of the target to be associated. The velocity similarity score quantifies the similarity between the velocity features of the candidate detection box and the detection box of the target to be associated. Consistent category information indicates that the candidate object corresponding to the candidate detection box and the target to be associated belong to the same category, such as both being vehicles or both being pedestrians.

[0060] The similarity score is determined based on the overlap score and the speed similarity score. It can be calculated using a preset weighted summation method to obtain the similarity score.

[0061] In some specific examples, determining the type of a sample may also include the following steps: The sample type is determined based on the overlap score, velocity similarity score, and category information between the target to be associated and the candidate detection box in the sample.

[0062] If any candidate detection box in a sample has an overlap score greater than the overlap threshold, a velocity similarity greater than the preset velocity threshold, and consistent category information with the target to be associated, then the sample is a simple sample; otherwise, it is a difficult sample.

[0063] The overlap threshold and speed threshold can be set according to actual needs, and no special restrictions are imposed here.

[0064] In some embodiments, such as Figure 3 As shown, the multi-target association method may further include steps S301 and S302, and step S204 may include steps S303 to S306.

[0065] Step S301: Pair the target to be associated with multiple candidate detection boxes one by one to obtain multiple association groups.

[0066] Step S302: Obtain the basic detection box features of multiple related groups.

[0067] Step S303: Perform feature fusion on the basic detection box features of the association group corresponding to the simple sample to obtain the first feature vector.

[0068] Step S304: Perform feature fusion on the basic detection box features and appearance features of the association group corresponding to the difficult sample to obtain the second feature vector.

[0069] Step S305: Input the first feature vector and the second feature vector into the classification model to obtain the association probability data of multiple association groups.

[0070] Specifically, feature fusion can be achieved through feature concatenation or weighted fusion.

[0071] In this embodiment, the detection phase focuses on identifying simple and difficult samples and extracting appearance features, while the tracking phase performs feature fusion and random forest classification association. This architecture design breaks through the limitations of related technologies that use a single stage for association judgment, allowing each stage to focus on a specific task, thus improving the overall performance and maintainability of the system. Furthermore, for difficult samples, by concatenating or weighting the basic detection box features and appearance features, the advantages of different features can be fully utilized, compensating for the limitations of a single feature, enabling a more comprehensive and accurate judgment of association relationships, and improving the accuracy and robustness of multi-target association.

[0072] In some embodiments, the classification model is a random forest model, and the matching algorithm is the Hungarian algorithm.

[0073] In some specific examples, the binary classification model can be an XGBoost (Extreme Gradient Boosting, an improved ensemble learning algorithm based on gradient boosting decision trees) model.

[0074] like Figure 4 As shown, step S205 may include steps S401 to S403.

[0075] Step S401: Construct the association cost matrix based on the association probability data.

[0076] Step S402: Solve for the optimal solution of the association cost matrix based on the Hungarian algorithm to obtain preliminary matching results.

[0077] Step S403: Based on the preliminary matching results and the preset confidence threshold, determine the target detection box that is the same as the target to be associated from the candidate detection boxes corresponding to the preliminary matching results.

[0078] In some embodiments, such as Figure 5 As shown, step S201 may include steps S501 to S502.

[0079] Step S501: Determine the candidate object region from the current frame data based on the location information and trajectory information of the target to be associated.

[0080] Step S502: Based on the preset detection model, extract multiple candidate detection boxes corresponding to multiple candidate objects from the candidate object region.

[0081] The detection model can be different from or the same as the target detection model mentioned in the previous embodiments.

[0082] In some specific examples, based on the location and trajectory information of the target to be associated, the region in the image where the target object may appear is determined, i.e., the candidate object region. According to the detection model, target extraction is performed on each candidate object in the candidate object region, resulting in multiple candidate detection boxes corresponding to multiple candidate objects. The candidate object region can be determined by the area of ​​a circle drawn with the current location of the target to be associated as the center and the farthest trajectory position corresponding to the trajectory information as the radius. Alternatively, it can be determined by starting from the current location, determining the possible movement path of the target to be associated based on the trajectory information, and extending the movement path as the starting edge outwards by a preset distance.

[0083] In some specific examples, step S201 may also include: determining multiple candidate objects from the current frame data based on the category information and velocity information of the target to be associated; extracting detection boxes from the multiple candidate objects according to a preset target detection model; and obtaining multiple candidate detection boxes corresponding to the candidate objects.

[0084] Among them, determining multiple candidate objects from the current frame data can be done by selecting objects with consistent category information and speed information similarity higher than a threshold from all objects included in the current frame data.

[0085] It should be understood that, although Figures 2 to 5 The steps in the flowchart are shown sequentially according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Figures 2 to 5 Unless otherwise expressly stated herein, the steps illustrated and other steps involved in the embodiments are not subject to strict order restrictions and may be performed in other orders. Furthermore, at least some steps in the foregoing embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but may be performed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0086] In a second aspect, embodiments of this disclosure provide a multi-objective association system, such as Figure 6 As shown, the multi-target association system 600 may include a sensor 610 and a processor 620. The sensor 610 is used to collect road traffic data and send the collected road traffic data to the processor 620. The processor 620 is used to receive the road traffic data and implement the steps of the multi-target association method provided in any embodiment of the first aspect of this disclosure.

[0087] In some specific examples, sensor 610 may be a sensor cluster installed on an autonomous vehicle, a non-autonomous vehicle, or a road setting. The sensor cluster may include radar sensors (such as lidar, millimeter-wave radar, etc.) and camera sensors.

[0088] In some embodiments, processor 620 includes a first processor and a second processor.

[0089] The first processor is used to obtain multiple candidate detection boxes corresponding to multiple targets to be associated in the current frame data, take each target to be associated and its corresponding multiple candidate detection boxes as a sample, determine the sample type of each sample, and extract the appearance features of the candidate detection boxes and the targets to be associated in the hard samples.

[0090] The second processor is used to obtain association probability data based on the basic detection box features of simple samples, the basic detection box features of difficult samples, and appearance features, according to a preset classification model. Based on the association probability data and a preset matching algorithm, it determines the target detection box that is associated with each target to be associated from multiple candidate detection boxes.

[0091] The target to be associated is a dynamic target in historical frame data. Historical frame data and current frame data are road traffic data from different times, including image data and / or point cloud data. Sample types include simple samples indicating that any candidate detection box has a similarity score greater than a preset similarity threshold with the target to be associated, and hard samples indicating that no candidate detection box has a similarity score greater than a preset similarity threshold with the target to be associated. Basic detection box features include at least attribute features, motion trajectory features, and spatiotemporal features.

[0092] In some specific application scenarios, the first processor corresponds to the system's detection module, used to perform steps related to target detection and appearance feature extraction, while the second processor can correspond to the system's tracking module, used to perform steps related to the target tracking stage. The first and second processors are two completely independent hardware devices; for example, the first processor can be a first chip, and the second processor can be a second chip.

[0093] For more specific limitations regarding the multi-objective association system, please refer to the limitations of the multi-objective association method above. The processor in the multi-objective association system can also be used to execute more steps of the multi-objective association method in the embodiments of this disclosure, which will not be repeated here.

[0094] In a third aspect, embodiments of this disclosure provide a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the multi-target association method provided in any embodiment of the first aspect of this disclosure.

[0095] In some embodiments, the computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. This computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. It can be, for example... Figure 6The processor 620 shown includes a computer device whose memory comprises a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The computer device's database stores image frames of road traffic images, object information of targets to be associated, and feature data of candidate detection boxes. The computer device's network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps of the multi-target association method provided in any embodiment of the first aspect of this disclosure.

[0096] Those skilled in the art will understand that Figure 7 The structures shown are merely block diagrams of some structures related to the embodiments of this disclosure and do not constitute a limitation on the computer devices to which the embodiments of this disclosure are applied. Specific computer devices may include more or fewer components than those shown in the figures, or combine certain components, or have different component arrangements.

[0097] In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multi-target association method provided in any embodiment of the first aspect of the present disclosure.

[0098] The computer-readable storage medium may be Figure 7 The computer-readable storage medium in the computer device shown.

[0099] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The aforementioned computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments of this disclosure can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0100] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this disclosure.

[0101] The above embodiments merely illustrate several implementation methods of this disclosure, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of protection of this disclosure. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this disclosure, and these all fall within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the appended claims.

Claims

1. A multi-objective association method, characterized in that, The method includes: Obtain multiple candidate detection boxes corresponding to multiple targets to be associated in the current frame data. The targets to be associated are dynamic targets in the historical frame data. The historical frame data and the current frame data are road traffic data at different times, including image data and / or point cloud data. Each target to be associated and its corresponding multiple candidate detection boxes are taken as a sample. The sample type of each sample is determined. The sample type includes simple samples, which indicate that there is a similarity score between any candidate detection box and the target to be associated that is greater than a preset similarity threshold, and difficult samples, which indicate that there is no similarity score between any candidate detection box and the target to be associated that is greater than a preset similarity threshold. Extract the appearance features of candidate detection boxes and the target to be associated from the difficult samples; Based on a preset classification model, association probability data is obtained according to the basic detection box features of the simple samples, the basic detection box features of the difficult samples, and the appearance features. The basic detection box features include at least attribute features, motion trajectory features, and spatiotemporal features. Based on the association probability data and the preset matching algorithm, target detection boxes that are associated with each of the targets to be associated are determined from the multiple candidate detection boxes.

2. The method according to claim 1, characterized in that, The extraction of the appearance features of candidate detection boxes and the target to be associated in the difficult samples includes: Based on a preset re-identification model, the appearance features of candidate detection boxes and targets to be associated in the difficult samples are extracted. The re-identification model runs in parallel with the target detection model used to identify and detect static targets in the current frame data.

3. The method according to claim 1, characterized in that, Determining the sample type for each sample includes: The sample type is determined based on the similarity score between the target to be associated in the sample and the candidate detection box; If any candidate detection box in the sample has a similarity score greater than a preset similarity threshold with the target to be associated, the sample is a simple sample. The sample is considered a difficult sample if no candidate detection box in the sample has a similarity score greater than a preset similarity threshold with the target to be associated.

4. The method according to claim 1, characterized in that, Determining the sample type of each sample also includes: The similarity score is determined based on the overlap score and velocity similarity score between the target to be associated and the candidate detection box in the sample. The sample type is determined based on the similarity score, the category information of the target to be associated and the candidate detection box; If any candidate detection box in the sample has a similarity score greater than a preset similarity threshold with the target to be associated, and the category information is consistent, the sample is a simple sample. The sample is considered a difficult sample if no candidate detection box in the sample has a similarity score greater than a preset similarity threshold and consistent category information with the target to be associated.

5. The method according to claim 1, characterized in that, The method further includes: The target to be associated is paired with the multiple candidate detection boxes one by one to obtain multiple association groups; Obtain the basic detection box features of the multiple associated groups; The pre-defined classification model, based on the basic detection box features of the simple samples, the basic detection box features of the difficult samples, and appearance features, obtains association probability data, including: The basic detection box features of the association group corresponding to the simple sample are fused to obtain the first feature vector; The basic detection box features and appearance features of the association group corresponding to the difficult sample are fused to obtain the second feature vector; The first feature vector and the second feature vector are input into the classification model to obtain the association probability data of the multiple association groups.

6. The method according to claim 1, characterized in that, The classification model is a random forest model, and the matching algorithm is the Hungarian algorithm; the step of determining target detection boxes that are associated with each of the targets to be associated from the plurality of candidate detection boxes according to the association probability data and the preset matching algorithm includes: Based on the association probability data, construct an association cost matrix; The optimal solution of the association cost matrix is ​​obtained based on the Hungarian algorithm to obtain preliminary matching results; Based on the preliminary matching results and the preset confidence threshold, a target detection box that is the same as the target to be associated is determined from the candidate detection boxes corresponding to the preliminary matching results.

7. The method according to claim 1, characterized in that, The step of obtaining multiple candidate detection boxes corresponding to multiple targets to be associated in the current frame data includes: Based on the location and trajectory information of the multiple targets to be associated, candidate object regions are determined from the current frame data; Based on a preset detection model, multiple candidate detection boxes corresponding to multiple candidate objects are extracted from the candidate object region.

8. A multi-objective association system, characterized in that, The system includes a sensor and a processor. The sensor is used to acquire road traffic data and send it to the processor. The processor is used to receive the road traffic data and execute the steps of the method according to any one of claims 1 to 7.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images