A processing method for matching newly detected targets with targets that have disappeared

Abstract

The embodiment of the present application relates to a kind of processing methods for matching new detection target to disappeared target, the method comprises: step 1 is carried out pedestrian target detection to starting frame image;And carry out tracking target object initialization to constitute tracking target set;And initialize disappeared target set;Step 2 is carried out pedestrian target detection to new frame image;And carry out timeout target object filtering to disappeared target set;And carry out target object matching;Step 3 is carried out disappeared target object conversion to disappeared target set by the tracking target object that has failed to be listed in any matching pair, and is deleted from tracking target set;Step 4 is initialized corresponding tracking target object to the new target detection frame that has failed to be listed in any matching pair, add in tracking target set and return step 2 when adding is completed.Through the method of the present application, as many as possible known pedestrian target can be matched at each moment.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a processing method for matching newly detected targets with those that have disappeared. Background Technology

[0002] In an autonomous driving system, the perception module can perform pedestrian target detection and matching processing on the perception data continuously output by the vision sensor, i.e., the visual images captured by the camera, to obtain the target tracking trajectory of each pedestrian target around the vehicle. Then, using conventional state analysis tools (such as Kalman filters) with the target tracking trajectory as input observations, the motion state of each pedestrian target at each time point can be obtained.

[0003] Under normal circumstances, the typical processing steps for pedestrian target detection and matching based on visual images by the perception module are as follows:

[0004] The first step is to perform target detection on the first frame image at time t1 to obtain target detection boxes bbox1 (Bounding Box, bbox) for multiple pedestrian targets; and initialize a corresponding tracking target object for each pedestrian target, which has at least two object attributes: target identifier and target detection box; and set the target detection box of the corresponding tracking target object based on the target detection box bbox1 of each pedestrian target.

[0005] Step 2, for t i At time i (i>1), the i-th frame image is used to perform object detection to obtain new object detection boxes (bboxes) for multiple pedestrian targets. i ; and for each new target detection bounding box obtained this time i The target is matched pairwise with each existing target detection bounding box to obtain multiple new target-target matching pairs, and a new target detection bounding box (bbox) is generated based on each new target-target matching pair. i Update the target detection bounding box settings for the matched tracking target object;

[0006] The third step is to treat the pedestrian targets corresponding to the existing tracked target objects that were not included in the new target-target object matching pairs in the second step as disappeared targets, and to delete the tracked target objects corresponding to each disappeared target from the existing tracked target object set.

[0007] Step 4: Add the new target detection bounding boxes (bboxes) that were not included in the new target-target matching pairs in Step 2. iThe corresponding pedestrian target is regarded as a new pedestrian target, and a corresponding tracking target object is initialized for each new pedestrian target based on a similar processing method as in the first step, and a new target detection bounding box (bbox) is generated based on the new pedestrian target. i Configure the target detection bounding box for the newly created tracking target object.

[0008] Here, through the four steps described above, we can see that if, in the second step, no matching new target detection box (bbox) is found in the current i-th frame of the image for an existing tracking target object... i The pedestrian target corresponding to the tracked target object will be deleted as a disappeared target. However, in practical applications, we have found two reasons that may cause premature deletion of pedestrian targets still around the vehicle: 1) Due to the camera's angle at the current moment, the camera may fail to capture the pedestrian target still around the vehicle in the current i-th frame image. In this case, the pairwise matching method in the second step above cannot find a matching new target detection box (bbox) for the tracked target object corresponding to the pedestrian target. i Therefore, based on the third step mentioned above, the pedestrian target will be regarded as a disappeared target and deleted prematurely; 2) Due to obstacle occlusion, the camera may fail to capture the pedestrian target that is still around the vehicle at the current moment but is temporarily occluded by the obstacle in the current frame image. At this time, the pairwise matching method based on the second step mentioned above cannot find a matching new target detection box (bbox) for the tracking target object corresponding to the pedestrian target. i Therefore, based on the third step mentioned above, the pedestrian target will also be regarded as a disappeared target and deleted prematurely.

[0009] Furthermore, as explained in the four steps above, if a pedestrian target is deleted at any point due to the two reasons mentioned above, the perception module will treat it as a new pedestrian target even if it detects the target again at a later time. It will then create a new tracking target object and perform a new round of target tracking processing based on this new object to obtain a new target tracking trajectory. However, because this new target tracking trajectory loses the previous historical tracking trajectory, using it for the target motion state analysis at the current moment will naturally reduce the accuracy of the current state analysis. Therefore, to minimize the impact of premature deletion of pedestrian targets on the accuracy of state analysis, the processing steps of the perception module need to be improved, which is the problem that this invention aims to solve. Summary of the Invention

[0010] The purpose of this invention is to address the shortcomings of existing technologies by providing a processing method, electronic device, and computer-readable storage medium for matching newly detected targets with already disappeared targets. First, in addition to using a conventional set of tracking targets to track all known pedestrian targets, a new set of disappeared targets is added to cache historical target information for all temporarily disappeared pedestrian targets. Second, at each time point, besides the conventional pairwise matching of each newly detected target detection box (hereinafter referred to as the new detected target) with the target detection box of each tracking target object (hereinafter referred to as the tracking target) to obtain the corresponding new detected target-tracking target matching pair, an additional step is added to the conventional matching: pairwise matching of each remaining unmatched new detected target with the target detection box of each already disappeared target to obtain the corresponding new detected target-disappeared target. The target matching pair processing mechanism, after obtaining a new detected target-disappeared target matching pair, evaluates its reasonableness based on the appearance / disappearance type matching relationship between the two, and updates the tracking target set based on the evaluation results. Then, the new target-target object matching pair set at the current moment is composed of all new detected target-tracking target matching pairs plus all reasonable new detected target-disappeared target matching pairs. Next, all tracking target objects in the current tracking target set that are not included in this new target-target object matching pair set are removed, and each removed tracking target object is added as a temporarily disappeared target object to the disappeared target set. Finally, similar to conventional processing, corresponding tracking targets are created for all new detected targets that are not included in this new target-target object matching pair set and added to the tracking target set. This invention first associates the newly detected targets and tracking targets obtained at the current moment based on conventional processing, and then associates the newly detected targets that have not formed an association with temporarily disappeared targets based on a supplementary mechanism. This minimizes the addition of new pedestrian targets and maximizes the tracking of known pedestrian targets at each moment, thereby avoiding the premature deletion of known pedestrian targets.

[0011] To achieve the above objectives, a first aspect of the present invention provides a method for matching newly detected targets with disappeared targets, the method comprising:

[0012] Step 1: Take the first frame image obtained at the start time as the corresponding starting frame image; perform pedestrian target detection processing on the starting frame image to obtain multiple first newly added target detection boxes; perform tracking target object initialization processing on each first newly added target detection box to obtain a corresponding first tracking target object; and form a corresponding first tracking target set by all the obtained first tracking target objects; and initialize the first disappeared target set to empty.

[0013] Step 2: Take the next frame image as the corresponding new frame image; perform pedestrian target detection processing on the new frame image to obtain multiple first new target detection boxes; perform timeout target object filtering processing on the first set of disappeared targets based on a preset absolute time difference threshold; and perform target object matching processing on each of the first new target detection boxes according to the first tracking target set and the first set of disappeared targets to obtain multiple sets of first new target-target object matching pairs.

[0014] Step 3: Record each first tracking target object in the first tracking target set that fails to be included in any of the first new target-target object matching pairs as the corresponding second tracking target object; perform a disappearing target object conversion process on each second tracking target object to obtain the corresponding first disappeared target object and add it to the first disappeared target set; and delete each second tracking target object from the first tracking target set.

[0015] Step 4: Each of the first new target detection boxes that fails to be included in any of the first new target-target object matching pairs is taken as a new first newly added target detection box; and a corresponding first tracking target object is obtained by performing tracking target object initialization processing based on each new first newly added target detection box and added to the first tracking target set; and when the tracking target set is completed, return to step 2 to continue processing the next frame image obtained at the next time moment.

[0016] Preferably, the first newly added target detection box and the first new target detection box each correspond to a set of target detection box parameters. The target detection box parameters include the detection box center position coordinates, the detection box size, and the detection box orientation angle. The detection box center position coordinates are pixel coordinates, and the detection box size includes height and width.

[0017] The first tracking target object includes a first target identifier and a first object detection box;

[0018] The first set of disappeared targets includes multiple first disappeared target objects; the first disappeared target objects include a first disappearance method, a first disappearance time, a second target identifier, and a second object detection box; the first disappearance method includes disappearing from the left side of the field of view, disappearing from the right side of the field of view, disappearing from the distance of the field of view, and disappearing due to occlusion.

[0019] Preferably, the initialization process for the tracking target object specifically includes:

[0020] Assign a corresponding first tracking target object to the current first newly added target detection box; assign a unique identifier to the current first tracking target object as the corresponding first target identifier, and set the first object detection box of the current first tracking target object as the current first newly added target detection box.

[0021] Preferably, the step of filtering out timed-out target objects from the first set of disappeared targets based on a preset absolute time difference threshold specifically includes:

[0022] When the first set of disappeared targets is not empty, the corresponding time of the current moment is extracted as the corresponding first time; and the absolute time difference between the first disappearance time and the first time of each first disappeared target object in the first set of disappeared targets is calculated to obtain the corresponding first absolute time difference; and the first disappeared target objects in the first set of disappeared targets whose first absolute time difference exceeds the absolute time difference threshold are deleted.

[0023] Preferably, the step of performing target object matching processing on each of the first new target detection boxes based on the first tracking target set and the first disappeared target set to obtain multiple sets of first new target-target object matching pairs specifically includes:

[0024] For each first new target detection box and the first object detection box of each first tracked target object in the first tracking target set, pairwise matching is performed to obtain multiple sets of first detection box matching pairs. Each first new target detection box in each set of first detection box matching pairs and the first tracked target object corresponding to the first object detection box in this matching pair form a corresponding first new target-target object matching pair. Using the first object detection box of the first tracked target object in each first new target-target object matching pair as the target detection box state quantity at the previous moment and the first new target detection box as the current target detection box observation quantity, a preset Kalman filter is used to predict the target detection box state quantity at the current moment based on the target detection box state quantity at the previous moment and the current target detection box observation quantity. The prediction result is used as the corresponding current predicted target detection box, and the first object detection box of the first tracked target object in the current matching pair is updated to the current predicted target detection box. Each first detection box matching pair includes one first new target detection box and one matched first object detection box.

[0025] When the first set of disappeared targets is not empty, the first new target detection box that fails to be included in any set of first detection box matching pairs is recorded as the corresponding second new target detection box; and pairwise matching is performed on each second new target detection box and the second object detection box of each first disappeared target object in the first set of disappeared targets to obtain multiple sets of corresponding second detection box matching pairs; and the second new target detection box in each set of second detection box matching pairs and the first disappeared target object corresponding to the second object detection box in this matching pair form a corresponding first matching pair; and a reasonableness evaluation is performed on each of the obtained first matching pairs to obtain the corresponding first evaluation result; and the first The first matching pair deemed unreasonable by the evaluation result is deleted; and the corresponding first tracking target object is added to the first tracking target set based on the remaining first matching pairs after target object transformation processing; and the corresponding first disappeared target object in the first disappeared target set is deleted based on the remaining first matching pairs; and a corresponding first new target-target object matching pair is formed by the first new target detection box and the first tracking target object corresponding to the remaining first matching pairs; the second detection box matching pair includes a second new target detection box and a matching second object detection box; the first evaluation result includes reasonable and unreasonable.

[0026] Output all the first new target-target object matching pairs obtained.

[0027] Furthermore, the step of performing a reasonableness assessment on each of the obtained first matching pairs to obtain the corresponding first assessment result specifically includes:

[0028] The second new target detection box and the first disappeared target object of the first matching pair are recorded as the corresponding first detection box and first object; the center position coordinates of the first detection box are extracted as the corresponding first center coordinates; and the first center coordinates are transformed from pixel coordinate system to world coordinate system to obtain the corresponding second center coordinates.

[0029] The new frame image corresponding to the current moment is taken as the corresponding first image; and the left and right boundary regions of the first image are confirmed according to the preset left boundary width and right boundary width of the image to obtain the corresponding first left boundary region and first right boundary region.

[0030] When the first center coordinate is within the first left boundary region, the corresponding first appearance mode is set to appear from the left side of the field of view; when the first center coordinate is within the first right boundary region, the corresponding first appearance mode is set to appear from the right side of the field of view; when the first center coordinate is neither within the first left boundary region nor within the first right boundary region, the corresponding first front-to-back distance is calculated by the front-to-back distance between the second center coordinate and the current positioning coordinate of the vehicle, and it is identified whether the first front-to-back distance has exceeded the preset front-to-back distance threshold. If it has exceeded the threshold, the corresponding first appearance mode is set to appear from a distance in the field of view; if it has not exceeded the threshold, the corresponding first appearance mode is set to appear after leaving the obstruction.

[0031] A first matching result is obtained by identifying whether the first appearance method matches the first disappearance method of the first object; the first matching result includes matching and non-matching.

[0032] When the first matching result is a match, the current time is taken as the corresponding first end time, and the time obtained by subtracting the absolute time difference threshold from the first end time is taken as the corresponding first start time; the vehicle's orientation angle at the first start time and the first end time is obtained as the corresponding first start and first end angles; the vehicle's average speed during the time period from the first start time to the first end time is obtained as the corresponding first average speed; the absolute angle difference between the first start and the first end angles is calculated to obtain the corresponding first absolute angle difference; when the first average speed is greater than a preset low speed threshold and the first absolute angle difference is less than a preset large steering angle difference threshold, the corresponding first evaluation result is set to reasonable; when the first average speed is less than or equal to the low speed threshold or the first absolute angle difference is greater than or equal to the large steering angle difference threshold, the corresponding first evaluation result is set to unreasonable.

[0033] When the first matching result is a mismatch, the corresponding first evaluation result is set to unreasonable.

[0034] More preferably, the step of identifying whether the first appearance mode matches the first disappearance mode of the first object to obtain a corresponding first matching result specifically includes:

[0035] If the first appearance method is to appear from the left side of the field of view, then the first matching result is set to match only when the first disappearance method is to disappear from the left side of the field of view; for the other methods of the first disappearance method, the corresponding first matching result is set to non-match.

[0036] If the first appearance method is from the right side of the field of view, then the first matching result is set to match only when the first disappearance method is from the right side of the field of view; otherwise, the first matching result is set to non-match.

[0037] If the first appearance method is to appear from a distance in the field of vision, then the first matching result is set to match only when the first disappearance method is to disappear from a distance in the field of vision; otherwise, the first matching result is set to non-match.

[0038] If the first appearance method is to appear after leaving the occlusion, then the corresponding first matching result is set to match only when the first disappearance method is to disappear while being occluded. For all other disappearance methods, the corresponding first matching result is set to non-match.

[0039] Furthermore, the step of adding the corresponding first tracking target object obtained by performing tracking target object transformation processing on the remaining first matching pairs to the first tracking target set specifically includes:

[0040] Create a new first tracking target object for the current first matching pair; set the first target identifier of the newly created first tracking target object as the second target identifier of the first disappeared target object of the current first matching pair; use the second object detection box of the first disappeared target object of the current first matching pair as the historical target detection box state quantity, and the second new target detection box of the current first matching pair as the current target detection box observation quantity. Based on a preset Kalman filter, predict the target detection box state quantity at the current moment according to the historical target detection box state quantity and the current target detection box observation quantity, and use the prediction result as the corresponding current predicted target detection box. Set the first object detection box of the newly created first tracking target object as the current predicted target detection box. Add the first tracking target object that has been set to the first tracking target set.

[0041] Furthermore, the step of performing the disappearance target object conversion process on each of the second tracking target objects to obtain the corresponding first disappearance target object and adding it to the first disappearance target set specifically includes:

[0042] Record the current second tracking target object as the corresponding historical tracking target object; and create a new first disappeared target object as the corresponding latest disappeared target object;

[0043] The coordinates of the center position of the detection box of the first object detection box of the historical tracking target object are extracted as the corresponding third center coordinates; the third center coordinates are transformed from pixel coordinate system to world coordinate system to obtain the corresponding fourth center coordinates; the vehicle positioning coordinates of the historical tracking target object at the corresponding time are obtained as the corresponding first vehicle coordinates; the front-to-back distance between the fourth center coordinates and the first vehicle coordinates is calculated to obtain the corresponding second front-to-back distance; the starting frame image or the new frame image of the historical tracking target object at the corresponding time is used as the corresponding second image; and the left and right boundary regions of the second image are confirmed according to the preset left and right boundary widths of the image to obtain the corresponding second left boundary region and second right boundary region.

[0044] When the third center coordinate is in the second left boundary region, the first disappearance mode of the latest disappearing target object is set to disappear from the left side of the field of view; when the third center coordinate is in the second right boundary region, the first disappearance mode of the latest disappearing target object is set to disappear from the right side of the field of view; when the third center coordinate is neither in the second left boundary region nor in the second right boundary region, it is identified whether the second front-back distance exceeds a preset front-back distance threshold. If it exceeds the threshold, the first disappearance mode of the latest disappearing target object is set to disappear from the distance of the field of view. If it does not exceed the threshold, the first disappearance mode of the latest disappearing target object is set to disappear by occlusion.

[0045] Set the first disappearance time of the latest disappeared target object to the corresponding time of the current moment;

[0046] Set the second target identifier of the latest disappeared target object as the first target identifier of the historically tracked target object;

[0047] The first object detection box of the historical tracking target object is taken as the target detection box state quantity of the previous moment. Based on the preset Kalman filter, the target detection box state quantity of the current moment is predicted in one step according to the target detection box state quantity of the previous moment, and the prediction result is used as the corresponding current predicted target detection box. The second object detection box of the latest disappeared target object is set as the current predicted target detection box.

[0048] The newly disappeared target object, once configured, is added to the first set of disappeared targets.

[0049] A second aspect of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0050] The processor is used to couple with the memory, read and execute instructions in the memory to implement the steps of the method described in the first aspect above;

[0051] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0052] A third aspect of the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a computer, cause the computer to perform the instructions described in the first aspect.

[0053] This invention provides a method, electronic device, and computer-readable storage medium for matching newly detected targets with previously disappeared targets. First, in addition to using a conventional set of tracking targets to track all known pedestrian targets, a new set of disappeared targets is added to cache historical target information for all temporarily disappeared pedestrian targets. Second, at each time step, besides the conventional pairwise matching of the target detection boxes of each newly detected target with each tracking target to obtain corresponding new detected target-tracking target matching pairs, a new processing mechanism is added to the conventional matching process: pairwise matching of the target detection boxes of each remaining unmatched newly detected target with each previously disappeared target to obtain corresponding new detected target-disappeared target matching pairs. Furthermore, upon obtaining the new detected target... After a target-disappeared target matching pair is established, its reasonableness is evaluated based on the appearance / disappearance type matching relationship between the two. The tracking target set can be updated based on the evaluation results. This results in a new target-target object matching pair set at the current moment, composed of all newly detected target-tracking target matching pairs plus all reasonable newly detected target-disappeared target matching pairs. Then, all tracking target objects in the current tracking target set that are not included in this new target-target object matching pair set are removed, and each removed tracking target object is added as a temporarily disappeared target object to the disappeared target set. Finally, similar to conventional processing, corresponding tracking targets are created for all newly detected targets that are not included in this new target-target object matching pair set and added to the tracking target set. This invention first associates newly detected targets and tracking targets obtained at the current moment using conventional processing methods, and then associates newly detected targets that have not formed an association with temporarily disappeared targets using a supplementary mechanism. This minimizes the addition of new pedestrian targets and maximizes the tracking of known pedestrian targets at each moment, thus avoiding the problem of premature deletion of known pedestrian targets. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of a processing method for matching newly detected targets with already disappeared targets, provided in Embodiment 1 of the present invention;

[0055] Figure 2 This is a schematic diagram of the structure of an electronic device provided in Embodiment 2 of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0057] Embodiment 1 of the present invention provides a method for matching newly detected targets with those that have disappeared, such as... Figure 1 The diagram illustrates a method for matching newly detected targets with already disappeared targets, as provided in Embodiment 1 of the present invention. This method mainly includes the following steps:

[0058] Step 1: Take the first frame image obtained at the start time as the corresponding starting frame image; perform pedestrian target detection processing on the starting frame image to obtain multiple first newly added target detection boxes; perform tracking target object initialization processing according to each first newly added target detection box to obtain a corresponding first tracking target object; and form a corresponding first tracking target set by all the obtained first tracking target objects; and initialize the first disappeared target set to empty.

[0059] in,

[0060] The first newly added target detection box corresponds to a set of target detection box parameters, which include the center position coordinates of the detection box, the size of the detection box, and the orientation angle of the detection box. The center position coordinates of the detection box are pixel coordinates, and the size of the detection box includes the height and width.

[0061] The first tracking target set includes multiple first tracking target objects; each first tracking target object includes a first target identifier and a first object detection box; here, the first tracking target set in this embodiment of the invention is a set of objects used to cache pedestrian target objects that continuously appear before the current time, i.e., the first tracking target objects. Each first tracking target object has a first target identifier for uniquely marking the target and a first object detection box for marking the target shape of the target in the previous time.

[0062] The first set of disappeared targets includes multiple first disappeared target objects; each first disappeared target object includes a first disappearance method, a first disappearance time, a second target identifier, and a second object detection box; the first disappearance method includes disappearing from the left side of the field of view, disappearing from the right side of the field of view, disappearing from the distance of the field of view, and disappearing due to occlusion; here, the first set of disappeared targets in this embodiment of the invention is a set of objects used to cache temporarily disappeared pedestrian target objects that were previously discovered, i.e., the first disappeared target objects. Each first disappeared target object corresponds to a temporarily disappeared target, and each first disappeared target object has a first disappearance method (disappearing from the left side of the field of view, disappearing from the right side of the field of view, disappearing from the distance of the field of view, and disappearing due to occlusion) for marking the disappearance method of the target, a first disappearance time for marking the disappearance time of the target, a second target identifier for uniquely marking the target, and a second object detection box for marking the target shape at the time of disappearance;

[0063] The target object initialization process is as follows: assign a corresponding first target object to the current first newly added target detection box; assign a unique identifier to the current first target object as the corresponding first target identifier, and set the first object detection box of the current first target object as the current first newly added target detection box.

[0064] In this embodiment of the invention, the perception module of the autonomous driving system continuously acquires multiple frames of images from the perception sensor, i.e., the camera. The first frame of image acquired at the beginning is recorded as the starting frame image. The perception module also performs pedestrian target detection on each frame of image (the starting frame image of the current step and the new frame image in subsequent steps). The specific implementation of the detection is to use a visual image target detection model to perform detection and obtain multiple target detection boxes. The target detection boxes with the target type of pedestrian are taken as the pedestrian target detection results of interest in this embodiment of the invention. This embodiment of the invention does not specifically limit the visual image target detection model. Common models such as YOLO series models, R-CNN models, or Faster R-CNN models can be used, or other well-trained image-based target detection models can be used.

[0065] It should be noted that, in addition to initializing a corresponding pedestrian tracking target object, i.e., the first tracking target object, for each pedestrian target detection result obtained in the current detection, i.e., the first newly added target detection box, and thus forming the first tracking target set, the perception module of this embodiment of the invention will also add a first set of disappeared targets and initialize it.

[0066] Step 2: Take the next frame image as the corresponding new frame image; perform pedestrian target detection processing on the new frame image to obtain multiple first new target detection boxes; perform timeout target object filtering processing on the first set of disappeared targets based on a preset absolute time difference threshold; and perform target object matching processing on each first new target detection box according to the first tracking target set and the first set of disappeared targets to obtain multiple sets of first new target-target object matching pairs.

[0067] Among them, the first new target detection box corresponds to a set of target detection box parameters, which include the center position coordinates of the detection box, the size of the detection box, and the orientation angle of the detection box. The center position coordinates of the detection box are pixel coordinates, and the size of the detection box includes height, width, and / or depth.

[0068] Specifically, this includes: Step 21, taking the image obtained at the next moment as the corresponding new image;

[0069] Here, if the transition is from step 1 to the current step 21, then the "next moment" mentioned above refers to the next moment t2 corresponding to the starting moment t1; if the transition is from step 4 to the current step 21, then the "next moment" mentioned above refers to the moment t corresponding to the previous round of processing steps 2-4. i The next moment t i+1 , i≥2;

[0070] Step 22: Perform pedestrian target detection processing on the new frame image to obtain multiple first new target detection boxes;

[0071] Here, the processing method of pedestrian target detection in the current step of the perception module in this embodiment of the invention is the same as the processing method of pedestrian target detection in step 1, and will not be repeated here;

[0072] Step 23: Filter out timed-out target objects from the first set of disappeared targets based on a preset absolute time difference threshold;

[0073] Specifically, this includes: when the first set of disappeared targets is not empty, extracting the corresponding time of the current moment as the corresponding first time; calculating the absolute time difference between the first disappearance time and the first time for each first disappeared target object in the first set of disappeared targets to obtain the corresponding first absolute time difference; and deleting the first disappeared target objects in the first set of disappeared targets whose first absolute time difference exceeds the absolute time difference threshold.

[0074] Here, as mentioned above, the first disappeared target set in this embodiment of the invention is a set of objects used to cache temporarily disappeared pedestrian target objects discovered at previous times, i.e., the first disappeared target objects. This embodiment of the invention stipulates that the first disappeared target objects stored in the first disappeared target set should have a certain time limit, that is, the time interval between the disappearance time of each first disappeared target object in the first disappeared target set and the current time, i.e., the first time, should be kept within a predetermined time interval. In other words, the absolute time difference between the first disappearance time and the first time, i.e., the first absolute time difference, cannot exceed a preset time length threshold, i.e., the absolute time difference threshold. Based on this stipulation, this embodiment of the invention caches first disappeared target objects whose first absolute time difference does not exceed the absolute time difference threshold as temporarily disappeared pedestrian target objects in the first disappeared target set, and completely deletes first disappeared target objects whose first absolute time difference exceeds the absolute time difference threshold as permanently disappeared pedestrian target objects from the first disappeared target set.

[0075] Step 24: Based on the first set of tracked targets and the first set of disappeared targets, perform target object matching processing on each first new target detection box to obtain multiple sets of first new target-target object matching pairs;

[0076] Here, in each time step, in addition to the conventional target matching processing mechanism based on the first set of tracked targets, the present invention also adds a target matching processing mechanism based on the first set of disappeared targets as a supplement to the conventional target matching processing mechanism, and combines the target matching results of the two to form the final multiple sets of first new target-target object matching pairs;

[0077] Specifically, this includes: Step 241, performing pairwise matching on each first new target detection box and the first object detection box of each first tracking target object in the first tracking target set to obtain multiple sets of first detection box matching pairs; and forming a set of corresponding first new target-target object matching pairs by the first new target detection box in each set of first detection box matching pairs and the first tracking target object corresponding to the first object detection box in this matching pair; and using the first object detection box of the first tracking target object in each first new target-target object matching pair as the target detection box state quantity at the previous moment and the first new target detection box as the current target detection box observation quantity, predicting the target detection box state quantity at the current moment based on the target detection box state quantity at the previous moment and the current target detection box observation quantity according to the preset Kalman filter, and using the prediction result as the corresponding current predicted target detection box, and updating the first object detection box of the first tracking target object in the current matching pair to the current predicted target detection box;

[0078] The first detection box matching pair includes a first new target detection box and a matching first object detection box;

[0079] Here, as described above, the first tracking target set in this embodiment of the invention is a set of objects used to cache pedestrian target objects that continuously appear before the current time, i.e., the first tracking target objects; in this embodiment of the invention, in the current step 241, target matching processing is first performed on each first new target detection box based on the first tracking target set;

[0080] In this embodiment of the invention, when performing pairwise matching on each first new target detection box and the first object detection box of each first tracked target object in the first tracked target set: firstly, the first object detection box of each first tracked target object is used as the target detection box state quantity of the previous moment, and a one-step prediction is performed on the target detection box state quantity of the previous moment based on a preset Kalman filter to obtain the corresponding one-step predicted target detection box; then, each first new target detection box and each one-step predicted target detection box are pairedwise cross-intersection and union (CUI) calculations are performed; then, based on the Hungarian algorithm, the matching degree between each first new target detection box and each one-step predicted target detection box is calculated according to all the obtained pairwise cross-intersections and unions; then, the maximum value is selected from the multiple matching degrees corresponding to each first new target detection box as the maximum matching degree, and when the maximum matching degree is greater than the preset matching degree threshold, the first object detection box corresponding to the one-step predicted target detection box with the current first new target detection box is combined into a corresponding first detection box matching pair;

[0081] It should be noted that the Kalman filter used in the embodiments of the present invention is a filter that uses the target detection box as the state quantity object and the observation object and predicts the state quantity at any time based on the set state equation and state-observation transition equation. The design principle of the Kalman filter's filtering equation and the implementation principle of one-step state prediction based on the state quantity at the previous time, the implementation principle of predicting the state quantity at the current time based on the state quantity at the previous time and the observation at the current time, and the implementation principle of multi-step state prediction based on the state quantity at the historical time and the observation at the current time mentioned in the embodiments of the present invention are all publicly available technical implementations, and will not be described in detail here.

[0082] In this embodiment of the invention, after obtaining multiple sets of first detection box matching pairs, a corresponding first new target detection box and the first tracked target object are formed by each first detection box matching pair.

[0083] In this embodiment of the invention, after obtaining multiple sets of first new target-target object matching pairs, the first object detection box of the first tracked target object in each set of first new target-target object matching pairs is updated. The update method is as follows: the first object detection box of the first tracked target object in each matching pair is taken as the target detection box state quantity at the previous moment, the first new target detection box is taken as the current target detection box observation quantity, and the target detection box state quantity at the current moment is predicted based on the target detection box state quantity at the previous moment and the current target detection box observation quantity according to the preset Kalman filter, and the prediction result is used to update the first object detection box of the first tracked target object.

[0084] It should be noted that when performing pairwise matching on each new target detection box and each first object detection box of the first tracked target object, it cannot be guaranteed that a matching first object detection box will be found for each new target detection box, nor can it be guaranteed that a matching new target detection box will be found for each first object detection box. The final number of matching pairs for the first detection boxes, X, is thus determined. match It is less than or equal to the total number of the first new object detection boxes X nbbox X trackOB The minimum value between, i.e., X match ≤min(X nbbox ,X trackOB For example, it is known that step 22 above performs pedestrian target detection processing on a new frame image to obtain 5 first new target detection boxes, namely X. nbbox =5, the first set of tracking targets includes 3 first tracking target objects, namely X trackOB =3, then, the number of first detection box matching pairs obtained after pairwise matching of the 5 first new target detection boxes with the 3 first object detection boxes of the first tracked target objects is X. match ≤3, meaning that at most 3 sets of first detection box matching pairs can be obtained through the current step; at this time, if the conventional solution mentioned in the previous technical background is followed, the remaining 2 or more first new target detection boxes will be regarded as new pedestrian targets and corresponding tracking target objects will be recreated for each new pedestrian target. As can be seen from the analysis of the previous technical background, this conventional processing method may cause the premature deletion of known pedestrian targets, thereby reducing the accuracy of the corresponding target state analysis; in order to improve this conventional processing method, the embodiment of the present invention specifically adds a subsequent step 242, which continues to perform target matching processing on the remaining 2 or more first new target detection boxes based on the first disappeared target set, in order to add as few new pedestrian targets as possible and track as many known pedestrian targets as possible;

[0085] Step 242: When the first set of disappeared targets is not empty, the first new target detection box that fails to be included in any set of first detection box matching pairs is recorded as the corresponding second new target detection box; and pairwise matching is performed on each second new target detection box and the second object detection box of each first disappeared target object in the first set of disappeared targets to obtain multiple sets of corresponding second detection box matching pairs; and a corresponding first matching pair is formed by the second new target detection box in each set of second detection box matching pairs and the first disappeared target object corresponding to the second object detection box in this matching pair; and a reasonableness evaluation is performed on each of the obtained first matching pairs to obtain a matching pair. The first evaluation result is obtained; the first matching pair that is deemed unreasonable is deleted; the target object transformation process is performed on the remaining first matching pairs to obtain the corresponding first tracking target object, which is then added to the first tracking target set; the first disappeared target object in the first disappeared target set is deleted based on the remaining first matching pairs; and a set of corresponding first new target-target object matching pairs is formed by the first new target detection box and the first tracking target object corresponding to the remaining first matching pairs; the second detection box matching pair includes a second new target detection box and a matching second object detection box.

[0086] Specifically, it includes: step 2421, when the first set of disappeared targets is not empty, the first new target detection box that failed to be included in any first detection box matching pair is recorded as the corresponding second new target detection box;

[0087] Here, the second new target detection box is the first new target detection box that failed to find a matching first tracking target object in step 241 above;

[0088] Step 2422: Perform pairwise matching on each second new target detection box and the second object detection box of each first disappeared target object in the first disappeared target set to obtain multiple sets of corresponding second detection box matching pairs; and form a corresponding first matching pair by the second new target detection box in each set of second detection box matching pairs and the first disappeared target object corresponding to the second object detection box in this matching pair.

[0089] The second detection box matching pair includes a second new target detection box and a matching second object detection box;

[0090] Here, as can be seen from the preceding text, the first set of disappeared targets in this embodiment of the invention is a set of objects used to cache the temporarily disappeared pedestrian target objects discovered at previous times, i.e., the first disappeared target objects; in this embodiment of the invention, the first new target detection box, i.e. the second new target detection box, which failed to find a matching first tracking target object in the aforementioned step 241, is used to continue target matching processing based on the first set of disappeared targets in the current step;

[0091] In this embodiment of the invention, when performing pairwise matching on each second new target detection box and the second object detection box of each first disappeared target object in the first disappeared target set: firstly, the second object detection box of each first disappeared target object is used as the target detection box state quantity at the starting time, and multiple one-step predictions are continuously performed from the target detection box state quantity at the starting time based on a preset Kalman filter until a one-step predicted target detection box (denoted as the first predicted target detection box) corresponding to the current time is obtained; then, pairwise intersection and union (OCU) calculations are performed on each second new target detection box and each first predicted target detection box; then, based on the Hungarian algorithm, the matching degree between each second new target detection box and each first predicted target detection box is calculated according to all the obtained pairwise intersections and unions; then, the maximum value is selected from the multiple matching degrees corresponding to each second new target detection box as the maximum matching degree, and when the maximum matching degree is greater than the preset matching degree threshold, the second object detection box corresponding to the first predicted target detection box with the maximum matching degree is combined with the current second new target detection box to form a corresponding second detection box matching pair;

[0092] In this embodiment of the invention, after obtaining multiple sets of second detection box matching pairs, a corresponding first matching pair is formed by the second new target detection box corresponding to each second detection box matching pair and the first disappeared target object;

[0093] It should be noted that after finding the first matching pair in the current step 2422, this embodiment of the invention will not directly regard it as a reliable first new target-target object matching pair, but will conduct a further rationality evaluation on each first matching pair in step 2423.

[0094] Step 2423: Perform a rationality evaluation on each of the obtained first matching pairs to obtain the corresponding first evaluation result;

[0095] The first assessment results include reasonable and unreasonable.

[0096] Specifically, this includes: step 24231, where the second new target detection box and the first disappeared target object of the first matching pair are recorded as the corresponding first detection box and the first object; the center position coordinates of the first detection box are extracted as the corresponding first center coordinates; the first center coordinates are transformed from pixel coordinate system to world coordinate system to obtain the corresponding second center coordinates; the new frame image corresponding to the current moment is taken as the corresponding first image; and the left and right boundary regions of the first image are confirmed according to the preset left boundary width and right boundary width of the image to obtain the corresponding first left boundary region and first right boundary region;

[0097] Step 24232: When the first center coordinate is within the first left boundary area, the corresponding first appearance mode is set to appear from the left side of the field of view; when the first center coordinate is within the first right boundary area, the corresponding first appearance mode is set to appear from the right side of the field of view; when the first center coordinate is neither within the first left boundary area nor the first right boundary area, the corresponding first front-to-back distance is calculated by the front-to-back distance between the second center coordinate and the current positioning coordinate of the vehicle, and it is identified whether the first front-to-back distance has exceeded the preset front-to-back distance threshold. If it has exceeded the threshold, the corresponding first appearance mode is set to appear from a distance in the field of view; if it has not exceeded the threshold, the corresponding first appearance mode is set to appear after leaving the obstruction.

[0098] Here, in this embodiment of the invention, the appearance pattern of the pedestrian target corresponding to the second new target detection box in each first matching pair in the current visual field is estimated, and the estimation rule is as follows:

[0099] If the center pixel coordinates of the target detection box corresponding to the pedestrian target, i.e. the first center coordinates, are on the left side of the first image corresponding to the current visual field, i.e. the first center coordinates are within the first left boundary area, then it is assumed that the pedestrian target appears from the left side of the current visual field, and the estimated result of the appearance mode of the pedestrian target, i.e. the first appearance mode, is set to appear from the left side of the field of vision.

[0100] If the first center coordinates are on the right side of the first image, i.e., the first center coordinates are within the first right boundary area, then it is assumed that the pedestrian target appears from the right side of the current visual field, and the estimated result of the first appearance mode is set to appear from the right side of the field of vision.

[0101] If the first center coordinates are neither within the first left boundary region nor the first right boundary region, then the longitudinal distance between the real-world coordinates of the pedestrian target (i.e., the second center coordinates) and the vehicle (i.e., the first front-to-back distance) is further determined. If the first front-to-back distance exceeds a preset ultra-large distance threshold (i.e., the front-to-back distance threshold), then it is assumed that the pedestrian target appears from a distance in the current visual field, and the estimated result of the first appearance mode is set to appear from a distance in the visual field. If the first front-to-back distance does not exceed the aforementioned front-to-back distance threshold, then it is assumed that the pedestrian target appears from behind an occluding object in the current visual field, and the estimated result of the first appearance mode is set to appear after leaving the occlusion.

[0102] Step 24233: Identify whether the first appearance mode matches the first disappearance mode of the first object to obtain the corresponding first matching result;

[0103] The first matching result includes both a match and a non-match.

[0104] Specifically, this includes: if the first appearance method is from the left side of the field of view, then the corresponding first matching result is set to match only when the first disappearance method is from the left side of the field of view; otherwise, the corresponding first matching result is set to non-match. If the first appearance method is from the right side of the field of view, then the corresponding first matching result is set to match only when the first disappearance method is from the right side of the field of view; otherwise, the corresponding first matching result is set to non-match. If the first appearance method is from a distance in the field of view, then the corresponding first matching result is set to match only when the first disappearance method is from a distance in the field of view; otherwise, the corresponding first matching result is set to non-match. If the first appearance method is after leaving the obstruction, then the corresponding first matching result is set to match only when the first disappearance method is disappeared while being obstructed; otherwise, the corresponding first matching result is set to non-match.

[0105] Here, the first appearance mode corresponding to the second new target detection of each first matching pair in the embodiments of the present invention has four types: appearing from the left side of the field of view, appearing from the right side of the field of view, appearing from the distance of the field of view, and appearing after leaving the occlusion. The first disappearance mode of the first object of each first matching pair also has four types: disappearing from the left side of the field of view, disappearing from the right side of the field of view, disappearing from the distance of the field of view, and disappearing after being occluded.

[0106] This invention specifies that the first matching result of the current first matching pair can only be determined to be a match if the first appearance mode and the first disappearance mode corresponding to a certain first matching pair satisfy the following four matching methods:

[0107] First matching method: First appearance method = appears from the left side of the field of view, first disappearance method = disappears from the left side of the field of view, in this case the first matching result = match;

[0108] The second matching method: First appearance method = appearing from the right side of the field of view, first disappearance method = disappearing from the right side of the field of view, in this case, the first matching result = match;

[0109] The third matching method: First appearance method = appearing from a distance in the field of vision, first disappearance method = disappearing from a distance in the field of vision, in this case, the first matching result = match;

[0110] The fourth matching method: First appearance method = appears after leaving the obscuration, first disappearance method = disappears when obscured, in this case the first matching result = match;

[0111] Step 24234: When the first matching result is a match, the current time is taken as the corresponding first end time, and the time obtained by subtracting the absolute time difference threshold from the first end time is taken as the corresponding first start time; the vehicle's orientation angle at the first start time and the first end time is obtained as the corresponding first start and first end angles; the average vehicle speed of the vehicle during the time period from the first start time to the first end time is obtained as the corresponding first average vehicle speed; the absolute angle difference between the first start and the first end angles is calculated to obtain the corresponding first absolute angle difference; when the first average vehicle speed is greater than the preset low speed threshold and the first absolute angle difference is less than the preset large steering angle difference threshold, the corresponding first evaluation result is set as reasonable; when the first average vehicle speed is less than or equal to the low speed threshold or the first absolute angle difference is greater than or equal to the large steering angle difference threshold, the corresponding first evaluation result is set as unreasonable.

[0112] Here, the low-speed threshold and the large steering angle difference threshold in this embodiment of the invention are two preset low-speed threshold and large-steering angle threshold;

[0113] In this embodiment of the invention, when the first matching result of a certain first matching pair is a match, it is necessary to further identify whether the vehicle has been driving at low speed or making a large-angle turn in the recent period:

[0114] If the vehicle does not travel at low speed and does not make large-angle turns in the most recent period, i.e., the first average vehicle speed is greater than the preset low speed threshold and the first absolute angle difference is less than the preset large steering angle difference threshold, then the rationality evaluation result corresponding to the first matching pair, i.e. the first evaluation result, is set as reasonable.

[0115] If the vehicle is traveling at low speed or making a large turn in the most recent period, i.e., the first average vehicle speed is less than or equal to the low speed threshold or the first absolute angle difference is greater than or equal to the large turn angle difference threshold, then the rationality assessment result corresponding to the first matching pair, i.e. the first assessment result, is set as unreasonable.

[0116] It should be noted that the "recent period" here refers to the time period between the first start time and the current time, which is the first end time, which is (current time - absolute time difference threshold). As can be seen from step 23 above, this absolute time difference threshold is the maximum cache time limit parameter for each first disappeared target object in the first disappeared target set.

[0117] Step 24235: When the first matching result is a mismatch, set the corresponding first evaluation result to unreasonable;

[0118] Here, in this embodiment of the invention, when the first matching result of a certain first matching pair is not a match, the reasonableness evaluation result corresponding to the first matching pair, i.e. the first evaluation result, is directly set to unreasonable.

[0119] Step 2424: Delete the first matching pair whose first evaluation result is unreasonable;

[0120] Here, after deleting all unreasonable first matching pairs in the current step, the remaining first matching pairs mentioned in subsequent steps are all reasonable first matching pairs for evaluation.

[0121] Step 2425: Perform target object transformation processing on the remaining first matching pairs to obtain the corresponding first target objects and add them to the first target object set;

[0122] Specifically, this includes: creating a new corresponding first tracking target object for the current first matching pair; setting the first target identifier of the newly created first tracking target object as the second target identifier of the first disappeared target object of the current first matching pair; using the second object detection box of the first disappeared target object of the current first matching pair as the historical target detection box state quantity and the second new target detection box of the current first matching pair as the current target detection box observation quantity, predicting the target detection box state quantity at the current moment based on the historical target detection box state quantity and the current target detection box observation quantity using a preset Kalman filter, and using the prediction result as the corresponding current predicted target detection box; setting the first object detection box of the newly created first tracking target object as the current predicted target detection box; and adding the completed first tracking target object to the first tracking target set.

[0123] Here, the remaining first matching pairs are reasonable matching pairs obtained through steps 2421-2424. The current step is to move the pedestrian target objects corresponding to these reasonable matching pairs back from the first set of disappeared targets to the first set of tracked targets.

[0124] Step 2426: Delete the corresponding first disappeared target objects in the first disappeared target set according to the remaining first matching pairs;

[0125] Specifically, this involves removing the first disappeared target object from the first disappeared target set for each first matching pair.

[0126] Step 2427: The remaining first matching pairs are composed of the first new target detection boxes and the first tracked target objects, forming a set of corresponding first new target-target object matching pairs;

[0127] Step 243: Output all the first new target-target object matching pairs obtained.

[0128] Here, all the first new target-target object matching pairs output in step 243 are the new target-target object matching pair sets obtained by combining the first tracking target set and the first disappeared target set for target matching in this embodiment of the invention.

[0129] Step 3: Record each first tracking target object in the first tracking target set that failed to be included in any first new target-target object matching pair as the corresponding second tracking target object; and perform the disappearance target object conversion process on each second tracking target object to obtain the corresponding first disappearance target object and add it to the first disappearance target set; and delete each second tracking target object from the first tracking target set.

[0130] Here, the second tracking target object is actually the pedestrian target object that appeared in the previous moment but has not appeared in the current moment; for the second tracking target object, this embodiment of the invention needs to perform two processing operations: first, add the corresponding disappeared target object to the first disappeared target set; second, delete it from the first tracking target set;

[0131] Specifically, the process of converting each second tracking target object into a disappeared target object to obtain the corresponding first disappeared target object is added to the first disappeared target set.

[0132] Step A1: Record the current second tracking target object as the corresponding historical tracking target object; and create a new first disappeared target object as the corresponding latest disappeared target object;

[0133] Step A2: Extract the center coordinates of the first object detection box of the historically tracked target object as the corresponding third center coordinates; perform coordinate transformation from pixel coordinate system to world coordinate system on the third center coordinates to obtain the corresponding fourth center coordinates; obtain the vehicle positioning coordinates of the historically tracked target object at the corresponding time as the corresponding first vehicle coordinates; calculate the front-to-back distance between the fourth center coordinates and the first vehicle coordinates to obtain the corresponding second front-to-back distance; take the starting frame image or the new frame image of the historically tracked target object at the corresponding time as the corresponding second image; and confirm the left and right boundary regions of the second image according to the preset left and right boundary widths of the image to obtain the corresponding second left boundary region and second right boundary region.

[0134] Step A3: When the third center coordinate is in the second left boundary region, set the first disappearance mode of the latest disappearing target object to disappear from the left side of the field of view; when the third center coordinate is in the second right boundary region, set the first disappearance mode of the latest disappearing target object to disappear from the right side of the field of view; when the third center coordinate is neither in the second left boundary region nor the second right boundary region, identify whether the second front-back distance exceeds the preset front-back distance threshold. If it exceeds the threshold, set the first disappearance mode of the latest disappearing target object to disappear from the distance of the field of view. If it does not exceed the threshold, set the first disappearance mode of the latest disappearing target object to disappear by occlusion.

[0135] Here, the embodiment of the present invention actually estimates the visual disappearance mode of the latest disappearing target object, i.e., the first disappearance mode, and the estimation rule is as follows:

[0136] If the center pixel coordinates of the target detection box of the target object corresponding to the latest disappeared target object, i.e. the historical tracked target object, are located on the left side of the second image corresponding to the visual field at the previous moment, i.e. the third center coordinates are within the second left boundary area, then it is assumed that the latest disappeared target object disappeared from the left side of the visual field, and the estimated result of the first disappearance method of the latest disappeared target object is set to disappear from the left side of the visual field.

[0137] If the third center coordinates are on the right side of the second image, i.e., the third center coordinates are within the second right boundary area, then it is assumed that the latest disappearing target object disappears from the right side of the visual field, and the estimation result of the first disappearance method is set to disappear from the right side of the visual field.

[0138] If the third center coordinates are neither within the second left boundary region nor the second right boundary region, then the real-world coordinates corresponding to the latest disappeared target object, i.e., the fourth center coordinates, and the longitudinal distance between the vehicle and the second front-to-back distance are further judged. If the second front-to-back distance exceeds a preset ultra-large distance threshold, i.e., the front-to-back distance threshold, then it is assumed that the latest disappeared target object disappeared from a distance in the visual field, and the estimation result of the first disappearance method is set to disappear from a distance in the visual field. If the second front-to-back distance does not exceed the aforementioned front-to-back distance threshold, then it is assumed that the latest disappeared target object disappeared from behind an occluding object in the visual field, and the estimation result of the first disappearance method is set to disappear due to occlusion.

[0139] Step A4: Set the first disappearance time of the latest disappeared target object to the corresponding time of the current moment;

[0140] Here, in this embodiment of the invention, the first disappearance time of the latest disappearing target object is set;

[0141] Step A5: Set the second target identifier of the latest disappeared target object as the first target identifier of the historically tracked target object;

[0142] Here, in this embodiment of the invention, a unique target identifier is set for the latest disappeared target object;

[0143] Step A6: Using the first object detection box of the current historical tracking target object as the target detection box state quantity of the previous moment, the target detection box state quantity of the current moment is predicted in one step based on the preset Kalman filter according to the target detection box state quantity of the previous moment, and the prediction result is used as the corresponding current predicted target detection box. The second object detection box of the latest disappeared target object is set as the current predicted target detection box.

[0144] Here, in this embodiment of the invention, the latest disappearing target object is the next moment target object corresponding to the historically tracked target object, and the second object detection box of the latest disappearing target object is the next moment target detection box corresponding to the first object detection box of the historically tracked target object. The corresponding second object detection box can be obtained by sending the first object detection box into the Kalman filter preset by the invention for one-step prediction.

[0145] Step A7: Add the latest disappeared target object that has been set up to the first set of disappeared targets.

[0146] Step 4: Take each new target detection box that fails to be included in any first new target-target object matching pair as a new first newly added target detection box; and perform target object initialization processing based on each new first newly added target detection box to obtain a corresponding first tracking target object and add it to the first tracking target set; and when the tracking target set is completed, return to step 2 to continue processing the next frame image obtained at the next time step.

[0147] Here, each new target detection box that fails to be included in any first new target-target object matching pair is actually a new target detection box that fails to find a matching object in both the first tracking target set and the first disappeared target set. In this embodiment of the invention, the pedestrian target corresponding to such a first new target detection box is regarded as a new pedestrian target discovered this time. Therefore, the same tracking target object initialization processing method as in step 1 is adopted. Based on the first new target detection box of each new pedestrian target, i.e., the first newly added target detection box, a corresponding first tracking target object is initialized for each new pedestrian target and added to the first tracking target set. After the start time, the perception module of this embodiment of the invention will repeat the above steps 2-4 at each time. Therefore, when the tracking target set is completed, the perception module will return to step 2 to continue processing the next frame image obtained at the next time.

[0148] Figure 2This is a schematic diagram of an electronic device provided in Embodiment 2 of the present invention. This electronic device can be the aforementioned terminal device or server, or it can be a terminal device or server connected to the aforementioned terminal device or server that implements the method of the embodiments of the present invention. Figure 2 As shown, the electronic device may include: a processor 301 (e.g., CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transmission and reception operations of the transceiver 303. The memory 302 may store various instructions for performing various processing functions and implementing the processing steps described in the foregoing method embodiments. Preferably, the electronic device involved in the embodiments of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The communication port 306 is used for communication between the electronic device and other peripherals.

[0149] exist Figure 2 The system bus 305 mentioned can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 2 The symbol is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk drive.

[0150] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0151] It should be noted that the embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to perform the methods and processes provided in the above embodiments.

[0152] This invention also provides a chip for executing instructions, which is used to perform the processing steps described in the foregoing method embodiments.

[0153] This invention provides a method, electronic device, and computer-readable storage medium for matching newly detected targets with previously disappeared targets. First, in addition to using a conventional set of tracking targets to track all known pedestrian targets, a new set of disappeared targets is added to cache historical target information for all temporarily disappeared pedestrian targets. Second, at each time step, besides the conventional pairwise matching of the target detection boxes of each newly detected target with each tracking target to obtain corresponding new detected target-tracking target matching pairs, a new processing mechanism is added to the conventional matching process: pairwise matching of the target detection boxes of each remaining unmatched newly detected target with each previously disappeared target to obtain corresponding new detected target-disappeared target matching pairs. Furthermore, upon obtaining the new detected target... After a target-disappeared target matching pair is established, its reasonableness is evaluated based on the appearance / disappearance type matching relationship between the two. The tracking target set can be updated based on the evaluation results. This results in a new target-target object matching pair set at the current moment, composed of all newly detected target-tracking target matching pairs plus all reasonable newly detected target-disappeared target matching pairs. Then, all tracking target objects in the current tracking target set that are not included in this new target-target object matching pair set are removed, and each removed tracking target object is added as a temporarily disappeared target object to the disappeared target set. Finally, similar to conventional processing, corresponding tracking targets are created for all newly detected targets that are not included in this new target-target object matching pair set and added to the tracking target set. This invention first associates newly detected targets and tracking targets obtained at the current moment using conventional processing methods, and then associates newly detected targets that have not formed an association with temporarily disappeared targets using a supplementary mechanism. This minimizes the addition of new pedestrian targets and maximizes the tracking of known pedestrian targets at each moment, thus avoiding the problem of premature deletion of known pedestrian targets.

[0154] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0155] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0156] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for matching newly detected targets with those that have disappeared, characterized in that, The method includes: Step 1: Take the first frame image obtained at the start time as the corresponding starting frame image; perform pedestrian target detection processing on the starting frame image to obtain multiple first newly added target detection boxes; perform tracking target object initialization processing on each first newly added target detection box to obtain a corresponding first tracking target object; and form a corresponding first tracking target set by all the obtained first tracking target objects; and initialize the first disappeared target set to empty. Step 2: Take the next frame image as the corresponding new frame image; perform pedestrian target detection processing on the new frame image to obtain multiple first new target detection boxes; perform timeout target object filtering processing on the first set of disappeared targets based on a preset absolute time difference threshold; and perform target object matching processing on each of the first new target detection boxes according to the first tracking target set and the first set of disappeared targets to obtain multiple sets of first new target-target object matching pairs. Step 3: Record each first tracking target object in the first tracking target set that fails to be included in any of the first new target-target object matching pairs as the corresponding second tracking target object; perform a disappearing target object conversion process on each second tracking target object to obtain the corresponding first disappeared target object and add it to the first disappeared target set; and delete each second tracking target object from the first tracking target set. Step 4: Each of the first new target detection boxes that fails to be included in any of the first new target-target object matching pairs is taken as a new first newly added target detection box; and a tracking target object initialization process is performed based on each new first newly added target detection box to obtain a corresponding first tracking target object, which is added to the first tracking target set; and when the tracking target set is completed, return to step 2 to continue processing the next frame image obtained at the next time step. Wherein, the first newly added target detection box and the first new target detection box each correspond to a set of target detection box parameters. The target detection box parameters include the detection box center position coordinates, the detection box size and the detection box orientation angle. The detection box center position coordinates are pixel coordinates and the detection box size includes height and width. The first tracking target object includes a first target identifier and a first object detection box; The first set of disappeared targets includes multiple first disappeared target objects; the first disappeared target object includes a first disappearance method, a first disappearance time, a second target identifier, and a second object detection box; the first disappearance method includes disappearing from the left side of the field of view, disappearing from the right side of the field of view, disappearing from the distance of the field of view, and disappearing due to occlusion; The step of performing target object matching processing on each of the first new target detection boxes based on the first tracking target set and the first disappeared target set to obtain multiple sets of first new target-target object matching pairs specifically includes: For each first new target detection box and the first object detection box of each first tracked target object in the first tracking target set, pairwise matching is performed to obtain multiple sets of first detection box matching pairs. Each first new target detection box in each set of first detection box matching pairs and the first tracked target object corresponding to the first object detection box in this matching pair form a corresponding first new target-target object matching pair. Using the first object detection box of the first tracked target object in each first new target-target object matching pair as the target detection box state quantity at the previous moment and the first new target detection box as the current target detection box observation quantity, a preset Kalman filter is used to predict the target detection box state quantity at the current moment based on the target detection box state quantity at the previous moment and the current target detection box observation quantity. The prediction result is used as the corresponding current predicted target detection box, and the first object detection box of the first tracked target object in the current matching pair is updated to the current predicted target detection box. Each first detection box matching pair includes one first new target detection box and one matched first object detection box. When the first set of disappeared targets is not empty, the first new target detection box that fails to be included in any set of first detection box matching pairs is recorded as the corresponding second new target detection box; and pairwise matching is performed on each second new target detection box and the second object detection box of each first disappeared target object in the first set of disappeared targets to obtain multiple sets of corresponding second detection box matching pairs; and the second new target detection box in each set of second detection box matching pairs and the first disappeared target object corresponding to the second object detection box in this matching pair form a corresponding first matching pair; and a reasonableness evaluation is performed on each of the obtained first matching pairs to obtain the corresponding first evaluation result; and the first The first matching pair deemed unreasonable by the evaluation result is deleted; and the corresponding first tracking target object is added to the first tracking target set based on the remaining first matching pairs after target object transformation processing; and the corresponding first disappeared target object in the first disappeared target set is deleted based on the remaining first matching pairs; and a corresponding first new target-target object matching pair is formed by the first new target detection box and the first tracking target object corresponding to the remaining first matching pairs; the second detection box matching pair includes a second new target detection box and a matching second object detection box; the first evaluation result includes reasonable and unreasonable. Output all the first new target-target object matching pairs obtained.

2. The processing method for matching newly detected targets with disappeared targets according to claim 1, characterized in that, The initialization process for the tracking target object specifically includes: Assign a corresponding first tracking target object to the current first newly added target detection box; assign a unique identifier to the current first tracking target object as the corresponding first target identifier, and set the first object detection box of the current first tracking target object as the current first newly added target detection box.

3. The processing method for matching newly detected targets with disappeared targets according to claim 1, characterized in that, The step of filtering out timed-out target objects in the first set of disappeared targets based on a preset absolute time difference threshold specifically includes: When the first set of disappeared targets is not empty, the corresponding time of the current moment is extracted as the corresponding first time; and the absolute time difference between the first disappearance time and the first time of each first disappeared target object in the first set of disappeared targets is calculated to obtain the corresponding first absolute time difference; and the first disappeared target objects in the first set of disappeared targets whose first absolute time difference exceeds the absolute time difference threshold are deleted.

4. The processing method for matching newly detected targets with already disappeared targets according to claim 1, characterized in that, The step of evaluating the reasonableness of each of the first matching pairs to obtain the corresponding first evaluation result specifically includes: The second new target detection box and the first disappeared target object of the first matching pair are recorded as the corresponding first detection box and first object; the center position coordinates of the first detection box are extracted as the corresponding first center coordinates; and the first center coordinates are transformed from pixel coordinate system to world coordinate system to obtain the corresponding second center coordinates. The new frame image corresponding to the current moment is taken as the corresponding first image; and the left and right boundary regions of the first image are confirmed according to the preset left boundary width and right boundary width of the image to obtain the corresponding first left boundary region and first right boundary region. When the first center coordinate is within the first left boundary region, the corresponding first appearance mode is set to appear from the left side of the field of view; when the first center coordinate is within the first right boundary region, the corresponding first appearance mode is set to appear from the right side of the field of view; when the first center coordinate is neither within the first left boundary region nor within the first right boundary region, the corresponding first front-to-back distance is calculated by the front-to-back distance between the second center coordinate and the current positioning coordinate of the vehicle, and it is identified whether the first front-to-back distance has exceeded the preset front-to-back distance threshold. If it has exceeded the threshold, the corresponding first appearance mode is set to appear from a distance in the field of view; if it has not exceeded the threshold, the corresponding first appearance mode is set to appear after leaving the obstruction. A first matching result is obtained by identifying whether the first appearance method matches the first disappearance method of the first object; the first matching result includes matching and non-matching. When the first matching result is a match, the current time is taken as the corresponding first end time, and the time obtained by subtracting the absolute time difference threshold from the first end time is taken as the corresponding first start time; the vehicle's orientation angle at the first start time and the first end time is obtained as the corresponding first start and first end angles; the vehicle's average speed during the time period from the first start time to the first end time is obtained as the corresponding first average speed; the absolute angle difference between the first start and the first end angles is calculated to obtain the corresponding first absolute angle difference; when the first average speed is greater than a preset low speed threshold and the first absolute angle difference is less than a preset large steering angle difference threshold, the corresponding first evaluation result is set to reasonable; when the first average speed is less than or equal to the low speed threshold or the first absolute angle difference is greater than or equal to the large steering angle difference threshold, the corresponding first evaluation result is set to unreasonable. When the first matching result is a mismatch, the corresponding first evaluation result is set to unreasonable; Specifically, the step of identifying whether the first appearance method matches the first disappearance method of the first object to obtain a corresponding first matching result includes: If the first appearance method is to appear from the left side of the field of view, then the first matching result is set to match only when the first disappearance method is to disappear from the left side of the field of view; for the other methods of the first disappearance method, the corresponding first matching result is set to non-match. If the first appearance method is from the right side of the field of view, then the first matching result is set to match only when the first disappearance method is from the right side of the field of view; otherwise, the first matching result is set to non-match. If the first appearance method is to appear from a distance in the field of vision, then the first matching result is set to match only when the first disappearance method is to disappear from a distance in the field of vision; otherwise, the first matching result is set to non-match. If the first appearance method is to appear after leaving the occlusion, then the corresponding first matching result is set to match only when the first disappearance method is to disappear while being occluded. For all other disappearance methods, the corresponding first matching result is set to non-match.

5. The processing method for matching newly detected targets with already disappeared targets according to claim 1, characterized in that, The step of adding the corresponding first tracking target object obtained by performing tracking target object transformation processing on the remaining first matching pairs to the first tracking target set specifically includes: Create a new first tracking target object for the current first matching pair; set the first target identifier of the newly created first tracking target object as the second target identifier of the first disappeared target object of the current first matching pair; use the second object detection box of the first disappeared target object of the current first matching pair as the historical target detection box state quantity, and the second new target detection box of the current first matching pair as the current target detection box observation quantity. Based on a preset Kalman filter, predict the target detection box state quantity at the current moment according to the historical target detection box state quantity and the current target detection box observation quantity, and use the prediction result as the corresponding current predicted target detection box. Set the first object detection box of the newly created first tracking target object as the current predicted target detection box. Add the first tracking target object that has been set to the first tracking target set.

6. The processing method for matching newly detected targets with disappeared targets according to claim 1, characterized in that, The step of performing the disappearance target object conversion process on each of the second tracking target objects to obtain the corresponding first disappearance target object and adding it to the first disappearance target set specifically includes: Record the current second tracking target object as the corresponding historical tracking target object; and create a new first disappeared target object as the corresponding latest disappeared target object; The coordinates of the center position of the detection box of the first object detection box of the historical tracking target object are extracted as the corresponding third center coordinates; the third center coordinates are transformed from pixel coordinate system to world coordinate system to obtain the corresponding fourth center coordinates; the vehicle positioning coordinates of the historical tracking target object at the corresponding time are obtained as the corresponding first vehicle coordinates; the front-to-back distance between the fourth center coordinates and the first vehicle coordinates is calculated to obtain the corresponding second front-to-back distance; the starting frame image or the new frame image of the historical tracking target object at the corresponding time is used as the corresponding second image; and the left and right boundary regions of the second image are confirmed according to the preset left and right boundary widths of the image to obtain the corresponding second left boundary region and second right boundary region. When the third center coordinate is in the second left boundary region, the first disappearance mode of the latest disappearing target object is set to disappear from the left side of the field of view; when the third center coordinate is in the second right boundary region, the first disappearance mode of the latest disappearing target object is set to disappear from the right side of the field of view; when the third center coordinate is neither in the second left boundary region nor in the second right boundary region, it is identified whether the second front-back distance exceeds a preset front-back distance threshold. If it exceeds the threshold, the first disappearance mode of the latest disappearing target object is set to disappear from the distance of the field of view. If it does not exceed the threshold, the first disappearance mode of the latest disappearing target object is set to disappear by occlusion. Set the first disappearance time of the latest disappeared target object to the corresponding time of the current moment; Set the second target identifier of the latest disappeared target object as the first target identifier of the historically tracked target object; The first object detection box of the historical tracking target object is taken as the target detection box state quantity of the previous moment. Based on the preset Kalman filter, the target detection box state quantity of the current moment is predicted in one step according to the target detection box state quantity of the previous moment, and the prediction result is used as the corresponding current predicted target detection box. The second object detection box of the latest disappeared target object is set as the current predicted target detection box. The newly disappeared target object, once configured, is added to the first set of disappeared targets.

7. An electronic device, characterized in that, include: Memory, processor, and transceiver; The processor is configured to be coupled to the memory, read and execute instructions in the memory to implement the steps of the method according to any one of claims 1-6; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the instructions of any one of claims 1-6.

Images