Multi-category multi-target tracking method and device and medium

Abstract

The invention discloses a multi-category multi-target tracking method and device and a medium. The method comprises the following steps: performing target detection on each frame of image in acquired multi-category multi-target video stream data, and determining the target credibility, the target frame and the target category of each target; calculating a first matching cost of each target category according to the target frame, and determining a matching success track, a matching failure track, a first high-confidence matching failure target and a low-confidence matching failure target based on the first matching cost, the target credibility and the target category; performing feature matching on the first high-confidence matching failure target and the matching failure trajectory, and determining a high-confidence target matching success trajectory and a second high-confidence matching failure target; generating a new target trajectory of the second high-confidence matching failure target; and generating a multi-category multi-target tracking trajectory based on the matching success trajectory, the high-confidence target matching success trajectory and the new target trajectory.

Description

Technical Field

[0001] The present invention relates to the field of target tracking technology, and more specifically, to a multi-category multi-target tracking method, device and medium. Background Art

[0002] Multi-object tracking (MOT) plays an important role in the field of computer vision. The main purpose of multi-object tracking is to obtain the objects of interest and their motion trajectories by analyzing continuous video frames. MOT is used in many fields, such as autonomous driving, intelligent monitoring, eliminating false identification, passenger flow statistics, etc.

[0003] According to the different categories of the tracked objects, it can be divided into single-category tracking and multi-category tracking. Common tracking generally assumes that the intersection over union (IOU) of two adjacent frames is large. However, in the case of multi-category low frame rate, the IOU of some small targets is small or even zero. Using feature matching can to some extent solve the situation where IOU does not exist, but when the target scale and shape change greatly, the features are inaccurate. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a multi-category and multi-target tracking method, device and medium.

[0005] According to one aspect of the present invention, a multi-category multi-target tracking method is provided, comprising:

[0006] Perform target detection on each frame image in the acquired multi-category and multi-target video stream data to determine the target credibility, target frame and target category of each target;

[0007] Calculate the first matching cost of each target category according to the target frame, and determine the matching success trajectory, the matching failure trajectory, the first high confidence matching failure target and the low confidence matching failure target based on the first matching cost, the target credibility and the target category;

[0008] Perform feature matching on the first high-confidence match failure target and the match failure trajectory to determine the high-confidence target match success trajectory and the second high-confidence match failure target;

[0009] Generate a new target trajectory that matches the failed target with the second highest confidence;

[0010] Based on the matching successful trajectories, high confidence target matching successful trajectories and new target trajectories, multi-category and multi-target tracking trajectories are generated.

[0011] Optionally, calculating the first matching cost of each target category according to the target box includes:

[0012] The target frame that is smaller than the preset target frame threshold is enlarged by a preset multiple to obtain a new target frame set;

[0013] Based on the new target frame set, calculate the first matching cost of each target category.

[0014] Optionally, the expansion equation of the target box is:

[0015] b left =max(b src_left -b src_width / n,0)

[0016] b top =max(b src_top -b src_height / n,0)

[0017] b width =b src_width *n

[0018] b height =b src_height *n

[0019] Where b left is the left boundary of the target box; b top is the upper boundary of the target box; b width is the width of the target box; b height is the height of the target frame; n is the preset multiple.

[0020] Optionally, the first matching cost S ciou The calculation formula is:

[0021]

[0022] Where v is the similarity of the aspect ratio, b o and b t represents the center point of the detection box and the trajectory, ρ represents the Euclidean distance between the two center points, c represents the diagonal distance of the minimum closed area that can contain both the detection box and the predicted trajectory box, and d i,class Represents the distance between different types of attributes. The same type is 1, and different types are 0. o and w t Represents the width of the detection box and trajectory box, h o and h t Represents the height of the detection box and track box.

[0023] Optionally, determining a matching success trajectory, a matching failure trajectory, a first high-confidence matching failure target, and a low-confidence matching failure target based on the first matching cost, the target credibility, and the target category includes:

[0024] Based on the credibility threshold, the target credibility and the first matching cost, a high confidence similarity distance matching matrix is ​​constructed;

[0025] The Hungarian matching algorithm is used to match targets based on a high-confidence similarity distance matching matrix to determine the first successful matching trajectory, the first failed matching trajectory, the first successful matching target, and the first high-confidence failed matching target.

[0026] Based on the confidence threshold, target confidence, first matching failure trajectory and matching cost, a low confidence similarity distance matching matrix is ​​constructed;

[0027] The Hungarian matching algorithm is used to match targets based on the low-confidence similarity distance matching matrix to determine the second matching success trajectory, the second matching failure trajectory, the second matching success target, and the low-confidence matching failure target;

[0028] Determine a successful matching trajectory according to the first successful matching trajectory and the second successful matching trajectory;

[0029] According to the second matching failure trajectory, a matching failure trajectory is determined.

[0030] Optionally, performing feature matching on the first high-confidence match failure target and the match failure trajectory to determine the high-confidence target match success trajectory and the second high-confidence match failure target includes:

[0031] Extract the appearance features and color features of the target that failed to match with the first high confidence;

[0032] Calculating a second matching cost based on the appearance feature and the color feature, and generating a similarity matrix based on the second matching cost;

[0033] The first matching failure high confidence target and the matching failure trajectory are cascade matched according to the similarity matrix to obtain the high confidence target matching success trajectory and the second matching failure high confidence target.

[0034] Optionally, the second matching cost S feature The calculation formula is:

[0035] S feature =1-d i,class (ω i,feature d i,feature +ω i,color d i,color )

[0036] In the formula, d i,color is the color pixel, η o and η t Represent the color mean of the detection target and the color mean of the track target respectively; di,feature is the cosine distance similarity of two appearance features, ω i,feature is the weight of the appearance feature, ω i,color is the weight of the color feature.

[0037] According to another aspect of the present invention, there is provided a multi-category multi-target tracking device, comprising:

[0038] A detection module is used to perform target detection on each frame image in the acquired multi-category and multi-target video stream data, and determine the target credibility, target frame and target category of each target;

[0039] A first determination module is used to calculate a first matching cost of each target category according to the target frame, and determine a matching success trajectory, a matching failure trajectory, a first high-confidence matching failure target, and a low-confidence matching failure target based on the first matching cost, the target credibility, and the target category;

[0040] A second determination module is used to perform feature matching on the first high-confidence match failure target and the match failure trajectory to determine the high-confidence target match success trajectory and the second high-confidence match failure target;

[0041] A first generation module is used to generate a new target trajectory of the target that failed to match with the second highest confidence;

[0042] The second generation module is used to generate multi-category multi-target tracking trajectories based on the successfully matched trajectories, the high-confidence target matched trajectories and the new target trajectories.

[0043] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the method described in any one of the above aspects of the present invention.

[0044] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of the above aspects of the present invention.

[0045] Therefore, the present invention performs a first matching cost match on multi-category targets, and then performs a second match by combining appearance features with color features, which can effectively match suitable targets. By adding strategies, the matching is more accurate, which solves the problem of low multi-category tracking accuracy in the prior art, achieves better tracking effect in low frame rate multi-category tracking, realizes improved tracking performance in multi-category tracking of people, vehicles, pets, etc., and enhances the generalization ability of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] A more complete understanding of exemplary embodiments of the present invention may be obtained by referring to the following drawings:

[0047] Figure 1 It is a flowchart of a multi-category multi-target tracking method provided by an exemplary embodiment of the present invention;

[0048] Figure 2 is another flow chart of a multi-category multi-target tracking method provided by an exemplary embodiment of the present invention;

[0049] Figure 3 It is a schematic structural diagram of a multi-category multi-target tracking device provided by an exemplary embodiment of the present invention;

[0050] Figure 4 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0051] Below, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described here.

[0052] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

[0053] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present invention are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0054] It should also be understood that, in the embodiments of the present invention, “plurality” may refer to two or more than two, and “at least one” may refer to one, two or more than two.

[0055] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0056] In addition, the term "and / or" in the present invention is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects before and after are in an "or" relationship.

[0057] It should also be understood that the description of the various embodiments of the present invention focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0058] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0059] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0060] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0061] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0062] Embodiments of the present invention can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0063] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0064] Exemplary Methods

[0065] Figure 1 FIG. 1 is a flow chart of a multi-category multi-target tracking method provided by an exemplary embodiment of the present invention. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the multi-category multi-target tracking method 100 includes the following steps:

[0066] Step 101, performing target detection on each frame image in the acquired multi-category multi-target video stream data, and determining the target credibility, target frame and target category of each target;

[0067] Step 102, calculating the first matching cost of each target category according to the target frame, and determining a matching success trajectory, a matching failure trajectory, a first high confidence matching failure target, and a low confidence matching failure target based on the first matching cost, the target credibility, and the target category;

[0068] Step 103, performing feature matching on the first high-confidence match failure target and the match failure trajectory, and determining the high-confidence target match success trajectory and the second high-confidence match failure target;

[0069] Step 104, generating a new target trajectory of the target that failed to match with the second highest confidence;

[0070] Step 105 : generating a multi-category multi-target tracking trajectory based on the successfully matched trajectory, the high-confidence target matched trajectory, and the new target trajectory.

[0071] Specifically, in view of the fact that in the prior art, the IOU of some small targets is small or even zero in the case of multi-category low frame rate. The use of feature matching can to a certain extent solve the situation where the IOU does not exist, but when the target scale and morphology change greatly, the feature is inaccurate. The present application proposes a multi-category multi-target tracking method, which solves the problem of low multi-category tracking accuracy in the prior art, achieves better tracking effect in low frame rate multi-category tracking, realizes the improvement of tracking performance in multi-category tracking of people, cars, pets, etc., and enhances the generalization ability of target tracking.

[0072] The present invention can be applied to road video monitoring scenarios, and the monitoring targets include pedestrians, motor vehicles, non-motor vehicles, cats, dogs and other moving targets. Figure 2 As shown, the implementation process of the method proposed in the present invention is as follows:

[0073] In step 101: the target scene contains video stream data of multiple categories of targets, where the multiple categories of targets can be set by the user. Yolov5 can be used to detect the location of the target, that is, the target box, target category, and target credibility. The Yolov5 detection algorithm is an existing well-known technology and will not be elaborated in detail in this article. For example, target detection uses Yolov5, depth_multiple is set to 0.33, width_multiple is set to 0.15, and outputs box, class, conf.

[0074] In step 102, the first matching cost of each target category is calculated. Since the tracking target involves multiple categories, the intersection of the previous and next frames of a larger target, such as a motor vehicle, is relatively large when the frame rate is low. However, for small targets such as cats and dogs, if the frame rate is low, the CIOU may disappear. Therefore, the original small target frame is first expanded by n times. For example, the calculation of expanding the small target by 2 times is as follows:

[0075] b left =max(b src_left -b src_width / 2,0)

[0076] b top =max(b src_top -b src_height / 2,0)

[0077] b width =b src_width *2

[0078] b height =b src_height *2

[0079] Therefore, by expanding the box of small targets, the matching success rate of small targets is improved. The color features combined with the appearance features can more accurately describe the features of the target, thereby improving the similarity and making the matching more accurate.

[0080] Calculate the matching cost S ciou as follows:

[0081]

[0082]

[0083] In the formula, v is used to measure the similarity of aspect ratio, b o and b t represents the center point of the detection box and the trajectory, ρ represents the Euclidean distance between the two center points, and c represents the diagonal distance of the minimum closed area that can contain both the detection box and the predicted trajectory box. i,class Represents the distance between different types of attributes. The same type is 1, and different types are 0. This parameter mainly ensures direct matching of the same category. For example, pedestrians only match pedestrians, and motor vehicles only match motor vehicles.

[0084] The IOU calculation formula is as follows:

[0085]

[0086] Where IOU represents the area of ​​the intersection of the prediction box A and the detection box B and the area of ​​the union of the prediction box and the detection box.

[0087] After calculating the similarity between n detected targets and m trajectories, a similarity distance matching matrix is ​​generated:

[0088]

[0089] Among them, the similarity distance matrix is ​​calculated twice. First, the targets are sorted according to the confidence, and the targets are divided into two categories according to the preset target confidence threshold, high confidence targets and low confidence targets. High confidence targets are matched first, and a high confidence similarity distance matching matrix is ​​generated with all trajectories. The targets are matched using the Hungarian matching algorithm to obtain the first successful matching trajectory, the first failed matching trajectory, the first successful matching target, and the first high confidence failed matching target. The low confidence target and the failed matching trajectory generate a low confidence similarity distance matching matrix, and the Hungarian matching algorithm is used to match, and the second successful matching trajectory, the second failed matching trajectory, the second successful matching target, and the low confidence failed matching target are also obtained. The low confidence target that fails to match is directly discarded and does not enter the subsequent matching.

[0090] In step 103, the appearance feature model is trained using Resnet18, which is a well-known technology and will not be elaborated in detail in this article. It outputs a 256-dimensional feature vector. The original target frame is scaled to the same size, and the pre-trained appearance feature model is used to extract the features of the successfully matched targets (used for feature matching of high-confidence targets that fail to match later) and the features of high-confidence targets that fail to match. The target is converted into HSV color space representation, 0.2 times the bounding box of the target frame is removed, and the target color mean is calculated as the color feature.

[0091] S05: Cascade matching is performed on the targets with high confidence that failed to match and the trajectories that failed to match. Cascade matching means matching the nearest trajectory first according to the time when the trajectory disappears, and then matching the farther trajectory in a loop if the nearest trajectory fails to match. Here, the similarity is also calculated, the similarity matrix is ​​generated, and the Hungarian matching algorithm is used for matching.

[0092] For small targets, the box is only enlarged during CIOU matching, and the original box is used when extracting color features and appearance features.

[0093] The second matching cost calculation S feature :

[0094] S feature =1-d i,class (ω i,feature d i,feature +ω i,color d i,color )

[0095] Where, d i,featureBy calculating the cosine distance similarity of two features; d i,color is the color pixel, convert the target area into HSV representation, calculate the mean value of the data in the target area, η o and η t They represent the color mean of the detected target and the color mean of the track target respectively.

[0096] In step 104 and step 105, for the successfully matched trajectories, the position information, color features, and appearance features are updated. For the detection frames with high credibility that failed to match, new trajectories are generated, and the detection frames with low credibility that failed to match are deleted. For the trajectories that failed to match, the disappearance time of the trajectories is updated, and the trajectories are deleted if the maximum disappearance time is exceeded.

[0097] Therefore, the present invention performs a first matching cost match on multi-category targets, and then performs a second match by combining appearance features with color features, which can effectively match suitable targets. By adding strategies, the matching is more accurate, which solves the problem of low multi-category tracking accuracy in the prior art, achieves better tracking effect in low frame rate multi-category tracking, realizes improved tracking performance in multi-category tracking of people, vehicles, pets, etc., and enhances the generalization ability of target tracking.

[0098] Exemplary Devices

[0099] Figure 3 FIG. 1 is a schematic diagram of a multi-category multi-target tracking device provided by an exemplary embodiment of the present invention. Figure 3 As shown, the device 300 includes:

[0100] The detection module 310 is used to perform target detection on each frame image in the acquired multi-category multi-target video stream data, and determine the target credibility, target frame and target category of each target;

[0101] A first determination module 320 is used to calculate a first matching cost of each target category according to the target frame, and determine a matching success trajectory, a matching failure trajectory, a first high-confidence matching failure target, and a low-confidence matching failure target based on the first matching cost, the target credibility, and the target category;

[0102] A second determination module 330 is used to perform feature matching on the first high-confidence match failure target and the match failure trajectory to determine the high-confidence target match success trajectory and the second high-confidence match failure target;

[0103] A first generating module 340, configured to generate a new target trajectory of a target that failed to match with a second high confidence;

[0104] The second generating module 350 is used to generate multi-category multi-target tracking trajectories based on the successfully matched trajectories, the high-confidence target matched trajectories and the new target trajectories.

[0105] Optionally, the detection module 310 calculates the first matching cost of each target category according to the target box, including:

[0106] The expansion submodule is used to expand the target frame that is smaller than the preset target frame threshold by a preset multiple to obtain a new target frame set;

[0107] The calculation submodule is used to calculate the first matching cost of each target category according to the new target frame set.

[0108] Optionally, the expansion equation of the target box is:

[0109] b left =max(b src_left -b src_width / n,0)

[0110] b top =max(b src_top -b src_height / n,0)

[0111] b width =b src_width *n

[0112] b height =b src_height *n

[0113] Where b left is the left boundary of the target box; b top is the upper boundary of the target box; b width is the width of the target box; b height is the height of the target frame; n is the preset multiple.

[0114] Optionally, the first matching cost S ciou The calculation formula is:

[0115]

[0116] Where v is the similarity of the aspect ratio, b o and b t represents the center point of the detection box and the trajectory, ρ represents the Euclidean distance between the two center points, c represents the diagonal distance of the minimum closed area that can contain both the detection box and the predicted trajectory box, and d i,class Represents the distance between different types of attributes. The same type is 1, and different types are 0. o and w t Represents the width of the detection box and trajectory box, h oand h t Represents the height of the detection box and track box.

[0117] Optionally, the detection module 310 determines the matching success trajectory, the matching failure trajectory, the first high-confidence matching failure target, and the low-confidence matching failure target based on the first matching cost, the target credibility, and the target category, including:

[0118] A first construction submodule, used to construct a high-confidence similarity distance matching matrix based on a confidence threshold, a target confidence, and a first matching cost;

[0119] A first matching submodule is used to perform target matching based on a high-confidence similarity distance matching matrix using a Hungarian matching algorithm, and determine a first matching success trajectory, a first matching failure trajectory, a first matching success target, and a first high-confidence matching failure target;

[0120] The second construction submodule is used to construct a low confidence similarity distance matching matrix based on the confidence threshold, the target confidence, the first matching failure trajectory and the matching cost;

[0121] A second matching submodule is used to perform target matching based on a low-confidence similarity distance matching matrix using a Hungarian matching algorithm, and determine a second matching success trajectory, a second matching failure trajectory, a second matching success target, and a low-confidence matching failure target;

[0122] A first determination submodule, configured to determine a successful matching trajectory according to the first successful matching trajectory and the second successful matching trajectory;

[0123] The second determination submodule is used to determine the matching failure trajectory according to the second matching failure trajectory.

[0124] Optionally, the second determining module 330 includes:

[0125] An extraction submodule, used to extract the appearance features and color features of the first high confidence matching failure target;

[0126] A generating submodule, used for calculating a second matching cost based on the appearance feature and the color feature, and generating a similarity matrix based on the second matching cost;

[0127] The third matching submodule is used to perform cascade matching on the first matching failure high confidence target and the matching failure trajectory according to the similarity matrix to obtain the high confidence target matching success trajectory and the second matching failure high confidence target.

[0128] Optionally, the second matching cost S feature The calculation formula is:

[0129] S feature =1-di,class (ω i,feature d i,feature +ω i,color d i,color )

[0130] In the formula, d i,color is the color pixel, η o and η t Represent the color mean of the detection target and the color mean of the track target respectively; d i,feature is the cosine distance similarity of two appearance features, ω i,feature is the weight of the appearance feature, ω i,color is the weight of the color feature.

[0131] Exemplary Electronic Devices

[0132] Figure 4 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. Figure 4 As shown, the electronic device 40 includes one or more processors 41 and a memory 42 .

[0133] The processor 41 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0134] The memory 42 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 41 may run the program instructions to implement the methods of the software programs of the various embodiments of the present invention described above and / or other desired functions. In one example, the electronic device may also include: an input device 43 and an output device 44, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0135] In addition, the input device 43 may also include, for example, a keyboard, a mouse, etc.

[0136] The output device 44 can output various information to the outside, and can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto.

[0137] Of course, to simplify, Figure 4 Only some of the components related to the present invention in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application conditions.

[0138] Exemplary computer program products and computer-readable storage media

[0139] In addition to the above-mentioned methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present invention described in the above-mentioned "Exemplary Method" section of this specification.

[0140] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the embodiments of the present invention, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0141] In addition, an embodiment of the present invention may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present invention described in the above “Exemplary Method” section of this specification.

[0142] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, system or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0143] The basic principle of the present invention is described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details disclosed above are only for the purpose of illustration and facilitation of understanding, rather than limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0144] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0145] The block diagrams of the devices, systems, equipment, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, systems, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0146] The method and system of the present invention may be implemented in many ways. For example, the method and system of the present invention may be implemented by software, hardware, firmware or any combination of software, hardware, firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present invention are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present invention may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present invention. Thus, the present invention also covers a recording medium storing a program for executing the method according to the present invention.

[0147] It should also be noted that in the system, device and method of the present invention, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. The above description of the disclosed aspects is provided to enable any technician in the field to make or use the present invention. Various modifications to these aspects are very obvious to those skilled in the art, and the general principles defined here can be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown here, but in accordance with the widest range consistent with the principles and novel features disclosed here.

[0148] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present invention to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A multi-category multi-target tracking method, characterized in that: include: Perform target detection on each frame image in the acquired multi-category and multi-target video stream data to determine the target credibility, target frame and target category of each target; Calculating a first matching cost of each target category according to the target frame, and determining a matching success trajectory, a matching failure trajectory, a first high-confidence matching failure target, and a low-confidence matching failure target based on the first matching cost, the target credibility, and the target category; Performing feature matching on the first high-confidence match failure target and the match failure trajectory to determine a high-confidence target match success trajectory and a second high-confidence match failure target; generating a new target trajectory of the second high confidence match failure target; Based on the successful matching trajectory, the high-confidence target successful matching trajectory and the new target trajectory, a multi-category multi-target tracking trajectory is generated.

2. The method according to claim 1, characterized in that Calculating a first matching cost of each target category according to the target frame includes: Enlarging the target frames that are smaller than a preset target frame threshold by a preset multiple to obtain a new target frame set; The first matching cost of each target category is calculated according to the new target frame set.

3. The method according to claim 2, characterized in that The expansion equation of the target frame is: b left =max(b src_left -b src_width / n,0) b top =max(b src_top -b src_height / n,0) b width =b src_width *n b height =b src_height *n Where b left is the left boundary of the target box; b top is the upper boundary of the target box; b width is the width of the target box; b height is the height of the target frame; n is the preset multiple.

4. The method according to claim 3, characterized in that The first matching cost S ciou The calculation formula is: Where v is the similarity of the aspect ratio, b o and b t represents the center point of the detection box and the trajectory, ρ represents the Euclidean distance between the two center points, c represents the diagonal distance of the minimum closed area that can contain both the detection box and the predicted trajectory box, and d i,class Represents the distance between different types of attributes. The same type is 1, and different types are 0. o and w t Represents the width of the detection box and trajectory box, h o and h t Represents the height of the detection box and track box.

5. The method according to claim 1, characterized in that Determining a matching success trajectory, a matching failure trajectory, a first high-confidence matching failure target, and a low-confidence matching failure target based on the first matching cost, the target credibility, and the target category, including: Based on the credibility threshold, the target credibility and the first matching cost, construct a high confidence similarity distance matching matrix; Using the Hungarian matching algorithm to perform target matching based on the high-confidence similarity distance matching matrix, determine a first matching successful trajectory, a first matching failed trajectory, a first matching successful target, and the first high-confidence matching failed target; Constructing a low-confidence similarity distance matching matrix based on the confidence threshold, the target confidence, the first matching failure trajectory, and the matching cost; Using the Hungarian matching algorithm to perform target matching based on a low-confidence similarity distance matching matrix, determine a second matching success trajectory, a second matching failure trajectory, a second matching success target, and the low-confidence matching failure target; Determining the successful matching trajectory according to the first successful matching trajectory and the second successful matching trajectory; The matching failure trajectory is determined according to the second matching failure trajectory.

6. The method according to claim 1, characterized in that Performing feature matching on the first high-confidence match failure target and the match failure trajectory to determine a high-confidence target match success trajectory and a second high-confidence match failure target includes: Extracting appearance features and color features of the first high-confidence matching failure target; Calculating a second matching cost based on the appearance feature and the color feature, and generating a similarity matrix based on the second matching cost; The first matching failure high-confidence target and the matching failure trajectory are cascade matched according to the similarity matrix to obtain the high-confidence target matching success trajectory and the second matching failure high-confidence target.

7. The method according to claim 6, characterized in that The second matching cost S feature The calculation formula is: S feature =1-d i,class (oh i,feature d i,feature +oh i,color d i,color ) In the formula, d i,color is the color pixel, η o and η t Represent the color mean of the detection target and the color mean of the track target respectively; d i,feature is the cosine distance similarity of two appearance features, ω i,feature is the weight of the appearance feature, ω i,color is the weight of the color feature.

8. A multi-category multi-target tracking device, characterized in that: include: A detection module is used to perform target detection on each frame image in the acquired multi-category and multi-target video stream data, and determine the target credibility, target frame and target category of each target; A first determination module is used to calculate a first matching cost of each target category according to the target frame, and determine a matching success trajectory, a matching failure trajectory, a first high-confidence matching failure target, and a low-confidence matching failure target based on the first matching cost, the target credibility, and the target category; A second determination module is used to perform feature matching on the first high-confidence match failure target and the match failure trajectory to determine a high-confidence target match success trajectory and a second high-confidence match failure target; A first generating module, used to generate a new target trajectory of the second high confidence matching failed target; The second generating module is used to generate a multi-category multi-target tracking trajectory based on the successful matching trajectory, the high-confidence target successful matching trajectory and the new target trajectory.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1 to 7.

Images