Multi-target tracking and identification method, device, electronic device and medium

Abstract

The present invention relates to a multi-target tracking and recognition method, device, electronic device and medium, and relates to the field of recognition algorithms. The method includes continuously intercepting adjacent first target images and second target images from a video object to be processed, obtaining a first detection frame of the first target image and a second detection frame of the second target image, selecting a matching second detection frame from the second target image for each first detection frame in the first target image, and using the physical position matching score and feature matching score of the first detection frame and the matching second detection frame to accurately measure whether the targets framed by the two are the same, thereby achieving tracking of the target framed by the first detection frame in the second target image; using the second target image as a new first target image, and using the next frame image of the second target image as a new second target image, and looping the above processing process, thereby achieving simultaneous and accurate tracking of multiple targets in the video object.

Description

Technical Field

[0001] The present invention relates to the field of recognition algorithms, and in particular to a multi-target tracking and recognition method, device, electronic equipment and medium. Background Art

[0002] Monitoring platforms often require analysis of certain targets. For example, in some scenarios, human behavior needs to be monitored, including whether people in the kitchen wear hats, whether they wear masks during epidemic prevention, and whether they smoke in public places. Humans observing videos for violations in real time would be time-consuming and labor-intensive. This is why intelligent monitoring has emerged, introducing machine analysis to reduce human effort. Humans only need to review images identified as violations by machine analysis.

[0003] However, these intelligent monitoring methods typically perform static analysis on a single target within a single frame and are unable to track the target's dynamic process. Therefore, how to achieve accurate dynamic tracking of multiple targets simultaneously in a video has become a technical challenge that needs to be addressed within the industry. Summary of the Invention

[0004] The technical problem to be solved by the present invention is: how to design a multi-target tracking and recognition method that can achieve accurate dynamic tracking of multiple targets in a video at the same time.

[0005] To solve the above problems, in the first aspect, the present invention proposes a multi-target tracking and recognition method, which includes: intercepting a video frame image from a video object to be processed as a first target image; intercepting the next frame image of the first target image from the video object to be processed as a second target image; inputting the first target image and the second target image into a preset target detection model respectively to obtain a first detection frame group of the first target image and a second detection frame group of the second target image, the first detection frame group including at least one first detection frame, and the second detection frame group including at least one second detection frame; selecting a first detection frame from the first detection frame group as the first target detection frame; selecting a second detection frame matching the first target detection frame from the second detection frame group as the second target detection frame; obtaining geometric information between the first target detection frame and the second target detection frame, and obtaining a physical position matching score between the first detection frame and the second detection frame based on the geometric information.

[0006] According to a preset feature extraction algorithm, a first image feature vector of the image defined by the first target detection frame and a second image feature vector of the image defined by the second target detection frame are respectively obtained; based on the first image feature vector and the second image feature vector, a feature degree matching score between the image defined by the first target detection frame and the image defined by the second target detection frame is obtained; based on the physical position matching score and the feature degree matching score, a similarity score between the image defined by the first target detection frame and the image defined by the second target detection frame is obtained.

[0007] Determine whether the similarity score is greater than a preset score threshold; if the similarity score is greater than the score threshold, determine that the target framed by the first target detection frame is the same as the target framed by the second target detection frame; determine whether there is an unselected first detection frame in the first detection frame group; if there is an unselected first detection frame in the first detection frame group, proceed to the step of selecting a first detection frame from the first detection frame group as the first target detection frame; if there is no unselected first detection frame in the first detection frame group, use the second target image as the new first target image, and proceed to the step of capturing the next frame image of the first target image from the video object to be processed as the second target image.

[0008] A further technical solution is that obtaining the geometric information between the first target detection frame and the second target detection frame includes: obtaining the intersection ratio, center point distance ratio, width ratio and height ratio between the first target detection frame and the second target detection frame.

[0009] A further technical solution is that obtaining a physical position matching score between the first detection frame and the second detection frame according to the geometric information includes:

[0010] According to the formula P score =IOU-((C dis / C)+(w dis / W)+(h dis / H)) calculates a physical position matching score between the first detection frame and the second detection frame.

[0011] Among them, IOU is the intersection-over-union ratio, C dis / C is the center point distance ratio, w dis / W is the width ratio, h dis / H is the height ratio, P score Assign a score to the physical location match.

[0012] A further technical solution is that obtaining a feature matching score between the image defined by the first object detection frame and the image defined by the second object detection frame based on the first image feature vector and the second image feature vector includes:

[0013] According to the formula Calculate a feature matching score between the image defined by the first object detection frame and the image defined by the second object detection frame.

[0014] Among them, (x1,x2,...,x n ) is the first eigenvector, (y1,y2,...,y n ) is the second eigenvector.

[0015] Its further technical solution is that, based on the physical position matching score and the feature matching score, the similarity score between the image defined by the first target detection frame and the image defined by the second target detection frame is obtained, including: multiplying the physical position matching score by a preset first weighting coefficient to obtain a physical position weighted score; multiplying the feature matching score by a preset second weighting coefficient to obtain a feature weighted score; and adding the physical position weighted score to the feature weighted score to obtain the similarity score.

[0016] Its further technical solution is that the selecting of a second detection frame matching the first target detection frame from the second detection frame group as the second target detection frame includes: selecting at least one sub-detection frame from the second detection frame group, wherein the ratio of the area of ​​the sub-detection frame to the area of ​​the first target detection frame is greater than a preset area ratio threshold; and setting the sub-detection frame as the second target detection frame.

[0017] Its further technical solution is to determine whether the target framed by the first target detection frame is a preset violation target; if the target framed by the first target detection frame is a preset violation target, obtain all video frame images containing the target from the video object to be processed to obtain a violation image set; obtain the violation feature similarity of the video frame images in the violation image set respectively, and the violation feature similarity is the numerical value of the similarity between the features of the video frame images in the violation image set and the features of the preset violation standard image; select the video frame image with the largest violation feature similarity from the violation image set as the violation image, and upload the violation image to a preset monitoring terminal.

[0018] In a second aspect, the present invention provides a multi-target tracking and identification device, which includes a unit for executing the method described in the first aspect.

[0019] In a third aspect, the present invention proposes an electronic device comprising: a memory for storing a computer program; and a processor for implementing the steps of the method described in the first aspect when executing the program stored in the memory.

[0020] In a fourth aspect, the present invention proposes a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, it can implement the method described in the first aspect.

[0021] The beneficial effects of the present invention are:

[0022] In the present invention, adjacent first target images and second target images are continuously captured from the video object to be processed to obtain a first detection frame of the first target image and a second detection frame of the second target image. For each first detection frame in the first target image, a matching second detection frame is selected from the second target image, and the physical position matching score and feature matching score of the first detection frame and the matching second detection frame are used to accurately measure whether the targets framed by the two are the same, thereby achieving tracking of the target framed by the first detection frame in the second target image; thereafter, the second target image is used as a new first target image, and the next frame image of the second target image is used as a new second target image, and the above processing process is repeated, thereby achieving accurate tracking of multiple targets in the video object at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0025] Figure 1 A flowchart of a multi-target tracking and identification method provided by an embodiment of the present invention.

[0026] Figure 2 Another flowchart of a multi-target tracking and identification method provided by an embodiment of the present invention.

[0027] Figure 3 A block diagram of a multi-target tracking and identification device provided by an embodiment of the present invention.

[0028] Figure 4 Another block diagram of a multi-target tracking and identification device provided by an embodiment of the present invention.

[0029] Figure 5 A block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0031] It will be understood that when used in this specification and the appended claims, the terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or other features, integers, steps, operations, elements, components and / or collections thereof.

[0032] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used in the specification and appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0033] It should be further understood that the term "and / or" used in the present description and the appended claims refers to one or any combination and all possible combinations of the associated listed items, and includes these combinations.

[0034] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0035] Example 1

[0036] See also Figure 1-Figure 5 , Figure 1A flowchart of a multi-target tracking and identification method provided by an embodiment of the present invention. The embodiment of the present invention proposes a multi-target tracking and identification method, which can track the target framed by a first detection frame in a second target image, and track the dynamic process of the target; the first target image and the second target image of the method are not fixed, but are continuously iteratively updated as the processing process cycles. The multi-target tracking and identification method can be used for a camera or a surveillance camera, and the position of the camera or surveillance camera can be a fixed position. The method includes:

[0037] S101 , capturing a video frame image from a video object to be processed as a first target image.

[0038] In the above solution, the video object to be processed may be a real-time surveillance video stream or a stored video file. The step of intercepting a video frame image from the video object to be processed as the first target image may be intercepting a first frame of the video frame image from the video object to be processed as the first target image.

[0039] S102: intercepting the next frame image of the first target image from the video object to be processed as the second target image.

[0040] The first target image may be a video frame image of the first frame of the video to be processed, and the second target image may be a video frame image of the second frame of the video to be processed. In other words, the multi-target tracking and recognition method may start from the first frame of the entire video object to be processed. In one embodiment, the video to be processed is continuously captured to obtain two target images, and then adjacent video frame images are extracted from the two target images to obtain the first target image and the second target image.

[0041] From a temporal perspective, the frame corresponding to the first target image is adjacent to the frame corresponding to the second target image; from an image position perspective, the first target image is adjacent to the second target image, that is, from the user's perspective, the first target image is located in the front position, and the second target image is located in the back position and is close to the second target image.

[0042] In addition, the first target image and the second target image are variable. Specifically, the processing process of this scheme is based on the first target image and the second target image, and this embodiment will be continuously looped. After one loop ends, the second target image will be used as the new first target image, and the next frame image of the second target image will be used as the new second target image to loop the processing process of this embodiment. Therefore, the first target image and the second target image are not fixed, but are continuously iterated and updated as the processing process loops.

[0043] S103: Input the first target image and the second target image into a preset target detection model respectively to obtain a first detection frame group of the first target image and a second detection frame group of the second target image.

[0044] The first detection frame group includes at least one first detection frame, and the second detection frame group includes at least one second detection frame. The first detection frame can be a rectangular frame, and the second detection frame can be a rectangular frame. The number of the first detection frame group can be one or more, specifically determined by the recognition results of the object detection model. The number of the second detection frame group can also be one or more, specifically determined by the recognition results of the object detection model.

[0045] In a specific implementation, the target detection model is pre-trained so that it can obtain a first detection frame set after recognizing a first target image, and can also obtain a second detection frame set after recognizing a second target image. In this embodiment, the target detection model specifically uses the YOLOX model. Those skilled in the art may also use other types of target detection models, and this invention is not specifically limited to this.

[0046] S104: Select a first detection frame from the first detection frame group as a first target detection frame.

[0047] The first target detection frame is the frame that will be tracked during the tracking process. Selecting the first detection frame from the first detection frame group as the first target detection frame means selecting any one or more first detection frames as the target. All subsequent steps will revolve around the selected target, and subsequent steps will continue to track this target.

[0048] The multi-target tracking and recognition method described in this solution will eventually achieve the tracking of multiple targets. During the implementation process, according to the processing capability of the device, one first detection frame can be selected as the first target detection frame to execute the processing of this embodiment, or multiple first detection frames can be selected as the first target detection frames, and the processing of this embodiment can be executed for multiple first target detection frames at the same time. After completing the tracking of the above-mentioned first target detection frame, it is also necessary to determine whether there are any unselected first detection frames in the first detection frame group to ensure that there are no omissions in the first detection frame group; next, after the information in the first target image is processed, the next frame image of the first target image is intercepted from the video object to be processed as the second target image, and the second target image is continued to be tracked.

[0049] S105: Select a second detection frame matching the first target detection frame from the second detection frame group as a second target detection frame.

[0050] The step of selecting a second detection frame from the second detection frame group that matches the first target detection frame as the second target detection frame specifically includes selecting at least one sub-detection frame from the second detection frame group, wherein the ratio of parameters of the sub-detection frame to parameters of the first target detection frame is greater than a preset parameter ratio threshold; and setting the sub-detection frame as the second target detection frame. In the above steps, the parameter may be area, and the parameter may also be selected as required. The process of selecting a sub-detection frame based on the preset parameter ratio threshold is equivalent to performing a preliminary screening of the second detection frame group, selecting all frames that meet the conditions, one or more in number, as the second target detection frames.

[0051] In one embodiment, selecting a second detection frame from the second detection frame group that matches the first target detection frame as the second target detection frame specifically includes selecting at least one sub-detection frame from the second detection frame group, wherein the ratio of the area of ​​the sub-detection frame to the area of ​​the first target detection frame is greater than a preset area ratio threshold, which can be set to any value between 95% and 100%, that is, it can be set based on the engineer's experience, including a range of 80%-100%. In this way, the areas of all second target detection frames are close to the area of ​​the first target detection frame. Regardless of whether there are one or more frames that meet the conditions, the objects contained therein may be the same as the objects in the first target detection frame. Therefore, to avoid omissions, the one or more frames are defined as the second target detection frame.

[0052] S106: Obtain geometric information between the first target detection frame and the second target detection frame, and obtain a physical position matching score between the first detection frame and the second detection frame based on the geometric information.

[0053] Among them, obtaining the geometric information between the first target detection frame and the second target detection frame specifically includes obtaining the intersection-and-union ratio, center point distance ratio, width ratio and height ratio between the first target detection frame and the second target detection frame; the intersection-and-union ratio is the ratio of the intersection and union of the two frames, and the center point distance can be the ratio between the center point coordinate distance of the two frames and the diagonal distance of the minimum closure area of ​​the two frames.

[0054] The physical position matching score between the first detection frame and the second detection frame is obtained according to the geometric information, that is, the geometric information is obtained by calculating the intersection-over-union ratio, the center point distance ratio, the width ratio, and the height ratio. Specifically, the geometric information includes:

[0055] According to the formula P score =IOU-((C dis / C)+(w dis / W)+(hdis / H)) calculates a physical position matching score between the first detection frame and the second detection frame.

[0056] Among them, IOU is the intersection-over-union ratio, C dis / C is the center point distance ratio, w dis / W is the width ratio, h dis / H is the height ratio, P score Assign a score to the physical location match.

[0057] S107 , obtaining, according to a preset feature extraction algorithm, a first image feature vector of the image defined by the first object detection frame and a second image feature vector of the image defined by the second object detection frame.

[0058] Among them, the image defined by the first target detection frame is different from the first target detection frame, and specifically refers to the image within the frame; the image defined by the second target detection frame is different from the second target detection frame, and specifically also refers to the image within the frame; the preset feature extraction algorithm is well known to those skilled in the art, and specific feature extraction algorithms may include: HOG, LBP or Harr feature extraction algorithms.

[0059] S108 : Obtain a feature degree matching score between the image defined by the first object detection frame and the image defined by the second object detection frame based on the first image feature vector and the second image feature vector.

[0060] The above steps are to obtain the similarity of the image features in the frame. In the process of obtaining the feature matching score, the first image feature vector A with a fixed dimension can be extracted by a traditional image feature extraction algorithm. feature (x1,x2,...,x n ) and a second image feature vector B of fixed dimension feature (y1,y2,...,y n ), using cosine similarity to calculate the similarity between the first image feature vector and the second image feature vector, specifically including:

[0061] According to the formula Calculate a feature matching score between the image defined by the first object detection frame and the image defined by the second object detection frame.

[0062] Among them, x1,x2,...,x n is the coordinate of the first eigenvector, y1,y2,...,y n is the coordinate of the second eigenvector. (x1,x2,...,x n ) is the first eigenvector, (y1,y2,...,yn ) is the second eigenvector.

[0063] S109: Obtain a similarity score between the image defined by the first object detection frame and the image defined by the second object detection frame according to the physical position matching score and the feature matching score.

[0064] Among them, the process of obtaining the similarity score includes: multiplying the physical location matching score by a preset first weighting coefficient to obtain a physical location weighted score; multiplying the feature matching score by a preset second weighting coefficient to obtain a feature weighted score; and adding the physical location weighted score to the feature weighted score to obtain the similarity score.

[0065] The calculation process for obtaining the similarity score includes:

[0066] Total score =αP score +βF score ;

[0067] Among them, α and β represent the weight values ​​of position matching and feature similarity, and the position matching is P score , the feature similarity is F score , the similarity score is Total score .

[0068] S110: Determine whether the similarity score is greater than a preset score threshold.

[0069] Among them, the preset score threshold can be set by the engineer based on his own experience. The preset score threshold is used to measure whether the target identified for the first time and the target identified for the second time are the same target, eliminating the manpower cost consumption caused by manual tracking and identification.

[0070] S111: If the similarity score is greater than the score threshold, determine that the target framed by the first target detection frame is the same as the target framed by the second target detection frame.

[0071] The target framed by the first target detection frame mainly includes three parts, namely the first target detection frame, the image of the first target detection frame, and the target of the first target detection frame; the target framed by the second target detection frame mainly includes three parts, namely the second target detection frame, the image of the second target detection frame, and the target of the second target detection frame. The first target detection frame and the second target detection frame belong to different video frame images and are therefore obviously different; the image of the first target detection frame and the image of the second target detection frame have certain similarities due to the gap in the frame capture time, but this similarity cannot objectively determine whether the targets in the two images are the same; the target of the first target detection frame and the target of the second target detection frame should ideally be the same target, or the same object. The purpose of the above steps is to objectively determine whether the target of the first target detection frame and the target of the second target detection frame are the same as in the ideal state by judging the relationship between the similarity score and the score threshold.

[0072] S112: Determine whether there is any unselected first detection frame in the first detection frame group.

[0073] After selecting a second detection frame that matches the first target detection frame from the second detection frame group as the second target detection frame, it is possible to determine whether there are any unselected first detection frames in the first detection frame group. This is to ensure that all first detection frames in the first detection frame group are tracked to ensure that no first detection frames are missed before starting tracking of the second target image; otherwise, unprocessed first detection frames will remain in the first detection frame group, which will reduce the accuracy of the entire multi-target tracking and recognition method, hindering the technical effects of the present invention.

[0074] After step S112, if there is an unselected first detection frame in the first detection frame group, the process proceeds to step S104, i.e., the step of selecting a first detection frame from the first detection frame group as the first target detection frame. The process proceeds to the step of selecting a first detection frame from the first detection frame group as the first target detection frame, i.e., selecting a first detection frame that has not been selected, i.e., selecting a detection frame that has not been selected in the first detection frame group. This is because all first detection frames in the entire first detection frame group must be selected as first target detection frames in order to achieve the optimal effect of the multi-target tracking and recognition method.

[0075] After the above step S112, if there is no unselected first detection frame in the first detection frame group, step S113 is executed to use the second target image as a new first target image.

[0076] After the second target image is used as the new first target image, the process proceeds to step S102, which involves capturing the next frame of the first target image from the video object to be processed as the second target image. This is so that after all first detection frames in the first detection frame group have been traversed and tracked, the tracking work for the first detection frame group is complete, which is equivalent to the tracking work for the first target image. At this point, the focus of the tracking work shifts from the first target image to the second target image, meaning that the second target image is used as the new first target image.

[0077] In the technical solution of S101-S113 above, if there is no unselected first detection frame in the first detection frame group, the second target image is used as the new first target image, and the process proceeds to the step of intercepting the next frame of the first target image from the video object to be processed as the second target image, that is, continuing to intercept the next frame of the second target image as the third target image. Since the relationship between the third target image and the second target image is equivalent to the relationship between the second target image and the first target image, the entire method can be continuously advanced, and not only can it be determined whether the target framed by the first target detection frame is the same as the target framed by the second target detection frame, but it can also be determined whether the target framed by the second target detection frame is the same as the target framed by the third target detection frame, and further it can be determined whether the target framed by the third target detection frame is the same as the target framed by the fourth target detection frame, that is, the solution achieves the function of tracking and recognition. Among them, the first target image and the second target image are continuously captured to obtain the first target detection frame and the second target detection frame, and the relationship between the two target detection frames is accurately measured using the physical position matching score and the feature matching score. Finally, it is judged whether the measurement result meets the expected score threshold. Here, the physical position matching score and the feature matching score are more detailed and more accurate than the existing technology, which achieves the technical effect of reducing the probability of misidentification; when it meets expectations, it is determined that the target framed by the first target detection frame and the target framed by the second target detection frame are the same. Here, the determination of being the same means that there is no need to report repeatedly, which achieves the ability to accurately track multiple targets at the same time.

[0078] The beneficial effect of the above scheme is that the first target image and the second target image are continuously captured, the first target detection frame and the second target detection frame are obtained, the relationship between the two target detection frames is accurately measured using the physical position matching score and the feature matching score, and finally it is judged whether the measurement result meets the expected score threshold, and if it meets the expectations, it is determined that the target framed by the first target detection frame and the target framed by the second target detection frame are the same; that is, it is achieved that in the target tracking process, not only the probability of misidentification is reduced, but also multiple targets can be accurately tracked at the same time.

[0079] In one embodiment, the multi-target tracking and identification method further includes the following steps:

[0080] S201: Determine whether the target framed by the first target detection frame is a preset illegal target.

[0081] Among them, the preset violation target is customizable by those skilled in the art. Based on the characteristics of the preset violation target and the characteristics of the target framed by the first target detection frame, and whether the degree of matching between the two exceeds a threshold, it can be determined whether the target is a preset violation target. The specific judgment method is known to those skilled in the art, and the present invention does not make any specific limitations on this.

[0082] S202: If the target framed by the first target detection frame is a preset illegal target, all video frame images containing the target are obtained from the video object to be processed to obtain an illegal image set.

[0083] Among them, all video frame images containing the target are an image set with large data volume and high processing cost. That is to say, the processing cost of the illegal image set is high. Long-term uploading or storage of this type of image set is equivalent to increasing unnecessary workload.

[0084] S203 , obtaining violation feature similarities of the video frame images in the violation image set respectively, where the violation feature similarity is a numerical value of the similarity between features of the video frame images in the violation image set and features of a preset violation standard image.

[0085] Among them, the preset violation standard image is customizable by those skilled in the art, and its features can also be obtained according to existing technical means; the violation feature similarity is equivalent to a quantitative value, which is used to measure the degree of violation of the video frame image in the violation image set.

[0086] S204 , selecting a video frame image with the greatest violation feature similarity from the violation image set as a violation image, and uploading the violation image to a preset monitoring terminal.

[0087] Among them, the video frame image with the greatest similarity of violation features is selected as the violation image, that is, a representative image is selected, which fully covers the said violation image set. The uploading or storage cost of the representative image is low, avoiding the duplication and redundancy of the violation image set and causing waste of resources.

[0088] Existing target tracking and recognition methods experience a sharp decline in accuracy and significantly increase recognition costs when encountering more complex scenarios. Specifically, existing technologies are prone to misidentification and redundancy. For example, if a person violates a rule over a period of time, existing solutions will report all images of that person violating the rule during that period, resulting in high duplication and high costs.

[0089] The beneficial effect of the above scheme S201-S204 is that the first target image and the second target image are continuously captured, the first target detection frame and the second target detection frame are obtained, and the relationship between the two target detection frames is accurately measured using the physical position matching score and the feature matching score. Finally, it is judged whether the measurement result meets the expected score threshold, and if it meets the expectations, it is determined that the target framed by the first target detection frame and the target framed by the second target detection frame are the same. Finally, the video frame image with the largest similarity of the violation feature is selected from the violation image set as the violation image, and the violation image is uploaded to the preset monitoring terminal as the only image; that is, it is achieved that in the target tracking process, not only the probability of misidentification can be reduced, but also the waste of resources caused by duplication and redundancy can be reduced.

[0090] In one embodiment, steps S106-S108 may specifically include the following process, the technical effect of which is that the method can further accurately track the target framed by the first detection frame in the second target image. Specifically, adjacent first target images and second target images are continuously captured from the video object to be processed to obtain a first detection frame of the first target image and a second detection frame of the second target image. For each first detection frame in the first target image, a matching second detection frame is selected from the second target image, and the physical position matching score and feature matching score of the first detection frame and the matching second detection frame are used to accurately measure whether the targets framed by the two are the same. The calculation process of the physical position matching score and the feature matching score refers to the preferred parameters and the preferred calculation method. Therefore, compared with the prior art, the tracking of the target framed by the first detection frame in the second target image can be further accurately achieved; the second target image is used as the new first target image, and the next frame image of the second target image is used as the new second target image, and the above processing process is repeated, thereby achieving accurate tracking of multiple targets in the video object at the same time.

[0091] In addition, the following process specifically involved in the above steps S106-S108 is applied to the scenario where the target continues to appear in the video to be processed, including continuously capturing the first target image and the second target image, obtaining the first target detection frame and the second target detection frame, and accurately measuring the relationship between the two target detection frames using the physical position matching score and the feature matching score. Finally, it is determined whether the measurement result meets the expected score threshold, and if it meets the expectations, it is determined that the target framed by the first target detection frame and the target framed by the second target detection frame are the same; that is, it is achieved that in the target tracking process, not only the probability of misidentification is reduced, but also multiple targets can be accurately tracked at the same time.

[0092] The top left and bottom right corners of rectangle A of the first detection frame are:

[0093] p A0 =(x A0 ,y A0 ),p A1 =(x A1 ,y A1 );

[0094] The upper left and lower right corners of rectangle B of the second detection frame are:

[0095] p A0 =(x A0 ,y A0 ),p A1 =(x A1 ,y A1 );

[0096] The coordinates of the upper left corner of the intersection of rectangle A of the first detection frame and rectangle B of the second detection frame are:

[0097] x1=max(x A0 ,x B0 ),y1=max(y A0 ,y B0 );

[0098] The coordinates of the lower right corner of the intersection of rectangle A of the first detection frame and rectangle B of the second detection frame are:

[0099] x2=min(x A1 ,x B1 ),y2=min(y A1 ,y B1 );

[0100] Then, the area of ​​the intersection of rectangle A of the first detection frame and rectangle B of the second detection frame is calculated as follows:

[0101] intersection=max(x2-x1+1.0,0)×max(y2-y1+1.0,0);

[0102] The area of ​​rectangle A of the first detection frame is:

[0103] S A =(x A1 -x A0 +1.0)×(y A1 -y A0 +1.0);

[0104] The area of ​​rectangle B of the second detection frame is:

[0105] S B =(x B1 -x B0 +1.0)×(y B1 -y B0 +1.0);

[0106] Then, the area of ​​the portion between rectangle A of the first detection frame and rectangle B of the second detection frame is calculated as follows:

[0107] union=S A +S B -intersection;

[0108] To sum up, the calculation formula of intersection-over-union ratio is:

[0109]

[0110] Where IOU is the intersection-over-union ratio, intersection is the area of ​​the intersecting part, and union is the area of ​​the part.

[0111] After completing the calculation of the intersection ratio, it is also necessary to complete the calculation of the center distance ratio;

[0112] The center point of rectangle A of the first detection frame is (A cx ,A cy ), the center point of the B rectangle of the second detection frame is (B cx ,B cy ), the center point distance between the center point of rectangle A of the first detection frame and the center point of rectangle B of the second detection frame is C dis ;

[0113] A cx =(x A0 +x A1 ) / 2.0; A cy =(y A0 +y A1 ) / 2.0;

[0114] B cx =(x B0 +x B1 ) / 2.0; B cy =(y B0 +y B1 ) / 2.0;

[0115]

[0116] The diagonal distance between the minimum enclosing area of ​​rectangle A of the first detection frame and rectangle B of the second detection frame is C. The minimum enclosing area is the minimum circumscribed rectangle of the two frames. The width of the minimum enclosing area is W, and the height of the minimum enclosing area is H.

[0117] W=max(x A1 ,x B1 )-min(x A0 ,x B0 );

[0118] H=max(y A1 ,y B1 )-min(y A0 ,y B0 );

[0119]

[0120] The width difference between rectangle A of the first detection frame and rectangle B of the second detection frame is w dis The height difference between rectangle A of the first detection frame and rectangle B of the second detection frame is h dis .

[0121] w dis =(x A1 -x A0 )-(x B1 -x B0 );

[0122] h dis =(y A1 -y A0 )-(y B1 -y B0 );

[0123] To sum up, the final calculation formula for the physical location matching score is:

[0124] P score =IOU-((C dis / C)+(w dis / W)+(h dis / H));

[0125] In the above steps, the physical location matching score is equal to the intersection-and-union ratio minus the sum of the other three items, which include: C dis / C is the center point distance ratio, w dis / W is the width ratio, h dis / H is the height ratio; In addition, IOU is the intersection-union ratio, P score Assign a score to the physical location match.

[0126] In the process of obtaining the feature matching score, a first image feature vector A with a fixed dimension may be extracted by a traditional image feature extraction algorithm, such as HOG, LBP or Harr feature extraction algorithm. feature (x1,x2,...,x n ) and a second image feature vector B of fixed dimension feature (y1,y2,...,y n ), using cosine similarity to calculate the similarity between the first image feature vector and the second image feature vector, specifically including the following formula:

[0127]

[0128] Among them, x1,x2,...,x n is the coordinate of the first eigenvector, y1,y2,...,y n is the coordinate of the second eigenvector. (x1,x2,...,x n ) is the first eigenvector, (y1,y2,...,y n ) is the second eigenvector.

[0129] In the above calculation, the position matching degree P score The larger the value, the better the match. The feature similarity F score The value range can be [-1, 1]. The larger the value of feature similarity, the more similar it is.

[0130] Multiply the physical location matching score by the preset first weighting coefficient to obtain the physical location weighted score P score , i.e. position matching degree; multiplying the feature matching score by the preset second weighting coefficient to obtain the feature weighted score F score , i.e. feature similarity; add the weighted score of the physical location to the weighted score of the feature degree to obtain the similarity score Total score .

[0131] Position matching degree P score Similarity with feature F score Combining the two, we get the similarity score Total score , the specific formula can be expressed as:

[0132] Total score =αP score +βF score ;

[0133] Among them, α and β represent the weight values ​​of position matching and feature similarity, which are set according to the actual scenario.

[0134] The technical effect of the above scheme is to accurately measure the relationship between the two target detection frames using the physical position matching score and the feature matching score, and further improve the accuracy through the weight value. Finally, it is judged whether the measurement result meets the expected score threshold. This measurement result is more accurate than the existing technology, which is conducive to the further precise tracking of multiple targets in the video object at the same time.

[0135] Example 2

[0136] See also Figure 3 , Figure 3 This is a block diagram of a multi-target tracking and identification device provided by an embodiment of the present invention. Corresponding to the above multi-target tracking and identification method, the present invention also provides a multi-target tracking and identification device 10, which can track the target framed by the first detection frame in the second target image. The multi-target tracking and identification device 10 includes a unit for executing the above multi-target tracking and identification method. The device can be configured in a terminal such as a desktop computer, tablet computer, or laptop computer, and specifically includes:

[0137] The first video capture unit 11 is configured to capture a video frame image from the video object to be processed as a first target image.

[0138] The second video capture unit 12 is configured to capture the next frame of the first target image from the video object to be processed as the second target image.

[0139] The detection frame group acquisition unit 13 is used to input the first target image and the second target image into a preset target detection model respectively to obtain a first detection frame group of the first target image and a second detection frame group of the second target image, wherein the first detection frame group includes at least one first detection frame, and the second detection frame group includes at least one second detection frame.

[0140] The first target selection unit 14 is configured to select a first detection frame from the first detection frame group as a first target detection frame.

[0141] The second target selection unit 15 is configured to select a second detection frame that matches the first target detection frame from the second detection frame group as a second target detection frame.

[0142] The physical position matching unit 16 is used to obtain geometric information between the first target detection frame and the second target detection frame, and obtain a physical position matching score between the first detection frame and the second detection frame based on the geometric information.

[0143] The feature vector acquisition unit 17 is configured to acquire, according to a preset feature extraction algorithm, a first image feature vector of the image defined by the first target detection frame and a second image feature vector of the image defined by the second target detection frame.

[0144] The feature matching unit 18 is configured to obtain a feature matching score between the image defined by the first object detection frame and the image defined by the second object detection frame based on the first image feature vector and the second image feature vector.

[0145] The similarity acquisition unit 19 is configured to acquire a similarity score between the image defined by the first object detection frame and the image defined by the second object detection frame according to the physical position matching score and the feature matching score.

[0146] The score threshold determination unit 20 is configured to determine whether the similarity score is greater than a preset score threshold.

[0147] The target determination unit 21 is configured to determine that the target framed by the first target detection frame is the same as the target framed by the second target detection frame if the similarity score is greater than the score threshold.

[0148] The detection frame judgment unit 22 is used to judge whether there is an unselected first detection frame in the first detection frame group; if there is an unselected first detection frame in the first detection frame group, go to the step of selecting the first detection frame from the first detection frame group as the first target detection frame; if there is no unselected first detection frame in the first detection frame group, use the second target image as the new first target image, and go to the step of intercepting the next frame image of the first target image from the video object to be processed as the second target image.

[0149] In one embodiment, obtaining geometric information between the first target detection frame and the second target detection frame includes: obtaining an intersection-over-union ratio, a center point distance ratio, a width ratio, and a height ratio between the first target detection frame and the second target detection frame.

[0150] In one embodiment, obtaining a physical position matching score between the first detection frame and the second detection frame according to the geometric information includes:

[0151] P score =IOU-((C dis / C)+(wdis / W)+(h dis / H));

[0152] Among them, IOU is the intersection-over-union ratio, C dis / C is the center point distance ratio, w dis / W is the width ratio, h dis / H is the height ratio, P score Assign a score to the physical location match.

[0153] In one embodiment, obtaining a feature matching score between the image defined by the first object detection frame and the image defined by the second object detection frame based on the first image feature vector and the second image feature vector includes:

[0154]

[0155] Among them, x1,x2,...,x n is the coordinate of the first eigenvector, y1,y2,...,y n is the coordinate of the second eigenvector. (x1,x2,...,x n ) is the first eigenvector, (y1,y2,...,y n ) is the second eigenvector.

[0156] In one embodiment, obtaining the similarity score between the image defined by the first target detection frame and the image defined by the second target detection frame based on the physical position matching score and the feature matching score includes: multiplying the physical position matching score by a preset first weighting coefficient to obtain a physical position weighted score; multiplying the feature matching score by a preset second weighting coefficient to obtain a feature weighted score; and adding the physical position weighted score to the feature weighted score to obtain the similarity score.

[0157] In one embodiment, selecting a second detection frame that matches the first target detection frame from the second detection frame group as the second target detection frame includes: selecting at least one sub-detection frame from the second detection frame group, wherein the ratio of the area of ​​the sub-detection frame to the area of ​​the first target detection frame is greater than a preset area ratio threshold; and setting the sub-detection frame as the second target detection frame.

[0158] The technical effect is that a multi-target tracking and identification device is proposed. On the one hand, the device can reduce the probability of misidentification and avoid repeated reporting resulting in high costs. On the other hand, it can continuously capture adjacent first target images and second target images from the video object to be processed, obtain the first detection frame of the first target image and the second detection frame of the second target image, and for each first detection frame in the first target image, a matching second detection frame is selected from the second target image, and the physical position matching score and feature matching score of the first detection frame and the matching second detection frame are used to accurately measure whether the targets framed by the two are the same, thereby realizing the tracking of the target framed by the first detection frame in the second target image; the second target image is used as the new first target image, and the next frame image of the second target image is used as the new second target image, and the above processing process is repeated, thereby realizing the simultaneous accurate tracking of multiple targets in the video object.

[0159] Example 3

[0160] See also Figure 4 , Figure 4 This is a block diagram of a multi-target tracking and identification device according to another embodiment of the present invention. Corresponding to the above multi-target tracking and identification method, the present invention further provides a multi-target tracking and identification device 70, which can be configured in a terminal such as a desktop computer, tablet computer, or laptop computer. The multi-target tracking and identification device 70, based on the multi-target tracking and identification device 10 of Example 2, further includes:

[0161] The target judgment unit 71 is used to judge whether the target framed by the first target detection frame is a preset illegal target.

[0162] The set acquisition unit 72 is configured to acquire all video frame images containing the target from the video object to be processed to obtain a set of illegal images if the target framed by the first target detection frame is a preset illegal target.

[0163] The similarity acquisition unit 73 is used to respectively acquire the violation feature similarity of the video frame images in the violation image set, where the violation feature similarity is a numerical value of the similarity between the features of the video frame images in the violation image set and the features of the preset violation standard images.

[0164] The selection and uploading unit 74 is configured to select a video frame image with the greatest similarity of violation features from the violation image set as a violation image, and upload the violation image to a preset monitoring terminal.

[0165] In one embodiment, it is determined whether the target framed by the first target detection frame is a preset illegal target. The preset illegal target can be customized by those skilled in the art. Based on the characteristics of the preset illegal target and the characteristics of the target framed by the first target detection frame, and whether the degree of match between the two exceeds a threshold, it is determined whether the target is a preset illegal target. The specific determination method is known to those skilled in the art and is not specifically limited in the present invention.

[0166] In one embodiment, if the target identified by the first target detection frame is a pre-set illegal target, all video frames containing the target are obtained from the video object to be processed to obtain an illegal image set. The set of all video frames containing the target is a large data set with high processing costs. This means that the processing cost of the illegal image set is high, and uploading or storing such an image set for a long time increases unnecessary workload.

[0167] In one embodiment, the violation feature similarity of each video frame image in the violation image set is obtained. The violation feature similarity is a numerical value representing the degree of similarity between the features of the video frame images in the violation image set and the features of a preset standard violation image. The preset standard violation image can be customized by those skilled in the art, and its features can also be obtained using existing techniques. The violation feature similarity is equivalent to a quantitative value used to measure the degree of violation of the video frame images in the violation image set.

[0168] In one embodiment, a video frame image with the greatest similarity in violation features is selected from the set of violation images as the violation image, and the violation image is uploaded to a pre-set monitoring terminal. Selecting the video frame image with the greatest similarity in violation features as the violation image, i.e., selecting a representative image that fully covers the set of violation images, reduces the cost of uploading or storing the representative image, and avoids redundant and redundant violation images that waste resources.

[0169] The beneficial effect of the multi-target tracking and identification device 70 is that the device can continuously capture the first target image and the second target image, obtain the first target detection frame and the second target detection frame, and accurately measure the relationship between the two target detection frames using the physical position matching score and the feature matching score. Finally, it is judged whether the measurement result meets the expected score threshold, and if it meets the expectations, it is judged that the target framed by the first target detection frame and the target framed by the second target detection frame are the same. Finally, the video frame image with the largest violation feature similarity is selected from the violation image set as the violation image, and the violation image is uploaded to the preset monitoring terminal as the only image; that is, it is achieved that in the target tracking process, not only the probability of misidentification can be reduced, but also the waste of resources caused by duplication and redundancy can be reduced.

[0170] Example 4

[0171] See also Figure 5 , Figure 5 This is a block diagram of an electronic device provided by the present invention. The electronic device can be a terminal or a server, wherein the terminal can be an electronic device with communication functions such as a smartphone, tablet computer, laptop computer, desktop computer, personal digital assistant, and wearable device. The electronic device includes a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other via the communication bus 114;

[0172] Memory 113, for storing computer programs;

[0173] In one embodiment of the present invention, the processor 111 is configured to implement the method provided by any one of the aforementioned method embodiments when executing a program stored in the memory 113 .

[0174] It should be understood that in the embodiment of the present application, the processor 111 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0175] Those skilled in the art will appreciate that all or part of the steps in the method of the above-described embodiment can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the steps in the method of the above-described embodiment.

[0176] Therefore, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method provided in any one of the aforementioned method embodiments are implemented.

[0177] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, etc. Any physical storage medium capable of storing program code. The computer-readable storage medium may be non-volatile or volatile.

[0178] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0179] In the several embodiments provided herein, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the various units is merely a logical functional division, and actual implementation may employ other division methods. For example, units or components may be combined or integrated into another system, or some features may be omitted or not implemented.

[0180] The steps in the methods of the embodiments of the present invention may be adjusted in order, combined, or deleted as needed. The units in the devices of the embodiments of the present invention may be combined, divided, or deleted as needed. Furthermore, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit.

[0181] If this integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the existing technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present invention.

[0182] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0183] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, to the extent such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to encompass such changes and modifications.

[0184] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A multi-target tracking and recognition method, characterized in that: The method comprises: intercepting a video frame image from the video object to be processed as a first target image; intercepting the next frame image of the first target image from the video object to be processed as the second target image; Inputting the first target image and the second target image into a preset target detection model respectively to obtain a first detection frame group for the first target image and a second detection frame group for the second target image, wherein the first detection frame group includes at least one first detection frame and the second detection frame group includes at least one second detection frame; Selecting a first detection frame from the first detection frame group as a first target detection frame; Selecting a second detection frame matching the first object detection frame from the second detection frame group as a second object detection frame; Obtaining geometric information between the first object detection frame and the second object detection frame, and obtaining a physical position matching score between the first detection frame and the second detection frame based on the geometric information; Obtaining, according to a preset feature extraction algorithm, a first image feature vector of the image defined by the first object detection frame and a second image feature vector of the image defined by the second object detection frame; Obtaining a feature matching score between the image defined by the first object detection frame and the image defined by the second object detection frame based on the first image feature vector and the second image feature vector; Obtaining a similarity score between the image defined by the first object detection frame and the image defined by the second object detection frame according to the physical position matching score and the feature matching score; Determining whether the similarity score is greater than a preset score threshold; If the similarity score is greater than the score threshold, it is determined that the target framed by the first target detection frame is the same as the target framed by the second target detection frame; Determining whether there is an unselected first detection frame in the first detection frame group; If there is an unselected first detection frame in the first detection frame group, proceed to the step of selecting a first detection frame from the first detection frame group as the first target detection frame; If there is no unselected first detection frame in the first detection frame group, the second target image is used as the new first target image, and the process proceeds to the step of capturing the next frame of the first target image from the video object to be processed as the second target image.

2. The multi-target tracking and identification method according to claim 1, characterized in that: The acquiring geometric information between the first target detection frame and the second target detection frame includes: Obtain an intersection-over-union ratio, a center point distance ratio, a width ratio, and a height ratio between the first object detection frame and the second object detection frame.

3. The multi-target tracking and identification method according to claim 1, characterized in that: The obtaining, according to the geometric information, a physical position matching score between the first detection frame and the second detection frame includes: According to the formula P score =IOU-((C dis / C)+(w dis / W)+(h dis / H)) calculating a physical position matching score between the first detection frame and the second detection frame; Among them, IOU is the intersection-over-union ratio, C dis / C is the center point distance ratio, w dis / W is the width ratio, h dis / H is the height ratio, P score Assign a score to the physical location match.

4. The multi-target tracking and identification method according to claim 1, characterized in that: The obtaining, based on the first image feature vector and the second image feature vector, a feature degree matching score between the image defined by the first object detection frame and the image defined by the second object detection frame includes: According to the formula Calculating a feature matching score between the image defined by the first object detection frame and the image defined by the second object detection frame; Among them, (x1,x2,...,x n ) is the first eigenvector, (y1,y2,...,y n ) is the second eigenvector.

5. The multi-target tracking and identification method according to claim 1, characterized in that: The obtaining, according to the physical location matching score and the feature matching score, a similarity score between the image defined by the first object detection frame and the image defined by the second object detection frame includes: Multiplying the physical location matching score by a preset first weighting coefficient to obtain a physical location weighted score; Multiplying the characteristic matching score by a preset second weighting coefficient to obtain a characteristic weighted score; The physical location weighted score is added to the characteristic weighted score to obtain the similarity score.

6. The multi-target tracking and identification method according to claim 1, characterized in that: The selecting, from the second detection frame group, a second detection frame that matches the first target detection frame as the second target detection frame includes: selecting at least one sub-detection frame from the second detection frame group, wherein a ratio of an area of ​​the sub-detection frame to an area of ​​the first object detection frame is greater than a preset area ratio threshold; The sub-detection frame is set as a second object detection frame.

7. The multi-target tracking and identification method according to claim 1, characterized in that: The method further comprises: Determining whether the target framed by the first target detection frame is a preset illegal target; If the target framed by the first target detection frame is a preset illegal target, all video frame images containing the target are obtained from the video object to be processed to obtain an illegal image set; Obtaining violation feature similarities of the video frame images in the violation image set respectively, wherein the violation feature similarity is a numerical value of the degree of similarity between features of the video frame images in the violation image set and features of a preset violation standard image; A video frame image with the greatest similarity of violation features is selected from the violation image set as the violation image, and the violation image is uploaded to a preset monitoring terminal.

8. A multi-target tracking and identification device, characterized in that: The multi-target tracking and identification device includes a unit for executing the method according to any one of claims 1-7.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method according to any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the computer program can implement the method according to any one of claims 1 to 7.

Images