Portrait tracking method and device, non-volatile storage medium, and electronic device

Abstract

The present application discloses a portrait tracking method and device, a non-volatile storage medium, and an electronic device. The method includes: obtaining a current image frame; determining a first face recognition area and a first head and shoulder recognition area, determining a second face recognition area and a second head and shoulder recognition area; combining the first face recognition area and the first head and shoulder recognition area to obtain a first area, adding the second face recognition area and the second head and shoulder recognition area to the first area to obtain a second area; obtaining a historical image frame corresponding to the current image frame; determining a predicted area of ​​the tracked object in the current image frame; matching the second area with the predicted area to obtain a similarity index, determining a target area, and updating tracking information of the predicted area according to the target area. The present application solves the technical problem of low accuracy of portrait tracking caused by factors such as lighting, occlusion, and posture having a significant impact on portrait tracking.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a portrait tracking method and device, a non-volatile storage medium, and an electronic device. Background Art

[0002] Human tracking is a computer vision technology that detects, identifies, and tracks the position and movement of people in video streams. This technology has wide applications in many fields such as security monitoring, smart home, autonomous driving, and augmented reality.

[0003] In the monitoring system, portrait tracking first identifies the portrait in the monitoring area through face detection, body detection, head and shoulder detection and other technologies, and tracks the target in real time as it moves, ensuring continuous tracking of the target's position and dynamics. Subsequently, the captured face and body images will be quality filtered and optimized to ensure clear, high-quality snapshots. These processed images are then sent to the face recognition or body recognition system for further face and body comparison and recognition.

[0004] However, if the portrait tracking technology does not work well, it may cause the following problems: 1. Frequent ID switching: The ID of the same person may switch frequently, and the same target may be regarded as multiple different individuals by the system, affecting the accuracy and continuity of recognition. 2. Repeated capture: The system may capture multiple repeated images of the same person, increasing the complexity and resource consumption of subsequent processing, thereby reducing the overall processing efficiency and response speed. 3. Low capture rate: It depends on the accuracy of face detection or body detection. For people who are far away from the camera, if the face or body is too small, it is easy to fail to track, resulting in a low capture rate.

[0005] Therefore, ensuring the efficiency and accuracy of portrait tracking technology is crucial to the normal operation of the intelligent monitoring system. This not only depends on the accuracy and stability of the technology itself, but also requires full consideration of various complex factors and challenges in actual application scenarios.

[0006] At present, some portrait tracking technologies are based on face frame tracking, some are based on body frame tracking, and some are based on a combination of face and body frame tracking. However, the monitoring scenes are complex and changeable, and people are easily affected by lighting, occlusion, posture, etc. Each method has its own shortcomings: 1. When using body tracking, when the person is close to the camera or the lower body is blocked, the body cannot be detected, resulting in tracking failure. 2. When using face tracking, when the person lowers his head, turns around, or blocks the body, the face cannot be seen, which will also cause the face tracking algorithm to fail. 3. In the case of combining face and body frame tracking, tracking failure will also occur due to problems with the matching method. When there is a detection error in a certain video image frame in the middle and the face or body is not detected, the ID will be lost.

[0007] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0008] The embodiments of the present application provide a portrait tracking method and device, a non-volatile storage medium, and an electronic device to at least solve the technical problem of low accuracy of portrait tracking caused by the significant impact of factors such as lighting, occlusion, and posture on portrait tracking.

[0009] According to one aspect of an embodiment of the present application, a portrait tracking method is provided, comprising: acquiring a current image frame, and determining tracking information of the current image frame, wherein the current image frame includes: a tracking object, and the tracking information includes at least: an activity trajectory of the tracking object; determining a target face recognition area in the current image frame, determining a first head and shoulder recognition area whose coincidence index with the target face recognition area meets a first preset condition, and a second head and shoulder recognition area whose coincidence index with the target face recognition area does not meet the first preset condition; combining the target face recognition area with the first head and shoulder recognition area to obtain a first area, and adding the second head and shoulder recognition area to the first area to obtain a second area; acquiring a historical image frame corresponding to the current image frame, wherein the historical image frame includes: a historical recognition area of ​​the tracking object; determining a predicted area of ​​the tracking object in the current image frame according to the historical recognition area, wherein the predicted area includes: a face recognition area and a head and shoulder recognition area; matching the second area with the predicted area to obtain a similarity index, determining a target area in the second area whose similarity index with the predicted area meets the second preset condition, and updating the tracking information of the predicted area according to the target area.

[0010] Optionally, matching the predicted area with the second area to obtain a similarity index includes: matching the first area and the second face recognition area in the second area with the face recognition area in the first predicted area, and matching the target head and shoulder recognition area in the second area with the head and shoulder recognition area in the first predicted area to obtain a first matching result, wherein the first predicted area is a predicted area in which tracking information is in a non-lost state; matching the second area in the first matching result where the matching fails with the first target area to obtain a second matching result, wherein the first target area includes: the second head and shoulder recognition area and the predicted area in the first matching result where the matching fails; matching the first area and the second face recognition area in the second area where the matching fails with the first matching result Matching with the second target area to obtain a third matching result, wherein the second target area includes: a predicted area where the matching fails in the second matching result and a predicted area where the tracking information is in a lost state in the first matching result; matching the second area where the matching fails in the third matching result with the second predicted area to obtain a fourth matching result, wherein the second predicted area is a predicted area where the matching fails and the tracking information is in a lost state for a duration not exceeding a preset duration in the third matching result, and the recognition quality index of the head and shoulder recognition area meets a third preset condition, and the recognition quality index of the head and shoulder recognition area is used to characterize the clarity of the head and shoulders of the tracked object; determining a similarity index based on the first matching result, the second matching result, the third matching result, and the fourth matching result.

[0011] Optionally, the first area and the second face recognition area in the second area are matched with the face recognition area in the first prediction area, and the target head and shoulder recognition area in the second area is matched with the head and shoulder recognition area in the first prediction area to obtain a first matching result, including: determining the first face feature of the first area and the second face recognition area in the second area, and determining the second face feature of the face recognition area; determining the first Euclidean distance between the first face feature and the second face feature; determining the first intersection-and-union ratio between the first head and shoulder recognition area and the head and shoulder recognition area; determining a first cost matrix based on the first Euclidean distance, the first intersection-and-union ratio, the first preset weight corresponding to the first Euclidean distance, and the second preset weight corresponding to the first intersection-and-union ratio; determining the first matching result based on the first cost matrix and a preset matching algorithm, wherein the preset matching algorithm at least includes: a Hungarian matching algorithm.

[0012] Optionally, the second area in the first matching result that fails to match is matched with the first target area to obtain a second matching result, including: determining a second intersection-and-union ratio between the second area in the first matching result that fails to match and the first target area; determining a second cost matrix based on the second intersection-and-union ratio; determining the second matching result based on the second cost matrix and a preset matching algorithm, wherein the preset matching algorithm includes at least: a Hungarian matching algorithm.

[0013] Optionally, the first area and the second face recognition area in the second area where the matching fails in the second matching result are matched with the second target area to obtain a third matching result, including: determining the third face feature of the first area and the second face recognition area in the second area where the matching fails in the second matching result, and determining the fourth face feature of the second target area; determining the second Euclidean distance between the third face feature and the fourth face feature; determining a third cost matrix based on the second Euclidean distance; determining the third matching result based on the third cost matrix and a preset matching algorithm, wherein the preset matching algorithm includes at least: a Hungarian matching algorithm.

[0014] Optionally, the second area in the third matching result that fails to match is matched with the second predicted area to obtain a fourth matching result, including: determining a third intersection-and-union ratio between the second area in the third matching result that fails to match and the second predicted area; determining a fourth cost matrix based on the third intersection-and-union ratio; determining a fourth matching result based on the fourth cost matrix and a preset matching algorithm, wherein the preset matching algorithm includes at least: a Hungarian matching algorithm.

[0015] Optionally, based on the target area, the tracking information of the predicted area is updated, including: if the matching success rate of the first matching result is greater than the preset success rate, or the matching success rate of the second matching result is greater than the preset success rate, or the matching success rate of the third matching result is greater than the preset success rate, or the matching success rate of the fourth matching result is greater than the preset success rate, the target area is determined as the historical identification area corresponding to the current image frame, and the historical identification area corresponding to the current image frame is added to the tracking information of the predicted area.

[0016] Optionally, after determining the similarity index, the method also includes: if the matching success rate of the first matching result, the matching success rate of the second matching result, the matching success rate of the third matching result and the matching success rate of the fourth matching result are all less than the preset success rate, determining whether the recognition quality index of the face area and / or the recognition quality index of the head and shoulder area in the second area meets the fourth preset condition; if the recognition quality index of the face area and / or the recognition quality index of the head and shoulder area in the second area meets the fourth preset condition, establishing tracking information corresponding to the second area, and adding the tracking information corresponding to the second area to the tracking information set corresponding to the current image frame, wherein the tracking information set includes different tracking information corresponding to different tracking objects.

[0017] Optionally, after determining the similarity index, the method further includes: if the matching success rate of the first matching result, the matching success rate of the second matching result, the matching success rate of the third matching result, and the matching success rate of the fourth matching result are all less than a preset success rate, when the tracked object is an uncertain object, modifying the state of the tracking information of the predicted area to an ineffective state; if the matching success rate of the first matching result, the matching success rate of the second matching result, the matching success rate of the third matching result, and the matching success rate of the fourth matching result are all less than a preset success rate, when the tracked object is a valid tracking object, determining whether a target time length from the first moment to the zero moment is a preset time length, wherein the first moment is a moment when it is determined that the matching success rate of the first matching result, the matching success rate of the second matching result, the matching success rate of the third matching result, and the matching success rate of the fourth matching result are all less than the preset success rate, and the zero moment is a moment when the current image frame is acquired; if the target time length is the preset time length, modifying the state of the tracking information of the predicted area to a lost state, and if the target time length is less than the preset time length, modifying the state of the tracking information of the predicted area to an early lost state.

[0018] According to another aspect of the embodiment of the present application, a portrait tracking device is further provided, comprising: a first acquisition module, used to acquire a current image frame, and determine tracking information of the current image frame, wherein the current image frame includes: a tracking object, and the tracking information at least includes: an activity trajectory of the tracking object; a first determination module, used to determine a target face recognition area and a target head and shoulder recognition area in the current image frame, determine a target face recognition area and a target head and shoulder recognition area whose coincidence index meets a first preset condition, obtain a first face recognition area and a first head and shoulder recognition area, determine a target face recognition area and a target head and shoulder recognition area whose coincidence index does not meet the first preset condition, obtain a second face recognition area and a second head and shoulder recognition area; a combining module, used to combine the first face recognition area and the target head and shoulder recognition area with the first person; The face recognition area and the first head and shoulder recognition area are combined to obtain the first area, and the second face recognition area and the second head and shoulder recognition area are added to the first area to obtain the second area; a second acquisition module is used to obtain a historical image frame corresponding to the current image frame, wherein the historical image frame includes: a historical recognition area of ​​the tracked object; a second determination module is used to determine a predicted area of ​​the tracked object in the current image frame according to the historical recognition area, wherein the predicted area includes: a face recognition area and a head and shoulder recognition area; a matching module is used to match the second area with the predicted area to obtain a similarity index, determine a target area in the second area whose similarity index with the predicted area meets a second preset condition, and update the tracking information of the predicted area according to the target area.

[0019] According to another aspect of the embodiments of the present application, a non-volatile storage medium is further provided, the storage medium including a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the above portrait tracking method.

[0020] According to another aspect of the embodiments of the present application, an electronic device is provided, including: a memory and a processor, the processor being used to run a program stored in the memory, wherein the above portrait tracking method is executed when the program is run.

[0021] According to yet another aspect of the embodiments of the present application, a computer program is further provided, wherein the above portrait tracking method is implemented when the computer program is executed by a processor.

[0022] According to another aspect of the embodiments of the present application, a computer program product is also provided, which includes a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above portrait tracking method is implemented.

[0023] In an embodiment of the present application, a current image frame is obtained, and tracking information of the current image frame is determined, wherein the current image frame includes: a tracking object, and the tracking information includes at least: an activity trajectory of the tracking object; a target face recognition area in the current image frame is determined, and a first head and shoulder recognition area whose coincidence index with the target face recognition area meets a first preset condition and a second head and shoulder recognition area whose coincidence index with the target face recognition area does not meet the first preset condition is determined; the target face recognition area and the first head and shoulder recognition area are combined to obtain a first area, and the second head and shoulder recognition area is added to the first area to obtain a second area; a historical image frame corresponding to the current image frame is obtained, wherein the historical image frame includes: the tracking object The method comprises the following steps: determining a historical recognition area of ​​a portrait; determining a predicted area of ​​the tracked object in a current image frame according to the historical recognition area, wherein the predicted area includes: a face recognition area and a head and shoulder recognition area; matching the second area with the predicted area to obtain a similarity index, determining a target area in the second area whose similarity index with the predicted area meets a second preset condition, and updating the tracking information of the predicted area according to the target area, thereby achieving the purpose of reducing the influence of factors such as illumination, occlusion, and posture on portrait tracking, thereby achieving the technical effect of improving the accuracy of portrait tracking, and further solving the technical problem of low accuracy of portrait tracking caused by the great influence of factors such as illumination, occlusion, and posture on portrait tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1 is a flow chart of a portrait tracking method according to an embodiment of the present application;

[0026] Figure 2 is a flow chart of another portrait tracking method according to an embodiment of the present application;

[0027] Figure 3 This is a flow chart of tracking state jump and file drop according to an embodiment of the present application;

[0028] Figure 4 is a flow chart of tracking state transition according to an embodiment of the present application;

[0029] Figure 5 is a structural diagram of a portrait tracking device according to an embodiment of the present application;

[0030] Figure 6 It is a hardware structure block diagram of a computer terminal of a portrait tracking method according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] According to an embodiment of the present application, a method embodiment of a portrait tracking method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0034] Figure 1 is a flow chart of a portrait tracking method according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0035] Step S101, obtaining a current image frame and determining tracking information of the current image frame, wherein the current image frame includes: a tracking object, and the tracking information at least includes: an activity trajectory of the tracking object.

[0036] For example, the current image frame can be obtained by the following method: connect to a camera, or receive a video file, obtain a video sequence, in which 5 frames per second or uniformly sample according to actual work requirements to obtain multiple image frames, and determine the currently processed image frame as the current image frame.

[0037] It is worth noting that the tracking information of the image frame is Tracklet, where Tracklet is a professional term in the field of video or object tracking, which can be translated as "tracking clip" or "tracklet clip". Tracklet is an important component in the process of forming a complete trajectory, and is composed of multiple continuous tracking clips belonging to the same physical target. The following is a detailed explanation of Tracklet: Tracklet refers to a short clip that connects the detection results of the same object in consecutive frames through data association methods during object tracking. It is the basic unit for constructing a complete trajectory. The detection results in the Tracklet are continuous in time, that is, they correspond to consecutive frames in the video. The detection results in the Tracklet belong to the same tracking object and have the same or similar characteristics. Multiple Tracklets can form a complete trajectory through further data association and trajectory combination algorithms.

[0038] In Multiple Object Tracking (MOT), multiple targets are tracked in continuous video images, and each target forms a series of tracklets, which are finally combined into a complete track.

[0039] Suppose there is a video sequence containing multiple pedestrians. In the first frame of the video, the position of each pedestrian is detected using the object detection algorithm. Then, pedestrians are detected in subsequent frames, and the detection results of the same pedestrian are connected through the data association method to form a tracklet. Finally, multiple tracklets are combined into complete tracks to track each pedestrian in the entire video sequence.

[0040] Step S102, determine the target face recognition area and the target head and shoulder recognition area in the current image frame, determine the target face recognition area and the target head and shoulder recognition area whose overlap index meets the first preset condition, and obtain the first face recognition area and the first head and shoulder recognition area, determine the target face recognition area and the target head and shoulder recognition area whose overlap index does not meet the first preset condition, and obtain the second face recognition area and the second head and shoulder recognition area.

[0041] Determine the intersection-over-union (IOU) ratio between the target face recognition area and the target head and shoulder recognition area in the detected current image frame to obtain a cost matrix. Use the Hungarian matching algorithm to determine the matching result of the target face recognition area and the target head and shoulder recognition area, and then determine the target face recognition area and the target head and shoulder recognition area whose overlap index meets the first preset condition and the target face recognition area and the target head and shoulder recognition area whose overlap index does not meet the first preset condition.

[0042] The overlap index is, for example, Intersection over Union (IoU), where IoU is an important metric in target detection, used to evaluate the overlap between the predicted box (predicted area) and the true box (or true area). The value range of IoU is between 0 and 1. The closer the IoU value is to 1, the higher the overlap between the predicted area and the true area. The overlap index satisfies the first preset condition, for example, the intersection over union ratio is greater than 0.85.

[0043] Step S103: Combine the first face recognition area and the first head and shoulder recognition area to obtain a first area, and add the second face recognition area and the second head and shoulder recognition area to the first area to obtain a second area.

[0044] It should be noted that the above combination is Bind, that is, the attributes belonging to the same tracking object are bound together. In other words, the first face recognition area and the first head and shoulder recognition area are Bind to obtain TrackInput, and the unmatched target face recognition area and target head and shoulder recognition area are also added to TrackInput respectively.

[0045] Step S104, obtaining a historical image frame corresponding to the current image frame, wherein the historical image frame includes: a historical identification area of ​​the tracked object.

[0046] For example, the historical image frame is a frame before the current image frame.

[0047] Step S105, determining a predicted area of ​​the tracked object in the current image frame according to the historical recognition area, wherein the predicted area includes: a face recognition area and a head and shoulder recognition area.

[0048] Preferably, the predicted area of ​​the tracked object in the current image frame is determined by the following method: determine the tracking status of each tracking information in the tracking information set. It is worth noting that the tracking information set is the above-mentioned Tracklets, where one tracking information corresponds to one tracking object. Delete the tracking information in the tracking information set whose tracking status is Deleted and NotSet. That is, before the current image frame, the tracking target object corresponding to the tracking information has not been matched for n consecutive frames, where n is a positive integer. Perform Kalman filtering on the historical identification area of ​​the historical image frame in the remaining tracking information in the tracking information set, predict the active position of the tracking object corresponding to the tracking information in the current image frame, and obtain the predicted area.

[0049] It should be explained that the status of tracking information (tracking status) includes but is not limited to: 1. NotSet: The tracking ID (tracking object) is not effective. 2. Uncertain: The tracking ID is created for the first time, but it is not certain whether it is a valid tracking ID. 3. Confirmed: After multiple frames, it is confirmed that the current image frame is a valid tracking ID. 4. EarlyLost: This mode is switched to when the tracking ID disappears before the first push. This mode is added to complete two pushes. 5. Lost: The lost tracking ID. Deleted: The tracking ID that has ended.

[0050] Among them, the first push is the image frame that the tracker receives and starts to process for the first time during the tracking process. This is the starting point of the tracking task. The target tracking algorithm will initialize the tracking state based on the target detection results in this image, such as the position and appearance features of the target. Subsequently, the algorithm will use this information to continue tracking the target in subsequent frames of the video sequence. EarlyLost means that during the target tracking process, the target is lost within a short period of time after being tracked. This is usually caused by reasons such as occlusion of the target, rapid movement, or limitations of the algorithm itself. The target tracking algorithm needs to have a certain degree of robustness to handle such early loss situations, such as recapture and maintain tracking of the target through methods such as motion model prediction, historical information backtracking, or re-detection mechanisms.

[0051] Step S106, matching the second region with the predicted region to obtain a similarity index, determining a target region in the second region whose similarity index with the predicted region meets a second preset condition, and updating tracking information of the predicted region according to the target region.

[0052] Exemplarily, a cascade matching method is adopted, and a multi-stage matching strategy is adopted to realize face tracking, and the second area is matched with the predicted area. Specifically, four different matching methods can be adopted in succession. The first time, the face recognition area and the head and shoulder recognition area of ​​the current image frame are matched with the face recognition area and the head and shoulder recognition area corresponding to the ID of the tracking object whose trajectory is tracked by combining IoU and face feature weight adaptiveness to perform the first accurate match to ensure the clear tracking ID matching accuracy, and then supplementary matching is performed through the head and shoulder IOU respectively, and finally the third match is performed through the face features and the fourth match is performed through the IOU of the head and shoulder recognition area again to retrieve the lost tracking ID, which ensures the tracking accuracy while better reducing the tracking repetition rate.

[0053] According to the above steps, the current image frame is obtained, and the tracking information of the current image frame is determined, wherein the current image frame includes: a tracking object, and the tracking information at least includes: an activity trajectory of the tracking object; a target face recognition area in the current image frame is determined, and a first head and shoulder recognition area whose coincidence index with the target face recognition area meets a first preset condition and a second head and shoulder recognition area whose coincidence index with the target face recognition area does not meet the first preset condition are determined; the target face recognition area and the first head and shoulder recognition area are combined to obtain a first area, and the second head and shoulder recognition area is added to the first area to obtain a second area; the current image frame is obtained. The corresponding historical image frame includes: a historical identification area of ​​the tracked object; based on the historical identification area, determining a predicted area of ​​the tracked object in the current image frame, wherein the predicted area includes: a face recognition area and a head and shoulder recognition area; matching the second area with the predicted area to obtain a similarity index, determining in the second area a target area whose similarity index with the predicted area meets a second preset condition, and updating the tracking information of the predicted area based on the target area, thereby achieving the purpose of reducing the influence of factors such as lighting, occlusion, and posture on portrait tracking, thereby achieving the technical effect of improving the accuracy of portrait tracking.

[0054] The following Figure 1 The steps shown are exemplary and explanatory.

[0055] According to some optional embodiments of the present application, matching the predicted region with the second region to obtain a similarity index may be achieved by the following method.

[0056] Step S1061, matching the first area and the second face recognition area in the second area with the face recognition area in the first prediction area, and matching the target head and shoulder recognition area in the second area with the head and shoulder recognition area in the first prediction area, to obtain a first matching result, wherein the first prediction area is a prediction area in which tracking information is in a non-lost state.

[0057] Step S1062: Match the second area in the first matching result where the matching fails with the first target area to obtain a second matching result, wherein the first target area includes: the second head and shoulder recognition area and the predicted area in the first matching result where the matching fails.

[0058] Step S1063, matching the first area and the second face recognition area in the second area where the matching fails in the second matching result with the second target area to obtain a third matching result, wherein the second target area includes: the predicted area where the matching fails in the second matching result and the predicted area where the tracking information is lost in the first matching result.

[0059] Step S1064, matching the second area where the matching fails in the third matching result with the second predicted area to obtain a fourth matching result, wherein the second predicted area is a predicted area where the matching fails in the third matching result and the tracking information is lost for a period of time not exceeding a preset period, and the recognition quality index of the head and shoulder recognition area meets the third preset condition, and the recognition quality index of the head and shoulder recognition area is used to characterize the clarity of the head and shoulders of the tracked object.

[0060] Step S1065: determine a similarity index based on the first matching result, the second matching result, the third matching result, and the fourth matching result.

[0061] Specifically, step S1061 can be implemented by the following method: determine the first facial features of the first area and the second face recognition area in the second area, and determine the second facial features of the face recognition area; determine the first Euclidean distance between the first facial feature and the second facial feature; determine the first intersection-and-union ratio between the first head and shoulder recognition area and the head and shoulder recognition area; determine the first cost matrix based on the first Euclidean distance, the first intersection-and-union ratio, the first preset weight corresponding to the first Euclidean distance, and the second preset weight corresponding to the first intersection-and-union ratio; determine the first matching result based on the first cost matrix and the preset matching algorithm.

[0062] The cost matrix is ​​a two-dimensional array whose elements represent the cost of associating the prediction region with the detection region. The cost is the intersection-over-union ratio between the head and shoulder recognition regions and / or the Euclidean distance between the face recognition regions. By minimizing the total cost in the cost matrix, the best matching solution can be found, thereby achieving accurate target tracking.

[0063] Step S1062 can be implemented by the following method: matching the second area that fails to match in the first matching result with the first target area to obtain a second matching result, including: determining a second intersection-and-union ratio between the second area that fails to match in the first matching result and the first target area; determining a second cost matrix based on the second intersection-and-union ratio; determining the second matching result based on the second cost matrix and a preset matching algorithm.

[0064] Step S1063 can be implemented by the following method: matching the first area and the second face recognition area in the second area where the matching fails in the second matching result with the second target area to obtain a third matching result, including: determining the third face feature of the first area and the second face recognition area in the second area where the matching fails in the second matching result, and determining the fourth face feature of the second target area; determining the second Euclidean distance between the third face feature and the fourth face feature; determining a third cost matrix based on the second Euclidean distance; determining the third matching result based on the third cost matrix and a preset matching algorithm.

[0065] It is worth noting that through the above third matching, the prediction area temporarily lost due to occlusion and other factors can be retrieved.

[0066] Step S1064 can be implemented by the following method: matching the second area in the third matching result where the matching fails with the second predicted area to obtain a fourth matching result, including: determining a third intersection-and-union ratio between the second area in the third matching result where the matching fails and the second predicted area; determining a fourth cost matrix based on the third intersection-and-union ratio; and determining a fourth matching result based on the fourth cost matrix and a preset matching algorithm.

[0067] It is worth noting that the fourth matching mentioned above can serve as a supplement to the third matching to retrieve the temporarily lost prediction area.

[0068] The preset matching algorithm at least includes: Hungarian matching algorithm. The Hungarian matching algorithm, also known as the Hungarian Algorithm, is an algorithm used to solve allocation problems in operations research and graph theory.

[0069] The basic steps of the algorithm are as follows: First, confirm the detection boxes in the current frame that match the targets in the historical frames, and build a bipartite graph model. Then, match them in order of target IDs. For example, first match target 1 in the current frame that may match target 1 in the previous frame. If a conflict occurs during the matching process, that is, a potential match for a target in the current frame is already occupied by other targets, try to rematch through augmenting paths. An augmenting path is a path that passes through unmatched points and can increase the number of matches. If a match is still not found after the augmenting path, the weight adjustment is required. Repeat the above steps until matches are found for all targets or it is confirmed that the number of matches cannot be further increased.

[0070] In multi-target tracking, the Hungarian algorithm can effectively associate the target detected in the current frame with the historical trajectory, thereby maintaining the identity consistency of the target.

[0071] Furthermore, updating the tracking information of the predicted area according to the target area can be achieved by the following method: if the matching success rate of the first matching result is greater than the preset success rate, or the matching success rate of the second matching result is greater than the preset success rate, or the matching success rate of the third matching result is greater than the preset success rate, or the matching success rate of the fourth matching result is greater than the preset success rate, the target area is determined as the historical identification area corresponding to the current image frame, and the historical identification area corresponding to the current image frame is added to the tracking information of the predicted area.

[0072] For example, if the matching success rate of the first matching result is greater than 80%, or the matching success rate of the second matching result is greater than 80%, or the matching success rate of the third matching result is greater than 80%, or the matching success rate of the fourth matching result is greater than 80%, the target area is determined as the historical identification area corresponding to the current image frame to update the tracking information of the predicted area.

[0073] Furthermore, after determining the similarity index, the following steps can also be performed: if the matching success rate of the first matching result, the matching success rate of the second matching result, the matching success rate of the third matching result and the matching success rate of the fourth matching result are all less than the preset success rate, determine whether the recognition quality index of the face area and / or the recognition quality index of the head and shoulder area in the second area meets the fourth preset condition; if the recognition quality index of the face area and / or the recognition quality index of the head and shoulder area in the second area meets the fourth preset condition, establish tracking information corresponding to the second area, and add the tracking information corresponding to the second area to the tracking information set corresponding to the current image frame, wherein the tracking information set includes different tracking information corresponding to different tracking objects.

[0074] On the other hand, after determining the similarity index, the following steps may also be performed: if the matching success rate of the first matching result, the matching success rate of the second matching result, the matching success rate of the third matching result, and the matching success rate of the fourth matching result are all less than the preset success rate, when the tracked object is an uncertain object, the state of the tracking information of the predicted area is modified to an ineffective state; if the matching success rate of the first matching result, the matching success rate of the second matching result, the matching success rate of the third matching result, and the matching success rate of the fourth matching result are all less than the preset success rate, when the tracked object is a valid tracking object, determine whether the target time length from the first moment to the zero moment is the preset time length, wherein the first moment is the moment when it is determined that the matching success rate of the first matching result, the matching success rate of the second matching result, the matching success rate of the third matching result, and the matching success rate of the fourth matching result are all less than the preset success rate, and the zero moment is the moment when the current image frame is acquired; if the target time length is the preset time length, the state of the tracking information of the predicted area is modified to a lost state, and if the target time length is less than the preset time length, the state of the tracking information of the predicted area is modified to an early lost state.

[0075] The tracking object is an uncertain object, which means that the tracking status of the tracking information is Uncertain, that is, the tracking ID is created for the first time, but it is uncertain whether it is a valid tracking ID.

[0076] Figure 2 is a flowchart of another portrait tracking method according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0077] Step S201, connect a camera or receive a video file to obtain a video sequence. In the video sequence, 5 frames per second or uniform sampling according to actual work requirements is performed to obtain multiple image frames, and the multiple image frames are input into a single frame analysis module.

[0078] Step S202, the single frame analysis module analyzes the image frame to obtain the following information: face frame, face attributes, face features, head and shoulder frame, body frame, and body frame quality.

[0079] Step S203, converting the single frame result detected in step S202 into a format required for tracking, and the tracking requires information: face frame, face quality, face features, head and shoulder frame, body frame, and body frame quality.

[0080] Step S204, calculate the IOU between all detected head-shoulder frames and face frames, obtain the cost matrix, apply the Hungarian matching algorithm, calculate the matching result, bind the face frame and the corresponding head-shoulder frame, and obtain TrackInput. The unmatched face frame and head-shoulder frame are also added to TrackInput.

[0081] Step S205, delete the ended Tracklets in the tracking list Tracklets (status is Deleted or NotSet), then traverse each Tracklet in the Tracklets, apply Kalman filtering, predict the motion trajectory of each Tracklet based on its historical trajectory, and obtain the estimated head and shoulder frame for subsequent IOU matching.

[0082] Step S206, executing four matching processes.

[0083] Step S2061, first matching: there are M TrackInputs with face frames and N Tracklets in the Tracklets whose status is not Lost. The IOU value of the head and shoulder frame and the Euclidean distance of the facial features are calculated. Then, different weight values ​​are applied to combine the two values ​​to calculate the cost matrix (the weight setting is based on actual debugging), an N*M cost matrix is ​​generated, and the Hungarian matching algorithm is applied to calculate the first matching result.

[0084] Step S2062, second matching: add M TrackInputs that were not matched in the first matching, N TrackInputs that do not contain face frames, and L Tracklets that were not matched in the first matching (a total of (N+L)), then calculate the IOU values ​​of the head-shoulder frames between the M TrackInputs and the (N+L) Tracklets, obtain the (N+L)*M cost matrix, and apply the Hungarian algorithm to calculate the second matching result.

[0085] Step S2063, third matching: select the M TrackInputs with face frames that were not matched in the second matching, the N Tracklets that were not matched in the second matching, and the L Tracklets with the Lost status in the first matching, and then calculate the Euclidean distance between the facial features of these M TrackInputs and (N+L) Tracklets, and obtain the (N+L)*M matrix as the cost matrix, and apply the Hungarian algorithm to calculate the third matching result. The main purpose of the third matching is to retrieve the Tracklets that are temporarily lost due to occlusion and other factors.

[0086] Step S2064, fourth matching: the M Tracklets that were not matched in the third time and were in the lost state within 5 seconds and whose head-shoulder box quality was greater than the quality threshold are again matched with the N unmatched TrackInputs to calculate the IOU value of the head-shoulder box, and N*M matrices are obtained as the cost matrix. The Hungarian matching algorithm is applied to calculate the fourth matching result. The fourth matching is mainly used as a supplement to the third to retrieve the temporarily lost Tracklets.

[0087] Step S207, Figure 3 This is a flow chart of tracking state jump and file drop according to an embodiment of the present application; Figure 4 This is a flow chart of the tracking state transition according to the embodiment of the present application. For the TrackInput and Tracklets that are successfully matched in all four matches, the TrackInput is applied to update the corresponding Tracklet, according to Figure 4 The method shown updates the state of each matching Tracklet and Figure 3 The method shown is used for subsequent archiving. The archiving is to save the image with the best tracking ID quality according to the requirements.

[0088] Tracklet status (tracking status) includes but is not limited to: 1. NotSet: Tracking ID is not effective. 2. Uncertain: Tracking ID is created for the first time, but it is not sure whether it is a valid tracking ID. 3. Confirmed: After multiple frames, it is confirmed that the current frame is a valid tracking ID. 4. EarlyLost: When the tracking ID disappears before the first push time, it switches to this mode. This mode is added to complete two pushes. 5. Lost: Lost tracking ID. Deleted: The tracking ID has ended.

[0089] Step S208, for the TrackInput that has not been matched with the Tracklets for four times, determine whether the size and quality of the face frame or the head and shoulder frame meet the quality threshold for creating a new Tracklet. If so, create a new Tracklet and add it to the Tracklets. If not, discard it.

[0090] Step S209: for the Tracklets that have not been matched with TrackInput for four times, Figure 4 Status update condition, change status, according to Figure 3 Follow the process in the figure to proceed.

[0091] Step S210, continue inputting the next frame of image.

[0092] Through the above steps, the adaptive fusion method of multiple features (head and shoulder frame, face frame, face features, face quality, body mass) is used for tracking, which better combines the advantages of both and reduces the tracking loss caused by face occlusion, blur, angle change and other factors when only face frame and face features are used for tracking, thereby improving the tracking accuracy.

[0093] In addition, a cascade matching method is used to implement face tracking with a multi-stage matching strategy. Four different matching methods are used in succession. The first accurate match is performed by inputting the face frame and head-shoulder frame of the current frame and combining the head-shoulder frame IOU and face feature weight adaptation of the tracked ID to ensure a clear ID matching accuracy. Then, supplementary matching is performed through the head-shoulder IOU respectively. Finally, the third match is performed through the face features and the fourth match is performed through the head-shoulder frame IOU to retrieve the lost ID. This ensures the tracking accuracy while better reducing the tracking repetition rate.

[0094] In summary, this embodiment can significantly improve the accuracy and robustness of face tracking, and effectively cope with challenges such as illumination changes, occlusions, and posture changes in complex surveillance scenes, thus having broad prospects in practical applications.

[0095] Figure 5 is a structural diagram of a portrait tracking device according to an embodiment of the present application, such as Figure 5 As shown, the device comprises:

[0096] The first acquisition module 51 is used to acquire the current image frame and determine the tracking information of the current image frame, wherein the current image frame includes: a tracking object, and the tracking information at least includes: an activity trajectory of the tracking object.

[0097] The first determination module 52 is used to determine the target face recognition area and the target head and shoulder recognition area in the current image frame, determine the target face recognition area and the target head and shoulder recognition area whose coincidence index meets the first preset condition, and obtain the first face recognition area and the first head and shoulder recognition area, determine the target face recognition area and the target head and shoulder recognition area whose coincidence index does not meet the first preset condition, and obtain the second face recognition area and the second head and shoulder recognition area.

[0098] The combining module 53 is used to combine the first face recognition area and the first head and shoulder recognition area to obtain a first area, and add the second face recognition area and the second head and shoulder recognition area to the first area to obtain a second area.

[0099] The second acquisition module 54 is used to acquire a historical image frame corresponding to the current image frame, wherein the historical image frame includes: a historical identification area of ​​the tracked object.

[0100] The second determination module 55 is used to determine the predicted area of ​​the tracked object in the current image frame according to the historical recognition area, wherein the predicted area includes: a face recognition area and a head and shoulder recognition area.

[0101] The matching module 56 is used to match the second area with the predicted area to obtain a similarity index, determine a target area in the second area whose similarity index with the predicted area meets a second preset condition, and update the tracking information of the predicted area according to the target area.

[0102] Optionally, the matching module 56 is further configured to perform the following steps: matching the first area and the second face recognition area in the second area with the face recognition area in the first prediction area, and matching the target head and shoulder recognition area in the second area with the head and shoulder recognition area in the first prediction area to obtain a first matching result, wherein the first prediction area is a prediction area in which tracking information is in a non-lost state; matching the second area in the first matching result where the matching fails with the first target area to obtain a second matching result, wherein the first target area includes: the second head and shoulder recognition area and the prediction area in the first matching result where the matching fails; matching the first area and the second face recognition area in the second area where the matching fails with the second target area to obtain a second matching result. The first matching result is matched with the target area to obtain a third matching result, wherein the second target area includes: a predicted area where the matching fails in the second matching result and a predicted area where the tracking information is in a lost state in the first matching result; the second area where the matching fails in the third matching result is matched with the second predicted area to obtain a fourth matching result, wherein the second predicted area is a predicted area where the matching fails and the tracking information is in a lost state in the third matching result for a duration not exceeding a preset duration, and the recognition quality index of the head and shoulder recognition area meets a third preset condition, and the recognition quality index of the head and shoulder recognition area is used to characterize the clarity of the head and shoulders of the tracked object; and a similarity index is determined according to the first matching result, the second matching result, the third matching result, and the fourth matching result.

[0103] Optionally, the matching module 56 is also used to perform the following steps: determine the first facial features of the first area and the second face recognition area in the second area, and determine the second facial features of the face recognition area; determine the first Euclidean distance between the first facial feature and the second facial feature; determine the first intersection-and-union ratio between the first head-and-shoulder recognition area and the head-and-shoulder recognition area; determine a first cost matrix based on the first Euclidean distance, the first intersection-and-union ratio, the first preset weight corresponding to the first Euclidean distance, and the second preset weight corresponding to the first intersection-and-union ratio; determine a first matching result based on the first cost matrix and a preset matching algorithm, wherein the preset matching algorithm at least includes: a Hungarian matching algorithm.

[0104] Optionally, the matching module 56 is also used to perform the following steps: determine a second intersection-and-union ratio between a second area in the first matching result where the match fails and the first target area; determine a second cost matrix based on the second intersection-and-union ratio; determine a second matching result based on the second cost matrix and a preset matching algorithm, wherein the preset matching algorithm includes at least: a Hungarian matching algorithm.

[0105] Optionally, the matching module 56 is also used to perform the following steps: including: determining the third facial feature of the first area and the second facial recognition area in the second area where the match fails in the second matching result, and determining the fourth facial feature of the second target area; determining the second Euclidean distance between the third facial feature and the fourth facial feature; determining a third cost matrix based on the second Euclidean distance; determining a third matching result based on the third cost matrix and a preset matching algorithm, wherein the preset matching algorithm at least includes: a Hungarian matching algorithm.

[0106] Optionally, the matching module 56 is also used to perform the following steps: determine a third intersection-and-union ratio between the second area where the match fails in the third matching result and the second predicted area; determine a fourth cost matrix based on the third intersection-and-union ratio; determine a fourth matching result based on the fourth cost matrix and a preset matching algorithm, wherein the preset matching algorithm includes at least: a Hungarian matching algorithm.

[0107] Optionally, the matching module 56 is also used to perform the following steps: if the matching success rate of the first matching result is greater than the preset success rate, or the matching success rate of the second matching result is greater than the preset success rate, or the matching success rate of the third matching result is greater than the preset success rate, or the matching success rate of the fourth matching result is greater than the preset success rate, the target area is determined as the historical identification area corresponding to the current image frame, and the historical identification area corresponding to the current image frame is added to the tracking information of the predicted area.

[0108] Optionally, the portrait tracking device also includes a third determination module, which is used to perform the following steps after determining the similarity index: if the matching success rate of the first matching result, the matching success rate of the second matching result, the matching success rate of the third matching result and the matching success rate of the fourth matching result are all less than the preset success rate, determine whether the recognition quality index of the face area and / or the recognition quality index of the head and shoulder area in the second area meets the fourth preset condition; if the recognition quality index of the face area and / or the recognition quality index of the head and shoulder area in the second area meets the fourth preset condition, establish tracking information corresponding to the second area, and add the tracking information corresponding to the second area to the tracking information set corresponding to the current image frame, wherein the tracking information set includes different tracking information corresponding to different tracking objects.

[0109] Optionally, the portrait tracking device also includes a fourth determination module, which is used to perform the following steps after determining the similarity index: if the matching success rate of the first matching result, the matching success rate of the second matching result, the matching success rate of the third matching result, and the matching success rate of the fourth matching result are all less than the preset success rate, when the tracked object is an uncertain object, the state of the tracking information of the predicted area is modified to an ineffective state; if the matching success rate of the first matching result, the matching success rate of the second matching result, the matching success rate of the third matching result, and the matching success rate of the fourth matching result are all less than the preset success rate, when the tracked object is a valid tracking object, determine whether the target time length from the first moment to the zero moment is a preset time length, wherein the first moment is a moment when it is determined that the matching success rate of the first matching result, the matching success rate of the second matching result, the matching success rate of the third matching result, and the matching success rate of the fourth matching result are all less than the preset success rate, and the zero moment is a moment when the current image frame is acquired; if the target time length is the preset time length, the state of the tracking information of the predicted area is modified to a lost state, and if the target time length is less than the preset time length, the state of the tracking information of the predicted area is modified to an early lost state.

[0110] It should be noted that the above Figure 5 The modules in the embodiment may be program modules (e.g., a set of program instructions for implementing a specific function) or hardware modules. For the latter, they may be expressed in the following forms, but are not limited thereto: the expression form of the above modules is a processor, or the functions of the above modules are implemented by a processor.

[0111] It should be noted that Figure 5 The preferred implementation of the illustrated embodiment can be found in Figure 1 The relevant description of the illustrated embodiment will not be repeated here.

[0112] Figure 6 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a portrait tracking method. Figure 6 As shown, the computer terminal 60 may include one or more (602a, 602b, ..., 602n are used to illustrate) processors 602 (the processor 602 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 604 for storing data, and a transmission module 606 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 6The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 6 More or fewer components as shown, or with Figure 6 Different configurations are shown.

[0113] It should be noted that the one or more processors 602 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 60. As involved in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0114] The memory 604 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the portrait tracking method in the embodiment of the present application. The processor 602 executes various functional applications and data processing by running the software programs and modules stored in the memory 604, that is, the above-mentioned portrait tracking method is realized. The memory 604 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 604 may further include a memory remotely arranged relative to the processor 602, and these remote memories may be connected to the computer terminal 60 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0115] The transmission module 606 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 60. In one example, the transmission module 606 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 606 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0116] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 60 .

[0117] It should be noted that, in some optional embodiments, the above Figure 6The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 6 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0118] It should be noted that Figure 6 The computer terminal shown is used to execute Figure 1 The portrait tracking method shown, therefore the relevant explanations in the execution method of the above command are also applicable to the electronic device and will not be repeated here.

[0119] An embodiment of the present application further provides a non-volatile storage medium, which includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the above-mentioned portrait tracking method.

[0120] A program for a non-volatile storage medium to perform the following functions: obtaining a current image frame and determining tracking information of the current image frame, wherein the current image frame includes: a tracking object, and the tracking information includes at least: an activity trajectory of the tracking object; determining a target face recognition area in the current image frame, determining a first head and shoulder recognition area whose coincidence index with the target face recognition area meets a first preset condition, and a second head and shoulder recognition area whose coincidence index with the target face recognition area does not meet the first preset condition; combining the target face recognition area with the first head and shoulder recognition area to obtain a first area, and adding the second head and shoulder recognition area to the first area to obtain a second area; obtaining a historical image frame corresponding to the current image frame, wherein the historical image frame includes: a historical recognition area of ​​the tracking object; determining a predicted area of ​​the tracking object in the current image frame according to the historical recognition area, wherein the predicted area includes: a face recognition area and a head and shoulder recognition area; matching the second area with the predicted area to obtain a similarity index, determining a target area in the second area whose similarity index with the predicted area meets the second preset condition, and updating the tracking information of the predicted area according to the target area.

[0121] An embodiment of the present application further provides an electronic device, including: a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the above portrait tracking method is executed when the program is run.

[0122] The processor is used to run a program that performs the following functions: obtaining a current image frame and determining tracking information of the current image frame, wherein the current image frame includes: a tracking object, and the tracking information includes at least: an activity trajectory of the tracking object; determining a target face recognition area in the current image frame, determining a first head and shoulder recognition area whose coincidence index with the target face recognition area meets a first preset condition, and a second head and shoulder recognition area whose coincidence index with the target face recognition area does not meet the first preset condition; combining the target face recognition area with the first head and shoulder recognition area to obtain a first area, and adding the second head and shoulder recognition area to the first area to obtain a second area; obtaining a historical image frame corresponding to the current image frame, wherein the historical image frame includes: a historical recognition area of ​​the tracking object; determining a predicted area of ​​the tracking object in the current image frame according to the historical recognition area, wherein the predicted area includes: a face recognition area and a head and shoulder recognition area; matching the second area with the predicted area to obtain a similarity index, determining a target area in the second area whose similarity index with the predicted area meets the second preset condition, and updating the tracking information of the predicted area according to the target area.

[0123] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0124] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0125] In the above-mentioned embodiments of the present application, the collected information is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary protection measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0126] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0127] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0128] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0129] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0130] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A portrait tracking method, characterized in that: include: Acquire a current image frame, and determine tracking information of the current image frame, wherein the current image frame includes: a tracking object, and the tracking information at least includes: an activity trajectory of the tracking object; Determine a target face recognition region and a target head and shoulder recognition region in the current image frame, determine a target face recognition region and a target head and shoulder recognition region whose overlap index satisfies a first preset condition, and obtain a first face recognition region and a first head and shoulder recognition region, determine a target face recognition region and a target head and shoulder recognition region whose overlap index does not satisfy the first preset condition, and obtain a second face recognition region and a second head and shoulder recognition region; Combine the first face recognition area and the first head and shoulder recognition area to obtain a first area, and add the second face recognition area and the second head and shoulder recognition area to the first area to obtain a second area; Acquire a historical image frame corresponding to the current image frame, wherein the historical image frame includes: a historical identification area of ​​the tracked object; Determine, according to the historical recognition area, a predicted area of ​​the tracked object in the current image frame, wherein the predicted area includes: a face recognition area and a head and shoulder recognition area; The second region is matched with the predicted region to obtain a similarity index, a target region whose similarity index with the predicted region satisfies a second preset condition is determined in the second region, and tracking information of the predicted region is updated according to the target region.

2. The method according to claim 1, characterized in that: Matching the predicted region with the second region to obtain a similarity index includes: Matching the first area and the second face recognition area in the second area with the face recognition area in the first prediction area, and matching the target head and shoulder recognition area in the second area with the head and shoulder recognition area in the first prediction area to obtain a first matching result, wherein the first prediction area is a prediction area in which the tracking information is in a non-lost state; Matching the second area in the first matching result where the matching fails with the first target area to obtain a second matching result, wherein the first target area includes: the second head and shoulder recognition area and the predicted area in the first matching result where the matching fails; Matching the first area and the second face recognition area in the second area where the matching fails in the second matching result with a second target area to obtain a third matching result, wherein the second target area includes: the predicted area where the matching fails in the second matching result and the predicted area where the tracking information is lost in the first matching result; Matching the second area where the matching fails in the third matching result with the second predicted area to obtain a fourth matching result, wherein the second predicted area is a predicted area where the matching fails in the third matching result and the duration of the tracking information being in a lost state does not exceed a preset duration, and the recognition quality index of the head and shoulder recognition area meets a third preset condition, and the recognition quality index of the head and shoulder recognition area is used to characterize the clarity of the head and shoulders of the tracked object; The similarity index is determined according to the first matching result, the second matching result, the third matching result, and the fourth matching result.

3. The method according to claim 2, characterized in that Matching the first area and the second face recognition area in the second area with the face recognition area in the first prediction area, and matching the target head and shoulder recognition area in the second area with the head and shoulder recognition area in the first prediction area to obtain a first matching result, including: Determine first facial features of the first area and the second facial recognition area in the second area, and determine second facial features of the facial recognition area; Determining a first Euclidean distance between the first facial feature and the second facial feature; determining a first intersection-over-union ratio between the first head-shoulder recognition region and the head-shoulder recognition region; Determine a first cost matrix according to the first Euclidean distance, the first intersection-and-union ratio, a first preset weight corresponding to the first Euclidean distance, and a second preset weight corresponding to the first intersection-and-union ratio; The first matching result is determined according to the first cost matrix and a preset matching algorithm, wherein the preset matching algorithm at least includes: a Hungarian matching algorithm.

4. The method according to claim 2, characterized in that: Matching the second area in the first matching result where the matching fails with the first target area to obtain a second matching result includes: Determine a second intersection-over-union ratio between a second region in the first matching result that fails to match and the first target region; Determining a second cost matrix according to the second intersection-to-union ratio; The second matching result is determined according to the second cost matrix and a preset matching algorithm, wherein the preset matching algorithm at least includes: a Hungarian matching algorithm.

5. The method according to claim 2, characterized in that: Matching the first area and the second face recognition area in the second area where the matching fails in the second matching result with the second target area to obtain a third matching result, including: Determine a third facial feature of the first area and the second facial recognition area in the second area where the match fails in the second matching result, and determine a fourth facial feature of the second target area; Determining a second Euclidean distance between the third facial feature and the fourth facial feature; determining a third cost matrix according to the second Euclidean distance; The third matching result is determined according to the third cost matrix and a preset matching algorithm, wherein the preset matching algorithm at least includes: a Hungarian matching algorithm.

6. The method according to claim 2, characterized in that Matching the second region in the third matching result where the matching fails with the second predicted region to obtain a fourth matching result includes: Determine a third intersection-over-union ratio between the second region that fails to match in the third matching result and the second predicted region; Determining a fourth cost matrix according to the third intersection-union ratio; The fourth matching result is determined according to the fourth cost matrix and a preset matching algorithm, wherein the preset matching algorithm at least includes: a Hungarian matching algorithm.

7. The method according to claim 2, characterized in that According to the target area, updating the tracking information of the predicted area includes: If the matching success rate of the first matching result is greater than the preset success rate, or the matching success rate of the second matching result is greater than the preset success rate, or the matching success rate of the third matching result is greater than the preset success rate, or the matching success rate of the fourth matching result is greater than the preset success rate, the target area is determined as the historical identification area corresponding to the current image frame, and the historical identification area corresponding to the current image frame is added to the tracking information of the predicted area.

8. The method according to claim 2, characterized in that: After determining the similarity index, the method further includes: If the matching success rate of the first matching result, the matching success rate of the second matching result, the matching success rate of the third matching result and the matching success rate of the fourth matching result are all less than the preset success rate, determine whether the recognition quality index of the face area and / or the recognition quality index of the head and shoulder area in the second area meets the fourth preset condition. If the recognition quality index of the face area and / or the recognition quality index of the head and shoulder area in the second area meets the fourth preset condition, establish tracking information corresponding to the second area, and add the tracking information corresponding to the second area to the tracking information set corresponding to the current image frame, wherein the tracking information set includes different tracking information corresponding to different tracking objects.

9. The method according to claim 8, characterized in that After determining the similarity index, the method further includes: If the matching success rate of the first matching result, the matching success rate of the second matching result, the matching success rate of the third matching result, and the matching success rate of the fourth matching result are all lower than a preset success rate, when the tracking object is an uncertain object, modifying the state of the tracking information of the predicted area to an ineffective state; If the matching success rate of the first matching result, the matching success rate of the second matching result, the matching success rate of the third matching result, and the matching success rate of the fourth matching result are all less than the preset success rate, when the tracking object is a valid tracking object, determining whether the target time length from the first moment to the zero moment is the preset time length, wherein the first moment is the moment when it is determined that the matching success rate of the first matching result, the matching success rate of the second matching result, the matching success rate of the third matching result, and the matching success rate of the fourth matching result are all less than the preset success rate, and the zero moment is the moment when the current image frame is acquired; If the target duration is the preset duration, the state of the tracking information of the predicted area is modified to a lost state; if the target duration is less than the preset duration, the state of the tracking information of the predicted area is modified to an early lost state.

10. A portrait tracking device, characterized in that: include: A first acquisition module is used to acquire a current image frame and determine tracking information of the current image frame, wherein the current image frame includes: a tracking object, and the tracking information at least includes: an activity trajectory of the tracking object; A first determination module is used to determine a target face recognition area and a target head and shoulder recognition area in the current image frame, determine a target face recognition area and a target head and shoulder recognition area whose overlap index satisfies a first preset condition, obtain a first face recognition area and a first head and shoulder recognition area, determine a target face recognition area and a target head and shoulder recognition area whose overlap index does not satisfy the first preset condition, and obtain a second face recognition area and a second head and shoulder recognition area; a combining module, configured to combine the first face recognition area and the first head and shoulder recognition area to obtain a first area, and add the second face recognition area and the second head and shoulder recognition area to the first area to obtain a second area; A second acquisition module is used to acquire a historical image frame corresponding to the current image frame, wherein the historical image frame includes: a historical identification area of ​​the tracked object; A second determination module is used to determine a predicted area of ​​the tracked object in the current image frame according to the historical recognition area, wherein the predicted area includes: a face recognition area and a head and shoulder recognition area; The matching module is used to match the second area with the predicted area to obtain a similarity index, determine a target area in the second area whose similarity index with the predicted area meets a second preset condition, and update the tracking information of the predicted area according to the target area.

11. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the portrait tracking method according to any one of claims 1 to 9.

12. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the portrait tracking method described in any one of claims 1 to 9 when running.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the portrait tracking method described in any one of claims 1 to 9 is implemented.

Images