Multi-target identification tracking method and system based on monitoring camera

Abstract

The invention relates to the technical field of video recognition, in particular to a multi-target recognition tracking method and system based on a monitoring camera, and solves the problems that in the prior art, cross-frame matching is carried out on a moving target by adopting a fixed computing power distribution strategy, the dynamic complexity change of a tracking scene cannot be adapted, and the tracking accuracy is poor. And the resource utilization efficiency is low. The method comprises the following steps: acquiring a video frame sequence; identifying a moving area, and dividing the video frame sequence into a moving time period and a static time period containing the moving area; for the motion time period, performing preliminary trajectory analysis according to the time sequence change of the moving area to determine the motion trend consistency of the moving area relative to the own historical motion trend and the overlapping degree between the moving areas, and further determining an identification demand parameter for evaluating the complexity of the tracking scene; and allocating corresponding computing resources according to the identification demand parameters, performing cross-frame identification and tracking on the moving area, and outputting a tracking identification result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video recognition, in particular to a multi-target recognition and tracking method and system based on a surveillance camera. BACKGROUND

[0002] In the fields of public safety and traffic management, surveillance cameras have become the core security monitoring equipment and are widely used in public scenes such as traffic intersections and streets. They are usually installed in fixed positions to capture images in the monitoring range in real time, realize full coverage monitoring of various moving objects, and provide basic data support for subsequent analysis of target moving track and public safety protection.

[0003] In order to determine the moving path of the moving object, the existing multi-target recognition and tracking method generally adopts the idea of precise matching, that is, by extracting the shape similarity and the consistency of the category of the moving object, the moving objects captured at different times are correspondingly associated, and then the moving track of each object is sorted based on the association result to complete the tracking and recognition task.

[0004] The existing multi-target tracking scheme mainly focuses on improving the matching accuracy of a single algorithm in complex scenes or using a fixed calculation framework to process all video frames. Such methods are difficult to adapt to the differentiated processing needs at different times: in simple scenes where the target is clear and the motion is regular, the fixed algorithm mode will cause idle and waste of computing resources; while in complex scenes where the target is severely occluded and the motion is chaotic, the instantaneous computing power may be insufficient, resulting in tracking loss or errors. Therefore, how to make the allocation of system computing resources adaptive to the dynamic changes of the scene, so as to optimize the overall computing efficiency while ensuring tracking accuracy, is a technical problem to be solved. SUMMARY

[0005] In order to solve the technical problem of low resource utilization efficiency caused by the fixed computing power allocation strategy for cross-frame matching of moving targets in the prior art, which cannot adapt to the dynamic complexity changes of the tracking scene, the purpose of the present application is to provide a multi-target recognition and tracking method and system based on a surveillance camera, and the technical solution adopted is as follows: In a first aspect, a multi-target recognition and tracking method based on a monitoring camera is provided, which includes: acquiring a video frame sequence collected by the monitoring camera; recognizing a moving area in the video frame sequence, and dividing the video frame sequence into a motion period containing the moving area and a static period not containing the moving area; for the motion period, performing preliminary trajectory analysis according to the time sequence change of the moving area, determining the motion trend consistency of the moving area relative to its historical motion trend at each time, and the overlapping degree between the moving areas, and combining the motion trend consistency and the overlapping degree to determine an identification demand parameter for evaluating the tracking scene complexity at each time; according to the identification demand parameter at each time, assigning corresponding computing resources to each time, and calling the assigned computing resources to perform cross-frame identification and tracking on the moving area, and outputting a tracking and identification result of the moving area.

[0006] Based on the above technical solution, in the multi-target recognition and tracking method based on the monitoring camera provided by the application, the video frame sequence collected by the monitoring camera is acquired, the moving area is accurately recognized and the motion period and the static period are divided, the identification demand parameter corresponding to the tracking scene complexity is determined in combination with the consistency of the moving area relative to its historical motion trend in the motion period and the overlapping degree between the areas, and then the parameter is used to assign computing resources on demand and perform cross-frame identification and tracking, which not only ensures the accuracy of multi-target tracking in a complex scene with overlapping moving areas and unstable motion trend, but also avoids waste of computing resources in a clear moving path scene, and effectively improves the overall efficiency of multi-target recognition and tracking and the rationality of resource utilization.

[0007] In addition, the motion trend and the area overlapping degree are simultaneously evaluated by a preliminary trajectory analysis process which is integrated and has low overhead, independent and repeated computing power overhead for obtaining the two types of information is avoided, and computing power is dynamically allocated accordingly, so that the application realizes accurate adaptation to changes in scene complexity and net saving of overall system computing resources under the premise of significantly reducing the cost of decision-making itself.

[0008] In combination with the first aspect, in a possible implementation manner, the method of recognizing the moving area in the video frame sequence and dividing the video frame sequence into the motion period containing the moving area and the static period not containing the moving area, specifically includes: performing difference processing on adjacent frames in the video frame sequence to detect moving pixel points in each frame of image; performing connected domain analysis on the moving pixel points in each frame of image to form the moving area; dividing video frames in the video frame sequence which are continuous and do not contain the moving area into a static period; and dividing video frames in the video frame sequence which are continuous and contain the moving area into a motion period.

[0009] With the first aspect above, in a possible implementation, the method for determining the motion trend consistency of the moving region at each time according to the time sequence change of the moving region, for the motion period, specifically comprises: fitting a reference moving path according to the positions of the center points of the moving region in multiple historical continuous frames that have a preliminary matching relationship; and determining the motion trend consistency by analyzing the deviation between the actual motion state and the predicted motion state of the moving region at each time according to the reference moving path.

[0010] With the first aspect above, in a possible implementation, the method for determining the overlap degree between the moving regions, specifically comprises: determining the moving region at the current time as an overlap region if the moving region at the current time and the multiple moving regions at the previous time all have a preliminary matching relationship; and determining the overlap degree parameter according to the ratio of the number of the overlap regions at the current time to the total number of the moving regions, and the ratio of the area of the overlap regions to the total area of the moving regions.

[0011] With the first aspect above, in a possible implementation, the method for determining the identification demand parameter for evaluating the tracking scene complexity at each time according to the motion trend consistency and the overlap degree, specifically comprises: determining the identification demand parameter according to the motion trend consistency and the overlap degree parameter for each time.

[0012] With the first aspect above, in a possible implementation, the method for allocating the corresponding computing resource for each time according to the identification demand parameter at each time, specifically comprises: determining an algorithm power allocation priority parameter for each time according to the identification demand parameter at each time; the tracking scene complexity represented by the identification demand parameter is higher, and the algorithm power allocation priority parameter is higher; and allocating the computing resource for the corresponding time in the order from high to low according to the algorithm power allocation priority parameter.

[0013] With the first aspect above, in a possible implementation, the method for calling the allocated computing resource to perform the cross-frame identification and tracking of the moving region, specifically comprises: performing target matching of the moving region according to the motion trend continuity of the moving region if the algorithm power allocation priority parameter is lower than a preset priority threshold; performing target matching of the moving region according to the feature point matching of the moving region if the algorithm power allocation priority parameter is higher than or equal to the preset priority threshold; and the computing resource required for the target matching according to the feature point matching is more than the computing resource required for the target matching according to the motion trend continuity.

[0014] In a possible implementation manner of the first aspect, the method of obtaining the video frame sequence captured by the monitoring camera specifically comprises: receiving, by an edge computing node deployed in a position adjacent to the monitoring camera, an original video stream captured by the monitoring camera; and pre-processing the original video stream to obtain the video frame sequence.

[0015] In a possible implementation manner of the first aspect, the method of outputting the tracking and identification result of the moving area specifically comprises: labeling an external figure for each identified moving area in the video frame; and associating the external figures of the moving areas determined as the same object in different video frames with the same identifier.

[0016] In a possible implementation manner of the first aspect, the method of outputting the tracking and identification result of the moving area specifically comprises: labeling an external figure for each identified moving area in the video frame; and associating the external figures of the moving areas determined as the same object in different video frames with the same identifier.

[0017] The present application has the following advantages: By obtaining the video frame sequence captured by the monitoring camera, the moving area is accurately identified and the motion period and the static period are divided, the identification demand parameter corresponding to the tracking scene complexity is determined in combination with the consistency of the moving area in the motion period relative to the historical motion trend and the overlapping degree between the areas, and then the calculation resource is allocated on demand and the cross-frame identification and tracking are carried out based on the parameter, which not only guarantees the accuracy of multi-target tracking in the complex scene with overlapping moving areas and unstable motion trend, but also avoids the waste of calculation resource in the clear moving path scene, and effectively improves the overall efficiency and resource utilization rationality of multi-target identification and tracking.

[0018] In addition, the motion trend and the area overlapping degree are synchronously evaluated by a integrated and low-overhead preliminary trajectory analysis process, the independent and repeated calculation resource overhead for obtaining the two types of information is avoided, and the calculation resource is dynamically allocated accordingly, so that the accurate adaptation to the scene complexity change and the net saving of the overall calculation resource of the system are realized on the premise of significantly reducing the cost of the decision itself.BRIEF DESCRIPTION OF DRAWINGS BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings required to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1 A system structure diagram of a multi-target recognition tracking system based on a monitoring camera is provided for an embodiment of the present application. Figure 2 A method flowchart of a multi-target recognition tracking method based on a monitoring camera is provided for an embodiment of the present application. Figure 3 A hardware structure schematic diagram of a multi-target recognition tracking device based on a monitoring camera is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined object, the specific embodiments, structures, features and effects of the multi-target recognition tracking method and system based on a monitoring camera according to the present application are described in detail as follows. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0023] The specific scheme of the multi-target recognition tracking method and system based on a monitoring camera provided by the present application is specifically described below with reference to the drawings.

[0024] Please refer to Figure 1 which shows a system structure diagram of a multi-target recognition tracking system based on a monitoring camera provided for an embodiment of the present application. The multi-target recognition tracking system based on a monitoring camera includes a data acquisition module 1, a time period division module 2, a demand evaluation module 3 and a computing power allocation and execution module 4.

[0025] The data acquisition module 1 is the data input core of the system, and is implemented by relying on the edge computing node deployed in the vicinity of the monitoring camera. The output video frame sequence is directly transmitted to the time period division module 2, thereby providing a high-quality data basis for subsequent mobile area identification and time period division. The data acquisition module 1 can include a video receiving submodule 11 and a video preprocessing submodule 12. The video receiving submodule 11 receives the original video stream collected by the monitoring camera in real time through the Ethernet interface, wireless communication module and other entity communication devices configured by the edge computing node, has a data transmission verification function, can filter abnormal data such as packet loss and disorder in the transmission process, and ensures the integrity of the original video stream.

[0026] The video preprocessing submodule 12 is borne by the embedded processor of the edge computing node, extracts single frames from the received original video stream, and then performs grayscale, image equalization and denoising processing, thereby eliminating the interference of light changes and image noise on subsequent identification, and finally generating a standardized video frame sequence.

[0027] The time period division module 2 is the scene classification core of the system, relies on the special image processing unit of the edge computing node to implement data processing, and transmits the output motion period data to the demand assessment module 3 to provide a scene basis for subsequent tracking scene complexity assessment; the static period information is directly sent to the computing power allocation and execution module 4, which is used for quickly outputting the tracking result of no moving object. The time period division module 2 can include a moving pixel point detection submodule 21, a mobile area formation submodule 22 and a time period division submodule 23. The moving pixel point detection submodule 21 performs difference processing on adjacent frames in the video frame sequence output by the data acquisition module 1, detects the moving pixel points with position changes in each frame of image by calculating the grayscale difference of the corresponding pixel points of the two frames.

[0028] The mobile area formation submodule 22 performs connected domain analysis on the detected moving pixel points, aggregates the spatially continuous moving pixel points to form an initial area, and then fills the holes in the initial area through morphological closing operation, while judging whether there are other moving pixel points in the neighborhood of each pixel point to exclude isolated points, and finally forms a complete mobile area.

[0029] The time period division submodule 23 judges the video frame sequence frame by frame, divides the video frames that are continuous and do not contain mobile areas into a static period, and divides the video frames that are continuous and contain mobile areas into a motion period.

[0030] The demand evaluation module 3 is the basis for the calculation power allocation of the system, carried by the data analysis unit of the edge computing node, and the identified demand parameters output are directly transmitted to the calculation power allocation and execution module 4, which becomes the core basis for the selection of calculation resource allocation and tracking mode. The demand evaluation module 3 can include a preliminary trajectory fitting sub-module 31, a motion trend consistency analysis sub-module 32, an overlap degree parameter calculation sub-module 33 and an identified demand parameter determination sub-module 34: The preliminary trajectory fitting sub-module 31 extracts the center point position of the mobile area in multiple continuous frames that has a preliminary matching relationship for the motion period output by the time period division module 2, and obtains the reference moving path of the mobile area through a straight line fitting algorithm.

[0031] The motion trend consistency analysis sub-module 32 calculates the deviation between the actual motion state and the predicted motion state of the mobile area at each time, determines the motion trend consistency of the mobile area relative to its historical motion trend according to the deviation size, and the smaller the deviation, the higher the motion trend consistency.

[0032] The overlap degree parameter calculation sub-module 33 judges whether the mobile area at the current time has a preliminary matching relationship with multiple mobile areas at the previous time for each time image, determines the mobile area that meets the condition as an overlapping area, and then calculates the overlap degree parameter through the ratio of the number of overlapping areas to the total number of mobile areas and the ratio of the area of the overlapping area to the total area of the mobile area. The parameter is positively correlated with the overlap number and the overlap area.

[0033] The identified demand parameter determination sub-module 34 integrates the motion trend consistency and the overlap degree parameter at each time, determines the identified demand parameter for evaluating the tracking scene complexity at the time through a preset comprehensive evaluation logic, and the more complex the tracking scene, the higher the identified demand parameter.

[0034] The calculation power allocation and execution module 4 is the tracking execution core of the system, which is realized by relying on the calculation power scheduling unit, image processing unit and output interface of the edge computing node. The calculation power allocation and execution module 4 can include a calculation power priority determination sub-module 41, a calculation resource allocation sub-module 42, a cross-frame tracking sub-module 43 and a tracking result output sub-module 44: The calculation power priority determination sub-module 41 receives the identified demand parameter output by the demand evaluation module 3, determines the calculation power allocation priority parameter at each time according to the parameter size, and the higher the tracking scene complexity represented by the identified demand parameter, the higher the corresponding calculation power allocation priority parameter.

[0035] The calculation resource allocation sub-module 42 allocates corresponding calculation resources for each time of the motion period in the order from high to low according to the calculation power allocation priority parameter, and the high-priority time obtains more processor operation resources and memory resources to ensure tracking accuracy.

[0036] The cross-frame tracking sub-module 43 performs differentiated tracking for different scenarios: for the stationary period output by the period division module 2, it is directly determined that there is no moving object and the corresponding tracking result is generated; for the time when the calculation power allocation priority parameter is lower than the preset priority threshold in the motion period, target matching is performed according to the motion trend continuity of the moving area; for the time when the calculation power allocation priority parameter is higher than or equal to the preset priority threshold, target matching is performed according to the feature point matching of the moving area, and the required computing resources of the latter are more than those of the former.

[0037] The tracking result output sub-module 44 labels the bounding graph for each identified moving area in the video frame through the display interface, storage interface or network transmission interface of the edge computing node, and associates the same identifier to the bounding graphs of the moving areas determined as the same object in different video frames, and finally outputs the visual tracking identification result.

[0038] Please refer to Figure 2 , which shows a method flowchart of a multi-target identification and tracking method based on a monitoring camera provided by an embodiment of the application. The multi-target identification and tracking method based on a monitoring camera comprises: S1, acquiring a video frame sequence collected by a monitoring camera.

[0039] In some implementations, first, the edge computing node deployed in the vicinity of the monitoring camera receives the original video stream collected by the monitoring camera. During the receiving process, the edge computing node performs integrity check on each frame of data, and if the check result is consistent with the sending end, it confirms the reception, and if it is not consistent, it triggers a retransmission request, avoiding data distortion caused by transmission abnormalities and ensuring the reliability of the original data.

[0040] Then, the original video stream is preprocessed. Specifically, the received original video stream is single-frame extracted, and then grayscale, image equalization and denoising processing are performed to eliminate the interference of light changes and image noise on subsequent identification, and finally a standardized video frame sequence is generated.

[0041] S2, identifying the moving area in the video frame sequence, and dividing the video frame sequence into a motion period containing the moving area and a stationary period not containing the moving area.

[0042] In some implementations, the method of identifying the moving area in the video frame sequence can be realized by the following steps: First, the adjacent frames in the video frame sequence are differentially processed to detect the moving pixel points in each frame of image. Specifically, for two adjacent frames, the gray scale diagram corresponding to the frame with larger time sequence is subtracted from the gray scale diagram corresponding to the frame with smaller time sequence to obtain the gray scale difference value of each pixel point between the two adjacent frames. Then, the mean value of the gray scale difference values of all pixel points of the two frames is calculated to correct the overall gray scale offset caused by light changes; then, through absolute value operation and normalization mapping, the gray scale difference deviation degree (i.e. the deviation of the gray scale difference value of the pixel from the overall mean value) of each pixel point is obtained, which is represented as: In the formula, represents the gray scale difference value of the i-th pixel point between the t-th frame and the t-1-th frame; represents the mean value of the gray scale difference values of all pixel points between the t-th frame and the t-1-th frame; represents the gray scale offset degree after eliminating the light interference (taking the absolute value to ignore the offset direction); the offset degree is normalized to the [0, 1] interval through the norm function to obtain the gray scale difference deviation degree of the i-th pixel point in the t-th frame .

[0043] The normalization function norm(*) mentioned in the embodiments of the present application adopts maximum-minimum value normalization unless otherwise specified. Among them, the maximum and minimum values are preset empirical extreme values based on a large amount of historical experimental data. If the calculation result exceeds the [0, 1] interval, it is limited in the [0, 1] range through the truncation function (i.e. if the result is less than 0, 0 is taken, and if the result is greater than 1, 1 is taken), so as to eliminate the influence of abnormal values on the result.

[0044] In particular, if the absolute values of the gray scale offsets of all pixels are the same, the denominator (maximum value-minimum value) of the normalization function is zero, and the gray scale difference deviation degree of the pixel point is directly defined as zero.

[0045] Then, a preset determination threshold such as 0.5 is used to mark the pixel points with a normalized result greater than 0.5 as the moving pixel points of the current frame. Among them, the determination threshold of the moving pixel points is determined by combining the frame rate of the monitoring scene, the conventional speed of the moving object and the conversion relationship between the pixel and the actual distance, calculating the pixel number corresponding to the reasonable actual displacement of the moving object between the adjacent frames, and then calibrating through the interframe difference experiment of the actual scene.

[0046] Then, the moving pixel points in each frame of image are analyzed by connected domain analysis, and the spatially continuous moving pixel points are aggregated into initial regions, and morphological closing operation (for example, the preset operation kernel size is 3*3) is used to fill the holes in the initial regions and exclude isolated points (i.e. the pixels without other moving pixel points in the 8-neighborhood) to filter scattered noise points, thereby forming the moving region.

[0047] Finally, consecutive video frames in the video frame sequence that do not contain moving areas are divided into a static time period; consecutive video frames in the video frame sequence that contain moving areas are divided into a moving time period.

[0048] S3. For the movement period, perform preliminary trajectory analysis based on the temporal changes of the moving area to determine the consistency of the movement trend of the moving area relative to its own historical movement trend at each moment, as well as the degree of overlap between moving areas. Combine the consistency of movement trend and the degree of overlap to determine the recognition requirement parameters used to evaluate the complexity of the tracking scenario at each moment.

[0049] In some implementations, the method for determining the consistency of the movement trend of the moving region at each moment relative to its own historical movement trend can be specifically implemented through the following steps S31 to S32, which are explained in detail below: S31. Based on the position of the center point where the moving region has a preliminary matching relationship in multiple consecutive frames, a reference moving path is fitted to obtain the reference moving path.

[0050] In some implementations, firstly, for a single moving region within the motion period, consecutive frames (such as 10-20 frames) within a preset sliding time window before the current moment are selected as the target segment. This stage fits the dynamic scene requirements of multi-target tracking, and the reference path fitted based on this can more accurately reflect the preceding motion characteristics at the current moment.

[0051] The center point is the spatial center of the moving region, possessing uniqueness and stability. Regardless of whether the region is a regular vehicle rectangle or an irregular pedestrian outline, a unique center coordinate can be obtained through a unified algorithm (the midpoint of the circumscribed rectangle or the centroid), accurately representing the spatial position of the entire region. When extracting the center point coordinates of a moving region, it is necessary to identify the boundary pixel coordinates of the region (connected component) after morphological closing operations have been completed and isolated points have been excluded, using Canny edge detection combined with contour tracking algorithms to define the spatial range of the moving region. If the region is close to a regular shape such as a rectangle, its smallest circumscribed rectangle is calculated, and the intersection of the horizontal and vertical midlines of the rectangle is taken as the center point coordinates. If it is an irregular shape such as a pedestrian, the x and y coordinates of all pixels within the region are statistically analyzed, and the average of the two types of coordinates is calculated. The combination of the "average x and y coordinates" is used as the center point coordinates. It is verified whether the pixel corresponding to the center point belongs to the current moving region. If it is not within the region, the edge detection threshold is adjusted, the contour is re-extracted, and the coordinates are recalculated to ensure that the center point is located inside the moving region, thus obtaining accurate center point coordinates that can support subsequent path analysis and inter-frame matching.

[0052] Next, frame-by-frame matching is performed on the center points of the moving regions within the target segment: the Euclidean distance between the center points of moving regions with similar contour sizes in adjacent frames is calculated (for example, the ratio of the areas of two regions is calculated, and if the ratio is within the range of [0.7, 1.3], they are considered similar, and this range is related to the distance and angle between the surveillance camera and the moving object). Two moving regions with a distance less than the distance threshold are determined as the matching regions of the moving regions in consecutive frames. The distance threshold is set based on the video frame interval and the normal speed of the moving object. For example, when existing image recognition technology identifies a pedestrian as the moving area, taking the common 25fps frame rate of surveillance video as an example, the video frame interval is 0.04 seconds, and the normal walking speed of a pedestrian is about 1.2m / s. Therefore, the actual moving distance of the pedestrian between adjacent frames is 1.2 × 0.04 = 0.048m. Combining the conversion relationship between the pixels of the surveillance image and the actual distance (assuming 1920 pixels corresponds to an actual intersection width of 20 meters, i.e., 1 pixel ≈ 0.0104 meters), 4.8 centimeters corresponds to about 4.6 pixels. Therefore, the distance threshold in this scenario can be preset to 5 pixels. Similarly, when the recognition technology identifies a vehicle as the moving area, considering that the vehicle's speed at the intersection is about 5m / s, the distance threshold in this scenario can be preset to 20 pixels.

[0053] Then, the initial motion type is predicted: the motion vector between consecutive frames within the target segment is calculated (obtained by subtracting the coordinates of the center point of the previous frame from the coordinates of the matching center point of the next frame), and the direction angle of each motion vector is calculated (for example, with 0° to the right horizontally, the angle is counted counterclockwise). The maximum difference in direction angle between the corresponding motion vectors of two consecutive adjacent frames within the target segment is calculated, with a preset angle threshold of 15°. If the maximum difference is less than 15°, the motion trend within the target segment is determined to be linear; if it is greater than 15°, the motion trend is determined to be curvilinear (such as the turning motion at an intersection corner), thus avoiding the limitations of a single linear fitting and adapting to motion changes in the actual scene.

[0054] Finally, a reference path is fitted to suit the motion type: if the motion trend is linear, the least squares linear regression algorithm is used to fit the coordinates of the center point within the target segment to obtain a linear reference movement path; if the motion trend is curvilinear, a preset quadratic polynomial regression algorithm is used (fitting formula y=ax). 2 The center point coordinates are fitted with a curve using +bx+c to obtain a curved reference movement path. This makes the reference movement path more closely match the actual initial movement trajectory of the movement area, thus improving the accuracy of subsequent movement trend consistency judgment.

[0055] S32. Analyze the deviation between the actual motion state and the predicted motion state of the moving area at each moment based on the reference moving path, and determine the consistency of the motion trend.

[0056] In some implementations, firstly, for the moving area during the movement period, the coordinates of its center point at each moment are extracted as the measured position at the current moment. Simultaneously, the reference movement path of the moving area is used to determine the reference position corresponding to the current moment. Using a point-to-point distance formula, the distance between the measured position coordinates of the moving area at the current moment and the reference position coordinates is calculated. This distance represents the positional deviation between the measured position and the reference movement path, quantifying the degree of deviation between the current movement position and the initial movement trend.

[0057] Secondly, for the moving area within the movement period, the actual movement vector of its center point coordinates at each moment compared to the center point coordinates at the previous moment is extracted, and the direction of the vector is used as the actual movement direction at the current moment. Simultaneously, the reference movement path of the moving area is invoked, and the reference movement vector of the reference position at the current moment compared to the reference position at the previous moment is calculated. The direction of this vector is used as the reference movement direction for the current moment based on historical trend prediction. A vector angle cosine algorithm is employed to calculate the cosine value of the angle between the actual movement direction and the reference movement direction at the current moment, used to quantify the degree of deviation between the current movement direction and the initial movement trend.

[0058] Thirdly, the magnitude of the actual movement vector can characterize the actual movement speed of the moving area at the current moment, while the magnitude of the reference movement vector can characterize the reference movement speed of the moving area at the current moment, predicted based on historical trends. The ratio of the change in actual movement speed to reference movement speed (i.e., the absolute value of the difference between actual and reference movement speed divided by the reference movement speed) is calculated to quantify the degree of deviation between the current movement speed and the initial movement trend.

[0059] Combining the deviations in the above three aspects, the consistency of the motion trend between the moving region and the initially matched moving regions in consecutive frames is comprehensively determined and expressed as: In the formula, Indicates the first The distance between the measured location coordinates of each moving area and the reference location coordinates; The distance deviation normalization coefficient (a non-zero constant) is determined based on the environmental activity range of the moving object; for example, when the moving object is a pedestrian. It can be set to the pixel distance calculated from the width of the sidewalk in the current scene; It is the inverse normalized term of the positional deviation. The smaller the value (the closer the actual position is to the reference position), the larger this value becomes, and the closer it is to 0, thus achieving consistency between distance deviation and movement trend. Negative correlation; Indicates the first The actual movement vector of each moving region Represents the magnitude of the actual movement vector; Indicates the first Reference movement vector for each moving region This represents the magnitude of the reference movement vector (since it is a trajectory prediction of a moving object, the magnitude of the reference movement vector is not zero). This represents the cosine of the angle between the actual direction of movement and the reference direction of movement, ranging from -1 to 1. The closer the actual and reference directions of movement are to each other, the closer this value is to 1, reflecting consistency with the trend of movement. Positive correlation; The term represents the opposite of the rate of change in velocity. The smaller the velocity deviation, the larger this term is, and the closer it is to zero, thus achieving consistency between the velocity deviation and the motion trend. The negative correlation.

[0060] By comprehensively assessing the movement trend from three core dimensions—position, direction, and speed—we can avoid the one-sidedness of a single dimension and more accurately reflect the consistency of the movement trend.

[0061] The above formula is a quantitative example of the consistency of motion trends. In other embodiments, different weighting coefficients can be assigned to the distance deviation, direction deviation, and velocity deviation before fusion.

[0062] Furthermore, if multiple suspected matching regions exist in consecutive frames during the initial matching process, path fitting and motion trend consistency calculation are performed for each matching region, and the suspected matching region with the greatest motion trend consistency is determined as the matching region of the moving region.

[0063] In some implementations, the method for determining the degree of overlap between moving regions can be specifically implemented through the following steps S33 to S34, which are explained in detail below: S33. If the current moving region has a preliminary matching relationship with multiple moving regions in the previous time, the moving region at the current time is determined to be an overlapping region.

[0064] In some implementations, for each moving region at the current moment, the Euclidean distance between its center point coordinates and the center point coordinates of all moving regions (regardless of region outline size) at the previous moment is calculated. The distance threshold in S31 is called. If the Euclidean distance between the center points of multiple moving regions at the previous moment and one moving region at the current moment is less than the distance threshold, it is considered that the region has a preliminary matching relationship with multiple moving regions at the previous moment and is determined to be an overlapping region.

[0065] S34. Determine the degree of overlap parameters based on the ratio of the number of overlapping regions to the total number of moving regions at the current moment, and the ratio of the area of ​​overlapping regions to the total area of ​​moving regions.

[0066] In some implementations, the total number of moving areas and the number of overlapping areas at the current moment are counted, and the ratio of the number of overlapping areas to the total number of moving areas is calculated.

[0067] Meanwhile, the region boundary pixels are extracted by contour detection algorithm, and the number of pixels within the boundary is counted as the region area. The region area values ​​of all moving regions and all overlapping regions are summed to calculate the ratio of the total area of ​​the overlapping region to the total area of ​​the moving region.

[0068] Then, the overlap parameter is determined, expressed as: In the formula, This represents the number of overlapping regions at time t. This represents the number of regions moved at time t. This represents the proportion of overlapping regions. The larger the ratio, the more overlapping regions exist at the current time. It is also a parameter related to the degree of overlap with time t. Positive correlation; This represents the area representation value of the overlapping region at time t. This represents the area of ​​the moved region at time t. This represents the area ratio of the overlapping region. The larger the ratio, the wider the spatial extent of the overlapping region at the current time. It is also a parameter related to the degree of overlap with the region at time t. Positive correlation; The product accurately reflects the superposition effect of a large number of overlapping regions and a large proportion of overlapping area. Only when both the number of overlapping regions and the proportion of overlapping area are satisfied will the confusion risk of multi-target tracking increase significantly (e.g., multiple vehicles overlapping at an intersection at the same time, and the overlapping area covers the main features of the vehicles). If only one dimension is satisfied (e.g., a small number of regions but a large area or multiple regions but a small area), the increase in tracking difficulty is far less than the situation where both are satisfied at the same time. Finally, the overlap parameter at time t is obtained. The value range is [0, 1]. It is a quantitative value of the degree of overlap of the moving areas at the current time. The larger the value, the more overlapping areas there are and the wider the area.

[0069] Furthermore, the method for determining the identification requirements parameters can be specifically implemented through the following S35, which will be explained in detail below: S35. For each moment, determine the identification requirements parameters comprehensively based on the parameters of consistency and overlap of motion trends.

[0070] In some implementations, the identification requirement parameters are represented as: In the formula, This indicates the end of the motion period at time t. The mean of the consistency of movement trends in each moving region; The inverse term of the mean representing the consistency of motion trends indicates long-term path deviation, avoiding misjudgment due to short-term fluctuations. In regions of long-term instability (even if the deviation is small at the current moment), subsequent frames are more prone to loss of tracking or confusion, requiring higher recognition accuracy. The inverse term of the mean representing the consistency of motion trends is mapped to the [0, 1] interval using the normalization function norm(*), thus avoiding the inverse term representing the consistency of motion trends. With overlap parameter The uneven influence of parameters due to the inconsistent magnitudes; and These are the weights for the parameters of consistency and overlap in movement trends, respectively. Specifically, the calibration is based on the actual impact of overlap and motion complexity in the scene. For example, in a general scene, let... =0.5、 =0.5, balancing the influence of both; if the scene mainly involves complex single-target motion (such as a park or pedestrian street), set... =0.3、 =0.7, which enhances the contribution of motion complexity.

[0071] The normalized result is compared with the overlap parameter. We can obtain the weighted summation at time t. Parameters for identifying a mobile area It integrates the consistency of long-term movement trends with the current regional overlap, and the higher the value, the higher the demand.

[0072] S4. Based on the recognition requirement parameters at each time moment, allocate corresponding computing resources for each time moment, and call the allocated computing resources to perform cross-frame recognition and tracking of the moving area, and output the tracking and recognition results of the moving area.

[0073] In some implementations, the method for allocating computing resources includes: first, determining the computing power allocation priority parameter for each time step based on the identification requirement parameters at each time step. The higher the complexity of the tracking scenario represented by the identification requirement parameters, the higher the computing power allocation priority parameter. Then, computing resources are allocated to the corresponding time steps in descending order of the computing power allocation priority parameters.

[0074] Firstly, the system iterates through all moments within the current motion period. For each moving region at each moment, its recognition requirement parameters are calculated individually. If at least one region at a given moment has a recognition requirement parameter greater than a preset high recognition requirement threshold, and the proportion of regions meeting the threshold is greater than or equal to a preset percentage (e.g., 30%), then that moment is considered a high recognition requirement moment. Then, consecutive high recognition requirement moments are merged into a single test period, clustering moments in high-complexity scenarios to avoid the inefficiency of fragmented computing power allocation and improve resource targeting. The preset high recognition requirement threshold for merging test periods is determined by statistically analyzing the tracking complexity and matching failure rate corresponding to historical recognition requirement parameter intervals, combined with the constraint of the proportion of high-requirement periods supported by computing power resources. This initial value is then verified and fine-tuned through actual scenario operation (e.g., 0.5). The duration of the test period to which the current moment belongs is recorded, and the proportion of the test period's duration in the total duration of the current motion period is calculated. A higher proportion indicates a stronger persistence of the high-complexity scenario, which is positively correlated with the priority of computing power allocation.

[0075] Secondly, for all moving regions within the test period to which the current moment belongs, the area difference between adjacent moments is calculated (the area of ​​the next moment minus the area of ​​the previous moment). The standard deviation and mean of the area differences of all regions are compared to obtain the coefficient of variation, which is used to represent the degree of disorder of morphological changes within the test period. The larger the coefficient of variation, the higher the risk of feature confusion during tracking, and the higher the corresponding computing power requirement.

[0076] Thirdly, extract the motion vector magnitude of all moving regions at the current moment (representing the motion speed of each region), take the average of the magnitudes to obtain the average motion speed at the current moment. The higher the speed, the faster the inter-frame feature changes, the greater the tracking difficulty, and the more computing power is required.

[0077] By integrating the above three aspects with the identification requirement parameters at the current moment, the computing power allocation priority parameters at the current moment are determined, as follows: In the formula, This represents the maximum value among the identification requirements parameters for the moving region at time t, accurately capturing the worst-case scenario within that time and ensuring that highly complex regions receive sufficient computing power. This represents the duration of the time period to be measured at time t. The duration from the start of the current motion period to time t. The proportion in; This represents the standard deviation of the area difference among all regions at time t. This represents the average area difference of all regions at time t; For values ​​to be taken as minimum values ​​(e.g., 10) -6The parameter is used to avoid abnormal results caused by a denominator of zero; The coefficient of variation represents the area difference of all regions at time t; This represents the average moving speed of all moving areas at time t, in pixels per frame. This represents a normalization constant with velocity dimensions, which can be set to 1 pixel / frame, and is used to eliminate the velocity dimension.

[0078] By integrating multi-dimensional complexity features, the one-sidedness of single-dimensional evaluation is avoided, providing a more accurate priority basis for computing power allocation. Then, the computing power allocation priority parameter at time t is obtained by mapping to the [0,1] interval through the normalization function norm(*).

[0079] The above formula is a quantitative example of the priority parameters for computing power allocation. In other embodiments, different weighting coefficients can be assigned to the three factors—duration percentage, morphological fluctuation, and movement speed—before they are fused.

[0080] In some implementations, a preset priority threshold is set based on the upper limit of computing power resources in the actual scenario and the target matching accuracy requirements. Specifically, this includes: first, statistically analyzing the distribution of computing power allocation priority parameters for multiple consecutive frames in the current monitoring scenario, and simultaneously recording the success rate of motion trend continuity matching and the proportion of computing power consumption for feature point matching under different parameter ranges; then, based on the standard that "the motion trend matching accuracy of low computing power meets the standard (e.g., success rate ≥ 85%) and the proportion of feature point matching scenarios with high computing power does not exceed the upper limit of computing power resources (e.g., 30%)", the corresponding computing power priority threshold is selected as the initial threshold; finally, the implementation is verified in the actual scenario, and the threshold is fine-tuned based on the matching failure rate in the low threshold range and the computing power occupancy rate in the high threshold range until the matching accuracy and computing power consumption reach a long-term balance.

[0081] If the computing power allocation priority parameter is lower than the preset priority threshold, the target matching of the moving area is performed based on the continuity of the movement trend of the moving area. Specifically, this includes: associating and matching the area with the continuous movement trend at the current moment with the area with the same movement trajectory at the previous moment. Matching is completed only by relying on motion features, and the computing power consumption is lower than that of feature point matching, which is suitable for scenarios with low complexity and low computing power requirements.

[0082] If the computing power allocation priority parameter is higher than or equal to the preset priority threshold, target matching of the moving region is performed based on feature point matching of the moving region. Specifically, this includes retrieving the initial number of matching feature points corresponding to the moving region at the current moment, which is the result of a computer vision algorithm combining an accelerated segment test (FAST) feature detector and a binary robust independent elementary features (BRIEF) descriptor (oriented FAST and rotated BRIEF). (If using the default values ​​from the standard algorithm library), priority parameters are allocated based on computing power. The number of feature points is dynamically adjusted based on the tracking difficulty of the region. , represented as: Based on the adjusted number of feature points The ORB algorithm is used to extract feature points of the current and previous time regions for region matching to assist in target tracking. This can improve matching accuracy and is suitable for highly complex scenarios with many overlapping regions and unstable motion. However, the computational resources required are greater than those for motion trend continuity matching.

[0083] In some implementations, a bounding graphic is labeled for each identified moving region in the video frame. Then, the bounding graphics of moving regions identified as the same object in different video frames are associated with the same identifier to generate the tracking and recognition results of the moving regions.

[0084] Based on the above technical solution, by acquiring video frame sequences captured by surveillance cameras, the moving areas are accurately identified and the moving and stationary periods are divided. The consistency of the moving areas within the moving period relative to their historical movement trends and the degree of overlap between areas are combined to determine the recognition requirement parameters corresponding to the complexity of the tracking scenario. Then, based on these parameters, computing resources are allocated as needed and cross-frame recognition and tracking are carried out. This not only ensures the accuracy of multi-target tracking in complex scenarios with overlapping moving areas and unstable movement trends, but also avoids the waste of computing resources in scenarios with clear moving paths, effectively improving the overall efficiency and rational utilization of resources in multi-target recognition and tracking.

[0085] Furthermore, by simultaneously assessing motion trends and regional overlap through an integrated and low-overhead preliminary trajectory analysis process, the independent and repetitive computational overhead of obtaining these two types of information is avoided. Computational power is dynamically allocated accordingly, thereby achieving precise adaptation to changes in scene complexity and net savings in overall system computing resources while significantly reducing the cost of decision-making itself.

[0086] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0087] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0088] In this embodiment of the invention, the multi-target recognition and tracking device based on a surveillance camera can be divided into functional units according to the above method example. For example, each function can be divided into its own functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0089] This invention also provides a hardware structure diagram of a multi-target recognition and tracking device based on a surveillance camera, see [link / reference]. Figure 3 The multi-target recognition and tracking device 300 based on a surveillance camera includes a processor 301, and optionally, a memory 302 connected to the processor 301.

[0090] In the first possible implementation, see Figure 3 The multi-target recognition and tracking device 300 based on a surveillance camera also includes a transceiver 303. The processor 301, memory 302, and transceiver 303 are connected via a bus. The transceiver 303 is used to communicate with other devices or communication networks. Optionally, the transceiver 303 may include a transmitter and a receiver. The device in the transceiver 303 that implements the receiving function can be considered as a receiver, which is used to perform the receiving steps in the embodiments of the present invention. The device in the transceiver 303 that implements the transmitting function can be considered as a transmitter, which is used to perform the transmitting steps in the embodiments of the present invention.

[0091] Based on the first possible implementation method Figure 3 The structural diagram shown can be used to illustrate the structure of the multi-target recognition and tracking device based on a surveillance camera involved in the above embodiments.

[0092] in, Figure 3The diagram also illustrates the system chip in a multi-target recognition and tracking device based on a surveillance camera. In this case, the actions performed by the aforementioned multi-target recognition and tracking device based on a surveillance camera can be implemented by this system chip. The specific actions performed can be found above and will not be repeated here.

[0093] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings and the disclosure, will understand and implement other variations of the disclosed embodiments in carrying out the claimed invention. In this invention, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several of the functions listed in this invention.

[0094] Although the invention has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made therein without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely illustrative of the invention and are to be considered as covering any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if such modifications and modifications of the invention fall within the scope of the invention and its equivalents, the invention is also intended to include such modifications and modifications.

Claims

1. A multi-target recognition and tracking method based on surveillance cameras, characterized in that, The method comprises the following steps: acquiring a video frame sequence captured by a monitoring camera; identifying a moving region in the video frame sequence, and dividing the video frame sequence into a motion period containing the moving region and a static period not containing the moving region; for the motion period, performing preliminary trajectory analysis according to the time sequence change of the moving region, determining the motion trend consistency of the moving region at each time relative to the historical motion trend of the moving region, and determining the overlap degree between the moving regions, and combining the motion trend consistency and the overlap degree to determine an identification demand parameter for evaluating the tracking scene complexity at each time; allocating corresponding computing resources for each time according to the identification demand parameter at each time, and calling the allocated computing resources to perform cross-frame identification and tracking on the moving region, and outputting a tracking identification result of the moving region.

2. The multi-target recognition tracking method according to claim 1, characterized in that, The method of identifying a moving region in the video frame sequence and dividing the video frame sequence into a motion period containing the moving region and a static period not containing the moving region comprises: performing difference processing on adjacent frames in the video frame sequence to detect moving pixel points in each frame of image; performing connected domain analysis on the moving pixel points in each frame of image to form the moving region; dividing video frames in the video frame sequence that are continuous and do not contain the moving region into one static period; dividing video frames in the video frame sequence that are continuous and contain the moving region into one motion period.

3. The multi-target recognition tracking method according to claim 1, characterized in that, For the motion period, the preliminary trajectory analysis according to the time sequence change of the moving region determines the motion trend consistency of the moving region at each time relative to the historical motion trend of the moving region, which comprises: fitting a reference moving path according to the center point positions of the moving region that have a preliminary matching relationship in multiple historical continuous frames; determining the motion trend consistency by analyzing the deviation between the actual motion state and the predicted motion state of the moving region at each time according to the reference moving path.

4. The multi-target recognition tracking method according to claim 3, characterized in that, The determination of the overlap degree between the moving regions comprises: if the moving region at the current time and the multiple moving regions at the previous time all have a preliminary matching relationship, the moving region at the current time is determined as an overlapping region; determining the overlap degree parameter according to the ratio of the number of overlapping regions to the total number of moving regions at the current time, and the ratio of the area of the overlapping regions to the total area of the moving regions.

5. The multi-target recognition tracking method according to claim 4, characterized in that, The identification demand parameter for evaluating the tracking scene complexity at each time is determined in combination with the motion trend consistency and the overlap degree, which comprises: for each time, the identification demand parameter is comprehensively determined according to the motion trend consistency and the overlap degree parameter.

6. The multi-target recognition tracking method of claim 1, wherein, The method of allocating corresponding computing resources for each time according to the identification demand parameter at each time comprises: determining an algorithm power allocation priority parameter for each time according to the identification demand parameter at each time; the higher the tracking scene complexity represented by the identification demand parameter, the higher the algorithm power allocation priority parameter; allocating computing resources for the corresponding time in the order from high to low according to the algorithm power allocation priority parameter.

7. The multi-target recognition tracking method according to claim 6, characterized in that, The method of calling the allocated computing resources to perform cross-frame identification and tracking on the moving region comprises: If the computing power allocation priority parameter is lower than a preset priority threshold, target matching of the moving region is performed according to motion trend continuity of the moving region; If the computing power allocation priority parameter is higher than or equal to the preset priority threshold, target matching of the moving region is performed according to feature point matching of the moving region; and the computing resources required for target matching of the feature point matching are more than the computing resources required for target matching of the motion trend continuity.

8. The multi-target recognition tracking method of claim 1, wherein, Obtaining a video frame sequence collected by a monitoring camera, comprising: Receiving an original video stream collected by the monitoring camera through an edge computing node deployed in a position adjacent to the monitoring camera; Pretreating the original video stream to obtain the video frame sequence.

9. The multi-target recognition tracking method of claim 1, wherein, Outputting a tracking and identification result of the moving region, comprising: Labeling an external figure for each moving region identified in the video frame; Associating the external figures of the moving regions determined as the same object in different video frames with the same identifier.

10. A multi-target recognition tracking system based on surveillance cameras, characterized in that, Comprising: A data acquisition module, a time period division module, a demand evaluation module, and a computing power allocation and execution module; The data acquisition module is configured to acquire a video frame sequence collected by a monitoring camera; The time period division module is configured to identify moving regions in the video frame sequence, and divide the video frame sequence into a motion time period containing the moving regions and a static time period not containing the moving regions; The demand evaluation module is configured to, for the motion time period, perform preliminary trajectory analysis according to time sequence changes of the moving regions, determine motion trend consistency of each moving region itself at each time and an overlapping degree between the moving regions, and determine an identification demand parameter for evaluating tracking scene complexity at each time in combination with the motion trend consistency and the overlapping degree; The computing power allocation and execution module is configured to allocate corresponding computing resources for each time according to the identification demand parameter at each time, and call the allocated computing resources to perform cross-frame identification and tracking on the moving regions, and output a tracking and identification result of the moving regions.

Images