Multi-target recognition and tracking method and system based on surveillance camera

Abstract

The application relates to the technical field of video recognition, in particular to a multi-target recognition and tracking method and system based on a monitoring camera, which solves the technical problem of low resource utilization efficiency caused by the fact that a fixed computing power allocation strategy is adopted to perform cross-frame matching on a moving target, which cannot adapt to the dynamic complexity change of a tracking scene. The method comprises the following steps: acquiring a video frame sequence; recognizing a moving area, and dividing the video frame sequence into a motion period containing the moving area and a static period; for the motion period, performing preliminary trajectory analysis according to the time sequence change of the moving area to determine the motion trend consistency of the moving area relative to the historical motion trend of the moving area and the overlapping degree between the moving areas, and further determining an identification demand parameter for evaluating the complexity of the tracking scene; allocating corresponding computing resources according to the identification demand parameter, performing cross-frame identification and tracking on the moving area, and outputting a tracking and identification result.

Description

Technical Field

[0001] This invention relates to the field of video recognition technology, and specifically to a multi-target recognition and tracking method and system based on surveillance cameras. Background Technology

[0002] In fields such as public safety and traffic management, surveillance cameras have become core security monitoring equipment, widely used in public scenes such as traffic intersections and streets. They are usually installed in fixed locations to capture images within the monitoring range in real time, achieving full coverage monitoring of various moving objects and providing basic data support for subsequent analysis of target movement trajectories and ensuring public safety.

[0003] To determine the movement path of moving objects, existing multi-target recognition and tracking methods generally adopt the approach of precise matching. This involves extracting precise features such as the similarity of the shape of moving objects and the consistency of their categories, associating moving objects captured at different times, and then sorting out the movement trajectories of each object based on the association results to complete the tracking and recognition task.

[0004] Existing multi-target tracking solutions primarily focus on improving the matching accuracy of a single algorithm in complex scenes, or employ a fixed computational framework to process all video frames. These methods struggle to adapt to the varying processing needs at different times: in simple scenes with clear targets and predictable motion, a fixed computational power model leads to idle and wasted computing resources; while in complex scenes with severe target occlusion and chaotic motion, insufficient instantaneous computing power may result in tracking loss or errors. Therefore, how to enable the system's computational resource allocation to adapt to dynamic scene changes, thereby optimizing overall computational efficiency while ensuring tracking accuracy, is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] To address the problem of low resource utilization efficiency in existing technologies that employ fixed computing power allocation strategies for cross-frame matching of moving targets, which cannot adapt to the dynamic complexity changes in tracking scenarios, the present invention aims to provide a multi-target recognition and tracking method and system based on surveillance cameras. The specific technical solution adopted is as follows:

[0006] Firstly, a multi-target recognition and tracking method based on a surveillance camera is provided, comprising: acquiring a video frame sequence captured by the surveillance camera; identifying moving regions in the video frame sequence, dividing the video frame sequence into motion periods containing moving regions and stationary periods not containing moving regions; for the motion periods, performing preliminary trajectory analysis based on the temporal changes of the moving regions, determining the consistency of the motion trend of the moving regions relative to their own historical motion trends at each moment, as well as the degree of overlap between the moving regions, and combining the consistency of motion trend and the degree of overlap to determine the recognition requirement parameters used to evaluate the complexity of the tracking scenario at each moment; allocating corresponding computing resources for each moment according to the recognition requirement parameters at each moment, and calling the allocated computing resources to perform cross-frame recognition and tracking of the moving regions, and outputting the tracking and recognition results of the moving regions.

[0007] Based on the above technical solution, in the multi-target recognition and tracking method based on a surveillance camera provided by this invention, by acquiring the video frame sequence collected by the surveillance camera, the moving area is accurately identified and the moving time period and the stationary time period are divided. The recognition requirement parameters corresponding to the complexity of the tracking scene are determined by combining the consistency of the moving area with its own historical movement trend and the degree of overlap between the areas within the moving time period. Then, based on the parameters, computing resources are allocated as needed and cross-frame recognition and tracking are carried out. This not only ensures the accuracy of multi-target tracking in complex scenarios with overlapping moving areas and unstable movement trends, but also avoids the waste of computing resources in scenarios with clear moving paths, effectively improving the overall efficiency and rational utilization of resources in multi-target recognition and tracking.

[0008] Furthermore, by simultaneously assessing motion trends and regional overlap through an integrated and low-overhead preliminary trajectory analysis process, the independent and repetitive computational overhead of obtaining these two types of information is avoided. Computational power is dynamically allocated accordingly, thereby achieving precise adaptation to changes in scene complexity and net savings in overall system computing resources while significantly reducing the cost of decision-making itself.

[0009] In conjunction with the first aspect above, in one possible implementation, the method for identifying moving regions in a video frame sequence and dividing the video frame sequence into motion periods containing moving regions and stationary periods not containing moving regions specifically includes: performing differential processing on adjacent frames in the video frame sequence to detect moving pixels in each frame; performing connected component analysis on the moving pixels in each frame to form moving regions; dividing consecutive video frames in the video frame sequence that do not contain moving regions into a stationary period; and dividing consecutive video frames in the video frame sequence that contain moving regions into a motion period.

[0010] In conjunction with the first aspect mentioned above, in one possible implementation, the method for determining the consistency of the motion trend of the moving area relative to its own historical motion trend at each moment by performing preliminary trajectory analysis based on the temporal changes of the moving area during the motion period specifically includes: fitting a reference motion path based on the center point positions of the moving area that have preliminary matching relationships in multiple historical consecutive frames; and analyzing the deviation between the actual motion state and the predicted motion state of the moving area at each moment based on the reference motion path to determine the consistency of the motion trend.

[0011] In conjunction with the first aspect above, in one possible implementation, the method for determining the degree of overlap between moving regions specifically includes: if the moving region at the current moment has a preliminary matching relationship with multiple moving regions at the previous moment, the moving region at the current moment is determined to be an overlapping region; and the degree of overlap parameter is determined based on the ratio of the number of overlapping regions to the total number of moving regions at the current moment, and the ratio of the area of ​​the overlapping region to the total area of ​​the moving regions.

[0012] In conjunction with the first aspect above, in one possible implementation, the method for determining the identification requirement parameters for evaluating the complexity of the tracking scenario at each moment by combining motion trend consistency and overlap specifically includes: for each moment, comprehensively determining the identification requirement parameters based on motion trend consistency and overlap parameters.

[0013] In conjunction with the first aspect above, in one possible implementation, the method of allocating corresponding computing resources for each moment based on the identification requirement parameters at each moment specifically includes: determining the computing power allocation priority parameter for each moment based on the identification requirement parameters at each moment; the higher the complexity of the tracking scenario represented by the identification requirement parameters, the higher the computing power allocation priority parameter; and allocating computing resources to the corresponding moments in descending order of the computing power allocation priority parameters.

[0014] In conjunction with the first aspect above, in one possible implementation, the method of using allocated computing resources to perform cross-frame identification and tracking of moving regions specifically includes: if the computing power allocation priority parameter is lower than a preset priority threshold, target matching of the moving region is performed based on the continuity of the movement trend of the moving region; if the computing power allocation priority parameter is higher than or equal to the preset priority threshold, target matching of the moving region is performed based on feature point matching of the moving region; the computing resources required for target matching based on feature point matching are greater than the computing resources required for target matching based on the continuity of the movement trend.

[0015] In conjunction with the first aspect above, in one possible implementation, the method for obtaining a video frame sequence captured by a surveillance camera specifically includes: receiving the raw video stream captured by the surveillance camera through an edge computing node deployed near the surveillance camera; and preprocessing the raw video stream to obtain a video frame sequence.

[0016] In conjunction with the first aspect above, in one possible implementation, the method for outputting the tracking and identification results of the moving region specifically includes: labeling each identified moving region with a circumscribed graphic in the video frame; associating the circumscribed graphics of moving regions determined to be the same object in different video frames with the same identifier.

[0017] Secondly, a multi-target recognition and tracking system based on a surveillance camera is provided, comprising: a data acquisition module, a time period segmentation module, a demand assessment module, and a computing power allocation and execution module; the data acquisition module is used to acquire video frame sequences captured by the surveillance camera; the time period segmentation module is used to identify moving regions in the video frame sequence, dividing the video frame sequence into motion periods containing moving regions and stationary periods not containing moving regions; the demand assessment module is used to perform preliminary trajectory analysis based on the temporal changes of moving regions during motion periods, determine the consistency of the motion trend of the moving regions at each moment, as well as the degree of overlap between moving regions, and determine the recognition demand parameters for evaluating the complexity of the tracking scenario at each moment by combining the consistency of motion trend and the degree of overlap; the computing power allocation and execution module is used to allocate corresponding computing resources for each moment according to the recognition demand parameters at each moment, and call the allocated computing resources to perform cross-frame recognition and tracking of the moving regions, outputting the tracking and recognition results of the moving regions.

[0018] The present invention has the following beneficial effects:

[0019] By acquiring video frame sequences captured by surveillance cameras, the system accurately identifies moving areas and divides them into moving and stationary periods. Combining the consistency of moving areas within a moving period with their historical movement trends and the degree of overlap between areas, the system determines the recognition requirement parameters corresponding to the complexity of the tracking scenario. Based on these parameters, computing resources are allocated as needed, and cross-frame recognition and tracking are carried out. This approach ensures the accuracy of multi-target tracking in complex scenarios with overlapping moving areas and unstable movement trends, while avoiding the waste of computing resources in scenarios with clear moving paths. This effectively improves the overall efficiency and rational utilization of resources in multi-target recognition and tracking.

[0020] Furthermore, by simultaneously assessing motion trends and regional overlap through an integrated and low-overhead preliminary trajectory analysis process, the independent and repetitive computational overhead of obtaining these two types of information is avoided. Computational power is dynamically allocated accordingly, thereby achieving precise adaptation to changes in scene complexity and net savings in overall system computing resources while significantly reducing the cost of decision-making itself. Attached Figure Description

[0021] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A system structure diagram of a multi-target recognition and tracking system based on a surveillance camera is provided in one embodiment of the present invention;

[0023] Figure 2 A flowchart of a multi-target recognition and tracking method based on a surveillance camera is provided as an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of the hardware structure of a multi-target recognition and tracking device based on a surveillance camera, provided as an embodiment of the present invention. Detailed Implementation

[0025] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a multi-target recognition and tracking method and system based on a surveillance camera proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0027] The following description, in conjunction with the accompanying drawings, details the specific scheme of a multi-target recognition and tracking method and system based on a surveillance camera provided by the present invention.

[0028] Please see Figure 1The diagram illustrates a system architecture of a multi-target recognition and tracking system based on a surveillance camera, according to an embodiment of the present invention. The multi-target recognition and tracking system based on a surveillance camera includes: a data acquisition module 1, a time period segmentation module 2, a demand assessment module 3, and a computing power allocation and execution module 4.

[0029] The data acquisition module 1 is the core of the system's data input, relying entirely on edge computing nodes deployed near the surveillance cameras. The output video frame sequence is directly transmitted to the time segmentation module 2, providing a high-quality data foundation for subsequent motion area identification and time segmentation. The data acquisition module 1 can include two functional sub-modules: a video receiving sub-module 11 and a video preprocessing sub-module 12.

[0030] The video receiving submodule 11 receives the raw video stream collected by the surveillance camera in real time through physical communication devices such as the Ethernet interface and wireless communication module configured on the edge computing node. It has a data transmission verification function, which can filter abnormal data such as packet loss and disorder during transmission to ensure the integrity of the raw video stream.

[0031] The video preprocessing submodule 12 is carried by the embedded processor of the edge computing node. It extracts single frames from the received raw video stream, and then performs grayscale conversion, image equalization and noise reduction processing to eliminate the interference of light changes and image noise on subsequent recognition, and finally generates a standardized video frame sequence.

[0032] The time period segmentation module 2 is the core of the system's scene classification. It relies on the dedicated image processing unit of the edge computing node to process data, and the output motion time period data is transmitted to the demand assessment module 3, providing a scene basis for subsequent tracking scene complexity assessment. Static time period information is directly sent to the computing power allocation and execution module 4 for quickly outputting tracking results for objects without movement. The time period segmentation module 2 may include a moving pixel detection submodule 21, a moving region formation submodule 22, and a time period segmentation submodule 23.

[0033] The moving pixel detection submodule 21 performs differential processing on adjacent frames in the video frame sequence output by the data acquisition module 1. By calculating the grayscale difference of corresponding pixels in two frames, it detects moving pixels with positional changes in each frame image.

[0034] The moving region forming submodule 22 performs connected component analysis on the detected moving pixels, aggregates spatially continuous moving pixels to form an initial region, and then fills the holes in the initial region through morphological closing operations. At the same time, it determines whether there are other moving pixels in the neighborhood of each pixel, eliminates isolated points, and finally forms a complete moving region.

[0035] The time segmentation submodule 23 performs frame-by-frame determination on the video frame sequence, dividing continuous video frames that do not contain moving areas into a static time segment, and dividing continuous video frames that contain moving areas into a moving time segment.

[0036] The demand assessment module 3 is the core basis for the system's computing power allocation. It is carried by the data analysis unit of the edge computing node, and the output identification demand parameters are directly transmitted to the computing power allocation and execution module 4, becoming the core basis for computing resource allocation and tracking method selection. The demand assessment module 3 may include a preliminary trajectory fitting submodule 31, a motion trend consistency analysis submodule 32, an overlap degree parameter calculation submodule 33, and an identification demand parameter determination submodule 34.

[0037] The preliminary trajectory fitting submodule 31 extracts the center point positions of the moving region that have preliminary matching relationships in multiple consecutive frames based on the motion time period output by the time period segmentation module 2, and obtains the reference movement path of the moving region through a straight line fitting algorithm.

[0038] The motion trend consistency analysis submodule 32 calculates the deviation between the actual motion state and the predicted motion state of the moving area at each moment, based on the reference moving path analysis. It determines the motion trend consistency of the moving area relative to its own historical motion trend based on the magnitude of the deviation. The smaller the deviation, the higher the motion trend consistency.

[0039] The overlap degree parameter calculation submodule 33 determines whether the moving region at the current moment has a preliminary matching relationship with multiple moving regions at the previous moment for each time moment of the image. Moving regions that meet the condition are identified as overlapping regions. Then, the overlap degree parameter is calculated by the ratio of the number of overlapping regions to the total number of moving regions and the ratio of the area of ​​overlapping regions to the total area of ​​moving regions. This parameter is positively correlated with the number of overlapping regions and the overlapping area.

[0040] The identification requirement parameter determination submodule 34 integrates the consistency and overlap parameters of the motion trend at each moment, and determines the identification requirement parameters used to evaluate the complexity of the tracking scene at that moment through a preset comprehensive evaluation logic. The more complex the tracking scene, the higher the identification requirement parameters.

[0041] The computing power allocation and execution module 4 is the core of the system's tracking and execution, implemented using the computing power scheduling unit, image processing unit, and output interface of the edge computing node. The computing power allocation and execution module 4 may include a computing power priority determination submodule 41, a computing resource allocation submodule 42, a cross-frame tracking submodule 43, and a tracking result output submodule 44.

[0042] The computing power priority determination submodule 41 receives the identification requirement parameters output by the requirement assessment module 3, and determines the computing power allocation priority parameters at each time according to the parameter size. The higher the complexity of the tracking scenario represented by the identification requirement parameters, the higher the corresponding computing power allocation priority parameters.

[0043] The computing resource allocation submodule 42 allocates corresponding computing resources to each moment of the motion period in descending order of computing power allocation priority parameters. Higher priority moments receive more processor computing resources and memory resources to ensure tracking accuracy.

[0044] The cross-frame tracking submodule 43 performs differentiated tracking for different scenarios: for the static period output by the time period segmentation module 2, it directly determines that there are no moving objects and generates the corresponding tracking results; for the moment when the computing power allocation priority parameter is lower than the preset priority threshold during the motion period, it performs target matching based on the continuity of the motion trend of the moving area; for the moment when the computing power allocation priority parameter is higher than or equal to the preset priority threshold, it performs target matching based on the feature point matching of the moving area, the latter requiring more computing resources than the former.

[0045] The tracking result output submodule 44 uses the display interface, storage interface or network transmission interface of the edge computing node to annotate the circumscribed graphics of each identified moving region in the video frame, and associates the circumscribed graphics of the moving regions determined to be the same object in different video frames with the same identifier, and finally outputs the visualized tracking and recognition results.

[0046] Please see Figure 2 The diagram illustrates a flowchart of a multi-target recognition and tracking method based on a surveillance camera, according to an embodiment of the present invention. This multi-target recognition and tracking method based on a surveillance camera includes:

[0047] S1. Obtain the video frame sequence captured by the surveillance camera.

[0048] In some implementations, edge computing nodes deployed near the surveillance cameras first receive the raw video streams captured by the cameras. During reception, the edge computing nodes perform integrity checks on each frame of data. If the check result matches that of the sender, reception is confirmed; otherwise, a retransmission request is triggered to avoid data distortion due to transmission anomalies and to ensure the reliability of the original data.

[0049] Then, the original video stream is preprocessed, specifically by extracting single frames from the received original video stream, and then performing grayscale conversion, image equalization, and noise reduction to eliminate the interference of lighting changes and image noise on subsequent recognition, and finally generating a standardized video frame sequence.

[0050] S2. Identify moving regions in the video frame sequence and divide the video frame sequence into motion periods containing moving regions and stationary periods not containing moving regions.

[0051] In some implementations, the method for identifying moving regions in a video frame sequence can be specifically implemented through the following steps:

[0052] First, differential processing is performed on adjacent frames in the video frame sequence to detect moving pixels in each frame. Specifically, for two adjacent frames, the grayscale image corresponding to the frame with the longer time sequence is taken, and the grayscale image corresponding to the frame with the shorter time sequence is subtracted to obtain the grayscale difference of each pixel between the two adjacent frames. Next, the mean of the grayscale differences of all pixels in the two frames is calculated to correct for the overall grayscale shift caused by changes in lighting. Then, through absolute value calculation and normalization mapping, the degree of grayscale difference deviation of each pixel (i.e., the deviation of the grayscale difference of that pixel from the overall mean) is obtained, expressed as:

[0053]

[0054] In the formula, This represents the grayscale difference of the i-th pixel between frame t and frame t-1; This represents the average grayscale difference of all pixels between frame t and frame t-1. This represents the degree of grayscale shift after eliminating light interference (absolute value, ignoring the shift direction); this deviation is normalized to the [0, 1] interval using the norm function to unify the judgment standard, thus obtaining the degree of grayscale difference deviation of the i-th pixel in frame t. .

[0055] Unless otherwise specified, the normalization function norm(*) mentioned in the embodiments of this invention uses maximum and minimum value normalization. The maximum and minimum values ​​are preset empirical extreme values ​​derived from a large amount of historical experimental data. If the calculation result exceeds the interval [0, 1], it is restricted to the range [0, 1] by a truncation function (i.e., if the result is less than 0, it is taken as 0; if it is greater than 1, it is taken as 1) to eliminate the influence of outliers on the result.

[0056] In special cases, if the absolute value of the grayscale offset of all pixels is the same, then the denominator (maximum value - minimum value) of the normalization function is zero, and the degree of grayscale difference deviation of the pixel is directly defined as zero.

[0057] Next, a preset judgment threshold, such as 0.5, is set, and pixels with a normalization result greater than 0.5 are marked as moving pixels in the current frame. The judgment threshold for moving pixels is determined by combining the frame rate of the monitored scene, the normal speed of the moving object, and the conversion relationship between pixels and actual distance. The number of pixels corresponding to the reasonable actual displacement of the moving object between adjacent frames is calculated, and then determined after calibration through inter-frame difference experiments in the actual scene.

[0058] Then, connected component analysis is performed on the moving pixels in each frame of the image to aggregate spatially continuous moving pixels into an initial region. Morphological closing operations (e.g., a preset kernel size of 3×3) are then used to fill the holes in the initial region, while isolated points (i.e., pixels without other moving pixels in their 8-neighborhood) are excluded to filter out scattered noise points and form the moving region.

[0059] Finally, consecutive video frames in the video frame sequence that do not contain moving areas are divided into a static time period; consecutive video frames in the video frame sequence that contain moving areas are divided into a moving time period.

[0060] S3. For the movement period, perform preliminary trajectory analysis based on the temporal changes of the moving area to determine the consistency of the movement trend of the moving area relative to its own historical movement trend at each moment, as well as the degree of overlap between moving areas. Combine the consistency of movement trend and the degree of overlap to determine the recognition requirement parameters used to evaluate the complexity of the tracking scenario at each moment.

[0061] In some implementations, the method for determining the consistency of the movement trend of the moving region at each moment relative to its own historical movement trend can be specifically implemented through the following steps S31 to S32, which are explained in detail below:

[0062] S31. Based on the position of the center point where the moving region has a preliminary matching relationship in multiple consecutive frames, a reference moving path is fitted to obtain the reference moving path.

[0063] In some implementations, firstly, for a single moving region within the motion period, consecutive frames (such as 10-20 frames) within a preset sliding time window before the current moment are selected as the target segment. This stage fits the dynamic scene requirements of multi-target tracking, and the reference path fitted based on this can more accurately reflect the preceding motion characteristics at the current moment.

[0064] The center point is the spatial center of the moving region, possessing uniqueness and stability. Regardless of whether the region is a regular vehicle rectangle or an irregular pedestrian outline, a unique center coordinate can be obtained through a unified algorithm (the midpoint of the circumscribed rectangle or the centroid), accurately representing the spatial position of the entire region. When extracting the center point coordinates of a moving region, it is necessary to identify the boundary pixel coordinates of the region (connected component) after morphological closing operations have been completed and isolated points have been excluded, using Canny edge detection combined with contour tracking algorithms to define the spatial range of the moving region. If the region is close to a regular shape such as a rectangle, its smallest circumscribed rectangle is calculated, and the intersection of the horizontal and vertical midlines of the rectangle is taken as the center point coordinates. If it is an irregular shape such as a pedestrian, the x and y coordinates of all pixels within the region are statistically analyzed, and the average of the two types of coordinates is calculated. The combination of the "average x and y coordinates" is used as the center point coordinates. It is verified whether the pixel corresponding to the center point belongs to the current moving region. If it is not within the region, the edge detection threshold is adjusted, the contour is re-extracted, and the coordinates are recalculated to ensure that the center point is located inside the moving region, thus obtaining accurate center point coordinates that can support subsequent path analysis and inter-frame matching.

[0065] Next, frame-by-frame matching is performed on the center points of the moving regions within the target segment: the Euclidean distance between the center points of moving regions with similar contour sizes in adjacent frames is calculated (for example, the ratio of the areas of two regions is calculated, and if the ratio is within the range of [0.7, 1.3], they are considered similar, and this range is related to the distance and angle between the surveillance camera and the moving object). Two moving regions with a distance less than the distance threshold are determined as the matching regions of the moving regions in consecutive frames. The distance threshold is set based on the video frame interval and the normal speed of the moving object. For example, when existing image recognition technology identifies a pedestrian as the moving area, taking the common 25fps frame rate of surveillance video as an example, the video frame interval is 0.04 seconds, and the normal walking speed of a pedestrian is about 1.2m / s. Therefore, the actual moving distance of the pedestrian between adjacent frames is 1.2 × 0.04 = 0.048m. Combining the conversion relationship between the pixels of the surveillance image and the actual distance (assuming 1920 pixels corresponds to an actual intersection width of 20 meters, i.e., 1 pixel ≈ 0.0104 meters), 4.8 centimeters corresponds to about 4.6 pixels. Therefore, the distance threshold in this scenario can be preset to 5 pixels. Similarly, when the recognition technology identifies a vehicle as the moving area, considering that the vehicle's speed at the intersection is about 5m / s, the distance threshold in this scenario can be preset to 20 pixels.

[0066] Then, the initial motion type is predicted: the motion vector between consecutive frames within the target segment is calculated (obtained by subtracting the coordinates of the center point of the previous frame from the coordinates of the matching center point of the next frame), and the direction angle of each motion vector is calculated (for example, with 0° to the right horizontally, the angle is counted counterclockwise). The maximum difference in direction angle between the corresponding motion vectors of two consecutive adjacent frames within the target segment is calculated, with a preset angle threshold of 15°. If the maximum difference is less than 15°, the motion trend within the target segment is determined to be linear; if it is greater than 15°, the motion trend is determined to be curvilinear (such as the turning motion at an intersection corner), thus avoiding the limitations of a single linear fitting and adapting to motion changes in the actual scene.

[0067] Finally, a reference path is fitted to suit the motion type: if the motion trend is linear, the least squares linear regression algorithm is used to fit the coordinates of the center point within the target segment to obtain a linear reference movement path; if the motion trend is curvilinear, a preset quadratic polynomial regression algorithm is used (fitting formula y=ax). 2 The center point coordinates are fitted with a curve using +bx+c to obtain a curved reference movement path. This makes the reference movement path more closely match the actual initial movement trajectory of the movement area, thus improving the accuracy of subsequent movement trend consistency judgment.

[0068] S32. Analyze the deviation between the actual motion state and the predicted motion state of the moving area at each moment based on the reference moving path, and determine the consistency of the motion trend.

[0069] In some implementations, firstly, for the moving area during the movement period, the coordinates of its center point at each moment are extracted as the measured position at the current moment. Simultaneously, the reference movement path of the moving area is used to determine the reference position corresponding to the current moment. Using a point-to-point distance formula, the distance between the measured position coordinates of the moving area at the current moment and the reference position coordinates is calculated. This distance represents the positional deviation between the measured position and the reference movement path, quantifying the degree of deviation between the current movement position and the initial movement trend.

[0070] Secondly, for the moving area within the movement period, the actual movement vector of its center point coordinates at each moment compared to the center point coordinates at the previous moment is extracted, and the direction of the vector is used as the actual movement direction at the current moment. Simultaneously, the reference movement path of the moving area is invoked, and the reference movement vector of the reference position at the current moment compared to the reference position at the previous moment is calculated. The direction of this vector is used as the reference movement direction for the current moment based on historical trend prediction. A vector angle cosine algorithm is employed to calculate the cosine value of the angle between the actual movement direction and the reference movement direction at the current moment, used to quantify the degree of deviation between the current movement direction and the initial movement trend.

[0071] Thirdly, the magnitude of the actual movement vector can characterize the actual movement speed of the moving area at the current moment, while the magnitude of the reference movement vector can characterize the reference movement speed of the moving area at the current moment, predicted based on historical trends. The ratio of the change in actual movement speed to reference movement speed (i.e., the absolute value of the difference between actual and reference movement speed divided by the reference movement speed) is calculated to quantify the degree of deviation between the current movement speed and the initial movement trend.

[0072] Combining the deviations in the above three aspects, the consistency of the motion trend between the moving region and the initially matched moving regions in consecutive frames is comprehensively determined and expressed as:

[0073]

[0074] In the formula, Indicates the first The distance between the measured location coordinates of each moving area and the reference location coordinates; The distance deviation normalization coefficient (a non-zero constant) is determined based on the environmental activity range of the moving object; for example, when the moving object is a pedestrian. It can be set to the pixel distance calculated from the width of the sidewalk in the current scene; It is the inverse normalized term of the positional deviation. The smaller the value (the closer the actual position is to the reference position), the larger this value becomes, and the closer it is to 0, thus achieving consistency between distance deviation and movement trend. Negative correlation;

[0075] Indicates the first The actual movement vector of each moving region Represents the magnitude of the actual movement vector; Indicates the first Reference movement vector for each moving region This represents the magnitude of the reference movement vector (since it is a trajectory prediction of a moving object, the magnitude of the reference movement vector is not zero).

[0076] This represents the cosine of the angle between the actual direction of movement and the reference direction of movement, ranging from -1 to 1. The closer the actual and reference directions of movement are to each other, the closer this value is to 1, reflecting consistency with the trend of movement. Positive correlation;

[0077] The term represents the opposite of the rate of change in velocity. The smaller the velocity deviation, the larger this term is, and the closer it is to zero, thus achieving consistency between the velocity deviation and the motion trend. The negative correlation.

[0078] By comprehensively assessing the movement trend from three core dimensions—position, direction, and speed—we can avoid the one-sidedness of a single dimension and more accurately reflect the consistency of the movement trend.

[0079] The above formula is a quantitative example of the consistency of motion trends. In other embodiments, different weighting coefficients can be assigned to the distance deviation, direction deviation, and velocity deviation before fusion.

[0080] Furthermore, if multiple suspected matching regions exist in consecutive frames during the initial matching process, path fitting and motion trend consistency calculation are performed for each matching region, and the suspected matching region with the greatest motion trend consistency is determined as the matching region of the moving region.

[0081] In some implementations, the method for determining the degree of overlap between moving regions can be specifically implemented through the following steps S33 to S34, which are explained in detail below:

[0082] S33. If the current moving region has a preliminary matching relationship with multiple moving regions in the previous time, the moving region at the current time is determined to be an overlapping region.

[0083] In some implementations, for each moving region at the current moment, the Euclidean distance between its center point coordinates and the center point coordinates of all moving regions (regardless of region outline size) at the previous moment is calculated. The distance threshold in S31 is called. If the Euclidean distance between the center points of multiple moving regions at the previous moment and one moving region at the current moment is less than the distance threshold, it is considered that the region has a preliminary matching relationship with multiple moving regions at the previous moment and is determined to be an overlapping region.

[0084] S34. Determine the degree of overlap parameters based on the ratio of the number of overlapping regions to the total number of moving regions at the current moment, and the ratio of the area of ​​overlapping regions to the total area of ​​moving regions.

[0085] In some implementations, the total number of moving areas and the number of overlapping areas at the current moment are counted, and the ratio of the number of overlapping areas to the total number of moving areas is calculated.

[0086] Meanwhile, the region boundary pixels are extracted by contour detection algorithm, and the number of pixels within the boundary is counted as the region area. The region area values ​​of all moving regions and all overlapping regions are summed to calculate the ratio of the total area of ​​the overlapping region to the total area of ​​the moving region.

[0087] Then, the overlap parameter is determined, expressed as:

[0088]

[0089] In the formula, This represents the number of overlapping regions at time t. This represents the number of regions moved at time t. This represents the proportion of overlapping regions. The larger the ratio, the more overlapping regions exist at the current time. It is also a parameter related to the degree of overlap with time t. Positive correlation;

[0090] This represents the area representation value of the overlapping region at time t. This represents the area of ​​the moved region at time t. This represents the area ratio of the overlapping region. The larger the ratio, the wider the spatial extent of the overlapping region at the current time. It is also a parameter related to the degree of overlap with the region at time t. Positive correlation;

[0091] The product accurately reflects the superposition effect of a large number of overlapping regions and a large proportion of overlapping area. Only when both the number of overlapping regions and the proportion of overlapping area are satisfied will the confusion risk of multi-target tracking increase significantly (e.g., multiple vehicles overlapping at an intersection at the same time, and the overlapping area covers the main features of the vehicles). If only one dimension is satisfied (e.g., a small number of regions but a large area or multiple regions but a small area), the increase in tracking difficulty is far less than the situation where both are satisfied at the same time.

[0092] Finally, the overlap parameter at time t is obtained. The value range is [0, 1]. It is a quantitative value of the degree of overlap of the moving areas at the current time. The larger the value, the more overlapping areas there are and the wider the area.

[0093] Furthermore, the method for determining the identification requirements parameters can be specifically implemented through the following S35, which will be explained in detail below:

[0094] S35. For each moment, determine the identification requirements parameters comprehensively based on the parameters of consistency and overlap of motion trends.

[0095] In some implementations, the identification requirement parameters are represented as:

[0096]

[0097] In the formula, This indicates the end of the motion period at time t. The mean of the consistency of movement trends in each moving region; The inverse term of the mean representing the consistency of motion trends indicates long-term path deviation, avoiding misjudgment due to short-term fluctuations. In regions of long-term instability (even if the deviation is small at the current moment), subsequent frames are more prone to loss of tracking or confusion, requiring higher recognition accuracy. The inverse term of the mean representing the consistency of motion trends is mapped to the [0, 1] interval using the normalization function norm(*), thus avoiding the inverse term representing the consistency of motion trends. With overlap parameter The uneven influence of parameters due to inconsistent magnitudes;

[0098] and These are the weights for the parameters of consistency and overlap in movement trends, respectively. Specifically, the calibration is based on the actual impact of overlap and motion complexity in the scene. For example, in a general scene, let... =0.5、 =0.5, balancing the influence of both; if the scene mainly involves complex single-target motion (such as a park or pedestrian street), set... =0.3、 =0.7, which enhances the contribution of motion complexity.

[0099] The normalized result is compared with the overlap parameter. We can obtain the weighted summation at time t. Parameters for identifying a mobile area It integrates the consistency of long-term movement trends with the current regional overlap, and the higher the value, the higher the demand.

[0100] S4. Based on the recognition requirement parameters at each time moment, allocate corresponding computing resources for each time moment, and call the allocated computing resources to perform cross-frame recognition and tracking of the moving area, and output the tracking and recognition results of the moving area.

[0101] In some implementations, the method for allocating computing resources includes: first, determining the computing power allocation priority parameter for each time step based on the identification requirement parameters at each time step. The higher the complexity of the tracking scenario represented by the identification requirement parameters, the higher the computing power allocation priority parameter. Then, computing resources are allocated to the corresponding time steps in descending order of the computing power allocation priority parameters.

[0102] Firstly, the system iterates through all moments within the current motion period. For each moving region at each moment, its recognition requirement parameters are calculated individually. If at least one region at a given moment has a recognition requirement parameter greater than a preset high recognition requirement threshold, and the proportion of regions meeting the threshold is greater than or equal to a preset percentage (e.g., 30%), then that moment is considered a high recognition requirement moment. Then, consecutive high recognition requirement moments are merged into a single test period, clustering moments in high-complexity scenarios to avoid the inefficiency of fragmented computing power allocation and improve resource targeting. The preset high recognition requirement threshold for merging test periods is determined by statistically analyzing the tracking complexity and matching failure rate corresponding to historical recognition requirement parameter intervals, combined with the constraint of the proportion of high-requirement periods supported by computing power resources. This initial value is then verified and fine-tuned through actual scenario operation (e.g., 0.5). The duration of the test period to which the current moment belongs is recorded, and the proportion of the test period's duration in the total duration of the current motion period is calculated. A higher proportion indicates a stronger persistence of the high-complexity scenario, which is positively correlated with the priority of computing power allocation.

[0103] Secondly, for all moving regions within the test period to which the current moment belongs, the area difference between adjacent moments is calculated (the area of ​​the next moment minus the area of ​​the previous moment). The standard deviation and mean of the area differences of all regions are compared to obtain the coefficient of variation, which is used to represent the degree of disorder of morphological changes within the test period. The larger the coefficient of variation, the higher the risk of feature confusion during tracking, and the higher the corresponding computing power requirement.

[0104] Thirdly, extract the motion vector magnitude of all moving regions at the current moment (representing the motion speed of each region), take the average of the magnitudes to obtain the average motion speed at the current moment. The higher the speed, the faster the inter-frame feature changes, the greater the tracking difficulty, and the more computing power is required.

[0105] By integrating the above three aspects with the identification requirement parameters at the current moment, the computing power allocation priority parameters at the current moment are determined, as follows:

[0106]

[0107] In the formula, This represents the maximum value among the identification requirements parameters for the moving region at time t, accurately capturing the worst-case scenario within that time and ensuring that highly complex regions receive sufficient computing power.

[0108] This represents the duration of the time period to be measured at time t. The duration from the start of the current motion period to time t. The proportion in;

[0109] This represents the standard deviation of the area difference among all regions at time t. This represents the average area difference of all regions at time t; For values ​​to be taken as minimum values ​​(e.g., 10) -6 The parameter is used to avoid abnormal results caused by a denominator of zero; The coefficient of variation represents the area difference of all regions at time t;

[0110] This represents the average moving speed of all moving areas at time t, in pixels per frame. This represents a normalization constant with velocity dimensions, which can be set to 1 pixel / frame, and is used to eliminate the velocity dimension.

[0111] By integrating multi-dimensional complexity features, the one-sidedness of single-dimensional evaluation is avoided, providing a more accurate priority basis for computing power allocation. Then, the computing power allocation priority parameter at time t is obtained by mapping to the [0,1] interval through the normalization function norm(*).

[0112] The above formula is a quantitative example of the priority parameters for computing power allocation. In other embodiments, different weighting coefficients can be assigned to the three factors—duration percentage, morphological fluctuation, and movement speed—before they are fused.

[0113] In some implementations, a preset priority threshold is set based on the upper limit of computing power resources in the actual scenario and the target matching accuracy requirements. Specifically, this includes: first, statistically analyzing the distribution of computing power allocation priority parameters for multiple consecutive frames in the current monitoring scenario, and simultaneously recording the success rate of motion trend continuity matching and the proportion of computing power consumption for feature point matching under different parameter ranges; then, based on the standard that "the motion trend matching accuracy of low computing power meets the standard (e.g., success rate ≥ 85%) and the proportion of feature point matching scenarios with high computing power does not exceed the upper limit of computing power resources (e.g., 30%)", the corresponding computing power priority threshold is selected as the initial threshold; finally, the implementation is verified in the actual scenario, and the threshold is fine-tuned based on the matching failure rate in the low threshold range and the computing power occupancy rate in the high threshold range until the matching accuracy and computing power consumption reach a long-term balance.

[0114] If the computing power allocation priority parameter is lower than the preset priority threshold, the target matching of the moving area is performed based on the continuity of the movement trend of the moving area. Specifically, this includes: associating and matching the area with the continuous movement trend at the current moment with the area with the same movement trajectory at the previous moment. Matching is completed only by relying on motion features, and the computing power consumption is lower than that of feature point matching, which is suitable for scenarios with low complexity and low computing power requirements.

[0115] If the computing power allocation priority parameter is higher than or equal to the preset priority threshold, target matching of the moving region is performed based on feature point matching of the moving region. Specifically, this includes retrieving the initial number of matching feature points corresponding to the moving region at the current moment, which is the result of a computer vision algorithm combining an accelerated segment test (FAST) feature detector and a binary robust independent elementary features (BRIEF) descriptor (oriented FAST and rotated BRIEF). (If using the default values ​​from the standard algorithm library), priority parameters are allocated based on computing power. The number of feature points is dynamically adjusted based on the tracking difficulty of the region. , represented as:

[0116]

[0117] Based on the adjusted number of feature points The ORB algorithm is used to extract feature points of the current and previous time regions for region matching to assist in target tracking. This can improve matching accuracy and is suitable for highly complex scenarios with many overlapping regions and unstable motion. However, the computational resources required are greater than those for motion trend continuity matching.

[0118] In some implementations, a bounding graphic is labeled for each identified moving region in the video frame. Then, the bounding graphics of moving regions identified as the same object in different video frames are associated with the same identifier to generate the tracking and recognition results of the moving regions.

[0119] Based on the above technical solution, by acquiring video frame sequences captured by surveillance cameras, the moving areas are accurately identified and the moving and stationary periods are divided. The consistency of the moving areas within the moving period relative to their historical movement trends and the degree of overlap between areas are combined to determine the recognition requirement parameters corresponding to the complexity of the tracking scenario. Then, based on these parameters, computing resources are allocated as needed and cross-frame recognition and tracking are carried out. This not only ensures the accuracy of multi-target tracking in complex scenarios with overlapping moving areas and unstable movement trends, but also avoids the waste of computing resources in scenarios with clear moving paths, effectively improving the overall efficiency and rational utilization of resources in multi-target recognition and tracking.

[0120] Furthermore, by simultaneously assessing motion trends and regional overlap through an integrated and low-overhead preliminary trajectory analysis process, the independent and repetitive computational overhead of obtaining these two types of information is avoided. Computational power is dynamically allocated accordingly, thereby achieving precise adaptation to changes in scene complexity and net savings in overall system computing resources while significantly reducing the cost of decision-making itself.

[0121] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0122] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0123] In this embodiment of the invention, the multi-target recognition and tracking device based on a surveillance camera can be divided into functional units according to the above method example. For example, each function can be divided into its own functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0124] This invention also provides a hardware structure diagram of a multi-target recognition and tracking device based on a surveillance camera, see [link / reference]. Figure 3 The multi-target recognition and tracking device 300 based on a surveillance camera includes a processor 301, and optionally, a memory 302 connected to the processor 301.

[0125] In the first possible implementation, see Figure 3 The multi-target recognition and tracking device 300 based on a surveillance camera also includes a transceiver 303. The processor 301, memory 302, and transceiver 303 are connected via a bus. The transceiver 303 is used to communicate with other devices or communication networks. Optionally, the transceiver 303 may include a transmitter and a receiver. The device in the transceiver 303 that implements the receiving function can be considered as a receiver, which is used to perform the receiving steps in the embodiments of the present invention. The device in the transceiver 303 that implements the transmitting function can be considered as a transmitter, which is used to perform the transmitting steps in the embodiments of the present invention.

[0126] Based on the first possible implementation method Figure 3The structural diagram shown can be used to illustrate the structure of the multi-target recognition and tracking device based on a surveillance camera involved in the above embodiments.

[0127] in, Figure 3 The diagram also illustrates the system chip in a multi-target recognition and tracking device based on a surveillance camera. In this case, the actions performed by the aforementioned multi-target recognition and tracking device based on a surveillance camera can be implemented by this system chip. The specific actions performed can be found above and will not be repeated here.

[0128] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings and the disclosure, will understand and implement other variations of the disclosed embodiments in carrying out the claimed invention. In this invention, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several of the functions listed in this invention.

[0129] Although the invention has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made therein without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely illustrative of the invention and are to be considered as covering any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if such modifications and modifications of the invention fall within the scope of the invention and its equivalents, the invention is also intended to include such modifications and modifications.

Claims

1. A multi-target recognition and tracking method based on a surveillance camera, characterized in that, include: Obtain the video frame sequence captured by the surveillance camera; Identify moving regions in the video frame sequence and divide the video frame sequence into motion periods containing moving regions and stationary periods not containing moving regions; For the aforementioned movement period, preliminary trajectory analysis is performed based on the temporal changes of the moving area to determine the consistency of the movement trend of the moving area relative to its own historical movement trend at each moment, as well as the degree of overlap between the moving areas. The recognition requirement parameters for evaluating the complexity of the tracking scenario at each moment are determined by combining the consistency of the movement trend and the degree of overlap. Based on the identification requirement parameters at each time moment, determine the computing power allocation priority parameters at each time moment; The higher the complexity of the tracking scenario represented by the identification requirement parameters, the higher the computing power allocation priority parameter. Computing resources are allocated to the corresponding time points in descending order of the computing power allocation priority parameters. If the computing power allocation priority parameter is lower than the preset priority threshold, target matching in the moving area is performed based on the continuity of the movement trend of the moving area. If the computing power allocation priority parameter is higher than or equal to the preset priority threshold, target matching in the moving area is performed based on feature point matching in the moving area; the computing resources required for target matching based on feature point matching are greater than the computing resources required for target matching based on the continuity of movement trends.

2. The multi-target recognition and tracking method according to claim 1, characterized in that, Identifying moving regions in the video frame sequence and dividing the video frame sequence into motion periods containing moving regions and stationary periods not containing moving regions includes: Differential processing is performed on adjacent frames in the video frame sequence to detect moving pixels in each frame image; Connectivity analysis is performed on the moving pixels in each frame of the image to form the moving region; The video frame sequence is divided into a static time period by consecutive video frames that do not contain moving areas; The video frame sequence is divided into a motion period by consecutive video frames that contain moving areas.

3. The multi-target recognition and tracking method according to claim 1, characterized in that, For the aforementioned movement period, preliminary trajectory analysis is performed based on the temporal changes of the movement area to determine the consistency of the movement trend of the movement area relative to its own historical movement trend at each moment, including: A reference movement path is obtained by fitting the center point location of the movement region that has a preliminary matching relationship in multiple historical consecutive frames. Based on the analysis of the reference movement path, the deviation between the actual movement state and the predicted movement state of the movement area at each moment is analyzed to determine the consistency of the movement trend.

4. The multi-target recognition and tracking method according to claim 3, characterized in that, Determine the degree of overlap between the moving areas, including: If the current movement region has a preliminary matching relationship with multiple movement regions from the previous time, the current movement region is determined to be an overlapping region. The degree of overlap is determined by the ratio of the number of overlapping regions to the total number of moving regions at the current moment, and the ratio of the area of ​​the overlapping regions to the total area of ​​the moving regions.

5. The multi-target recognition and tracking method according to claim 4, characterized in that, The recognition requirement parameters for evaluating the complexity of the tracking scene at each time step are determined by combining the consistency and overlap of motion trends, including: For each moment, the identification requirement parameters are determined comprehensively based on the parameters of consistency and overlap of movement trends.

6. The multi-target recognition and tracking method according to claim 1, characterized in that, Obtain the video frame sequence captured by the surveillance camera, including: The raw video stream captured by the surveillance camera is received by an edge computing node deployed near the surveillance camera. The original video stream is preprocessed to obtain the video frame sequence.

7. The multi-target recognition and tracking method according to claim 1, characterized in that, Output the tracking and identification results of the moving area, including: Annotate the outer graph for each identified moving region in the video frame; For moving regions identified as the same object, the same identifier is associated with the circumscribed graphics in different video frames.

8. A multi-target recognition and tracking system based on a surveillance camera, characterized in that, include: Data acquisition module, time period segmentation module, demand assessment module, and computing power allocation and execution module; The data acquisition module is used to acquire video frame sequences captured by the surveillance camera; The time segmentation module is used to identify moving regions in the video frame sequence and divide the video frame sequence into motion periods containing moving regions and stationary periods not containing moving regions. The requirement assessment module is used to perform preliminary trajectory analysis based on the temporal changes of the moving area for the movement period, determine the consistency of the movement trend of the moving area itself at each moment, as well as the degree of overlap between the moving areas, and combine the consistency of the movement trend and the degree of overlap to determine the identification requirement parameters used to assess the complexity of the tracking scenario at each moment. The computing power allocation and execution module is used to determine the computing power allocation priority parameters for each moment based on the identification requirement parameters for each moment. The higher the complexity of the tracking scenario represented by the identification requirement parameters, the higher the computing power allocation priority parameter; computing resources are allocated to the corresponding time points in descending order of the computing power allocation priority parameters. If the computing power allocation priority parameter is lower than the preset priority threshold, target matching in the moving area is performed based on the continuity of the movement trend of the moving area. If the computing power allocation priority parameter is higher than or equal to the preset priority threshold, target matching in the moving area is performed based on feature point matching in the moving area; the computing resources required for target matching based on feature point matching are greater than the computing resources required for target matching based on the continuity of movement trends.

Images