An object recognition detection and real-time tracking system

Abstract

The present invention belongs to the technical field of object recognition and tracking, and discloses an object recognition detection and real-time tracking system; the system comprises the following steps: acquiring a video image sequence, preprocessing each frame of the video image sequence to obtain a plurality of standard image frames; constructing an image directed graph model based on each standard image frame, clustering the image directed graph model to obtain n potential target objects and corresponding feature descriptions; based on the n potential target objects and the corresponding feature descriptions, performing target object detection in the current frame standard image to obtain the target object; and tracking the detected target object, thereby optimizing the accuracy and robustness of detection and tracking, and enabling the target object to be reliably locked and continuously tracked.

Description

Technical Field

[0001] The present invention relates to the technical field of object recognition and tracking, and more particularly to an object recognition, detection and real-time tracking system. Background Art

[0002] The patent application publication number CN107564037A discloses a multi-robot detection and tracking method based on local features, which mainly solves the problems of large errors and "out of control" in the detection of moving objects in the intelligent monitoring technology in the existing technology. The steps are as follows: S1, select appropriate local features; S2, image preprocessing; S3, feature extraction; S4, pattern recognition; S5, extract the center of mass; S6, Kalman filter algorithm for tracking and filtering; S7, calculate the world coordinates; use the more ideal local features instead of the irregular robot as a whole as the detection object to greatly reduce the detection error and the error of the center of mass extraction; use the method of mutual feedback between detection and tracking to reduce the detection range and improve the real-time and stability of the system; when the detection result of the current image frame is not ideal, make full use of the most recent valid detection result to narrow the detection range of the next frame, with a strong error correction mechanism to avoid the "out of control" phenomenon caused by too long detection time.

[0003] However, existing methods still usually rely on artificially designed features and are difficult to adapt to the diversity and changes of target objects in complex environments. Taking robot navigation as an example, when a robot moves in indoor and outdoor environments, it needs to detect and track various objects in real time, such as people, vehicles, obstacles, etc. The shapes, sizes, and colors of these target objects vary greatly, and artificially designed features are difficult to cover all situations. Secondly, when processing video image sequences, existing methods need to re-perform feature extraction and target detection for each frame of the image, which requires huge computational effort and poor real-time performance. Taking video surveillance as an example, when multiple cameras work at the same time, a large amount of video data needs to be processed in real time. If complex feature extraction and detection are performed for each frame, it is easy to cause system overload and fail to meet the needs of real-time response. Furthermore, the detection and tracking accuracy of existing methods needs to be improved. When encountering bad weather, congested road conditions, etc., there will be various noises and complex backgrounds in the video images, and the existing methods are easily interfered with, resulting in failure in target object detection and tracking.

[0004] In view of this, the present invention proposes an object recognition, detection and real-time tracking system to solve the above problems. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art and achieve the above-mentioned objectives, the present invention provides the following technical solution: an object recognition detection and real-time tracking system, comprising: an image acquisition and processing module, a cluster analysis module, and a detection and tracking module, wherein the modules are connected to each other via wired and / or wireless means;

[0006] The image acquisition and processing module includes an acquisition unit and a processing unit. The acquisition unit is used to obtain a video image sequence, and the processing unit is used to pre-process each frame of the video image sequence to obtain several frames of standard images.

[0007] The cluster analysis module includes a model construction unit and an analysis unit. The construction unit constructs an image directed graph model based on each frame of the standard image. The analysis unit is used to perform cluster analysis on the image directed graph model to obtain n potential target objects and corresponding feature descriptions.

[0008] The detection and tracking module includes a detection unit and a tracking module. The detection unit performs target object detection in the current frame standard image based on n potential target objects and corresponding feature descriptions to obtain the target object; the tracking module is used to track the detected target object.

[0009] Furthermore, the pre-processing method includes:

[0010] Convert the image to a grayscale image, traverse each pixel in the grayscale image, count the number of times each grayscale value appears, and use an array of length 256 to store the number of times each grayscale value appears; draw a histogram based on the array; the horizontal axis of the histogram represents the grayscale value, and the vertical axis of the histogram represents the number of times the corresponding grayscale value appears;

[0011] The number of occurrences of grayscale values ​​in the histogram is used as the histogram data, and the histogram data is normalized; the normalized histogram data is fitted to obtain the mean of its Gaussian distribution. and standard deviation ;

[0012] Mean based on Gaussian distribution and standard deviation , calculate the peak height of the Gaussian distribution and peak width Peak height ;in, is the initial scale parameter, For time, is the preset growth rate; is the adaptive shape parameter;

[0013] Adaptive shape parameters ;in, and is a constant coefficient, Represents time Histogram data observed at each moment; is the kernel function;

[0014] Peak width ;

[0015] Count the number of pixels with grayscale value 0 s_0 and the number of pixels with grayscale value 255 s_255 in the histogram, and calculate the ratio of s_0 to s_255 as the noise intensity quantification index;

[0016] Perform Fourier transform on the normalized histogram data to obtain the frequency spectrum, find the peak in the frequency spectrum, and count the peak amplitude and peak period of the peak;

[0017] Preset Gaussian threshold ranges corresponding to the peak height and peak width, and if the calculated peak height and peak width are both within the corresponding Gaussian threshold ranges, it is determined that Gaussian noise exists;

[0018] A preset intensity threshold range is set. If the noise intensity quantification index is within the threshold range, it is determined that salt and pepper noise exists.

[0019] Preset the period threshold ranges corresponding to the peak amplitude and peak period, respectively. If the statistically obtained peak amplitude and peak period are within the corresponding period threshold ranges, it is determined that periodic noise exists.

[0020] If there is Gaussian noise, a Gaussian filter or a mean filter is used for filtering. If there is salt and pepper noise, a median filter or an adaptive median filter is used for filtering. If there is periodic noise, an adaptive special filter is performed. After filtering is completed, a denoised image is obtained. The denoised image is enhanced to obtain a standard image.

[0021] Furthermore, the method of performing adaptive special filtering includes:

[0022] Perform a two-dimensional discrete Fourier transform on the image to convert it from the spatial domain to the frequency domain to obtain the two-dimensional frequency spectrum of the image ,in, Indicates the frequency in the horizontal direction, Indicates the frequency in the vertical direction; initially define an adaptive band-stop filter and define the transfer function of the adaptive band-stop filter ;

[0023] ;in, Indicates frequency point The Euclidean distance to the center of the adaptive bandstop filter, is the normalization constant, and is the order control constant coefficient;

[0024] Initialize normalization constant and the order control constant coefficient; and by collecting a training data set, using cross validation or grid search methods to optimize the normalization constant on the training data set and the order control constant coefficient, find the optimal normalization constant and a combination of order control constant coefficients;

[0025] The two-dimensional frequency spectrum of the image Perform a point multiplication operation with the transfer function to obtain the two-dimensional frequency spectrum after filtering ;right A two-dimensional inverse discrete Fourier transform is performed to convert it from the frequency domain back to the spatial domain, and adaptive special filtering is completed.

[0026] Furthermore, the method of performing image enhancement on the denoised image includes:

[0027] Defining Mapping Values , and construct a correction lookup table, for each input pixel value in the denoised image , calculate the corresponding output pixel value ;in, 、 and is a nonlinear parameter; the input pixel value and output pixel value The corresponding images are stored in the correction lookup table; a mapping operation is performed based on the correction lookup table to obtain a standard image;

[0028] The mapping operation is performed as follows: for each pixel in the image, the pixel value of the three channels R, G and B is converted from the input pixel value to the corrected lookup table. Mapped to output pixel values; recombining the mapped pixel values ​​into a new image, which is the standard image.

[0029] Furthermore, the mapping value The definitions include:

[0030] Preset mapping value The search range is discretized into a series of discrete mapping value points, recorded as mapping points; the evaluation index function is defined ;

[0031] ;in, 、 and is the weight coefficient of each item, is the maximum pixel value in the image after image enhancement, is the minimum pixel value in the image after image enhancement, is the total number of gray levels of the image after image enhancement, is the index of gray level, For the The probability of gray levels, Represents the horizontal gradient of the image after image enhancement, Represents the vertical gradient of the image after image enhancement;

[0032] Within the search range, a grid containing all mapping points is constructed. For each mapping point, the value of the corresponding evaluation index function is initialized; each mapping point in the grid is traversed; for the current mapping point, it is used as the mapping value, and a mapping operation is performed to obtain the enhanced image. The value of the evaluation index function of the enhanced image is calculated, and the value of the corresponding evaluation index function in the grid is updated;

[0033] After traversing all grid points, find the mapping point with the largest value of the evaluation index function in the grid and use it as the optimal mapping point. Then build a new grid near the optimal mapping point. Repeat the search for the mapping point with the largest value of the evaluation index function in the new grid until the accuracy of the grid reaches the preset accuracy threshold.

[0034] The calculation formula of the grid accuracy is:

[0035] ;in, The mapping value corresponding to the optimal mapping point is The accuracy of the grid, is the upper limit of the search range, is the lower limit of the search range; is the number of intervals into which the search range is divided; The mapping value corresponding to the optimal mapping point is The value of the evaluation index function when for Relative to the mapped value gradient; for Relative to the mapped value The second derivative of and is the equilibrium parameter, and ; is the density control parameter, is the curvature control parameter.

[0036] Furthermore, the image directed graph model is constructed in the following manner:

[0037] The standard image is divided into several pixel blocks, and for each pixel block, its feature descriptor is extracted; the feature descriptor is a histogram of oriented gradients, a local binary pattern, or a scale-invariant feature transform;

[0038] For each pixel block in the standard image, treat it as a node in the image directed graph model and calculate the similarity between each pair of pixel blocks;

[0039] According to the spatial position of the pixel blocks in the standard image, a directed edge is connected between each pair of adjacent pixel blocks; the direction of the directed edge is from left to right or from top to bottom;

[0040] Assign a weight to each directed edge, and the weight value is equal to the similarity between the corresponding two pixel blocks; preset a similarity threshold, for two non-adjacent pixel blocks; if their similarity is greater than or equal to the similarity threshold, also connect a directed edge between the two, thus completing the construction of the directed graph model of the image.

[0041] Furthermore, the method of dividing the standard image into a plurality of pixel blocks includes:

[0042] Initialize a sliding window of fixed size; starting from the upper left corner of the standard image, slide the sliding window from left to right or from top to bottom on the image;

[0043] For the area of ​​the standard image covered by the current sliding window, the texture complexity of the pixel values ​​inside it is calculated; the calculation method of texture complexity includes:

[0044] For the area of ​​the standard image covered by the current sliding window, a gray-level co-occurrence matrix is ​​constructed, and the texture complexity is calculated based on the gray-level co-occurrence matrix. ;

[0045] ;in, is the contrast of the area, is the entropy of the pixels in the region, is the gray-level co-occurrence matrix Rank The element value of the column, is the distance weighted kernel function; is the distance attenuation factor; is the pixel weighting function;

[0046] Pixel weighting function ;in, For the The pixel corresponding to the row is The direction of the pixel corresponding to the column, is the preset main direction of interest;

[0047] A uniform threshold is preset. If the texture complexity of the area is less than the uniform threshold, the current sliding window is treated as a pixel block.

[0048] If the texture complexity of the region is greater than or equal to the uniform threshold, the sliding window is segmented;

[0049] The splitting is done as follows:

[0050] The current sliding window is divided into four equal sub-windows in the horizontal and vertical directions. For each sub-window, it is repeatedly determined whether it needs to be split again until the texture complexity of all sub-windows is less than the uniform threshold. These sub-windows are regarded as pixel blocks; the sliding window is continued until the entire standard image is covered and several pixel blocks are obtained.

[0051] Furthermore, the cluster analysis method includes:

[0052] Initialize the clustering parameters of cluster analysis, which include the number of clusters, the maximum number of local minimization q, the change rate adjustment parameter b1, and the neighborhood adjustment parameter b2;

[0053] Randomly select r nodes from the image directed graph model as the initial cluster centers; calculate the minimum distance from all nodes to these r cluster centers, and assign each node to the cluster where the nearest cluster center is located based on the calculated minimum distance; calculate the overall intra-cluster sum of squares and perform local minimization on the current cluster;

[0054] The way to perform local minimization is to randomly select a node , remove it from its cluster, define node Neighborhood ;

[0055] Neighborhood Contains nodes Distance less than Nodes, where is the maximum distance between two nodes in the image directed graph model, in the neighborhood In the calculation, the node Redistribute the total sum of squares to the other r-1 clusters, choose the redistribution method that minimizes the total sum of squares, and repeat until q local minimizations are performed; in this process, dynamically adjust the size of b2 so that the neighborhood The size of is gradually reduced; if a cluster with a smaller overall sum of squares than the current cluster is found in the process of local minimization, the current cluster is replaced by this cluster;

[0056] Repeat until the rate of change of the sum of squares within the population is less than b1. Finally, we get r clusters, each cluster corresponds to a potential target object, and a total of n potential target objects are obtained. The value of r is equal to the value of n.

[0057] For each cluster, calculate the weighted average feature descriptor of all nodes in it as the feature description of the corresponding potential target object;

[0058] The calculation formula of the weighted average feature descriptor is:

[0059] ;in, For the The weighted average feature descriptor of clusters, is the corresponding node in the cluster The weight of For nodes feature descriptor.

[0060] Furthermore, the target object detection method includes:

[0061] Define a detection sliding window, scan the current frame standard image using the detection sliding window to generate candidate target regions; for each candidate target region, extract its feature descriptor, record it as the candidate region feature descriptor, and match the candidate region feature descriptor with the obtained feature descriptions of n potential target objects; the matching method is:

[0062] Calculate the distance between the feature descriptor of the candidate region and the weighted average feature descriptor of each potential target object as the matching score between the corresponding potential target object and the candidate target region;

[0063] Mark the potential target object and candidate target area with the highest matching score, that is, mark the corresponding candidate target area as the potential target object;

[0064] A confidence threshold is preset, and candidate target areas whose matching scores with all potential target objects are less than the confidence threshold are determined to be non-target objects and eliminated; for several candidate target areas marked as the same potential target object, post-processing is performed to obtain the corresponding target object;

[0065] Post-processing methods include:

[0066] Based on the matching scores of several candidate target regions and the same potential target object, the aspect ratio of the candidate target region, and the area of ​​the candidate target region, an optimal candidate target region is screened out and marked as the potential target object;

[0067] Screening methods include:

[0068] Set the matching score threshold, eliminate candidate target areas whose matching scores are greater than the matching score threshold, set the upper and lower bounds of the aspect ratio [as_min, as_max], and for the remaining candidate areas, eliminate candidate target areas whose aspect ratios are not within the upper and lower bounds [as_min, as_max];

[0069] Set the upper and lower bounds of the area [ar_min, ar_max], and eliminate candidate target areas whose areas are not within the upper and lower bounds [ar_min, ar_max];

[0070] The candidate target area with the largest matching score among the remaining candidate target areas is taken as the most suitable candidate target area.

[0071] Furthermore, the method of tracking the detected target object includes:

[0072] Generate N2 particles, each particle represents a state of the target object, and assign an initial weight to each particle, usually set to 1 / N2; for each particle, predict the state at the next moment according to the motion model, that is, predict the particle; extract the observation features corresponding to each predicted particle, and calculate the observation likelihood of each predicted particle, that is, the similarity between the observation features and the motion model; update the weight of each particle according to the observation likelihood; calculate the optimal state of the target based on the states and corresponding weights of all particles, and take the mean of the sum of the weights as the optimal state; and resample; the resampling method is residual resampling, systematic resampling or stratified sampling; after resampling, the weights of all particles are reset to 1 / N2; repeat to track the detected target object.

[0073] The technical effects and advantages of the object recognition, detection and real-time tracking system of the present invention are as follows:

[0074] By constructing an image directed graph model and utilizing cluster analysis, the present invention can automatically discover potential target objects and their feature descriptions, avoiding the defects of manually designed features and greatly improving the adaptability to the diversity and changes of target objects in complex scenes; secondly, after performing a cluster analysis on the first frame image of the video sequence, subsequent frames only need to be detected and tracked based on the obtained potential target object features, which greatly reduces the amount of calculation and significantly improves the real-time performance of the system, enabling it to meet the strict requirements of real-time applications; thirdly, the input image is carefully preprocessed, including adaptive denoising, image enhancement and other links, which effectively improves the system's robustness to complex backgrounds and various noises, and ensures the quality of the input data; in addition, in the target detection and tracking stage, a variety of strategies are combined, such as sliding window scanning, candidate area screening, particle filtering, etc., to further optimize the accuracy and robustness of detection and tracking, so that the target object can be reliably locked and continuously tracked. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 A schematic diagram of an object recognition, detection and real-time tracking system of the present invention;

[0076] Figure 2 Schematic diagram of an object recognition, detection and real-time tracking method of the present invention. DETAILED DESCRIPTION

[0077] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0078] Example 1

[0079] See also Figure 1 As shown, this embodiment provides an object recognition, detection, and real-time tracking system, including: an image acquisition and processing module, a cluster analysis module, and a detection and tracking module. The modules are connected to each other via wired and / or wireless means to achieve data transmission between the modules;

[0080] The image acquisition and processing module includes an acquisition unit and a processing unit. The acquisition unit is used to obtain a video image sequence, and the processing unit is used to pre-process each frame of the video image sequence to obtain several frames of standard images.

[0081] The cluster analysis module includes a model construction unit and an analysis unit. The construction unit constructs an image directed graph model based on each frame of the standard image. The analysis unit is used to perform cluster analysis on the image directed graph model to obtain n potential target objects and corresponding feature descriptions.

[0082] The detection and tracking module includes a detection unit and a tracking module. The detection unit performs target object detection in the current frame standard image based on n potential target objects and corresponding feature descriptions to obtain the target object; the tracking module is used to track the detected target object.

[0083] Use a USB camera, webcam, etc. connected to the computer to capture video image frames in real time and obtain a video image sequence.

[0084] Further, the pre-processing method includes:

[0085] Convert the image to a grayscale image, traverse each pixel in the grayscale image, count the number of times each grayscale value appears, and use an array of length 256 to store the number of times each grayscale value appears; draw a histogram based on the array; the horizontal axis of the histogram represents the grayscale value (0-255), and the vertical axis of the histogram represents the number of times the corresponding grayscale value appears.

[0086] The number of grayscale values ​​in the histogram is taken as the histogram data, and the histogram data is normalized so that the sum of the number of grayscale values ​​is 1; the normalized histogram data is fitted to obtain the mean of its Gaussian distribution. and standard deviation .

[0087] Mean based on Gaussian distribution and standard deviation , calculate the peak height of the Gaussian distribution and peak width ;

[0088] Peak height ;in, is the initial scale parameter, For time, The preset growth rate makes the amplitude show exponential growth or exponential decay; is an adaptive shape parameter used to adjust the shape of the function.

[0089] Adaptive shape parameters ;in, and is a constant coefficient, is a scalar value representing the The data points observed at each moment (histogram data); is a kernel function, such as Gaussian kernel, Laplace kernel, etc.;

[0090] Peak width .

[0091] The count s_0 of the pixel points with a grayscale value of 0 and the count s_255 of the pixel points with a grayscale value of 255 in the histogram are counted, and the ratio of s_0 to s_255 is calculated as a noise intensity quantification indicator.

[0092] Perform Fourier transform on the normalized histogram data to obtain the frequency spectrum, find the peak in the frequency spectrum, and count the peak amplitude and peak period of the peak.

[0093] The peak height and peak width are preset to have corresponding Gaussian threshold ranges. If the calculated peak height and peak width are both within the corresponding Gaussian threshold ranges, it is determined that Gaussian noise exists.

[0094] A preset intensity threshold range is set. If the noise intensity quantification index is within the threshold range, it is determined that salt and pepper noise exists.

[0095] A period threshold range corresponding to each of the peak amplitude and the peak period is preset. If the statistically obtained peak amplitude and peak period are within the corresponding period threshold range, it is determined that periodic noise exists.

[0096] If there is Gaussian noise, a Gaussian filter or a mean filter is used for filtering. If there is salt and pepper noise, a median filter or an adaptive median filter is used for filtering. If there is periodic noise, an adaptive special filter is performed. After filtering is completed, a denoised image is obtained.

[0097] Methods for performing adaptive special filtering include:

[0098] Perform a two-dimensional discrete Fourier transform (2D DFT) on the image to convert it from the spatial domain to the frequency domain to obtain the two-dimensional frequency spectrum of the image ,in, Indicates the frequency in the horizontal direction, Indicates the frequency in the vertical direction.

[0099] Initially define an adaptive band-stop filter, determine the frequency range that needs to be filtered out based on periodic noise, and define the transfer function of the adaptive band-stop filter ;

[0100] ;in, Indicates frequency point The Euclidean distance to the center of the adaptive bandstop filter, is a normalization constant that controls the bandwidth of the filter, and is the order control constant coefficient, and the steepness of the adaptive band-stop filter increases or decays exponentially with the increase of frequency.

[0101] Initialize normalization constant and the order control constant coefficient; and by collecting a training data set (an image data set containing various periodic noises, which can be images collected from real scenes or artificially synthesized images), using cross-validation or grid search methods to optimize the normalization constant on the training data set The goal is to find the optimal normalization constant that maximizes the noise removal effect. and the order control constant coefficient.

[0102] The two-dimensional frequency spectrum of the image Perform a point multiplication operation with the transfer function to obtain the two-dimensional frequency spectrum after filtering ;right A two-dimensional inverse discrete Fourier transform (2D IDFT) is performed to convert it from the frequency domain back to the spatial domain, and the filtering is completed.

[0103] Furthermore, the denoised image is enhanced to obtain a standard image; specifically, the mapping value is defined , and construct a correction lookup table, for each input pixel value in the denoised image (usually 0 to 255), calculate the corresponding output pixel value ;in, 、 and Is a nonlinear parameter used to control the degree of nonlinearity; the input pixel value and output pixel value The corresponding values ​​are stored in a correction lookup table; a mapping operation is performed based on the correction lookup table to obtain a standard image.

[0104] The mapping operation is performed as follows: for each pixel in the image, the pixel value of the three channels R, G and B is converted from the input pixel value to the corrected lookup table. Mapped to output pixel values; recombining the mapped pixel values ​​into a new image, which is the standard image.

[0105] Mapping Values The definitions include:

[0106] Preset mapping value The search range is usually between [0.2, 5]. The search range is discretized into a series of discrete mapping value points, recorded as mapping points; define the evaluation index function ;

[0107] ;in, 、 and is the weight coefficient of each item, is the maximum pixel value in the image after image enhancement, is the minimum pixel value in the image after image enhancement, is the total number of gray levels of the image after image enhancement, is the index of gray level, For the The probability of gray levels, Represents the horizontal gradient of the image after image enhancement, Represents the vertical gradient of the image after image enhancement.

[0108] Within the search range, a grid containing all mapping points is constructed. For each mapping point, the value of the corresponding evaluation index function is initialized, such as setting it to 0 or other default values; each mapping point in the grid is traversed.

[0109] For the current mapping point, it is used as the mapping value, a mapping operation is performed to obtain the enhanced image, the value of the evaluation index function of the enhanced image is calculated, and the value of the corresponding evaluation index function in the grid is updated.

[0110] After traversing all grid points, find the mapping point with the largest value of the evaluation index function in the grid, which is used as the optimal mapping point. Then build a new grid near the optimal mapping point. Repeat the search for the mapping point with the largest value of the evaluation index function in the new grid until the accuracy of the grid reaches the preset accuracy threshold.

[0111] The calculation formula of the grid accuracy is:

[0112] ;in, The mapping value corresponding to the optimal mapping point is The accuracy of the grid, is the upper limit of the search range, is the lower limit of the search range; is the number of intervals into which the search range is divided; The mapping value corresponding to the optimal mapping point is The value of the evaluation index function when for Relative to the mapped value When the gradient of the value of the evaluation index function is large, the density of grid points increases, resulting in a finer search in the area; for Relative to the mapped value The second derivative of and is the equilibrium parameter, and ; is the density control parameter, are curvature control parameters, which are used to control the degree of change of grid density.

[0113] Furthermore, the image directed graph model is constructed in the following ways:

[0114] The standard image is divided into several pixel blocks, and for each pixel block, its feature descriptor is extracted; the feature descriptor is HOG (Histogram of Oriented Gradients), LBP (Local Binary Pattern) or SIFT (Scale Invariant Feature Transform).

[0115] For each pixel block in the standard image, it is treated as a node in the image directed graph model, and the similarity between each pair of pixel blocks (cosine similarity between feature descriptors) is calculated.

[0116] According to the spatial position of the pixel blocks in the standard image, a directed edge is connected between each pair of adjacent pixel blocks; the direction of the directed edge is from left to right or from top to bottom.

[0117] Assign a weight to each directed edge, and the weight value is equal to the similarity between the corresponding two pixel blocks; preset a similarity threshold, for two non-adjacent pixel blocks; if their similarity is greater than or equal to the similarity threshold, also connect a directed edge between the two, indicating that there is a similarity relationship between them, thus completing the construction of the image directed graph model.

[0118] Ways to divide a standard image into several pixel blocks include:

[0119] Initialize a sliding window of a fixed size, such as 8x8 pixels; starting from the upper left corner of the standard image, slide the sliding window from left to right or from top to bottom on the image. The step size of each slide can be fixed (such as 4 pixels) or adaptive.

[0120] For the area of ​​the standard image covered by the current sliding window, the texture complexity of the pixel values ​​inside it is calculated; the calculation method of texture complexity includes:

[0121] For the area of ​​the standard image covered by the current sliding window, a grayscale co-occurrence matrix is ​​constructed. Specifically, four main directions are selected, including 0° (horizontal), 45°, 90° (vertical) and 135°, and the distance is 1 pixel or other distances are set as required; the co-occurrence matrix of the grayscale values ​​of pixels in the area is calculated; any pixel is defined as a reference pixel, and for each pair of reference pixels and their neighboring pixels, the corresponding position of the combination of their grayscale values ​​in the grayscale co-occurrence matrix is ​​calculated, and the count value of the position is added by 1, and all element values ​​in the grayscale co-occurrence matrix are divided by the total number of pixel pairs to normalize the element values ​​of the grayscale co-occurrence matrix to the range of [0, 1] to obtain the grayscale co-occurrence matrix.

[0122] Texture complexity is calculated based on the gray-level co-occurrence matrix ;

[0123] ;in, is the contrast of the area, is the entropy of the pixels in the region, is the gray-level co-occurrence matrix Rank The element value of the column, is a distance-weighted kernel function (using a Gaussian kernel or other kernel function), which is used to weight pixel pairs with different distances; is the distance attenuation factor, which controls the effect of distance on homogeneity; It is a pixel weighting function, which is used to assign different weights to different element values.

[0124] Pixel weighting function ;in, For the The pixel corresponding to the row is The direction of the pixel corresponding to the column, is the preset main direction of interest.

[0125] A uniformity threshold is preset. If the texture complexity of the area is less than the uniformity threshold, it means that the area is relatively uniform, and the current sliding window is regarded as a pixel block. If the texture complexity of the area is greater than or equal to the uniformity threshold, it means that there are more details and edge information in the area, and the sliding window is segmented.

[0126] The splitting is done as follows:

[0127] The current sliding window is divided into four equal sub-windows in the horizontal and vertical directions. For each sub-window, it is repeatedly determined whether it needs to be split again until the texture complexity of all sub-windows is less than the uniform threshold. These sub-windows are regarded as pixel blocks; all the obtained pixel blocks are recorded, including their position, size and other information; the sliding window is continued until the entire standard image is covered and several pixel blocks are obtained.

[0128] In the above process, the image will be divided into multiple pixel blocks of varying sizes. In areas with simple textures, the pixel blocks will be larger, while in areas with complex textures, the pixel blocks will be smaller. This is conducive to balancing the trade-off between computational efficiency and detail retention.

[0129] Further, cluster analysis methods include:

[0130] Initialize the clustering parameters of the cluster analysis, which include the number of clusters, the maximum number of local minimization q, the change rate adjustment parameter b1, and the neighborhood adjustment parameter b2.

[0131] Randomly select r nodes (pixel blocks) from the image directed graph model as the initial cluster centers; calculate the minimum distance from all nodes to these r cluster centers, and assign each node to the cluster where the nearest cluster center is located based on the calculated minimum distance.

[0132] Calculate the total internal sum of squares of the current cluster (the sum of squares of the total internal distances) and minimize it locally for the current cluster.

[0133] The way to perform local minimization is to randomly select a node , remove it from its cluster, define node Neighborhood , neighborhood Contains nodes Distance less than Nodes, where is the maximum distance between two nodes in the image directed graph model, in the neighborhood In the calculation, the node Redistribute the total sum of squares to the other r-1 clusters, choose the redistribution method that minimizes the total sum of squares, and repeat until q local minimizations are performed; in this process, dynamically adjust the size of b2 so that the neighborhood The size of is gradually reduced (the degree of reduction is customized) to accelerate convergence; if a cluster with a smaller overall sum of squares than the current cluster is found during the local minimization process, the current cluster is replaced by this cluster.

[0134] Repeat until the rate of change of the sum of squares within the population is less than b1. Finally, we get r clusters, each cluster corresponds to a potential target object, and a total of n potential target objects are obtained. The value of r is equal to the value of n.

[0135] For each cluster, the weighted average feature descriptor of all nodes (pixel blocks) in it is calculated as the feature description of the corresponding potential target object.

[0136] The calculation formula of the weighted average feature descriptor is:

[0137] ;in, For the The weighted average feature descriptor of clusters, is the corresponding node in the cluster The weight (similarity) of For nodes feature descriptor.

[0138] Furthermore, methods for detecting target objects include:

[0139] Define a detection sliding window, scan the current frame standard image using the detection sliding window to generate candidate target areas; the size of the detection sliding window can be fixed or adaptive, and it usually slides from left to right and from top to bottom on the image with a certain step size.

[0140] For each candidate target region, extract its feature descriptor, record it as the candidate region feature descriptor, and match the candidate region feature descriptor with the obtained feature descriptions of the n potential target objects; the matching method is:

[0141] The distance (such as Euclidean distance, cosine similarity, etc.) between the feature descriptor of the candidate region and the weighted average feature descriptor of each potential target object is calculated as the matching score between the corresponding potential target object and the candidate target region.

[0142] The potential target object and candidate target area with the highest matching score are marked, that is, the corresponding candidate target area is marked as the potential target object.

[0143] A confidence threshold is preset, and candidate target areas whose matching scores with all potential target objects are less than the confidence threshold are judged as non-target objects and eliminated; for several candidate target areas marked as the same potential target object, post-processing is performed to obtain the corresponding target object.

[0144] Post-processing methods include:

[0145] Based on the matching scores of several candidate target regions and the same potential target object, the aspect ratio of the candidate target region and the area of ​​the candidate target region, an optimal candidate target region is screened out and marked as the potential target object.

[0146] Screening methods include:

[0147] Set the matching score threshold, eliminate candidate target areas whose matching scores are greater than the matching score threshold, set the upper and lower bounds of the aspect ratio [as_min, as_max], and for the remaining candidate areas, eliminate candidate target areas whose aspect ratios are not within the upper and lower bounds [as_min, as_max].

[0148] Set the upper and lower bounds of the area [ar_min, ar_max], and eliminate candidate target areas whose areas are not within the upper and lower bounds [ar_min, ar_max];

[0149] The candidate target area with the largest matching score among the remaining candidate target areas is taken as the most suitable candidate target area.

[0150] It can effectively eliminate a large number of interference areas that obviously do not conform to the characteristics of the target object and retain the most likely target candidate areas, thereby improving the accuracy and robustness of detection.

[0151] Furthermore, the method of tracking the detected target object includes:

[0152] Generate N2 particles (samples), each particle represents a state of the target object (position, scale, direction, etc.). Usually, Gaussian distribution is used to randomly sample around the target, and an initial weight is assigned to each particle, which is usually set to 1 / N2. For each particle, the state at the next moment is predicted based on the motion model (such as constant velocity model, random walk model, etc.), that is, the predicted particle. The observation features corresponding to each predicted particle (such as color histogram, HOG, etc.) are extracted, and the observation likelihood of each predicted particle is calculated, that is, the similarity between the observation feature and the motion model. According to the observation likelihood, the weight of each particle is updated (the weight of the previous iteration is added to the observation likelihood and normalized). Based on the states and corresponding weights of all particles, the optimal state of the target is calculated, and the mean of the sum of the weights is taken as the optimal state. Since the weight of the particles will gradually degenerate, resampling is performed to eliminate low-weight particles. The resampling method is residual resampling, systematic resampling or stratified sampling. After resampling, the weights of all particles are reset to 1 / N2. Repeated tracking of the detected target object is achieved.

[0153] This embodiment, by constructing an image directed graph model and utilizing cluster analysis, can automatically discover potential target objects and their feature descriptions, avoiding the defects of manually designed features and greatly improving the adaptability to the diversity and changes of target objects in complex scenes; secondly, after performing a cluster analysis on the first frame image of the video sequence, subsequent frames only need to be detected and tracked based on the obtained potential target object features, which greatly reduces the amount of calculation and significantly improves the real-time performance of the system, enabling it to meet the strict requirements of real-time applications; thirdly, the input image is carefully preprocessed, including adaptive denoising, image enhancement and other links, which effectively improves the system's robustness to complex backgrounds and various noises, and ensures the quality of the input data; in addition, in the target detection and tracking stage, a variety of strategies are combined, such as sliding window scanning, candidate area screening, particle filtering, etc., to further optimize the accuracy and robustness of detection and tracking, so that the target object can be reliably locked and continuously tracked.

[0154] Example 2

[0155] See also Figure 2 As shown, for the parts not described in detail in this embodiment, please refer to the description of Example 1. A method for object recognition, detection and real-time tracking is provided, including:

[0156] S1. Obtain a video image sequence, pre-process each frame of the video image sequence, and obtain several frames of standard images;

[0157] S2. Based on each frame of the standard image, an image directed graph model is constructed, and cluster analysis is performed on the image directed graph model to obtain n potential target objects and corresponding feature descriptions;

[0158] S3. Based on the n potential target objects and the corresponding feature descriptions, target object detection is performed in the current frame standard image to obtain the target object; and the detected target object is tracked.

[0159] Example 3

[0160] This embodiment discloses an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the operation mode of the object recognition, detection, and real-time tracking method provided above is implemented.

[0161] Since the electronic device described in this embodiment is an electronic device used to implement an object recognition detection and real-time tracking method in the embodiment of this application, based on the object recognition detection and real-time tracking method described in the embodiment of this application, those skilled in the art will be able to understand the specific implementation of the electronic device of this embodiment and its various variations, so how the electronic device implements the method in the embodiment of this application will not be described in detail here. As long as those skilled in the art implement the electronic device used in the object recognition detection and real-time tracking method in the embodiment of this application, it falls within the scope of protection to be provided by this application.

[0162] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters and thresholds in the formulas are set by technicians in this field according to actual conditions.

[0163] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that for users of ordinary skill in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. An object recognition detection and real-time tracking system, characterized in that: include: The image acquisition and processing module, the cluster analysis module, and the detection and tracking module are connected to each other via wired and / or wireless means; The image acquisition and processing module includes an acquisition unit and a processing unit. The acquisition unit is used to obtain a video image sequence. The processing unit is used to pre-process each frame of the video image sequence to obtain several frames of standard images; including: for the presence of Gaussian noise, using a Gaussian filter or a mean filter for filtering; for the presence of salt and pepper noise, using a median filter or an adaptive median filter for filtering; for the presence of periodic noise, performing adaptive special filtering to obtain a denoised image; defining the mapping value , and construct a correction lookup table, for each input pixel value in the denoised image , calculate the corresponding output pixel value, and input pixel value and output pixel value The corresponding values ​​are stored in the correction lookup table; a mapping operation is performed based on the correction lookup table to obtain a standard image; the mapping value The definitions include: Preset mapping value The search range is discretized to obtain mapping points; an evaluation index function is defined; within the search range, a grid containing all mapping points is constructed, the mapping points are traversed, a mapping operation is performed, an enhanced image is obtained, the value of the evaluation index function of the enhanced image is calculated, and the mapping point with the largest value of the evaluation index function in the grid is found as the optimal mapping point. A new grid is constructed near the optimal mapping point; and the search for the mapping point with the largest value of the evaluation index function in the new grid is repeated until the accuracy of the grid reaches a preset accuracy threshold. The cluster analysis module includes a model construction unit and an analysis unit. The construction unit constructs an image directed graph model based on each frame of the standard image. The analysis unit is used to perform cluster analysis on the image directed graph model to obtain n potential target objects and corresponding feature descriptions. The detection and tracking module includes a detection unit and a tracking module. The detection unit detects the target object in the current frame standard image based on n potential target objects and corresponding feature descriptions to obtain the target object; The tracking module is used to track the detected target objects, including: Generate N2 particles, each particle represents a state of the target object, and assign an initial weight to each particle, set to 1 / N2; for each particle, predict the state at the next moment according to the motion model, that is, predict the particle; extract the observation feature corresponding to each predicted particle, and calculate the observation likelihood of each predicted particle, that is, the similarity between the observation feature and the motion model; update the weight of each particle according to the observation likelihood; calculate the optimal state of the target based on the states and corresponding weights of all particles, and take the mean of the sum of the weights as the optimal state; and resample; the resampling method is residual resampling, systematic resampling or stratified sampling; after resampling, the weights of all particles are reset to 1 / N2; repeatedly implement tracking of the detected target object.

2. The object recognition detection and real-time tracking system according to claim 1, characterized in that: Convert the image to a grayscale image, traverse each pixel in the grayscale image, count the number of times each grayscale value appears, and use an array of length 256 to store the number of times each grayscale value appears; Draw a histogram based on the array; The horizontal axis of the histogram represents the grayscale value, and the vertical axis of the histogram represents the number of times the corresponding grayscale value appears; The number of occurrences of grayscale values ​​in the histogram is used as the histogram data, and the histogram data is normalized; the normalized histogram data is fitted to obtain the mean of its Gaussian distribution. and standard deviation ; Mean based on Gaussian distribution and standard deviation , calculate the peak height of the Gaussian distribution and peak width Peak height ;in, is the initial scale parameter, For time, is the preset growth rate; is the adaptive shape parameter; Adaptive shape parameters ;in, and is a constant coefficient, Represents time Histogram data observed at each moment; is the kernel function; Peak width ; Count the number of pixels with grayscale value 0 s_0 and the number of pixels with grayscale value 255 s_255 in the histogram, and calculate the ratio of s_0 to s_255 as the noise intensity quantification index; Perform Fourier transform on the normalized histogram data to obtain the frequency spectrum, find the peak in the frequency spectrum, and count the peak amplitude and peak period of the peak; Preset Gaussian threshold ranges corresponding to the peak height and peak width, and if the calculated peak height and peak width are both within the corresponding Gaussian threshold ranges, it is determined that Gaussian noise exists; A preset intensity threshold range is set. If the noise intensity quantification index is within the threshold range, it is determined that salt and pepper noise exists. A period threshold range corresponding to each of the peak amplitude and the peak period is preset. If the statistically obtained peak amplitude and peak period are within the corresponding period threshold range, it is determined that periodic noise exists.

3. The object recognition detection and real-time tracking system according to claim 2, characterized in that: The method for performing adaptive special filtering includes: Perform a two-dimensional discrete Fourier transform on the image to convert it from the spatial domain to the frequency domain to obtain the two-dimensional frequency spectrum of the image ,in, Indicates the frequency in the horizontal direction, Indicates the frequency in the vertical direction; initially define an adaptive band-stop filter and define the transfer function of the adaptive band-stop filter ; ;in, Indicates frequency point The Euclidean distance to the center of the adaptive bandstop filter, is the normalization constant, and is the order control constant coefficient; Initialize normalization constant and the order control constant coefficient; and by collecting a training data set, using cross validation or grid search methods to optimize the normalization constant on the training data set and the order control constant coefficient, find the optimal normalization constant and a combination of order control constant coefficients; The two-dimensional frequency spectrum of the image Perform a point multiplication operation with the transfer function to obtain the two-dimensional frequency spectrum after filtering ;right A two-dimensional inverse discrete Fourier transform is performed to convert it from the frequency domain back to the spatial domain, and adaptive special filtering is completed.

4. The object recognition detection and real-time tracking system according to claim 3, characterized in that: The mapping operation is performed as follows: for each pixel in the image, the pixel value of each of the three channels R, G and B is changed from the input pixel value to the corrected lookup table. Mapped to output pixel values; recombining the mapped pixel values ​​into a new image, which is the standard image.

5. The object recognition detection and real-time tracking system according to claim 4, characterized in that: The evaluation index function ;in, 、 and is the weight coefficient of each item, is the maximum pixel value in the image after image enhancement, is the minimum pixel value in the image after image enhancement, is the total number of gray levels of the image after image enhancement, is the index of gray level, For the The probability of gray levels, Represents the horizontal gradient of the image after image enhancement, Represents the vertical gradient of the image after image enhancement; The calculation formula of the grid accuracy is: ;in, The mapping value corresponding to the optimal mapping point is The accuracy of the grid, is the upper limit of the search range, is the lower limit of the search range; is the number of intervals into which the search range is divided; The mapping value corresponding to the optimal mapping point is The value of the evaluation index function when for Relative to the mapped value gradient; for Relative to the mapped value The second derivative of and is the equilibrium parameter, and ; is the density control parameter, is the curvature control parameter.

6. The object recognition detection and real-time tracking system according to claim 5, characterized in that: The method of constructing the image directed graph model includes: The standard image is divided into several pixel blocks, and for each pixel block, its feature descriptor is extracted; the feature descriptor is a histogram of oriented gradients, a local binary pattern, or a scale-invariant feature transform; For each pixel block in the standard image, treat it as a node in the image directed graph model and calculate the similarity between each pair of pixel blocks; According to the spatial position of the pixel blocks in the standard image, a directed edge is connected between each pair of adjacent pixel blocks; the direction of the directed edge is from left to right or from top to bottom; Assign a weight to each directed edge, and the weight value is equal to the similarity between the corresponding two pixel blocks; preset a similarity threshold, for two non-adjacent pixel blocks; if their similarity is greater than or equal to the similarity threshold, also connect a directed edge between the two, thus completing the construction of the directed graph model of the image.

7. The object recognition detection and real-time tracking system according to claim 6, characterized in that: Ways to divide a standard image into several pixel blocks include: Initialize a sliding window of fixed size; starting from the upper left corner of the standard image, slide the sliding window from left to right or from top to bottom on the image; For the area of ​​the standard image covered by the current sliding window, the texture complexity of the pixel values ​​inside it is calculated; the calculation method of texture complexity includes: For the area of ​​the standard image covered by the current sliding window, a gray-level co-occurrence matrix is ​​constructed, and the texture complexity is calculated based on the gray-level co-occurrence matrix. ; ;in, is the contrast of the area, is the entropy of the pixels in the region, is the gray-level co-occurrence matrix Rank The element value of the column, is the distance weighted kernel function; is the distance attenuation factor; is the pixel weighting function; Pixel weighting function ;in, For the The pixel point corresponding to the row is The direction of the pixel corresponding to the column, is the preset main direction of interest; A uniform threshold is preset. If the texture complexity of the area is less than the uniform threshold, the current sliding window is treated as a pixel block. If the texture complexity of the region is greater than or equal to the uniform threshold, the sliding window is segmented; The splitting is done as follows: The current sliding window is divided into four equal sub-windows in the horizontal and vertical directions. For each sub-window, it is repeatedly determined whether it needs to be split again until the texture complexity of all sub-windows is less than the uniform threshold. These sub-windows are regarded as pixel blocks; the sliding window is continued until the entire standard image is covered and several pixel blocks are obtained.

8. The object recognition, detection and real-time tracking system according to claim 7, characterized in that: The cluster analysis method includes: Initialize the clustering parameters of cluster analysis, which include the number of clusters, the maximum number of local minimization q, the change rate adjustment parameter b1, and the neighborhood adjustment parameter b2; Randomly select r nodes from the image directed graph model as the initial cluster centers; calculate the minimum distance from all nodes to these r cluster centers, and assign each node to the cluster where the nearest cluster center is located based on the calculated minimum distance; calculate the overall intra-cluster sum of squares and perform local minimization on the current cluster; The way to perform local minimization is to randomly select a node , remove it from its cluster, define node Neighborhood ; Neighborhood Contains nodes Distance less than Nodes, where is the maximum distance between two nodes in the image directed graph model, in the neighborhood In the calculation, the node Redistribute the total sum of squares to the other r-1 clusters, choose the redistribution method that minimizes the total sum of squares, and repeat until q local minimizations are performed; in this process, dynamically adjust the size of b2 so that the neighborhood The size of is gradually reduced; if a cluster with a smaller overall sum of squares than the current cluster is found in the process of local minimization, the current cluster is replaced by this cluster; Repeat until the rate of change of the sum of squares within the population is less than b1. Finally, we get r clusters, each cluster corresponds to a potential target object, and a total of n potential target objects are obtained. The value of r is equal to the value of n. For each cluster, calculate the weighted average feature descriptor of all nodes in it as the feature description of the corresponding potential target object; The calculation formula of the weighted average feature descriptor is: ;in, For the The weighted average feature descriptor of clusters, is the corresponding node in the cluster The weight of For nodes feature descriptor.

9. The object recognition detection and real-time tracking system according to claim 8, characterized in that: The method for detecting the target object includes: Define a detection sliding window, scan the current frame standard image using the detection sliding window to generate candidate target regions; for each candidate target region, extract its feature descriptor, record it as the candidate region feature descriptor, and match the candidate region feature descriptor with the obtained feature descriptions of n potential target objects; the matching method is: Calculate the distance between the feature descriptor of the candidate region and the weighted average feature descriptor of each potential target object as the matching score between the corresponding potential target object and the candidate target region; Mark the potential target object and candidate target area with the highest matching score, that is, mark the corresponding candidate target area as the potential target object; A confidence threshold is preset, and candidate target areas whose matching scores with all potential target objects are less than the confidence threshold are determined to be non-target objects and eliminated; for several candidate target areas marked as the same potential target object, post-processing is performed to obtain the corresponding target object; Post-processing methods include: Based on the matching scores of several candidate target regions and the same potential target object, the aspect ratio of the candidate target region, and the area of ​​the candidate target region, an optimal candidate target region is screened out and marked as the potential target object; Screening methods include: Set the matching score threshold, eliminate candidate target areas whose matching scores are greater than the matching score threshold, set the upper and lower bounds of the aspect ratio [as_min, as_max], and for the remaining candidate areas, eliminate candidate target areas whose aspect ratios are not within the upper and lower bounds [as_min, as_max]; Set the upper and lower bounds of the area [ar_min, ar_max], and eliminate candidate target areas whose areas are not within the upper and lower bounds [ar_min, ar_max]; The candidate target area with the largest matching score among the remaining candidate target areas is taken as the most suitable candidate target area.

Images