Image Data Retrieval Method and System Based on Big Data Surveillance

Abstract

This invention relates to the field of intelligent surveillance technology, and more particularly to an image data retrieval method and system based on surveillance big data. The method includes: extracting the visibility and color features of pedestrian key points from surveillance trajectory segments; calculating viewpoint similarity using the cosine similarity of visibility vectors combined with the ratio of frequency standard deviations; selecting comparable key points and weighting them by the minimum visible frequency; calculating appearance similarity by combining color differences; introducing time constraint parameters to optimize the initial clustering distance to obtain a significant clustering distance; clustering matching for the same pedestrian trajectory; and finally establishing a retrieval index to complete image data retrieval. This invention improves pedestrian information retrieval performance by extracting key point visibility and color information, fusing optimized viewpoint and appearance similarity to construct an initial clustering distance, and optimizing with time constraint parameters to avoid interference from multi-camera scenes.

Description

Technical Field

[0001] This invention relates to the field of intelligent surveillance technology, and in particular to an image data retrieval method and system based on large-scale surveillance data. Background Technology

[0002] With the application of video surveillance systems in fields such as public security management, the technology for acquiring and storing surveillance image data has developed rapidly, resulting in the continuous accumulation of massive amounts of surveillance big data containing time and spatial tags. This data covers various target information within the monitored area, is vast in scale and complex in content, providing a rich foundation for subsequent data analysis and retrieval, while also placing higher demands on related technologies.

[0003] Image retrieval is a crucial component of video surveillance systems, enabling the rapid location of relevant information about target pedestrians from massive amounts of surveillance data. This provides vital support for security maintenance, incident tracing, and other scenarios. Accurate and efficient retrieval results help staff promptly track the movements of target pedestrians, improving the targeting and efficiency of management and response, and playing an irreplaceable role in ensuring public safety.

[0004] Currently, in multi-camera surveillance networks, the same pedestrian may appear in the surveillance footage of different cameras at different times. However, due to differences in lighting and viewing angles under different cameras, as well as the uncertainty of the pedestrian's movement time between cameras, the complete trajectory of the same pedestrian is divided into multiple discrete segments. This makes it difficult for existing retrieval technologies to accurately correlate these segments, resulting in inaccurate results when retrieving information about the pedestrian. Summary of the Invention

[0005] To address the problem of inaccurate pedestrian information retrieval in existing technologies due to the segmentation of the same pedestrian's trajectory into discrete fragments caused by differences in camera lighting and viewing angles, as well as the uncertainty of pedestrian movement time across cameras in multi-camera surveillance networks, this invention provides solutions in the following aspects.

[0006] In the first aspect, the image data retrieval method based on big data surveillance includes: extracting pedestrian trajectory segments from surveillance equipment; extracting key point features of pedestrians in each trajectory segment using a preset pose model, wherein the key point features include visibility information and color information; calculating the visibility frequency of each key point in each trajectory segment based on the visibility information in the key point features and constructing a visibility vector; calculating the cosine similarity of the visibility vectors of two trajectory segments; combining the standard deviation ratio of the visibility frequencies of key points in two trajectory segments to obtain the viewpoint similarity between any two trajectory segments; and selecting a set of comparable key points in any two trajectory segments that meet the visibility threshold based on the color information in the key point features, and using the visibility frequency of each key point as the basis for the comparison. The minimum value is used as the weight, and the appearance similarity between any two trajectory segments is calculated by combining the difference in the mean chroma of key points in the trajectory segments. A time constraint is introduced, and the timestamp sequence corresponding to all frames contained in each trajectory segment is obtained. The time constraint parameter is determined by judging whether there is an intersection between the timestamp sequences of any two trajectory segments. The initial clustering distance is calculated by combining viewpoint similarity and appearance similarity. The initial clustering distance is adjusted and optimized using the time constraint parameter to obtain a significant clustering distance. All trajectory segments are clustered based on the significant clustering distance to complete the trajectory segment matching for the same pedestrian. A retrieval index is built based on the clustering results of the trajectory segments, and pedestrian-related image data is retrieved based on the retrieval index.

[0007] Preferably, the visibility frequency of each key point in each trajectory segment is calculated and a visibility vector is constructed, including:

[0008] Taking any trajectory segment as the segment to be analyzed, obtain the vector of visibility information of each frame in the segment to be analyzed. Traverse all frame images in the trajectory segment, take any key point as the target point, extract the visibility value of the target point in each frame, and accumulate all visibility values ​​to obtain the total number of visible frames of the target point. Divide the total number of visible frames by the total number of frames in the trajectory segment to obtain the visibility frequency of the target point. Preset the number of key points and construct the visibility vector according to the visibility frequency of the key points.

[0009] Preferably, the method for calculating the viewpoint similarity includes:

[0010] For any two trajectory segments, calculate the cosine similarity of the visibility vectors corresponding to the two trajectory segments, and perform a linear transformation on the calculation result to obtain the directional similarity; calculate the ratio of the standard deviations of the visibility vectors of the two trajectory segments, and take the absolute value of 1 minus the standard deviation ratio as the absolute deviation. Perform an exponential mapping on the negative absolute deviation through the natural exponential function to obtain the dispersion consistency; take the product of the directional similarity and the dispersion consistency as the viewpoint similarity.

[0011] Preferably, the step of obtaining the set of comparable key points includes:

[0012] For any two trajectory segments, based on the human pose detection algorithm, the complete set of human key points corresponding to the two trajectory segments is extracted respectively. If the visibility frequency of each human key point in the two trajectory segments is greater than or equal to the preset visibility threshold, the corresponding key point is included in the set of comparable key points. Otherwise, if each key point only meets one condition or does not meet any condition, the key point is excluded. All key points in the complete set of human key points are traversed to obtain the set of comparable key points.

[0013] Preferably, calculating the appearance similarity between any two trajectory segments includes:

[0014] For each keypoint in the comparable keypoint set, extract the keypoint at the 1st... The visible frequency in the first trajectory segment and the first The visible frequency of each trajectory segment is selected, and the minimum visible frequency of key points is selected as the weight coefficient of a single key point. The normalized weight coefficient is then obtained.

[0015] Calculate the degree of difference between the mean chromaticity values ​​of key points in two trajectory segments, and use the natural exponential function to perform an exponential mapping on the degree of difference to obtain the chromaticity similarity of individual key points;

[0016] The appearance similarity between any two trajectory segments is obtained by weighting and summing the chromatic similarity of all key points in the comparable key point set with the corresponding normalized weight coefficients.

[0017] Preferably, calculating the degree of difference between the mean chromaticity values ​​of key points in two trajectory segments includes:

[0018] Localization based on attitude estimation model The trajectory segment of the first The first frame of the image Pixel coordinates of key points The neighboring pixels within a preset range are selected with the coordinates as the center. The mean values ​​of the three channels within the neighboring pixels are calculated to obtain the original color feature values. The original color feature values ​​are then normalized and converted into the HSV color space to obtain the chromaticity of the HSV color space.

[0019] Calculate the difference between the mean chromaticity values ​​of key points in any two trajectory segments to obtain the degree of difference between the mean chromaticity values ​​of key points in the two trajectory segments.

[0020] Preferably, the calculation method for the significant cluster distance includes:

[0021] Get the The trajectory segment and the first The initial clustering distance of each trajectory segment is used to extract the effective time intervals corresponding to two trajectory segments respectively;

[0022] Calculate the intersection of two valid time intervals, and determine the intersection state using an indicator function. If the valid time intervals of the two trajectory segments do not overlap, the indicator function value is 1; if the valid time intervals of the two trajectory segments overlap, the indicator function value is 0.

[0023] The result of the exponential function is added to the preset adjustment coefficient, and the reciprocal is taken to obtain the time constraint weight factor. The initial cluster distance is multiplied by the time constraint weight factor to obtain the significant cluster distance.

[0024] Secondly, an image data retrieval system based on surveillance big data includes a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the above-mentioned image data retrieval method based on surveillance big data is implemented.

[0025] The present invention has the following effects:

[0026] 1. This invention constructs an initial clustering distance by fusing perspective similarity and appearance similarity. Perspective similarity is combined with the cosine similarity of visibility vectors and the consistency of frequency dispersion. Appearance similarity is calculated based on the color features of comparable key points. At the same time, a time constraint parameter is introduced to optimize the clustering distance. This effectively avoids the problem of trajectory segment dispersion caused by differences in lighting and perspectives of multiple cameras and uncertainties in cross-camera time. It significantly reduces the probability of misclassifying the trajectory of the same pedestrian and misclustering the trajectories of different pedestrians, and significantly improves the accuracy of pedestrian information retrieval.

[0027] 2. This invention extracts key point features and incorporates visibility and color information, calculates the visibility frequency of key points to construct a visibility vector, and filters a set of comparable key points that meet the threshold. This not only eliminates the interference of low-reliability key points, but also reduces the impact of lighting changes on features through HSV color space conversion, making the feature representation more adaptable to the complex environment under multi-camera monitoring, and improving the versatility and robustness of the method. Attached Figure Description

[0028] Figure 1 This is a flowchart of steps S1-S4 in the image data retrieval method based on big data monitoring according to an embodiment of the present invention.

[0029] Figure 2 This is a structural block diagram of an image data retrieval system based on big data surveillance, according to an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0031] The specific application scenario is as follows: In a multi-camera surveillance network scenario, the same pedestrian may appear in the surveillance videos of different cameras at different times. Due to the objective reasons that the shooting angle of the cameras changes and the time when the pedestrian appears in the monitoring range of different cameras is not continuous, the complete movement trajectory of the same pedestrian will be divided into multiple continuous trajectory segments. When retrieving the surveillance data, if the discrete trajectory segments belonging to the same pedestrian can be accurately spliced ​​and integrated, the reliability of the retrieval and the practical application value can be effectively improved.

[0032] Reference Figure 1 The image data retrieval method based on big data surveillance includes steps S1-S4, as detailed below:

[0033] S1: Extract the trajectory segments of pedestrians from the monitoring equipment, and use a preset posture model to extract the key point features of pedestrians in each trajectory segment. The key point features include: visibility information and color information.

[0034] In multi-camera surveillance network scenarios, to accurately acquire basic motion trajectory data of different pedestrians within the monitoring range of each camera, continuous monitoring image sequences are collected from the multi-camera monitoring devices. For each frame, YOLO (You Only Look Once, a real-time object detection algorithm) is used to identify and locate pedestrian targets in the image, thereby obtaining the bounding box corresponding to each pedestrian in each frame and clarifying the spatial location range of the pedestrian in the image. Considering that trajectory segments need to reflect the continuity of pedestrian motion between consecutive frames, the ByteTrack (ByteTrack: Multi-Object Tracking by Associating Every Detection Box, an algorithm that achieves multi-object tracking by associating each detection box) tracking algorithm is further employed. Based on the position, size, and other feature information of the pedestrian bounding boxes in each frame, the pedestrian data of consecutive frames is associated and matched, linking consecutive frames of the same pedestrian within the monitoring range of the same camera, and then extracting the independent trajectory segments corresponding to different pedestrians within the monitoring range of each camera. One pedestrian corresponds to one trajectory segment within the monitoring range of the same camera.

[0035] For each trajectory segment, a pre-defined HRNet (High-Resolution Network) pose estimation model is used to extract key point features of the pedestrian. Key point features include visibility information and color information of each key point. Visibility information is obtained by quantizing the occlusion state of key points in each frame, and color information is extracted after converting the RGB (Red, Green, Blue) channels of the image to the HSV (Hue, Saturation, Brightness) channels.

[0036] After acquiring trajectory segments of each pedestrian within the monitoring range of each camera, similarity calculations are used to identify trajectory segments belonging to the same pedestrian, thereby completing trajectory clustering and stitching, providing complete trajectory data support for cross-camera pedestrian retrieval. In existing technologies, similarity matching of pedestrian trajectory segments largely relies on pedestrian appearance features, such as clothing color and body contour features, for analysis. However, in multi-camera monitoring scenarios, there are objective differences in the shooting angles of different cameras, resulting in significant differences in the appearance features of the same pedestrian under different angles, directly affecting the accuracy of appearance feature matching, and thus causing mismatches of trajectory segments. Therefore, to avoid the interference of feature distortion caused by perspective differences on trajectory matching results, this solution proposes a trajectory segment similarity calculation method that integrates key point visibility perspective features and appearance features. By quantifying the visibility frequency of pedestrian key points in each trajectory segment, a perspective similarity model between trajectory segments is constructed. This perspective similarity is used as the confidence weight for appearance similarity calculation, thereby correcting the matching deviation of appearance features under different perspectives. At the same time, appearance features corresponding to high visibility key points are selected to participate in the similarity calculation, eliminating interference from low visibility and easily distorted features. Ultimately, this improves the accuracy of cross-camera pedestrian trajectory segment matching and achieves efficient clustering and stitching of the same pedestrian trajectory. The specific steps are as follows:

[0037] S2: Based on the visibility information in the keypoint features, calculate the visibility frequency of each keypoint in each trajectory segment and construct a visibility vector. Calculate the cosine similarity of the visibility vectors of two trajectory segments. Combine the ratio of the standard deviations of the visibility frequencies of keypoints in the two trajectory segments to obtain the viewpoint similarity between any two trajectory segments. Based on the color information in the keypoint features, filter the set of comparable keypoints in any two trajectory segments that meet the visibility threshold. Use the minimum visibility frequency of each keypoint as the weight, and combine the degree of difference in the mean chromaticity of the keypoints in the trajectory segments to calculate the appearance similarity between any two trajectory segments.

[0038] Taking a trajectory segment as an example, obtain the vector of visibility information for each frame in the trajectory segment. ,in, Indicates the first In the trajectory segment, the first The visibility vector of a frame image. Indicates the first The first frame of the image Visibility value of each key point This represents the total number of preset keypoints for pedestrians. If the keypoints are not occluded (visible), then... If it is obscured (invisible), then .

[0039] Traverse all frame images in the trajectory segment, take any key point as the target point, extract the visibility value of the target point in each frame, and accumulate all visibility values ​​to obtain the total number of visible frames of the target point. Divide the total number of visible frames by the total number of frames in the trajectory segment to obtain the visibility frequency of the target point.

[0040] Specifically, the visible frequencies satisfy the following relationship:

[0041] ;

[0042] In the formula, Indicates the first In the trajectory segment, the first The visibility frequency of each key point Indicates the first The total number of image frames contained in each trajectory segment Indicates the first The trajectory segment of the first The first frame of the image The visibility value of each key point.

[0043] With a preset number of key points, in this embodiment, a visibility vector is constructed based on the visibility frequency of the key points: This vector can structurally represent the viewpoint features of a trajectory segment, that is, the visible frequency distribution of each key point under different viewpoints is significantly different, providing a standardized data basis for subsequent calculation of the viewpoint similarity between two trajectory segments.

[0044] Similarly, repeat the above steps to traverse all trajectory segments and obtain the visibility frequency of each key point in each trajectory segment and the corresponding visibility vector.

[0045] For any two trajectory segments, calculate the cosine similarity of the visibility vectors corresponding to the two trajectory segments. Since the cosine similarity value ranges from... Perform a linear transformation on the calculation results. Map it to the [0,1] interval to obtain the directional similarity; calculate the ratio of the standard deviations of the visibility vectors of the two trajectory segments, take the absolute value of 1 minus the ratio of the standard deviations as the absolute deviation, and perform exponential mapping on the negative absolute deviation through the natural exponential function to obtain the dispersion consistency; take the product of directional similarity and dispersion consistency as the viewpoint similarity.

[0046] Specifically, the similarity of viewpoints satisfies the following relationship:

[0047] ;

[0048] In the formula, Indicates the first The trajectory segment and the first The visual similarity between trajectory segments; Indicates the first Visibility vector of each trajectory segment With the Visibility vector of each trajectory segment The cosine similarity between them reflects the degree of similarity between the visibility of key points of pedestrians under different camera segments. The greater the cosine similarity, the higher the visual similarity. Indicates the first Visibility vector of each trajectory segment The standard deviation of all components, characterizing the first Within the trajectory segment, the first The frequency dispersion can be seen at key points; This indicates the total number of preset pedestrian keypoints; Indicates the first In the trajectory segment, the first The visible frequency of each key point; Indicates the first In the trajectory segment, the first The visible frequency of each key point; Represented by natural constant An exponential function with base 0.

[0049] Current technologies rely heavily on physical features such as clothing color and body contours, which are highly susceptible to the influence of shooting angle (e.g., a frontal view clearly captures chest and facial features, while a side view more easily reveals the side features of the torso and limbs). The appearance of the same pedestrian can vary significantly from different angles, directly leading to mismatches of trajectory segments. The visibility of pedestrian key points is directly determined by the shooting angle: at a specific angle, some key points may be invisible due to body posture or occlusion, and this visibility distribution is unique to each angle (e.g., the head and chest key points are more frequently visible from a frontal view, while the side features of the torso are more frequently visible from a side view). Therefore, the distribution characteristics of the visibility frequency of key points can serve as a quantitative representation of viewpoint similarity.

[0050] By quantifying the consistency of shooting perspective between two trajectory segments through the distribution differences of key point visibility, that is, the more similar the perspective, the higher the confidence of appearance feature matching and the greater the weight; the greater the perspective difference, the lower the confidence and the smaller the weight, thereby correcting the interference of perspective difference on matching results.

[0051] This scheme integrates two core indicators—the cosine similarity of keypoint visibility vectors and the ratio of the standard deviation of the visibility vectors—to comprehensively quantify the viewpoint similarity between two trajectory segments, thereby characterizing the consistency level of the corresponding camera's shooting perspective. This viewpoint similarity is then used as a confidence weight in the appearance similarity calculation process, weighting and correcting the appearance feature matching results under different perspectives. Specifically, the higher the viewpoint similarity between two trajectory segments, the greater the corresponding confidence weight, and the higher the confidence level of the appearance feature matching result; conversely, the lower the viewpoint similarity, the smaller the corresponding confidence weight. This weakens the interference of appearance feature distortion caused by viewpoint differences on the matching results, thereby improving the reliability of appearance feature comparison across camera trajectory segments and effectively reducing the probability of mismatched trajectory segments due to viewpoint differences.

[0052] After completing the quantitative calculation of the viewpoint similarity of the two trajectory segments, it is necessary to further conduct a quantitative analysis of the appearance similarity of pedestrians in the two trajectory segments; among them, the quantification of appearance similarity is based on the color features of pedestrian key points as the core basic data.

[0053] It is important to note that the visibility features of pedestrian key points must still be included as a core constraint in the quantification of appearance similarity. This is because the color features of key points can only truly reflect the objective appearance attributes of a pedestrian when the key point is visible. Occluded key points have distorted color features and cannot be used as valid features in the calculation. Therefore, to avoid interference from distorted color features of occluded key points in the appearance similarity calculation, this scheme explicitly follows the rule of using visible key points in the appearance similarity quantification operation. This ensures the validity of the appearance feature data and the accuracy of the trajectory segment matching results. The specific operation steps are as follows:

[0054] For any two trajectory segments, construct a set of comparable key points based on the visible key points. ,in Indicates the first The trajectory segment and the first A set of comparable key points for each trajectory segment Indicates the sequence number of the key point. Indicates the first In the trajectory segment, the first The visibility frequency of each key point Indicates the first In the trajectory segment, the first The visibility frequency of each key point The visibility threshold is set to 0.5 in this scheme. The purpose of setting it to 0.5 is to ensure that key points are visible in half or more frames of the corresponding trajectory segment for their color features to be effective, thus avoiding feature distortion caused by occlusion. In short, the set of comparable key points is not the key points whose visible frequency exceeds the threshold in a single trajectory segment, but rather the intersection key points in two trajectory segments that meet the visible frequency threshold condition. This design ensures that the feature data for subsequent appearance similarity calculation is reliable in both trajectory segments; only key points that meet the above conditions can be included in the set of comparable key points.

[0055] First, the color information of the key points corresponding to the two trajectory segments is preprocessed by color space conversion, mapping the original RGB color space data to the HSV color space. Since the chromaticity H component in the HSV color space has illumination invariance, it can effectively avoid the interference of light intensity changes on color features. Therefore, this scheme uses the chromaticity H component obtained after conversion to quantitatively calculate the appearance similarity between any two trajectory segments.

[0056] Specifically, the color space conversion preprocessing steps include: localizing the first color space based on the HRNet pose estimation model. The trajectory segment of the first The first frame of the image Pixel coordinates of key points Select with this coordinate as the center The original color feature value is obtained by calculating the average values ​​of the RGB three channels within the neighboring pixels. After normalization, it is converted to the HSV color space according to the industry-standard conversion formula. The chromaticity H component, saturation S component, and lightness V component are extracted as the final color information of the key points. In this embodiment, only the chromaticity H component is used to reduce the interference of light changes and improve the stability and distinguishability of color features.

[0057] For each keypoint in the comparable keypoint set, extract the keypoint at the 1st... The visible frequency in the first trajectory segment and the first The visible frequency of each trajectory segment is selected, and the minimum visible frequency of each key point is used as the weight coefficient of a single key point. The weight coefficients of all key points in the comparable key point set are summed to obtain the total weight of the comparable key points. The ratio of the weight coefficient of a single key point to the total weight of the comparable key points is used as the normalized weight coefficient. If the visible frequency of a key point in any trajectory segment is low, it indicates that its features are easily interfered with by occlusion, and its contribution to the appearance similarity calculation needs to be reduced.

[0058] The degree of difference between the mean chromaticity values ​​of keypoints in two trajectory segments is calculated. The natural exponential function is used to exponentially map this difference to obtain the chromaticity similarity of individual keypoints. The smaller the chromaticity difference, the closer the exponential term output is to 1, indicating greater similarity in the color features of the keypoints; the larger the chromaticity difference, the closer the exponential term output is to 0.

[0059] The appearance similarity between any two trajectory segments is obtained by weighting and summing the chromatic similarity of all key points in the comparable key point set with the corresponding normalized weight coefficients.

[0060] Specifically, appearance similarity satisfies the following relationship:

[0061] ;

[0062] In the formula, Indicates the first The trajectory segment and the first Appearance similarity between trajectory segments The serial number identifier representing the key points of a pedestrian. Indicates the first The trajectory segment and the first A set of comparable key points for each trajectory segment Indicates the first In the trajectory segment, the first The visible frequency of each key point; Indicates the first In the trajectory segment, the first The visible frequency of each key point; Indicates the first The minimum visible frequency of a key point in two trajectory segments is considered the key point's importance. A lower visible frequency indicates that the key point is easily occluded in the trajectory segment, thus its importance is lower in the appearance similarity calculation. Indicates the first In the trajectory segment, the first The average value of the H component of chromaticity in the HSV color space at each key point. Indicates the first In the trajectory segment, the first The average value of the H component of chromaticity in the HSV color space at each key point. Indicates the first In the trajectory segment, the first The average H component of the HSV color space at each key point and the first In the trajectory segment, the first The degree of difference between the mean values ​​of the H component of the HSV color space at key points This represents an exponential function with the natural constant as its base.

[0063] In this embodiment, the color features of pedestrian key points are used as the core basis for quantitative calculation of appearance similarity. At the same time, the visibility parameter of key points is introduced as a weighting coefficient, and it is clearly limited that only key points that meet the visibility threshold condition can participate in the appearance similarity calculation. This eliminates the interference of distorted color features of occluded key points on the calculation results, improves the accuracy and reliability of appearance similarity quantification results, and provides high-quality feature basis for accurate matching of cross-camera trajectory segments.

[0064] After completing the quantitative calculation of the visual similarity and appearance similarity between two trajectory segments, it is only possible to determine the feature similarity of pairwise trajectory segments. However, in the actual application scenario of cross-camera pedestrian trajectory matching, the number of trajectory segments collected by the monitoring system is huge and the sources are scattered. Different trajectory segments may correspond to the movement trajectory of the same pedestrian at different times and in different monitoring areas, or they may correspond to the independent movement trajectory of different pedestrians.

[0065] If we rely solely on pairwise similarity judgment, we cannot achieve systematic classification of massive trajectory segments, making it difficult to form a complete movement trajectory chain for the same pedestrian. At the same time, some trajectory segments may have ambiguous matching due to large differences in perspective and moderate similarity in appearance features. A single pairwise comparison mode cannot effectively solve this kind of ambiguity problem.

[0066] Therefore, cluster analysis needs to be performed on multiple sets of trajectory segments. The clustering goal is to group trajectory segments originating from the same pedestrian into the same category. By leveraging the aggregation properties of clustering algorithms, discrete trajectory segments of the same pedestrian are integrated to form a complete cross-camera motion trajectory, while simultaneously eliminating matching ambiguities between trajectory segments from different pedestrians. The specific steps are as follows:

[0067] S3: Introduce time constraints, obtain the timestamp sequence corresponding to all frame images contained in each trajectory segment, determine the time constraint parameters by judging whether there is an intersection between the timestamp sequences of any two trajectory segments, calculate the initial clustering distance by combining viewpoint similarity and appearance similarity, adjust and optimize the initial clustering distance using the time constraint parameters to obtain the significant clustering distance, and perform clustering processing on all trajectory segments based on the significant clustering distance to complete the trajectory segment matching of the same pedestrian.

[0068] Calculate the initial cluster distance between any two trajectory segments by multiplying the viewpoint similarity and appearance similarity by applying a negative exponential mapping using the natural constant exponential function, and obtain the initial cluster distance between any two trajectory segments.

[0069] Specifically, the initial cluster distances satisfy the following relationship:

[0070] ;

[0071] In the formula, Indicates the first The trajectory segment and the first The initial cluster distance between trajectory segments Indicates the first The trajectory segment and the first The higher the visual similarity between two trajectory segments, the more consistent the shooting angle of the camera to which the two trajectory segments belong to the pedestrian, and the higher the confidence level of appearance similarity. Indicates the first The trajectory segment and the first The appearance similarity between trajectory segments reflects the degree of consistency in the color features of key points contained in the two trajectory segments. The higher the appearance similarity, the higher the probability that the two trajectory segments belong to the same pedestrian, and the shorter the clustering distance.

[0072] In the calculation of the clustering distance between arbitrary trajectory segment pairs, no time dimension constraint was included; while in the multi-camera monitoring system adopted in this solution, the monitoring coverage areas of each camera are independent and do not overlap spatially. Based on the physical movement law of pedestrians, the same pedestrian cannot appear in two non-overlapping monitoring areas at the same time, and it is even more impossible to complete the spatial transfer between different monitoring areas without a reasonable time interval.

[0073] Therefore, to avoid erroneous clustering caused by time dimension mismatch and improve the accuracy of trajectory segment matching, it is necessary to introduce time constraints to weight and adjust the initial clustering distance. The specific steps are as follows:

[0074] Get the The trajectory segment and the first The initial clustering distance of each trajectory segment is used to extract the effective time intervals corresponding to two trajectory segments. The effective time intervals are the start and end times of the trajectory segment being collected in the monitoring system.

[0075] Calculate the intersection of two valid time intervals and determine the intersection state using an indicator function. If the valid time intervals of the two trajectory segments do not overlap, the indicator function value is 1; if the valid time intervals of the two trajectory segments overlap, the indicator function value is 0.

[0076] The result of the exponential function is added to the preset adjustment coefficient, and the reciprocal is taken to obtain the time constraint weight factor. The initial cluster distance is multiplied by the time constraint weight factor to obtain the significant cluster distance.

[0077] Specifically, significant cluster distances satisfy the following relationship:

[0078] ;

[0079] In the formula, This indicates the first time constraint introduced. The trajectory segment and the first Significant clustering distance between trajectory segments Indicates the first The trajectory segment and the first The initial cluster distance between trajectory segments Indicates the first The effective time interval corresponding to each trajectory segment, that is, the time range during which the trajectory segment is collected by the monitoring system. Indicates the first The effective time interval corresponding to each trajectory segment Indicates an indicator function, Representing the empty set, when If true, take 1; otherwise, take 0. The first and the The trajectory segment contains the timestamp sequence of all frame images. If the intersection of the two is not empty, it indicates that the two trajectory segments overlap in time and do not belong to the same pedestrian, and their clustering distance needs to be increased. If the intersection is empty, it indicates that the two trajectory segments do not overlap in time and may belong to the same pedestrian. This represents the time constraint adjustment coefficient.

[0080] When the effective time intervals of two trajectory segments do not overlap, this conforms to the spatiotemporal pattern of the same pedestrian passing through different cameras sequentially, and the weighting factor is... This will reduce the initial clustering distance and increase the probability that two trajectory segments will be clustered into one class. When the effective time intervals of two trajectory segments overlap, the same pedestrian cannot appear in two non-overlapping monitoring areas at the same time. The weighting factor is... This will amplify the initial clustering distance and reduce the probability that two trajectory segments are incorrectly clustered.

[0081] S4: Establish a retrieval index based on the clustering results of trajectory segments, and retrieve pedestrian-related image data based on the retrieval index.

[0082] Based on the adjusted clustering distance constructed by fusing perspective similarity and appearance similarity, clustering operations are performed on all trajectory segments within the monitoring area to obtain several clusters.

[0083] In this system, a cluster represents the complete movement trajectory of the same pedestrian within the monitored area, and an identity identifier is assigned to each cluster.

[0084] The clustering results are integrated, each cluster is assigned a unique identifier ID, the pedestrian feature set corresponding to each cluster is extracted and a retrieval index is built. When it is necessary to retrieve pedestrian-related image data, the target features in the retrieval request are matched with the feature set in the retrieval index, and the complete trajectory and related image data corresponding to the target pedestrian are output to achieve accurate retrieval.

[0085] This invention also provides an image data retrieval system based on big data surveillance. For example... Figure 2 As shown, the system includes a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement the image data retrieval method based on surveillance big data according to the first aspect of the present invention. The system also includes other components well known to those skilled in the art, such as a communication bus and a communication interface, the settings and functions of which are known in the art and will not be described further here.

[0086] It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of this invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. An image data retrieval method based on big data surveillance, characterized in that, include: Extract pedestrian trajectory segments from monitoring equipment, and use a preset posture model to extract key point features of pedestrians in each trajectory segment. The key point features include: visibility information and color information. Based on the visibility information in the key point features, the visibility frequency of each key point in each trajectory segment is calculated and a visibility vector is constructed. The cosine similarity of the visibility vectors of two trajectory segments is calculated. Combined with the ratio of the standard deviation of the visibility frequencies of key points in two trajectory segments, the visual similarity between any two trajectory segments is obtained. Based on the color information in the key point features, a set of comparable key points in any two trajectory segments that meet the visibility threshold is selected. The minimum visibility frequency of each key point is used as the weight. Combined with the degree of difference in the mean chromaticity of the key points in the trajectory segments, the appearance similarity between any two trajectory segments is calculated. A time constraint is introduced, and the timestamp sequence corresponding to all frame images contained in each trajectory segment is obtained. The time constraint parameter is determined by judging whether there is an intersection between the timestamp sequences of any two trajectory segments. The initial clustering distance is calculated by combining the viewpoint similarity and appearance similarity. The initial clustering distance is adjusted and optimized using the time constraint parameter to obtain the significant clustering distance. All trajectory segments are clustered according to the significant clustering distance to complete the trajectory segment matching of the same pedestrian. A retrieval index is built based on the clustering results of trajectory segments, and pedestrian-related image data is retrieved based on the retrieval index; The calculation of the appearance similarity between any two trajectory segments includes: For each keypoint in the comparable keypoint set, extract the keypoint at the 1st... The visible frequency in the first trajectory segment and the first The visible frequencies of keypoints in each trajectory segment are analyzed, and the minimum visible frequency of each keypoint is selected as the weight coefficient of a single keypoint. This value is then normalized to obtain the normalized weight coefficient. The degree of difference between the mean chromaticity values ​​of keypoints in two trajectory segments is calculated, and the degree of difference is exponentially mapped using the natural exponential function to obtain the chromaticity similarity of a single keypoint. The chromaticity similarity of all keypoints in the comparable keypoint set is weighted and summed with the corresponding normalized weight coefficient to obtain the appearance similarity between any two trajectory segments.

2. The image data retrieval method based on big data surveillance according to claim 1, characterized in that, Calculate the visibility frequency of each key point in each trajectory segment and construct a visibility vector, including: Taking any trajectory segment as the segment to be analyzed, obtain the vector of visibility information of each frame in the segment to be analyzed. Traverse all frame images in the trajectory segment, take any key point as the target point, extract the visibility value of the target point in each frame, and accumulate all visibility values ​​to obtain the total number of visible frames of the target point. Divide the total number of visible frames by the total number of frames in the trajectory segment to obtain the visibility frequency of the target point. Preset the number of key points and construct the visibility vector according to the visibility frequency of the key points.

3. The image data retrieval method based on big data surveillance according to claim 1, characterized in that, The methods for calculating the similarity of viewpoints include: For any two trajectory segments, calculate the cosine similarity of the visibility vectors corresponding to the two trajectory segments, and perform a linear transformation on the calculation result to obtain the directional similarity; calculate the ratio of the standard deviations of the visibility vectors of the two trajectory segments, and take the absolute value of 1 minus the standard deviation ratio as the absolute deviation. Perform an exponential mapping on the negative absolute deviation through the natural exponential function to obtain the dispersion consistency; take the product of the directional similarity and the dispersion consistency as the viewpoint similarity.

4. The image data retrieval method based on big data surveillance according to claim 1, characterized in that, The steps for obtaining the set of comparable key points include: For any two trajectory segments, based on the human pose detection algorithm, the complete set of human key points corresponding to the two trajectory segments is extracted respectively. If the visibility frequency of each human key point in the two trajectory segments is greater than or equal to the preset visibility threshold, the corresponding key point is included in the set of comparable key points. Otherwise, if each key point only meets one condition or does not meet any condition, the key point is excluded. All key points in the complete set of human key points are traversed to obtain the set of comparable key points.

5. The image data retrieval method based on big data surveillance according to claim 1, characterized in that, Calculate the degree of difference between the mean chromaticity values ​​of key points in two trajectory segments, including: Localization based on attitude estimation model The first trajectory segment The first frame of the image Pixel coordinates of key points The neighboring pixels within a preset range are selected with the coordinates as the center. The mean values ​​of the three channels within the neighboring pixels are calculated to obtain the original color feature values. The original color feature values ​​are then normalized and converted into the HSV color space to obtain the chromaticity of the HSV color space. Calculate the difference between the mean chromaticity values ​​of key points in any two trajectory segments to obtain the degree of difference between the mean chromaticity values ​​of key points in the two trajectory segments.

6. The image data retrieval method based on big data surveillance according to claim 1, characterized in that, The calculation methods for the significant cluster distance include: Get the The trajectory segment and the first The initial clustering distance of each trajectory segment is used to extract the effective time intervals corresponding to two trajectory segments respectively; Calculate the intersection of two valid time intervals, and determine the intersection state using an indicator function. If the valid time intervals of the two trajectory segments do not overlap, the indicator function value is 1; if the valid time intervals of the two trajectory segments overlap, the indicator function value is 0. The result of the exponential function is added to the preset adjustment coefficient, and the reciprocal is taken to obtain the time constraint weight factor. The initial cluster distance is multiplied by the time constraint weight factor to obtain the significant cluster distance.

7. An image data retrieval system based on big data surveillance, characterized in that, include: A processor and a memory, wherein the memory stores computer program instructions that, when executed by the processor, implement the image data retrieval method based on surveillance big data according to any one of claims 1-6.

Images