Multi-target spatial relationship driven human recognition method in social scene images

Abstract

The application discloses a multi-target space relation driven human recognition method in a social scene image, relates to the technical field of image recognition, and effectively improves the quality and diversity of input data, significantly reduces the interference of redundant data and noise data on subsequent models, greatly improves the precision and robustness of target detection and identity recognition, and greatly improves the processing efficiency and automation degree of the whole system, solves the bottleneck problem of the image screening link in the prior art, and provides strong technical support for efficient and accurate human recognition in a complex social scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and particularly relates to a human recognition method in a multi-target spatial relationship driven social scene image. BACKGROUND

[0002] In the fields of intelligent monitoring, social network content analysis, human-computer interaction and public security, it is a crucial and fundamental work to accurately identify individual identities and understand their behavior patterns from complex social scene images. Such tasks usually need to process image sequences containing multiple targets, complex spatial relationships and interactive behaviors between targets. In order to improve the efficiency and accuracy of subsequent identification, it is usually necessary to preprocess a large number of original social scene images collected before target detection and identity confirmation, and image screening is a crucial and indispensable first step. At present, in practical applications or early research, the screening of images is usually carried out in the way of manual naked-eye identification. The operator compares a large number of original social scene images collected one by one through visual observation, and judges the similarity between images according to experience, so as to manually remove those obviously repeated or highly similar images and retain representative images. However, this screening method relying on manual naked-eye identification has significant disadvantages:

[0003] In the face of massive image data, the manual screening process is extremely slow, requiring a large amount of manpower and time cost, and it is difficult to meet the needs of real-time or large-scale data processing; and manual judgment is highly dependent on the experience, attention and subjective feelings of the operator. For images with critical similarity, such as slight changes in human posture, slight differences in lighting conditions, and partial occlusion of the background, different personnel or even the same personnel in different states may make inconsistent judgments, leading to the phenomenon of misjudgment that images that should not be removed are removed; at the same time, manual screening is difficult to quantitatively and systematically evaluate various uncertainty factors that affect image similarity judgment, such as image clarity, low contrast, etc. These factors will significantly increase the difficulty of judgment, but the current manual method cannot provide an objective and unified standard to measure the influence of these factors on similarity judgment, making the decision-making process more arbitrary and unreliable when dealing with such uncertain images; the above disadvantages directly lead to the complexity of image deduplication being artificially magnified, and the operator needs to repeatedly weigh a large number of visually similar but different images, which is easy to cause fatigue and judgment errors, therefore, the manual screening method inevitably increases the misjudgment rate of image deduplication, making it difficult to guarantee the accuracy and consistency of the screening results; therefore, if the existing manual method is still used for image screening, it is easy to cause the following serious consequences: the image set after screening may still contain a large number of redundant images, or important effective images are mistakenly removed, resulting in inconsistent data set quality input to the subsequent processing process, and low-quality input data will directly affect the performance of deep learning models such as convolutional neural networks, and redundant data is easy to cause model overfitting, ultimately reducing the recall rate of target detection, directly affecting the accuracy of identity recognition; in summary, the manual visual screening method widely used in the prior art is inefficient, highly subjective, prone to misjudgment, and lacks an effective evaluation mechanism for uncertainty factors in images, not only greatly increasing the complexity and misjudgment risk of image deduplication, but also seriously affecting the performance of subsequent core tasks such as target detection and identity recognition.

[0004] Therefore, the prior art urgently needs a multi-target spatial relationship driven human recognition method in a social scene image. SUMMARY

[0005] In order to solve the above technical problems, the present application provides a multi-target spatial relationship driven human recognition method in a social scene image, which specifically comprises the following steps:

[0006] Step 1, obtaining an original social scene image and screening to obtain an optimized social scene image, and pre-processing the optimized social scene image;

[0007] Step 11, calculating the spatial scene similarity value between any two original social scene images;

[0008] Step 111, extracting key points and descriptors of each original social scene image based on the SIFT algorithm;

[0009] Step 112, constructing a high-level semantic feature vector of each original social scene image using the key points and descriptors;

[0010] Step 113, calculating the semantic similarity between any two original social scene images based on the extracted high-level semantic feature vectors;

[0011] Step 114, generating a perceptual hash value of each image using the perceptual hashing algorithm, and calculating the hash difference between any two original social scene images;

[0012] Step 115, normalizing the hash difference to obtain the hash similarity between any two original social scene images;

[0013] Step 116, taking the average of the semantic similarity and the hash similarity between any two original social scene images to obtain the spatial scene similarity value between any two original social scene images;

[0014] Step 12, determining a spatial scene similarity threshold interval based on the spatial scene similarity value, the spatial scene similarity threshold interval including a first spatial scene similarity threshold interval and a second spatial scene similarity threshold interval;

[0015] Step 121, collecting all spatial scene similarity values, and constructing a probability density distribution curve of all spatial scene similarity values using a probability density estimation method based on the collection result;

[0016] Step 122, identifying all peak values and valley values on both sides of the peak values according to the probability density distribution curve;

[0017] Step 123, sorting all identified peak values in descending order;

[0018] Step 124, selecting the largest peak value according to the sorting result, and taking the leftmost valley value on the left side of the largest peak value as the first left boundary, and taking the rightmost valley value on the right side of the largest peak value as the first right boundary; the first left boundary is greater than or equal to the leftmost valley value on the left side of the largest peak value; the first right boundary is less than or equal to the rightmost valley value on the right side of the largest peak value;

[0019] Step 125, combining the first left boundary and the first right boundary to obtain the first spatial scene similarity threshold interval;

[0020] Step 126, according to the sorting result, selecting the minimum peak value, and taking the leftmost valley value of the minimum peak value as the second left boundary and taking the rightmost valley value of the minimum peak value as the second right boundary; the second left boundary is greater than or equal to the leftmost valley value of the minimum peak value; the second right boundary is less than or equal to the rightmost valley value of the minimum peak value;

[0021] Step 127, combining the second left boundary and the second right boundary to obtain a second spatial scene similarity threshold interval;

[0022] Step 13, according to the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval, combining the spatial scene similarity value to determine the original social scene image to obtain a reserved original social scene image;

[0023] Step 131, setting the valley value less than the first left boundary of the first spatial scene similarity threshold interval as a third right boundary, setting the valley value greater than the second right boundary of the second spatial scene similarity threshold interval as a third left boundary, and combining the third right boundary and the third left boundary to obtain a third spatial scene similarity threshold interval;

[0024] Step 132, extracting the spatial scene similarity value between each pair of original social scene images;

[0025] Step 133, counting the number of times that the spatial scene similarity value between the current original social scene image and the remaining original social scene images falls into the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval, respectively;

[0026] Step 134, based on the counting result, if the spatial scene similarity value between the current original social scene image and the remaining original social scene images falls into the first spatial scene similarity threshold interval, the current original social scene image is removed;

[0027] If the spatial scene similarity value between the current original social scene image and the remaining original social scene images falls into the second spatial scene similarity threshold interval, the current original social scene image is retained;

[0028] If the spatial scene similarity value between the current original social scene image and the remaining original social scene images falls into the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval, respectively, it is determined whether the number of times falling into the first spatial scene similarity threshold interval is greater than the number of times falling into the second spatial scene similarity threshold interval: if greater, the current original social scene image is removed; if less, the current original social scene image is retained; if equal, the current original social scene image is determined as an uncertain original social scene image;

[0029] if the spatial scene similarity values between the current original social scene image and the rest of the original social scene images fall into the first spatial scene similarity threshold interval, the second spatial scene similarity threshold interval and the third spatial scene similarity threshold interval respectively, it is determined that the current original social scene image is an uncertain original social scene image;

[0030] Step 135, analyzing the original social scene image determined as an uncertain original social scene image, and obtaining a reserved original social scene image according to the analysis result;

[0031] Step 1351, calculating an uncertainty factor index of each original social scene image determined as an uncertain original social scene image;

[0032] Step 13511, obtaining the definition, contrast and noise level of each original social scene image determined as an uncertain original social scene image, and determining the quality score of each original social scene image determined as an uncertain original social scene image based on the definition, contrast and noise level;

[0033] Step 13512, analyzing and extracting the content complexity features in each original social scene image determined as uncertain, the content complexity features including target quantity, target distribution density and background interference degree, and determining the content complexity score of each original social scene image determined as uncertain based on the content complexity features;

[0034] Step 13513, taking the mean of the quality score of each original social scene image determined as an uncertain original social scene image and the content complexity score of each original social scene image determined as uncertain, to obtain the comprehensive uncertainty component of each original social scene image determined as an uncertain original social scene image;

[0035] Step 13514, combining the number of times each original social scene image determined as an uncertain original social scene image falls into the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval in statistics, normalizing the comprehensive uncertainty component of each original social scene image determined as an uncertain original social scene image, and converting through a preset nonlinear mapping function, to obtain the uncertainty factor index of each original social scene image determined as an uncertain original social scene image;

[0036] Step 1352, setting a determination threshold for the uncertainty factor index;

[0037] Step 1353, based on the determination threshold, if the current determination is that the number of times that the spatial scene similarity value between the original social scene image determined as uncertain and the remaining original social scene images falls into the first spatial scene similarity threshold interval is greater than the number of times that falls into the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image determined as uncertain by the current determination is less than the determination threshold, the original social scene image determined as uncertain by the current determination is removed;

[0038] If the current determination is that the number of times that the spatial scene similarity value between the original social scene image determined as uncertain and the remaining original social scene images falls into the first spatial scene similarity threshold interval is greater than the number of times that falls into the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image determined as uncertain by the current determination is greater than or equal to the determination threshold, the original social scene image determined as uncertain by the current determination is retained;

[0039] If the current determination is that the number of times that the spatial scene similarity value between the original social scene image determined as uncertain and the remaining original social scene images falls into the first spatial scene similarity threshold interval is less than the number of times that falls into the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image determined as uncertain by the current determination is less than the determination threshold, the original social scene image determined as uncertain by the current determination is removed;

[0040] If the current determination is that the number of times that the spatial scene similarity value between the original social scene image determined as uncertain and the remaining original social scene images falls into the first spatial scene similarity threshold interval is less than the number of times that falls into the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image determined as uncertain by the current determination is greater than or equal to the determination threshold, the original social scene image determined as uncertain by the current determination is retained;

[0041] If the current determination is that the number of times that the spatial scene similarity value between the original social scene image determined as uncertain and the remaining original social scene images falls into the first spatial scene similarity threshold interval is equal to the number of times that falls into the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image determined as uncertain by the current determination is less than the determination threshold, the original social scene image determined as uncertain by the current determination is removed;

[0042] If the current determined original social scene image is uncertain, the number of times that the spatial scene similarity value between the original social scene image and the rest of the original social scene images falls into the first spatial scene similarity threshold interval is equal to the number of times that falls into the second spatial scene similarity threshold interval, and the uncertainty factor index of the current determined original social scene image is greater than or equal to the determination threshold, the current determined original social scene image is retained;

[0043] Step 14, the retained original social scene images are collected to obtain preferred social scene images;

[0044] Step 2, the preprocessed preferred social scene images are input into the pre-trained convolutional neural network model, all potential targets in the preferred social scene images are detected, each detection target is identified, and a bounding box is allocated to each detection target, the bounding box includes a position and a size;

[0045] Step 3, based on the position and size of each detection target, the relative position, distance and angle between each detection target are calculated, a spatial relationship graph between each detection target is constructed, the behavior features of each detection target are extracted in combination with the spatial relationship graph, the behavior mode of each detection target is analyzed based on the behavior features of each detection target, and a feature vector of each detection target is extracted, the feature vector, the behavior mode and the spatial relationship graph are input into the pre-trained deep learning model, and the identity of each detection target is confirmed.

[0046] The embodiment of the application has the following technical effects:

[0047] The present application can more comprehensively and robustly measure the comprehensive similarity between images by automatically calculating the spatial scene similarity value between any two original social scene images, which integrates the high-level semantic feature similarity based on the SIFT algorithm and the hash similarity based on the perceptual hash algorithm, effectively overcoming the limitations of single measurement method and laying a solid data foundation for subsequent accurate screening. Secondly, the present application innovatively proposes an adaptive threshold interval determination method based on probability density estimation. By analyzing the probability density distribution curve of all spatial scene similarity values, the peak and valley are automatically identified, and the first and second spatial scene similarity threshold intervals representing "high similarity" and "significant difference" are determined, so that the screening standard can be dynamically adjusted according to the actual data distribution, greatly improving the scientificity and adaptability of screening. On this basis, the present application designs a rigorous decision logic, which not only performs preliminary screening according to the statistical number of image pair similarity values falling into different threshold intervals, but more importantly, introduces the concept of uncertain original social scene images and builds a decision rule containing a third spatial scene similarity threshold interval, which can effectively identify images that are difficult to determine directly due to their similarity at the boundary. For these uncertain images, the present application further proposes a calculation and analysis method for the uncertainty factor index, which comprehensively considers the quality factors such as image clarity, contrast, noise level, and content complexity characteristics such as target quantity and distribution density, and combines their performance in similarity statistics to realize the objective quantification of image uncertainty. Finally, by setting a decision threshold and combining the index, the uncertain images are subjected to a fine secondary decision, i.e. retention or rejection, completely solving the randomness and high misjudgment rate problem in manual screening of such images. This series of innovations enables the present application to efficiently and accurately screen high-quality, low-redundancy, and information-rich preferred social scene images from a large number of original images. Finally, the preferred images that have undergone strict screening and preprocessing are input into a pre-trained deep learning model, which can more reliably confirm the identity of each detection target using its accurate bounding box detection, behavior feature extraction based on spatial relationship graph, and pattern analysis.

[0048] In summary, the present application effectively improves the quality and diversity of input data through an automated and intelligent multi-level screening mechanism, significantly reduces the interference of redundant and noisy data on subsequent models, not only greatly improves the accuracy and robustness of target detection and identity recognition, but also greatly improves the processing efficiency and automation level of the entire system, solving the bottleneck problem of image screening in the prior art and providing strong technical support for efficient and accurate human recognition in complex social scenes. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to make the technical solutions in the specific embodiments of the present application or the prior art clearer, the accompanying drawings needed in the specific embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and all other embodiments obtained by a person of ordinary skill in the art without creative work based on the embodiments in the present application shall fall within the protection scope of the present application.

[0050] Figure 1 is a flowchart of a human recognition method in a multi-target spatial relationship driven social scene image provided by an embodiment of the present application. Specific embodiments

[0051] In order to make the technical solutions in the specific embodiments of the present application or the prior art clearer, the accompanying drawings needed in the specific embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and all other embodiments obtained by a person of ordinary skill in the art without creative work based on the embodiments in the present application shall fall within the protection scope of the present application.

[0052] Embodiment one: as shown in the figure, the present application provides a human recognition method in a multi-target spatial relationship driven social scene image, comprising the following steps: Figure 1

[0053] Step 1, obtaining an original social scene image and performing screening to obtain an optimal social scene image, and pre-processing the optimal social scene image;

[0054] ​It is worth mentioning that the acquisition of original social scene images is mainly realized through the following two ways: 1. Fixed monitoring device collection, that is, fixed position cameras or video monitoring systems are deployed in specific social places such as shopping malls, squares, stations, school corridors, conference rooms, etc.; these devices continuously or on demand collect video streams or timed snapshot images of the area; through video frame extraction technology, single frame images can be extracted from continuous video streams as original social scene images; 2. Mobile device or social media upload, that is, from user-authorized mobile devices such as smartphones, tablets or social media platforms such as Weibo, WeChat Moments, Instagram, Facebook, etc., collect photos uploaded by users containing social scenes; these images are usually taken by users in various social activities and have characteristics such as diverse perspectives, rich scenes, and complex lighting conditions; among them, the original social scene images refer to static images containing two or more individuals, that is, humans interacting or coexisting in the same physical space; these images specifically include but are not limited to: group photos, such as family gatherings, friend gatherings, team photos, etc.; public space activity images, such as walking in the park, chatting in the coffee shop, discussing in the meeting, watching the game in the sports event, etc.; specific social event images, such as wedding, birthday party, award ceremony, etc.; monitoring scene images, obtained from fixed monitoring cameras, record the activities of multiple people in a specific area, such as crowds at the entrance, pedestrians in the corridor, etc.; the common characteristics of these images are: multiple targets, that is, people, exist in the image, and there may be certain spatial relationships between these targets, such as standing side by side, face-to-face talking, following behind, etc., and potential interactive behaviors;

[0055] After obtaining the original social scene images, a series of preprocessing operations need to be performed on the selected optimal social scene images to improve image quality, unify data format, and prepare for subsequent deep learning model processing. The preprocessing operations mainly include: image size normalization, i.e. adjusting all optimal social scene images to a unified resolution, such as 640x480 pixels or 1024x768 pixels, which ensures that the data input to the convolutional neural network model has a fixed size, avoiding the problem of increased computational complexity or model input mismatch caused by size differences; and color space conversion, i.e. converting the image from the original color space, such as RGB, to a more suitable color space for subsequent processing, such as BGR, to match the default input format of commonly used deep learning frameworks such as OpenCV, or to convert to a grayscale image to reduce computational load; and brightness and contrast adjustment, i.e. for images that are too dark or too bright, automatically or semi-automatically correcting the brightness and contrast to improve visual effects and feature extraction capabilities; and noise removal processing, i.e. applying image filtering techniques such as Gaussian filtering and median filtering to remove random noise in the image, especially for images taken in low light conditions, to improve image clarity, etc.

[0056] Step 11, calculating the spatial scene similarity value between any two original social scene images;

[0057] Step 111, extracting the key points and descriptors of each original social scene image based on the SIFT algorithm;

[0058] It is worth noting that for each original social scene image to be processed, the Scale-Invariant Feature Transform (SIFT) algorithm is applied for feature extraction. The specific operation is as follows: first, detect the key points in the image in different scale spaces through the Gaussian difference method, these key points are usually located at the corner points, edge intersection points, etc. with significant local features, and are invariant to scale and rotation, then assign a principal direction based on the local gradient direction to each detected key point to achieve rotation invariance, finally, extract the gradient information of the surrounding area centered on the key point to generate a 128-dimensional vector as the descriptor of the key point, which can robustly represent the local appearance features of the key point. This step generates a feature set containing multiple key points and their corresponding descriptors for each original social scene image.

[0059] Step 112, constructing a high-level semantic feature vector for each original social scene image using key points and descriptors;

[0060] It is worth mentioning that after the SIFT feature extraction is completed, the discrete key point-descriptor set needs to be converted into a high-level semantic feature vector capable of representing the whole original social scene image. In this embodiment, the "bag of words" method is used for construction. First, a large number of samples are randomly selected from the SIFT descriptors of all original social scene images, and a "visual dictionary" containing K cluster centers, i.e. "visual words", is generated through the K-means clustering algorithm. Then, for each original social scene image, the visual words to which all SIFT descriptors belong are counted. Specifically, the Euclidean distance between the descriptors and each cluster center is calculated to find the nearest cluster center, thereby counting the visual words to which all SIFT descriptors belong. Next, the frequency of each visual word in the image is counted. Finally, the K frequency values are arranged in order to form a K-dimensional vector, which is the high-level semantic feature vector of the original social scene image. This vector reflects the distribution of "visual words" in the image from a statistical point of view and contains the high-level semantic information of the image.

[0061] Step 113, based on the extracted high-level semantic feature vector, the semantic similarity between any two original social scene images is calculated;

[0062] It is worth mentioning that for any two original social scene images A and B, their high-level semantic feature vectors VA and VB constructed in step 112 are obtained respectively, and then the cosine similarity between the two vectors is calculated as the semantic similarity between them.

[0063] Step 114, applying the perceptual hashing algorithm to generate the perceptual hash value of each image, and calculating the hash difference between any two original social scene images;

[0064] It is worth mentioning that for each original social scene image, the average hashing algorithm is applied to generate its perceptual hash value. The specific process is as follows: first, the image is scaled to a fixed small size, such as 8x8 pixels, then it is converted to a grayscale image, then the average gray value of the 64 pixels is calculated, and finally the gray value of each pixel is compared with the average value, and the value greater than or equal to the average value is recorded as 1, and the value less than the average value is recorded as 0, thereby obtaining a 64-bit binary string, which is the perceptual hash value of the original social scene image. For any two original social scene images A and B, their perceptual hash values Hash_A and Hash_B are obtained respectively, and the Hamming distance between them is calculated, i.e. the number of different bits in the same position of the two binary strings is counted, which is the hash difference between the two original social scene images.

[0065] Step 115, the hash difference is normalized to obtain the hash similarity between any two original social scene images;

[0066] It is worth noting that since the range of hash difference, i.e. Hamming distance, is [0, 64] (for 64-bit hash), and the range of semantic similarity is [-1, 1], in order to facilitate subsequent fusion, it is necessary to normalize the hash difference to the interval [0, 1] and convert it to similarity, therefore, the linear normalization method is adopted in this embodiment, after this processing, the greater the hash difference, the smaller the hash similarity; when the hash difference is 0, the hash similarity is 1; when the hash difference is 64, the hash similarity is 0, the hash similarity obtained in this way is comparable with the semantic similarity.

[0067] Step 116, taking the average of the semantic similarity and the hash similarity between any two original social scene images to obtain the spatial scene similarity value between any two original social scene images;

[0068] It is worth noting that for any two original social scene images A and B, their semantic similarity SS and hash similarity HS are obtained respectively, and the arithmetic mean of the two similarity values is calculated, i.e. the spatial scene similarity value between the two original social scene images is obtained;

[0069] It is worth further noting that, among them, the spatial scene similarity value is calculated by fusing multi-modal features through steps 111 to 116, which effectively overcomes the limitations of single similarity measurement method and significantly improves the robustness and accuracy of image similarity evaluation. This method not only uses SIFT algorithm to extract key points and descriptors of images to construct high-level semantic feature vectors and calculate semantic similarity, capturing the content and structure information of images, but also applies perceptual hashing algorithm to generate hash value and calculate hash similarity, reflecting the overall perceptual consistency of images. By taking the average of the two, the final spatial scene similarity value is obtained, which can more comprehensively measure the similarity of images, considering both the semantic level of content and the overall visual perception level. This fusion strategy makes the similarity calculation have stronger anti-interference ability to common interference factors such as light change, small range content disturbance and slight compression, ensuring that the subsequent threshold interval determination and image screening decision based on this similarity value are based on more reliable and objective data, providing a core guarantee for the accuracy of the entire screening process.

[0070] Step 12, determining the spatial scene similarity threshold interval based on the spatial scene similarity value, the spatial scene similarity threshold interval including a first spatial scene similarity threshold interval and a second spatial scene similarity threshold interval;

[0071] Step 121, collecting all spatial scene similarity values, and constructing a probability density distribution curve of all spatial scene similarity values based on the collection result by using a probability density estimation method;

[0072] It is worth noting that the process of constructing the probability density distribution curve of all spatial scene similarity values is as follows: first, all the calculated spatial scene similarity values (SV) are taken as a set. In order to construct the probability density distribution curve of these values, a non-parametric method, Kernel Density Estimation (KDE), is used. KDE is a method of estimating the probability density function based on data samples without assuming that the data follows a specific distribution form. The specific operation is as follows:

[0073] It includes selecting a suitable bandwidth and applying the KDE algorithm, where the selection of the bandwidth is crucial to the result of KDE, which affects the degree of smoothing. Too small a bandwidth will lead to overfitting, resulting in too many peak values and valley values; too large a bandwidth will lose detailed information. The selection process usually uses the cross-validation method to select a suitable bandwidth; in addition, for each spatial scene similarity value, the selected bandwidth is used as a parameter to estimate the density using a Gaussian kernel function. Traverse the entire value set, accumulate the contribution of each point, and thus form a continuous probability density distribution curve.

[0074] Step 122, according to the probability density distribution curve, identify all peak values and valley values on both sides of the peak values;

[0075] It is worth noting that after obtaining the probability density distribution curve, the next step is to identify all the peak values and the valley values on both sides of the peak values. This step usually involves the following processes, including finding local maximum points and determining the valley values on both sides of the peak values. For finding local maximum points, the probability density values of adjacent points are compared to determine the local maximum points, which are the peak values; for determining the valley values on both sides of the peak values, starting from each peak value, search to the left and right until a position is found that is lower than the current point and the adjacent points are also lower than the current point, which are the valley values on both sides of the peak value.

[0076] Step 123, sort all the identified peak values in descending order;

[0077] It is worth noting that assuming that three peak values P1, P2, P3 are identified, their corresponding probability densities are PD(P1), PD(P2), and PD(P3), where PD(P1)>PD(P2)>PD(P3), then arrange them in descending order of probability density value, and the resulting order is [P1, P2, P3].

[0078] Step 124, according to the sorting result, selecting the maximum peak value, and taking the leftmost valley value of the maximum peak value as the first left boundary and the rightmost valley value of the maximum peak value as the first right boundary; the first left boundary is greater than or equal to the leftmost valley value of the maximum peak value; the first right boundary is less than or equal to the rightmost valley value of the maximum peak value;

[0079] Step 125, combining the first left boundary and the first right boundary to obtain the first spatial scene similarity threshold interval;

[0080] It is worth noting that the maximum peak value represents the most common spatial scene similarity range, which means that the similarity between images in this range is very high. Therefore, selecting the leftmost valley value of the maximum peak value as the first left boundary and the rightmost valley value as the first right boundary can ensure that the highly similar image pairs are included, while excluding some image pairs with slightly lower similarity but still at a high level. The setting of this interval helps to filter out the truly highly visually or semantically consistent image combinations.

[0081] Step 126, according to the sorting result, selecting the minimum peak value, and taking the leftmost valley value of the minimum peak value as the second left boundary and the rightmost valley value of the minimum peak value as the second right boundary; the second left boundary is greater than or equal to the leftmost valley value of the minimum peak value; the second right boundary is less than or equal to the rightmost valley value of the minimum peak value;

[0082] Step 127, combining the second left boundary and the second right boundary to obtain the second spatial scene similarity threshold interval;

[0083] It is worth noting that the minimum peak value corresponds to a relatively rare spatial scene similarity range, indicating a certain similarity pattern or noise under certain conditions. Selecting the boundary near the minimum peak value to define the second spatial scene similarity threshold interval is mainly used to capture image similarity under abnormal or edge conditions.

[0084] The first spatial scene similarity threshold interval is mainly used to screen out the most representative and typical image pairs because they share highly consistent spatial structures or contents, are suitable for application scenarios requiring accurate matching, and clearly defines the image pairs that are "highly similar" or "potentially redundant". The spatial scene similarity values of the image pairs falling into this interval are at the highest peak of the data distribution, meaning that they have the highest frequency of occurrence in the data set and are most likely to be the same scene or highly repeated perspectives. Therefore, this interval is used as the main basis for "removing" redundant images, and the logic is clear and the goal is clear; the second spatial scene similarity threshold interval is mainly used to help discover less common but still meaningful similar relationships, such as for exploring image similarity analysis under different perspectives or slight changes, and this interval defines the "significantly different" or "highly differentiated" image pairs. The similarity values of the image pairs falling into this interval are near the minimum peak of the distribution (around the minimum peak), meaning that they are "different" in the data set and are significantly different from the similarity patterns of most other image pairs. Using this interval as the basis for "retaining" key differentiated images can effectively ensure that the filtered image set contains a sufficient variety of perspectives and scene changes, avoiding information loss. By focusing on these two extreme and semantically clear intervals, the present application establishes a simple and efficient binary decision-making framework for "deduplication" and "diversity preservation", avoiding the decision-making logic complexity and rule conflicts brought by the introduction of multiple intermediate intervals.

[0085] It is worth further explaining that the maximum peak and the minimum peak are usually the most significant and stable feature points in the probability density distribution curve, and are relatively insensitive to small perturbations in the data. Selecting them as the reference makes the determination of the threshold interval more objective and repeatable. In contrast, the number, location and significance of the intermediate intervals can vary greatly depending on the data set, and may not be significant in some data sets. Relying on these unstable or variable intermediate intervals as the core screening basis can reduce the robustness and universality of the entire method. The present application enhances the stability of the method by relying only on the most stable (maximum peak) and most marginal (minimum peak) features.

[0086] Based on the above about the intermediate interval, it is worth further explaining that in the probability density distribution curve of the constructed spatial scene similarity value, in addition to being selected as the "first spatial scene similarity threshold interval" (corresponding to the maximum peak) and the "second spatial scene similarity threshold interval" (corresponding to the minimum peak), there are usually one or more local peaks and their corresponding intervals. These intermediate intervals, i.e. the setting process of the third spatial scene similarity threshold interval provided later, represent other similarity patterns in the data set other than "most common similarity" and "least common similarity"; Therefore, the ingenious part of the present application is that it does not ignore the image pairs falling outside the first and second intervals, i.e. the image pairs falling into the "intermediate interval" or the edge region, on the contrary, a mechanism is specially designed in the subsequent design process, i.e. the setting process of the third spatial scene similarity threshold interval, to judge these image pairs that cannot be simply classified as "uncertain original social scene images", these "uncertain" images are mainly distributed in the vast area between the first and second intervals, i.e. the range covered by those intermediate intervals. By not directly using these intermediate intervals for hard "retention / rejection" decisions, but using them as the source of "uncertain" state, the present application ingeniously converts the complex intermediate state problem into a special, quantifiable "uncertainty evaluation" problem, i.e. through the uncertainty factor index, which is more scientific and effective than trying to set complex, possibly conflicting retention or rejection rules for each intermediate interval; Further, if multiple intermediate intervals are introduced and different processing strategies are set for each interval, for example, some intermediate intervals are retained and some are rejected, it will greatly increase the complexity of the rules and may cause ambiguity in decision making. For example, an image pair may approach the boundaries of two different intermediate intervals at the same time, making it difficult to judge. More importantly, this fine-grained multi-interval division has diminishing marginal benefits for the core goal of "deduplication", and instead increases unnecessary calculation and logic overhead. The two-interval + uncertain processing mechanism of the present application, while ensuring the core goal of efficient deduplication + preserving key diversity, achieves effective management of complex situations with a relatively simple structure;

[0087] Further, the adaptive threshold interval determination method based on probability density estimation is proposed for steps 121 to 127, which realizes the dynamic and objective of the screening criteria. The method uses the statistical characteristics (peak and valley) of the data itself to define the boundaries of "high similarity" and "significant difference" by constructing the probability density distribution curve of all spatial scene similarity values. Selecting the valley value on both sides of the maximum peak as the boundary of the first spatial scene similarity threshold interval (high similarity interval) can effectively capture the similarity of the main body in the data; selecting the valley value on both sides of the minimum peak as the boundary of the second spatial scene similarity threshold interval (significant difference interval) can define the region of obvious difference. This method of automatically determining the threshold interval based on the distribution characteristics of the data enables the screening criteria to automatically adjust according to the actual similarity distribution of the image set in each processing task, avoiding the deviation and inadaptability caused by manual threshold setting, greatly enhancing the scientificity, objectivity and adaptability to different scenes of the screening process, and ensuring the rationality of the screening results.

[0088] Step 13, determining the original social scene images according to the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval, and combining the spatial scene similarity values to obtain the retained original social scene images;

[0089] Step 131, setting the valley value less than the first left boundary of the first spatial scene similarity threshold interval as the third right boundary, setting the valley value greater than the second right boundary of the second spatial scene similarity threshold interval as the third left boundary, and combining the third right boundary and the third left boundary to obtain the third spatial scene similarity threshold interval;

[0090] Step 132, extracting the spatial scene similarity value between each pair of original social scene images;

[0091] Step 133, counting the number of times that the spatial scene similarity value between the current original social scene image and the remaining original social scene images falls into the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval;

[0092] Step 134, based on the statistical result, if the spatial scene similarity value between the current original social scene image and the remaining original social scene images falls into the first spatial scene similarity threshold interval, the current original social scene image is removed;

[0093] If the spatial scene similarity value between the current original social scene image and the remaining original social scene images falls into the second spatial scene similarity threshold interval, the current original social scene image is retained;

[0094] If the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images fall into the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval respectively, then determine whether the number of times they fall into the first spatial scene similarity threshold interval is greater than the number of times they fall into the second spatial scene similarity threshold interval: if greater, then discard the current original social scene image; if less, then retain the current original social scene image; if equal, then determine that the current original social scene image is an uncertain original social scene image;

[0095] If the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval, the second spatial scene similarity threshold interval, and the third spatial scene similarity threshold interval, the current original social scene image is determined to be an uncertain original social scene image;

[0096] Step 135: Analyze the original social scene image determined to be an uncertain original social scene image, and obtain a retained original social scene image based on the analysis result;

[0097] It is worth noting that a well-organized and logically rigorous judgment system has been established for steps 131 to 135, enabling refined classification and decision-making of raw social scene images. This effectively handles complex situations that a simple "keep / remove" strategy cannot address. This system not only makes a preliminary judgment based on the number of times an image pair's similarity value falls into the first (highly similar) and second (significantly different) intervals (those falling entirely into the highly similar zone are eliminated to eliminate duplication, while those falling entirely into the significantly different zone are retained to preserve diversity), but more importantly, it introduces the concept of "uncertain raw social scene images." When the similarity status of an image pair fluctuates between highly similar and significantly different (with unequal or equal similarity) or involves borderline areas (the third interval), the system labels it as "uncertain." This design avoids arbitrary decisions made when information is insufficient or ambiguous, emphasizing prudent decision-making. By separating images that are difficult to judge directly for subsequent specialized analysis, steps 131 to 135 provide a structured framework for handling complex and boundary cases. They are the core steps for achieving refined screening and reducing the false positive rate, ensuring the integrity and rigor of the screening process.

[0098] Step 1351: Calculate the uncertainty factor index of each original social scene image determined to be an uncertain original social scene image;

[0099] Step 13511: Obtain the clarity, contrast, and noise level of each original social scene image determined to be an uncertain original social scene image, and determine a quality score for each original social scene image determined to be an uncertain original social scene image based on the clarity, contrast, and noise level;

[0100] It is worth mentioning that for each original social scene image judged as "uncertain", firstly, its basic image quality is evaluated. The specific process is as follows: firstly, the gradient variance of the image is calculated by using the Laplace operator, and the greater the variance, the clearer the image. The variance value is normalized to the interval [0, 1] as the definition score, then the standard deviation of the image gray histogram is calculated, and the greater the standard deviation, the higher the contrast. Similarly, it is normalized to the interval [0, 1] as the contrast score, then the noise intensity is estimated using the local variance method, that is, the average value of the square difference between each pixel and its neighborhood mean is calculated. The lower the noise intensity is, the better, so it is taken as the reciprocal and normalized to the interval [0, 1] as the noise level score; finally, the arithmetic mean of the above three scores is taken, so as to obtain the quality score Q of the original social scene image of each original social scene image judged as uncertain.

[0101] Step 13512, analyze and extract the content complexity features in each original social scene image judged as uncertain, the content complexity features including target quantity, target distribution density and background interference degree, and determine the content complexity score of each original social scene image judged as uncertain based on the content complexity features;

[0102] It is worth mentioning that the specific process of analyzing the content features of each "uncertain" image to evaluate its complexity is as follows: firstly, for the target quantity, the number of people in the image is counted, the more the number of people, the higher the complexity, and then the number of people is normalized to the interval [0, 1], then for the target distribution density, the ratio of the total area of all detected target bounding boxes to the total area of the image is calculated, the larger the ratio, the more dense the distribution, the higher the complexity, and the ratio is directly used as the density score, then for the background interference degree, an edge detection algorithm such as Canny is used to calculate the edge density of the background area, the higher the edge density, the more complex the background, and then the edge density is normalized to the interval [0, 1] as the interference degree score; finally, the arithmetic mean of the above three scores is taken, so as to obtain the content complexity score of each original social scene image judged as uncertain .

[0103] Step 13513, take the mean of the quality score of the original social scene image of each original social scene image judged as uncertain and the content complexity score of each original social scene image judged as uncertain, to obtain the comprehensive uncertainty component of the original social scene image of each original social scene image judged as uncertain ;

[0104] Step 13514: Based on the number of times that each original social scene image determined to be an uncertain original social scene image falls within the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval in the statistics, normalize the comprehensive uncertainty component of each original social scene image determined to be an uncertain original social scene image, and convert it through a preset nonlinear mapping function to obtain an uncertainty factor index for each original social scene image determined to be an uncertain original social scene image;

[0105] It is worth noting that the performance of the combined image in similarity statistics is When making adjustments, first count the number of times the image falls into the first spatial scene similarity threshold interval in the similarity statistics , and the number of times it falls into the second spatial scene similarity threshold interval , and define a normalization factor F. Specifically, the calculation formula for the normalization factor F is: in is a very small positive number to prevent the denominator from being zero; The value range of is close to [0,1], reflecting the degree of bias in the similarity judgment of the image;

[0106] Then the normalized combined uncertainty component Input a preset nonlinear mapping function. In this embodiment, the Sigmoid function is preferably used to enhance the discrimination: in The uncertainty factor index of the original social scene image that is determined to be uncertain; k is the steepness of the control curve; t is the threshold parameter; after this transformation, the value of the uncertainty factor index is more concentrated around 0 or 1, which is convenient for subsequent binary decision-making;

[0107] It is worth further explaining that, among them, for steps 13511 to steps 13514, a specific and implementable technical solution is provided for the calculation of the "uncertainty factor index", realizing the objective quantification of the inherent uncertainty of the image and being the key support for the landing of fine analysis. This scheme systematically evaluates multiple inherent attributes of the image: first, the quality score is determined by obtaining parameters such as sharpness, contrast, noise level, etc., directly measuring the usability basis of the image; second, the content complexity score is determined by analyzing the target quantity, distribution density, background interference, etc., reflecting the difficulty of image recognition; then, the quality score and the content complexity score are fused to obtain a comprehensive uncertainty component; finally, combined with the specific performance of this image in the similarity statistics (the number of times falling into each interval), the final uncertainty factor index is generated through normalization and nonlinear mapping. This calculation process is logically clear and closely related, and it converts the subjective "uncertainty" concept into an objective and comparable numerical value. The generation of this index fully considers the internal and external factors that affect the difficulty of image judgment, providing a solid and quantitative basis for the decision-making in steps 1351 to steps 1353, ensuring the scientificity, consistency and repeatability of the processing of "uncertain" images, and is the final technical closed loop for realizing fine and intelligent decision-making in the entire intelligent screening system.

[0108] Step 1352, setting a decision threshold for the uncertainty factor index;

[0109] It is worth mentioning that the specific process of setting a decision threshold for the uncertainty factor index is as follows: first, a part of representative samples are selected from the known data set as a reference sample set. This sample set should include images that are definitely retained and images that are definitely excluded, i.e. images directly determined by the first and second spatial scene similarity threshold intervals. These samples not only provide a reference standard in actual operation, but also provide basic data support for subsequent threshold setting. For each image in the selected reference sample set, calculate its uncertainty factor index according to the method defined earlier. This step ensures that all reference samples have a corresponding uncertainty factor index value, providing a quantitative basis for subsequent analysis. Next, analyze the distribution of the uncertainty factor index in the two groups of samples. Specifically: draw the uncertainty factor index histogram or density plot of the two groups of samples, observe their distribution patterns and overlap, and calculate and record the statistical characteristics of the uncertainty factor index of each group of samples, such as mean, median, standard deviation, etc. According to the analysis results, a decision threshold can be set. The common strategy is to use the mean or median, i.e. if the uncertainty factor index distributions of the retained group and the excluded group are clearly separated, a value between the mean or median of the two groups can be selected as the decision threshold.

[0110] Step 1353, based on the determination threshold, if the current determination is that the number of times that the spatial scene similarity value between the original social scene image determined as uncertain and the remaining original social scene images falls into the first spatial scene similarity threshold interval is greater than the number of times that the spatial scene similarity value falls into the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image determined as uncertain is less than the determination threshold, then the original social scene image determined as uncertain is removed;

[0111] If the current determination is that the number of times that the spatial scene similarity value between the original social scene image determined as uncertain and the remaining original social scene images falls into the first spatial scene similarity threshold interval is greater than the number of times that the spatial scene similarity value falls into the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image determined as uncertain is greater than or equal to the determination threshold, then the original social scene image determined as uncertain is retained;

[0112] If the current determination is that the number of times that the spatial scene similarity value between the original social scene image determined as uncertain and the remaining original social scene images falls into the first spatial scene similarity threshold interval is less than the number of times that the spatial scene similarity value falls into the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image determined as uncertain is less than the determination threshold, then the original social scene image determined as uncertain is removed;

[0113] If the current determination is that the number of times that the spatial scene similarity value between the original social scene image determined as uncertain and the remaining original social scene images falls into the first spatial scene similarity threshold interval is less than the number of times that the spatial scene similarity value falls into the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image determined as uncertain is greater than or equal to the determination threshold, then the original social scene image determined as uncertain is retained;

[0114] If the current determination is that the number of times that the spatial scene similarity value between the original social scene image determined as uncertain and the remaining original social scene images falls into the first spatial scene similarity threshold interval is equal to the number of times that the spatial scene similarity value falls into the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image determined as uncertain is less than the determination threshold, then the original social scene image determined as uncertain is removed;

[0115] If the current original social scene image is determined to be uncertain, and the number of times that the spatial scene similarity value between the original social scene image and the rest of the original social scene images falls into the first spatial scene similarity threshold interval is equal to the number of times that falls into the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image determined to be uncertain is greater than or equal to the determination threshold, the current original social scene image determined to be uncertain is retained;

[0116] It is worth noting that a set of refined analysis and decision mechanisms based on the "uncertainty factor index" are proposed for steps 1351 to 1353, which successfully solve the processing problem of such ambiguous images and realize the transformation from "uncertainty" to clear "retention / rejection". The mechanism calculates the uncertainty factor index of each uncertain image, sets a determination threshold, and combines the quality and complexity attributes of the image itself (quantified by indicators) with its relative relationship in the group (comparison of the number of times falling into different intervals) to make a comprehensive judgment. Whether in the case of high similarity, significant difference or equal, the final retention or rejection is determined according to whether the uncertainty factor index exceeds the threshold. This dual judgment standard (external relationship + internal attribute) makes the decision more comprehensive and reliable: even if the similarity statistics are biased to one side, if the uncertainty factor index is high (such as poor image quality and complex content), the image may still be retained, avoiding the rejection caused by a single statistical result; on the contrary, if the index is low, it is more inclined to be rejected. Therefore, this effectively utilizes the inherent characteristics of "uncertain" images, and changes them from the "blind area" of the screening process to quantifiable and decidable objects, significantly improving the intelligence and accuracy of the overall screening strategy.

[0117] Step 14, collecting the retained original social scene images to obtain preferred social scene images;

[0118] It is worth mentioning that, among them, by introducing a set of systematic automatic screening process, steps 11 to step 14 completely replace the traditional inefficient and easy to misjudge image screening method relying on manual naked eye identification, realize the automatic and intelligent conversion from the original mass image to the high quality preferred image set. The process first calculates the spatial scene similarity value between any two images to provide a data basis for objective quantification of the relationship between images; Then, based on these similarity values, the first (high similarity) and second (significantly different) spatial scene similarity threshold intervals are automatically determined, which eliminates the subjectivity of fixed threshold, so that the screening standard can be adapted to the actual data distribution; Finally, by combining the threshold interval to determine the original image and collect the reserved images, the preferred social scene image is formed. This series of operations not only significantly improves the efficiency and consistency of image screening, effectively removes a large number of redundant images, and retains representative key perspectives, but also provides purer and more diverse input data for subsequent target detection and identity recognition, fundamentally solving the problem of subsequent recognition accuracy decline caused by improper screening, and is the key pre-process link to improve the performance of the whole system.

[0119] Step 2, input the preprocessed preferred social scene image into the pre-trained convolutional neural network model, detect all potential targets in the preferred social scene image, identify each detection target, and assign a bounding box to each detection target, the bounding box includes position and size;

[0120] It is worth mentioning that the training process of the convolutional neural network model is as follows: first, a training set is obtained, which includes a large number of social scene images, which come from multiple channels, for example: public data sets such as COCO, Pascal VOC, etc., which contain rich annotation information, covering various complex scenes and diverse target categories, and self-defined collected data, which are images taken by fixed monitoring devices or mobile devices (such as smart phones) in different social occasions. These images need to be manually labeled to mark all potential targets and their bounding boxes; each image contains at least one target, and each target has detailed annotation information, including position coordinates, i.e. the left upper corner x, y coordinates of the bounding box, width, height and target category label, such as person, animal, object, etc.; then the training label is set, which mainly includes two types of information: 1. Bounding box coordinates: for each target, the left upper corner coordinates of its bounding box and the width and height are provided, which are used to locate the position and size of the target; 2. Category label: assign a category label to each target to indicate which category it belongs to, such as "person", "car", etc.; before obtaining the trained convolutional neural network model using the above training set and training label, an evaluation index needs to be selected, and the main evaluation indicators of the model performance include: mean average precision, which is one of the most commonly used evaluation indicators in target detection tasks, which calculates the average precision of each category and then takes the average of all categories, and recall and precision, wherein recall measures the proportion of correctly identified targets among all actual targets, and precision measures the proportion of actual correct targets among the predicted positive targets, and F1 score, which is the harmonic mean of precision and recall, is suitable for unbalanced classification problems; In addition, a cross-entropy loss function is used to measure the accuracy of the model's target category prediction; through the above steps, a trained convolutional neural network model can be obtained.

[0121] It is further worth mentioning that when using the trained convolutional neural network model for target detection, first, the preferred social scene image is input, i.e. the preprocessed preferred social scene image is input into the trained convolutional neural network model, at this time the model first extracts the appearance feature vector of the image through a series of convolutional layers , then generates candidate regions using the region proposal network, i.e. regions that may contain targets, and then further refines these candidate regions, finally outputs the specific position and category of each detection target; specifically, first input the image after convolutional layer, pooling layer, etc. operation, get multi-level appearance feature vector , based on the appearance feature vector The region proposal network generates a series of candidate boxes to preliminarily determine the region in which the target may exist, and then classifies the features in each candidate box while adjusting the position and size of the candidate box to more accurately match the real bounding box of the target. For each identified target, the model assigns a bounding box to the target, and the information of the bounding box includes the coordinates of the upper left corner of the bounding box, i.e., the position, and the width and height of the bounding box, i.e., the size.

[0122] Step 3, based on the position and size of each detected target, the relative position, distance and angle between each detected target are calculated, a spatial relationship graph between each detected target is constructed, the behavior features of each detected target are extracted in combination with the spatial relationship graph, the behavior mode of each detected target is analyzed based on the behavior features of each detected target, and a feature vector of each detected target is extracted. The feature vector, behavior mode and spatial relationship graph are input into the pre-trained deep learning model to confirm the identity of each detected target.

[0123] It is worth noting that the process of constructing the spatial relationship graph between the detected targets is as follows. In step 2, a bounding box has been assigned to each detected target in the preferred social scene image, and its position, i.e., the center coordinates of the bounding box and size, i.e., the width and height have been obtained. Based on this information, a spatial relationship graph between each detected target is constructed. Specifically, first, node definition is performed, i.e., each node in the spatial relationship graph represents a detected target, and the attributes of the node include the center coordinates , size and appearance feature vector extracted from the convolutional neural network model of the bounding box of the target; then edge definition is performed, i.e., there is an edge between any two nodes, i.e., detected targets i and j in the graph, and the weight of the edge is determined by the relative position, distance and angle between them. The relative position can be obtained by calculating the horizontal and vertical offsets of target i relative to target j, the spatial distance between the centers of the two targets can be obtained by using the Euclidean distance algorithm, and the angle can be obtained by calculating the angle between the vector from target j to target i and the horizontal axis. The weight of the edge can be a multi-dimensional vector or a scalar by fusing it through a simple function such as the inverse of the normalized distance. However, the embodiment adopts a multi-dimensional vector form to retain more rich spatial information. Finally, the spatial relationship graph is constructed, where V is the set of nodes and E is the set of edges, which completely describes the spatial topology between all detected targets in the image; then the behavior features of each detected target i are extracted in combination with the spatial relation graph, which include not only its own motion state, but more importantly its interaction behavior with other targets. The specific features include: local neighborhood features, i.e. the average distance, average angle deviation, nearest neighbor distance between target i and all its neighbor nodes, which reflect its position and interaction intensity in the local group; graph structure features, i.e. the degree centrality, closeness centrality of target i in graph G, which quantify its "center degree" or "isolation degree" in the social group;

[0124] Then based on the extracted behavior features, the behavior patterns of each detected target are analyzed, for example: if a target i has small average distance, small nearest neighbor distance and high degree centrality, its behavior pattern can be analyzed as "in the center of the group" or "interacting with multiple people"; if a target j has large average distance and low degree centrality, its behavior pattern is "on the edge" or "relatively isolated"; if a pair of targets k and l have very small distance and angle close to o or π, their behavior pattern is "standing side by side" or "talking face to face", these behavior patterns are qualitative or semi-quantitative descriptions, which provide context information for subsequent identity confirmation;

[0125] Then the feature vector is extracted, which fuses the following information of each detected target i into a high-dimensional feature vector , including the appearance feature vector , the behavior feature vector including local neighborhood features and graph structure features, and the encoding of behavior patterns , which uses one-hot encoding to represent "center", "edge", "pair interaction" and other behavior patterns; the final feature vector , where the semicolon represents vector splicing, which encodes the appearance, behavior and social context of the target, i.e. the encoding of behavior patterns;

[0126] It is worth further explaining that the training process of a deep learning model begins with obtaining a training set, which contains a large number of labeled social scene image sequences or single-frame images. Each image undergoes the same preprocessing and target detection process as described above. Crucially, each detected target in each image has a real identity label, which is usually obtained through manual annotation or other methods such as RFID in a controlled environment. Next, a training label is set, which is the real identity of each detected target. Before using the above training set and training labels to obtain a trained deep learning model, it is necessary to select model evaluation metrics, including identity recognition accuracy, which is the ratio of correctly recognized identities to the total number of detected targets; Top-K accuracy, which is the proportion of real identities that appear in the top K most likely identities predicted by the model, where K is usually 1 or 5; and a confusion matrix, which analyzes the confusion between different identities. Furthermore, the cross-entropy loss function is used as the main loss function. For each detected target i, the model outputs a probability distribution, representing the probability of it belonging to each identity category. Through the above steps, a trained deep learning model is obtained.

[0127] It is worth further explaining that when confirming the identity of each detected target, the extracted feature vector, behavior pattern, and spatial relationship graph of the entire image of each detected target i are input into the trained deep learning model. The model receives the feature vector, behavior pattern, and spatial relationship graph of the entire image as input. At this time, the deep learning model can directly process the graph structure and use graph convolution operations to aggregate neighbor information, so as to incorporate its social context when updating each node, that is, the representation of the detected target. Then, after multiple layers of calculation, the model outputs a probability distribution about the identity category, and finally selects the identity category with the highest probability as the final identity of the detected target i.

[0128] It should be noted that the terms used in the present invention are only for describing specific embodiments and are not intended to limit the scope of this application. As shown in the present specification, unless the context clearly indicates an exception, the words "one", "a", "a kind of" and / or "the" do not specifically refer to the singular and may also include the plural. The terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method or device comprising a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of further restrictions, the elements defined by the sentence "comprise a..." do not exclude the presence of other identical elements in the process, method or device comprising the elements.

[0129] It should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", and the like, indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are used only to facilitate the description of the present application and simplify the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. Unless otherwise specifically defined and limited, the terms "mount", "connect", "connect" and the like should be broadly understood, for example, can be fixedly connected, can also be detachably connected, or integrally connected; can be mechanically connected, can also be electrically connected; can be directly connected, can also be indirectly connected through an intermediate medium; can be internal communication of two elements. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0130] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present application.

Claims

1. A multi-objective spatial relationship driven human recognition method in social scene images, characterized by: The following steps are involved: Step 1: obtaining and screening original social scene images to obtain preferred social scene images, and preprocessing the preferred social scene images; Step 2: Input the pre-processed preferred social scene image into a pre-trained convolutional neural network model, detect all potential targets in the preferred social scene image, identify each detected target, and assign a bounding box to each detected target, the bounding box including its position and size; Step 3. Based on the position and size of each detection target, calculate the relative position, distance and angle between each detection target, construct a spatial relationship diagram between each detection target, extract the behavioral characteristics of each detection target in combination with the spatial relationship diagram, analyze the behavioral pattern of each detection target based on the behavioral characteristics of each detection target, and extract the feature vector of each detection target. Input the feature vector, behavioral pattern and spatial relationship diagram into the pre-trained deep learning model to confirm the identity of each detection target.

2. The multi-objective spatial relationship driven human recognition method in social scene images according to claim 1, characterized in that: The obtaining and screening the original social scene images to obtain the preferred social scene images includes: Step 11: Calculate the spatial scene similarity value between any two original social scene images; Step 12: determining a spatial scene similarity threshold interval based on the spatial scene similarity value, wherein the spatial scene similarity threshold interval includes a first spatial scene similarity threshold interval and a second spatial scene similarity threshold interval; Step 13: judging the original social scene image based on the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval in combination with the spatial scene similarity value to obtain a retained original social scene image; Step 14: Aggregate the retained original social scene images to obtain the preferred social scene images.

3. The multi-objective spatial relationship driven human recognition method in social scene images according to claim 2, characterized in that: The calculating of the spatial scene similarity value between any two original social scene images includes: Step 111: extract key points and descriptors of each original social scene image based on the SIFT algorithm; Step 112: construct a high-level semantic feature vector for each original social scene image using key points and descriptors; Step 113: Calculate the semantic similarity between any two original social scene images based on the extracted high-level semantic feature vectors; Step 114: Apply the perceptual hash algorithm to generate a perceptual hash value for each image, and calculate the hash difference between any two original social scene images; Step 115: normalize the hash difference to obtain the hash similarity between any two original social scene images; Step 116: Take the average of the semantic similarity and the hash similarity between any two original social scene images to obtain the spatial scene similarity value between any two original social scene images.

4. The multi-target spatial relationship driven human recognition method in social scene images according to claim 3, characterized in that: The determining of the spatial scene similarity threshold interval based on the spatial scene similarity value includes: Step 121: All spatial scene similarity values ​​are aggregated, and a probability density distribution curve of all spatial scene similarity values ​​is constructed using a probability density estimation method based on the aggregated results; Step 122: Identify all peak values ​​and valley values ​​on both sides of the peak values ​​according to the probability density distribution curve; Step 123: sort all identified peaks from largest to smallest; Step 124: Select the maximum peak value based on the sorting result, and use the nearest valley value to the left of the maximum peak value as the first left boundary, and the nearest valley value to the right of the maximum peak value as the first right boundary; the first left boundary is greater than or equal to the nearest valley value to the left of the maximum peak value; and the first right boundary is less than or equal to the nearest valley value to the right of the maximum peak value. Step 125: Combine the first left boundary and the first right boundary to obtain a first spatial scene similarity threshold interval; Step 126: Select the minimum peak value based on the sorting result, and use the nearest valley value to the left of the minimum peak value as the second left boundary, and use the nearest valley value to the right of the minimum peak value as the second right boundary; the second left boundary is greater than or equal to the nearest valley value to the left of the minimum peak value; and the second right boundary is less than or equal to the nearest valley value to the right of the minimum peak value. Step 127: Combine the second left boundary and the second right boundary to obtain a second spatial scene similarity threshold interval.

5. The multi-objective spatial relationship driven human recognition method in social scene images according to claim 4, characterized in that: The determining the original social scene image based on the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval in combination with the spatial scene similarity value to obtain the retained original social scene image includes: Step 131: Set the valley value of the first left boundary smaller than the first spatial scene similarity threshold interval as the third right boundary, set the valley value of the second right boundary larger than the second spatial scene similarity threshold interval as the third left boundary, and combine the third right boundary and the third left boundary to obtain a third spatial scene similarity threshold interval; Step 132: extracting the spatial scene similarity value between each pair of original social scene images; Step 133: Count the number of times that the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval respectively; Step 134: based on the statistical results, if the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images all fall within a first spatial scene similarity threshold range, then the current original social scene image is eliminated; If the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images all fall within the second spatial scene similarity threshold interval, retaining the current original social scene image; If the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images fall into the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval respectively, then determine whether the number of times they fall into the first spatial scene similarity threshold interval is greater than the number of times they fall into the second spatial scene similarity threshold interval: if greater, then discard the current original social scene image; if less, then retain the current original social scene image; if equal, then determine that the current original social scene image is an uncertain original social scene image; If the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval, the second spatial scene similarity threshold interval, and the third spatial scene similarity threshold interval, respectively, then the current original social scene image is determined to be an uncertain original social scene image; Step 135 : Analyze the original social scene image determined to be an uncertain original social scene image, and obtain a retained original social scene image based on the analysis result.

6. The multi-objective spatial relationship driven human recognition method in social scene images according to claim 5, characterized in that: The analyzing the current original social scene image determined to be an uncertain original social scene image and obtaining a retained original social scene image based on the analysis result includes: Step 1351: Calculate the uncertainty factor index of each original social scene image determined to be an uncertain original social scene image; Step 1352: Setting a determination threshold for the uncertainty factor index; Step 1353: Based on the determination threshold, if the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is greater than the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is less than the determination threshold, then the original social scene image currently determined to be an uncertain original social scene image is eliminated; If the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is greater than the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is greater than or equal to the determination threshold, then retaining the original social scene image currently determined to be an uncertain original social scene image; If the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is less than the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is less than the determination threshold, then the original social scene image currently determined to be an uncertain original social scene image is eliminated; If the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is less than the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is greater than or equal to the determination threshold, then retaining the original social scene image currently determined to be an uncertain original social scene image; If the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is equal to the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is less than the determination threshold, then the original social scene image currently determined to be an uncertain original social scene image is eliminated; If the number of times the spatial scene similarity value between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images falls into the first spatial scene similarity threshold interval is equal to the number of times it falls into the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is greater than or equal to the determination threshold, then the original social scene image currently determined to be an uncertain original social scene image is retained.

7. The multi-objective spatial relationship driven human recognition method in social scene images according to claim 6, characterized in that: The calculating of the uncertainty factor index of each original social scene image determined to be an uncertain original social scene image includes: Step 13511: Obtain the clarity, contrast, and noise level of each original social scene image determined to be an uncertain original social scene image, and determine a quality score for each original social scene image determined to be an uncertain original social scene image based on the clarity, contrast, and noise level; Step 13512: Analyze and extract content complexity features from each original social scene image determined to be uncertain, wherein the content complexity features include the number of objects, the density of object distribution, and the degree of background interference, and determine a content complexity score for each original social scene image determined to be uncertain based on the content complexity features. Step 13513: averaging the quality score of each original social scene image determined to be an uncertain original social scene image and the content complexity score of each original social scene image determined to be uncertain, to obtain a comprehensive uncertainty component of each original social scene image determined to be an uncertain original social scene image. Step 13514: Based on the number of times that each original social scene image determined to be an uncertain original social scene image falls into the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval in the statistics, the comprehensive uncertainty component of each original social scene image determined to be an uncertain original social scene image is normalized, and converted through a preset nonlinear mapping function to obtain the uncertainty factor index of each original social scene image determined to be an uncertain original social scene image.

Images