Method for human recognition in social scene image driven by multi-target spatial relationship

Abstract

The invention discloses a method for human recognition in a social scene image driven by a multi-target spatial relationship, relates to the technical field of image recognition, and effectively improves the quality and diversity of input data through an automatic and intelligent multi-stage screening mechanism, remarkably reduces the interference of redundant data and noise data on a subsequent model, and improves the recognition efficiency. The method greatly improves the precision and robustness of target detection and identity recognition, greatly improves the processing efficiency and automation degree of the whole system, solves a bottleneck problem of an image screening link in the prior art, and provides a powerful technical support for efficient and precise human recognition in a complex social scene.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method for human recognition in social scene images driven by multi-target spatial relationships. Background Art

[0002] In the fields of intelligent monitoring, social network content analysis, human-computer interaction, and public security, accurately identifying individuals and understanding their behavior patterns from complex social scene images has become a vital basic task. Such tasks usually require processing image sequences containing multiple targets with complex spatial relationships and interactive behaviors between targets. In order to improve the efficiency and accuracy of subsequent recognition, it is usually necessary to pre-process a large number of original social scene images before target detection and identity confirmation. Among them, image screening is a crucial and indispensable first step. Currently, in practical applications or early research, image screening is usually performed by manual visual identification. Operators compare the large number of original social scene images collected one by one through visual observation, and judge the similarity between images based on experience, so as to manually eliminate those obviously repeated or highly similar images and retain representative images. However, this screening method that relies on manual visual identification has significant disadvantages:

[0003] Faced with massive amounts of image data, the manual screening process is extremely slow, requiring a lot of manpower and time costs, and it is difficult to meet the needs of real-time or large-scale data processing; and manual judgment is highly dependent on the operator's experience, attention, and subjective feelings. For images with critical similarity, for example, slight changes in the person's posture, slightly different lighting conditions, partial occlusion of the background, etc., different people or even the same person in different states may make inconsistent judgments, resulting in the elimination of images that should not be eliminated, that is, frequent misjudgments; at the same time, manual screening is difficult to systematically quantify and evaluate the various uncertain factors that affect image similarity judgment, such as blurred image clarity and low contrast. These factors will significantly increase the difficulty of judgment, but current manual methods cannot provide an objective and unified standard to measure the impact of these factors on similarity judgment, resulting in a more arbitrary and unreliable decision-making process when processing such uncertain images; the above disadvantages directly lead to the artificial amplification of the complexity of image deduplication operations. Operators need to repeatedly weigh a large number of visually similar but different details of images, which is prone to fatigue and misjudgment. Therefore, manual screening methods inevitably increase The false positive rate of image deduplication operations has increased, making it difficult to ensure the accuracy and consistency of the screening results. Therefore, if the existing manual method is still used for image screening, it is very likely to lead to the following serious consequences: the screened image set is likely to still contain a large number of redundant images, or key valid images are mistakenly eliminated, resulting in uneven quality of the data set input to the subsequent processing flow, and low-quality input data will directly affect the performance of deep learning models such as convolutional neural networks. Redundant data can easily lead to model overfitting, ultimately reducing the recall rate of target detection and directly affecting the accuracy of identity recognition. In summary, the manual naked eye screening method commonly relied on in existing technologies is inefficient, highly subjective, prone to misjudgment, and lacks an effective evaluation mechanism for uncertainty factors in images. It not only greatly increases the complexity and misjudgment risk of image deduplication work, but also seriously affects the performance of subsequent core tasks such as target detection and identity recognition.

[0004] Therefore, the existing technology urgently needs a technical solution for human recognition methods in social scene images driven by multi-target spatial relationships. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides a method for human recognition in social scene images driven by multi-target spatial relationships, which specifically includes the following steps:

[0006] Step 1: obtaining and screening original social scene images to obtain preferred social scene images, and preprocessing the preferred social scene images;

[0007] Step 11: Calculate the spatial scene similarity value between any two original social scene images;

[0008] Step 111: extract key points and descriptors of each original social scene image based on the SIFT algorithm;

[0009] Step 112: construct a high-level semantic feature vector for each original social scene image using key points and descriptors;

[0010] Step 113: Calculate the semantic similarity between any two original social scene images based on the extracted high-level semantic feature vectors;

[0011] Step 114: Apply the perceptual hash algorithm to generate a perceptual hash value for each image, and calculate the hash difference between any two original social scene images;

[0012] Step 115: normalize the hash difference to obtain the hash similarity between any two original social scene images;

[0013] Step 116: taking the average of the semantic similarity and the hash similarity between any two original social scene images to obtain a spatial scene similarity value between any two original social scene images;

[0014] Step 12: determining a spatial scene similarity threshold interval based on the spatial scene similarity value, wherein the spatial scene similarity threshold interval includes a first spatial scene similarity threshold interval and a second spatial scene similarity threshold interval;

[0015] Step 121: All spatial scene similarity values ​​are aggregated, and a probability density distribution curve of all spatial scene similarity values ​​is constructed using a probability density estimation method based on the aggregated results;

[0016] Step 122: Identify all peak values ​​and valley values ​​on both sides of the peak values ​​according to the probability density distribution curve;

[0017] Step 123: sort all identified peaks from largest to smallest;

[0018] Step 124: Select the maximum peak value based on the sorting result, and use the nearest valley value to the left of the maximum peak value as the first left boundary, and the nearest valley value to the right of the maximum peak value as the first right boundary; the first left boundary is greater than or equal to the nearest valley value to the left of the maximum peak value; and the first right boundary is less than or equal to the nearest valley value to the right of the maximum peak value.

[0019] Step 125: Combine the first left boundary and the first right boundary to obtain a first spatial scene similarity threshold interval;

[0020] Step 126: Select the minimum peak value based on the sorting result, and use the nearest valley value to the left of the minimum peak value as the second left boundary, and use the nearest valley value to the right of the minimum peak value as the second right boundary; the second left boundary is greater than or equal to the nearest valley value to the left of the minimum peak value; and the second right boundary is less than or equal to the nearest valley value to the right of the minimum peak value.

[0021] Step 127: Combine the second left boundary and the second right boundary to obtain a second spatial scene similarity threshold interval;

[0022] Step 13: judging the original social scene image based on the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval in combination with the spatial scene similarity value to obtain a retained original social scene image;

[0023] Step 131: Set the valley value of the first left boundary smaller than the first spatial scene similarity threshold interval as the third right boundary, set the valley value of the second right boundary larger than the second spatial scene similarity threshold interval as the third left boundary, and combine the third right boundary and the third left boundary to obtain a third spatial scene similarity threshold interval;

[0024] Step 132: extracting the spatial scene similarity value between each pair of original social scene images;

[0025] Step 133: Count the number of times that the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval respectively;

[0026] Step 134: based on the statistical results, if the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images all fall within a first spatial scene similarity threshold range, then the current original social scene image is eliminated;

[0027] If the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images all fall within the second spatial scene similarity threshold interval, retaining the current original social scene image;

[0028] If the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images fall into the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval respectively, then determine whether the number of times they fall into the first spatial scene similarity threshold interval is greater than the number of times they fall into the second spatial scene similarity threshold interval: if greater, then discard the current original social scene image; if less, then retain the current original social scene image; if equal, then determine that the current original social scene image is an uncertain original social scene image;

[0029] If the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval, the second spatial scene similarity threshold interval, and the third spatial scene similarity threshold interval, the current original social scene image is determined to be an uncertain original social scene image;

[0030] Step 135: Analyze the original social scene image determined to be an uncertain original social scene image, and obtain a retained original social scene image based on the analysis result;

[0031] Step 1351: Calculate the uncertainty factor index of each original social scene image determined to be an uncertain original social scene image;

[0032] Step 13511: Obtain the clarity, contrast, and noise level of each original social scene image determined to be an uncertain original social scene image, and determine a quality score for each original social scene image determined to be an uncertain original social scene image based on the clarity, contrast, and noise level;

[0033] Step 13512: Analyze and extract content complexity features from each original social scene image determined to be uncertain, wherein the content complexity features include the number of objects, the density of object distribution, and the degree of background interference, and determine a content complexity score for each original social scene image determined to be uncertain based on the content complexity features.

[0034] Step 13513: averaging the quality score of each original social scene image determined to be an uncertain original social scene image and the content complexity score of each original social scene image determined to be uncertain, to obtain a comprehensive uncertainty component of each original social scene image determined to be an uncertain original social scene image.

[0035] Step 13514: Based on the number of times that each original social scene image determined to be an uncertain original social scene image falls within the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval in the statistics, normalize the comprehensive uncertainty component of each original social scene image determined to be an uncertain original social scene image, and convert it through a preset nonlinear mapping function to obtain an uncertainty factor index for each original social scene image determined to be an uncertain original social scene image;

[0036] Step 1352: Setting a determination threshold for the uncertainty factor index;

[0037] Step 1353: Based on the determination threshold, if the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is greater than the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is less than the determination threshold, then the original social scene image currently determined to be an uncertain original social scene image is eliminated;

[0038] If the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is greater than the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is greater than or equal to the determination threshold, then retaining the original social scene image currently determined to be an uncertain original social scene image;

[0039] If the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is less than the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is less than the determination threshold, then the original social scene image currently determined to be an uncertain original social scene image is eliminated;

[0040] If the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is less than the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is greater than or equal to the determination threshold, then retaining the original social scene image currently determined to be an uncertain original social scene image;

[0041] If the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is equal to the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is less than the determination threshold, then the original social scene image currently determined to be an uncertain original social scene image is eliminated;

[0042] If the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is equal to the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is greater than or equal to the determination threshold, then retaining the original social scene image currently determined to be an uncertain original social scene image;

[0043] Step 14: Aggregate the retained original social scene images to obtain the optimal social scene images;

[0044] Step 2: Input the pre-processed preferred social scene image into a pre-trained convolutional neural network model, detect all potential targets in the preferred social scene image, identify each detected target, and assign a bounding box to each detected target, the bounding box including its position and size;

[0045] Step 3. Based on the position and size of each detection target, calculate the relative position, distance and angle between each detection target, construct a spatial relationship diagram between each detection target, extract the behavioral characteristics of each detection target in combination with the spatial relationship diagram, analyze the behavioral pattern of each detection target based on the behavioral characteristics of each detection target, and extract the feature vector of each detection target. Input the feature vector, behavioral pattern and spatial relationship diagram into the pre-trained deep learning model to confirm the identity of each detection target.

[0046] The embodiments of the present invention have the following technical effects:

[0047] The present invention automatically calculates the spatial scene similarity value between any two original social scene images. This value combines the high-level semantic feature similarity extracted by the SIFT algorithm and the hash similarity obtained by the perceptual hashing algorithm. It can measure the comprehensive similarity between images more comprehensively and robustly, effectively overcome the limitations of a single measurement method, and lay a solid data foundation for subsequent accurate screening. Secondly, the present invention innovatively proposes an adaptive threshold interval determination method based on probability density estimation. By analyzing the probability density distribution curve of all spatial scene similarity values, the peaks and valleys are automatically identified, and then the first and second spatial scene similarity threshold intervals representing "high similarity" and "significant difference" are determined, so that the screening criteria can be dynamically adjusted according to the actual data distribution, greatly improving the scientificity and adaptability of the screening. On this basis, the present invention designs a rigorous judgment logic, which not only performs preliminary screening based on the statistical number of times the image similarity value falls into different threshold intervals, but more importantly, introduces the concept of uncertain original social scene images, and constructs a third space containing The judgment rule of the scene similarity threshold interval can effectively identify images that are at the similarity boundary and difficult to judge directly. For these uncertain images, the present invention further proposes a calculation and analysis method for the uncertainty factor index. This index comprehensively considers quality factors such as image clarity, contrast, and noise level, as well as content complexity characteristics such as the number of targets and distribution density, and combines its performance in similarity statistics to achieve objective quantification of the degree of image uncertainty. Finally, by setting a judgment threshold and combining it with the index, a refined secondary decision is made on the uncertain images, that is, whether to retain or eliminate them, which completely solves the problems of arbitrariness and high error rate in manual screening of such images. This series of innovations enables the present invention to efficiently and accurately screen high-quality, low-redundancy, and information-rich preferred social scene images from massive raw images. Finally, the strictly screened and pre-processed preferred images are input into the pre-trained deep learning model, and its precise bounding box detection, behavioral feature extraction and pattern analysis based on spatial relationship graphs can more reliably confirm the identity of each detected target.

[0048] In summary, the present invention effectively improves the quality and diversity of input data through an automated and intelligent multi-level screening mechanism, significantly reduces the interference of redundant data and noise data on subsequent models, and not only greatly improves the accuracy and robustness of target detection and identity recognition, but also greatly improves the processing efficiency and automation level of the entire system, solves the bottleneck problem of the image screening link in the existing technology, and provides strong technical support for efficient and accurate human recognition in complex social scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 This is a flowchart of a method for human recognition in social scene images driven by multi-target spatial relationships provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention are described clearly and completely below. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are also within the scope of protection of the present invention.

[0052] Example 1: Figure 1 As shown, the present invention provides a multi-target spatial relationship driven human recognition method in social scene images, comprising the following steps:

[0053] Step 1: obtaining and screening original social scene images to obtain preferred social scene images, and preprocessing the preferred social scene images;

[0054] It is worth noting that the acquisition of original social scene images is mainly achieved through the following two ways: First, fixed monitoring equipment collection, that is, deploying fixed-position cameras or video surveillance systems in specific social places, such as shopping malls, squares, stations, school corridors, conference rooms, etc.; these devices continuously or on demand collect video streams or timed snapshots of the area; through video frame extraction technology, single-frame images can be extracted from continuous video streams as original social scene images; Second, mobile device or social media upload, that is, from user-authorized mobile devices such as smartphones, tablets or social media platforms such as Weibo, WeChat Moments, Instagram, Facebook, etc., collect photos containing social scenes uploaded by users. These images are usually taken by users in various social activities. Shooting has the characteristics of diverse perspectives, rich scenes, and complex lighting conditions; among them, the original social scene images obtained specifically refer to static images containing two or more individuals, that is, humans interacting or coexisting in the same physical space. These images specifically include but are not limited to: group photos, such as photos of family gatherings, friends’ dinners, team photos, etc.; public space activity images, such as images taken in scenes such as walking in the park, talking in a cafe, discussing in a meeting, watching sports events, etc.; specific social event images, such as images containing multiple people taken in events such as weddings, birthday parties, and award ceremonies; surveillance scene images, obtained from fixed surveillance cameras, recording images of multiple people moving in a specific area, such as crowds at entrances, pedestrians in corridors, etc.; The common characteristics of these images are: there are multiple targets, that is, people, in the image, and there may be certain spatial relationships between these targets, such as standing side by side, talking face to face, following each other, etc., and potential interactive behaviors;

[0055] After obtaining the original social scene images, a series of preprocessing operations are required to improve the image quality, unify the data format, and prepare for subsequent deep learning model processing. The preprocessing operations mainly include: image size normalization, that is, adjusting all the preferred social scene images to a unified resolution, such as 640x480 pixels or 1024x768 pixels. Pixels, which ensures that the data input to the convolutional neural network model has a fixed size, avoiding the problem of increased computational complexity or model input mismatch due to size differences; and color space conversion, that is, converting the image from the original color space, such as RGB, to a color space more suitable for subsequent processing, such as BGR, to match the default input format of common deep learning frameworks such as OpenCV, or converting it to a grayscale image to reduce the amount of calculation; and brightness and contrast adjustment, that is, for images that are too dark or too bright, automatic or semi-automatic brightness and contrast correction is performed to improve visual effects and feature extraction capabilities; and denoising processing, that is, applying image filtering techniques, such as Gaussian filtering and median filtering, to remove random noise in the image, especially for images taken under low light conditions, to improve image clarity, etc.

[0056] Step 11: Calculate the spatial scene similarity value between any two original social scene images;

[0057] Step 111: extract key points and descriptors of each original social scene image based on the SIFT algorithm;

[0058] It is worth noting that for each raw social scene image to be processed, the scale-invariant feature transform (SIFT) algorithm is applied for feature extraction. The specific operation is as follows: First, the Gaussian difference method is used to detect key points in the image at different scales. These key points are usually located at locations with significant local features such as image corners and edge intersections, and are invariant to scale scaling and rotation. Next, each detected key point is assigned a main direction based on the local gradient direction to achieve rotation invariance. Finally, with the key point as the center, the gradient information of the surrounding area is extracted to generate a 128-dimensional vector as the descriptor of the key point. This descriptor can robustly characterize the local appearance characteristics of the key point. This step generates a feature set containing multiple key points and their corresponding descriptors for each raw social scene image.

[0059] Step 112: construct a high-level semantic feature vector for each original social scene image using key points and descriptors;

[0060] It is worth noting that after completing the SIFT feature extraction, the discrete key point-descriptor set needs to be converted into a high-level semantic feature vector that can represent the entire original social scene image. This embodiment adopts the "bag of words model" method for construction. First, a large number of samples are randomly selected from the SIFT descriptors of all original social scene images, and a "visual dictionary" containing K cluster centers, i.e., "visual vocabulary", is generated through the K-means clustering algorithm. Then, for each original social scene image, all its SIFT descriptors are counted to which visual vocabulary they belong. Specifically, by calculating the Euclidean distance between the descriptor and each cluster center, the nearest cluster center is found to complete the process of counting which visual vocabulary all its SIFT descriptors belong to. Then, the frequency of each visual vocabulary in the image is counted. Finally, these K frequency values ​​are arranged in order to form a K-dimensional vector, which is the high-level semantic feature vector of the original social scene image. This vector reflects the distribution of "visual vocabulary" contained in the image from a statistical perspective and contains high-level semantic information of the image.

[0061] Step 113: Calculate the semantic similarity between any two original social scene images based on the extracted high-level semantic feature vectors;

[0062] It is worth noting that, for any two original social scene images A and B, their high-level semantic feature vectors VA and VB constructed in step 112 are obtained respectively, and then the cosine similarity between the two vectors is calculated as the semantic similarity between them.

[0063] Step 114: Apply the perceptual hash algorithm to generate a perceptual hash value for each image, and calculate the hash difference between any two original social scene images;

[0064] It is worth noting that for each original social scene image, an average hashing algorithm is applied to generate its perceptual hash value. The specific process is as follows: first, the image is scaled to a fixed small size, such as 8x8 pixels, and then converted to a grayscale image. Next, the average grayscale value of these 64 pixels is calculated. Finally, the grayscale value of each pixel is compared with the average value. Pixels with a grayscale value greater than or equal to the average value are marked as 1, and those with a grayscale value less than the average value are marked as 0. This results in a 64-bit binary string, which is the perceptual hash value of the original social scene image. For any two original social scene images A and B, their perceptual hash values ​​Hash_A and Hash_B are obtained respectively, and the Hamming distance between the two is calculated. This value is the number of different bits at the same position in the two binary strings. This value is the hash difference between the two original social scene images.

[0065] Step 115: normalize the hash difference to obtain the hash similarity between any two original social scene images;

[0066] It is worth noting that since the hash difference, that is, the Hamming distance, has a value range of [0, 64] (for 64-bit hash), and the range of semantic similarity is [-1, 1], in order to facilitate subsequent fusion, the hash difference needs to be normalized to the interval [0, 1] and converted into similarity. Therefore, this embodiment adopts a linear normalization method. After this processing, the greater the hash difference, the smaller the hash similarity; when the hash difference is 0, the hash similarity is 1; when the hash difference is 64, the hash similarity is 0. The hash similarity obtained in this way is comparable to the semantic similarity.

[0067] Step 116: taking the average of the semantic similarity and the hash similarity between any two original social scene images to obtain a spatial scene similarity value between any two original social scene images;

[0068] It is worth noting that for any two original social scene images A and B, their semantic similarity SS and hash similarity HS are obtained respectively, and the arithmetic average of these two similarity values ​​is calculated to obtain the spatial scene similarity value between the two original social scene images;

[0069] It's worth noting that steps 111 through 116 calculate spatial scene similarity by fusing multimodal features, effectively overcoming the limitations of single similarity metric methods and significantly improving the robustness and accuracy of image similarity assessment. This method not only utilizes the SIFT algorithm to extract image keypoints and descriptors to construct high-level semantic feature vectors and calculate semantic similarity, capturing both image content and structural information, but also simultaneously applies a perceptual hashing algorithm to generate hash values ​​and calculate hash similarity, reflecting the overall perceptual consistency of the images. By taking the average of these two values ​​to obtain the final spatial scene similarity value, this method provides a more comprehensive measure of image similarity, taking into account both the semantic level of content and the overall visual perception. This fusion strategy makes the similarity calculation more resilient to common interference factors such as illumination changes, small-scale content perturbations, and slight compression. This ensures that subsequent threshold interval determination and image screening decisions based on this similarity value are based on more reliable and objective data, providing a key guarantee for the accuracy of the entire screening process.

[0070] Step 12: determining a spatial scene similarity threshold interval based on the spatial scene similarity value, wherein the spatial scene similarity threshold interval includes a first spatial scene similarity threshold interval and a second spatial scene similarity threshold interval;

[0071] Step 121: All spatial scene similarity values ​​are aggregated, and a probability density distribution curve of all spatial scene similarity values ​​is constructed using a probability density estimation method based on the aggregated results;

[0072] It's worth noting that the process for constructing the probability density distribution curve for all spatial scene similarity values ​​is to first treat all calculated spatial scene similarity values ​​(SVs) as a set. To construct the probability density distribution curve for these values, we use a non-parametric method called kernel density estimation (KDE). KDE is a method for estimating probability density functions based on data samples, without assuming that the data follows a specific distribution form. The specific steps are as follows:

[0073] This involves selecting an appropriate bandwidth and applying the KDE algorithm. The bandwidth selection is crucial to the KDE result, as it affects the degree of smoothness. A too small bandwidth can lead to overfitting, resulting in excessive peaks and valleys; a too large bandwidth can lose detailed information. Cross-validation is typically used to select an appropriate bandwidth. Furthermore, for each spatial scene similarity value, the selected bandwidth is used as a parameter to perform density estimation using a Gaussian kernel function. The entire value set is traversed, and the contribution of each point is accumulated to form a continuous probability density distribution curve.

[0074] Step 122: Identify all peak values ​​and valley values ​​on both sides of the peak values ​​according to the probability density distribution curve;

[0075] It is worth noting that after obtaining the probability density distribution curve, the next step is to identify all peaks and the valleys on both sides. This step usually involves the following process, including finding local maximum points and determining the valleys on both sides of the peak. For finding local maximum points, the local maximum point is determined by comparing the probability density values ​​of adjacent points, that is, the peak; for determining the valleys on both sides of the peak, starting from each peak, search to the left and right until a position is found that is lower than the current point and the adjacent point is also lower than the current point. These positions are the valleys on both sides of the peak.

[0076] Step 123: sort all identified peaks from largest to smallest;

[0077] It is worth noting that, assuming that three peaks P1, P2, and P3 are identified, their corresponding probability densities are PD(P1), PD(P2), and PD(P3), respectively, where PD(P1)>PD(P2)>PD(P3). Then, they are arranged from large to small according to the probability density values, and the resulting order is [P1, P2, P3].

[0078] Step 124: Select the maximum peak value based on the sorting result, and use the nearest valley value to the left of the maximum peak value as the first left boundary, and the nearest valley value to the right of the maximum peak value as the first right boundary; the first left boundary is greater than or equal to the nearest valley value to the left of the maximum peak value; and the first right boundary is less than or equal to the nearest valley value to the right of the maximum peak value.

[0079] Step 125: Combine the first left boundary and the first right boundary to obtain a first spatial scene similarity threshold interval;

[0080] It's worth noting that the maximum peak represents the most common range of spatial scene similarity, meaning that within this range, the similarity between images is very high. Therefore, selecting the nearest valley to the left of the maximum peak as the first left boundary and the nearest valley to the right as the first right boundary ensures that highly similar image pairs are included while excluding those with slightly lower but still high similarity. Setting this range helps screen for image combinations that truly exhibit a high degree of visual or semantic consistency.

[0081] Step 126: Select the minimum peak value based on the sorting result, and use the nearest valley value to the left of the minimum peak value as the second left boundary, and use the nearest valley value to the right of the minimum peak value as the second right boundary; the second left boundary is greater than or equal to the nearest valley value to the left of the minimum peak value; and the second right boundary is less than or equal to the nearest valley value to the right of the minimum peak value.

[0082] Step 127: Combine the second left boundary and the second right boundary to obtain a second spatial scene similarity threshold interval;

[0083] It is worth noting that the minimum peak corresponds to a relatively rare spatial scene similarity range, indicating similarity patterns or noise under certain specific conditions. The boundary near the minimum peak is selected to define the second spatial scene similarity threshold interval, which is mainly used to capture image similarity judgment in abnormal or edge cases.

[0084] The first spatial scene similarity threshold range is primarily used to select the most representative and typical image pairs, as they share highly consistent spatial structure or content, making them suitable for applications requiring precise matching. This range also clearly defines "highly similar" or "potentially redundant" image pairs. Image pairs falling within this range have spatial scene similarity values ​​at the peak of the data distribution, indicating that they appear most frequently in the dataset and are most likely to represent the same scene or highly repetitive viewpoints. Therefore, using this range as the primary basis for "removing redundant images" is logically clear and objectively targeted. The second spatial scene similarity threshold range is primarily used to identify less common but still meaningful similarities, such as exploring image similarity analysis under different viewpoints or slight variations. This range defines "significantly different" or "highly differentiated" image pairs. Image pairs falling within this range have similarity values ​​near the bottom of the distribution (around the minimum peak), indicating that they are "outliers" in the dataset, exhibiting a distinct similarity pattern from the majority of other image pairs. Using this range as the basis for "retaining" key differentiated images effectively ensures that the filtered image set contains a sufficient diversity of viewpoints and scene variations, avoiding information loss. By focusing on these two extreme and semantically clear intervals, the present invention establishes a concise and efficient binary decision framework for "deduplication" and "diversity preservation," avoiding the complexity of decision logic and rule conflicts caused by introducing multiple intermediate intervals.

[0085] It is worth further explaining that the maximum peak and the minimum peak are usually the most significant and stable feature points in the probability density distribution curve, and are relatively insensitive to small disturbances in the data. Selecting them as the benchmark makes the determination of the threshold interval more objective and repeatable. In contrast, the number, position and significance of the intermediate intervals may vary greatly from one data set to another, and may not even be obvious in some data sets. Relying on these unstable or variable intermediate intervals as the core screening basis will reduce the robustness and universality of the entire method. The present invention enhances the stability of the method by relying only on the most stable (maximum peak) and most marginal (minimum peak) features;

[0086] Based on the above-mentioned intermediate intervals, it is worth further explaining that in the constructed probability density distribution curve of the spatial scene similarity value, in addition to the "first spatial scene similarity threshold interval" (corresponding to the maximum peak) and the "second spatial scene similarity threshold interval" (corresponding to the minimum peak), there are usually one or more other local peaks and their corresponding intervals. These intermediate intervals, that is, the setting process of the third spatial scene similarity threshold interval provided subsequently, represent other similarity patterns in the data set except for the "most common similarity" and "most rare similarity"; therefore, the subtlety of the present invention lies in that it does not ignore the image pairs that fall outside the first and second intervals, that is, the image pairs that fall into the "intermediate interval" or the edge area. On the contrary, in the subsequent design process, a set of mechanisms is specially designed, namely, the setting process of the third spatial scene similarity threshold interval, to determine these image pairs that cannot be simply classified as "uncertain original social scene images". These "uncertain" images are mainly distributed in the vast area between the first and second intervals, that is, the range covered by those intermediate intervals. By not using these intermediate intervals directly for hard "retention / removal" decisions, but instead using them as the source of "uncertain" states, the present invention cleverly transforms the complex intermediate state problem into a specialized, quantifiable "uncertainty assessment" problem, that is, it is solved through an uncertainty factor index, which is more scientific and effective than trying to set complex, potentially conflicting retention or removal rules for each intermediate interval. Furthermore, if multiple intermediate intervals are introduced and different processing strategies are set for each interval, for example, some intermediate intervals are retained and some are removed, it will not only greatly increase the complexity of the rules, but may also lead to ambiguity in decision-making. For example, an image pair may be close to the boundaries of two different intermediate intervals at the same time, making it difficult to judge. More importantly, this refined multi-interval division has diminishing marginal benefits for the core goal of "deduplicate removal", and instead increases unnecessary computational and logical overhead. The two-interval + uncertainty processing mechanism of the present invention, while ensuring the core goal of efficient deduplication + maintaining key diversity, achieves effective management of complex situations with a relatively simple structure.

[0087] Furthermore, for steps 121 through 127, an adaptive threshold interval determination method based on probability density estimation is proposed, achieving dynamic and objective screening criteria. This method constructs a probability density distribution curve for all spatial scene similarity values ​​and leverages the statistical characteristics of the data (peaks and valleys) to define the boundary between "high similarity" and "significant difference." The valleys on either side of the maximum peak are selected as the boundaries of the first spatial scene similarity threshold interval (the "high similarity zone"), effectively capturing areas of concentrated similarity in the data. The valleys on either side of the minimum peak are selected as the boundaries of the second spatial scene similarity threshold interval (the "significant difference zone"), effectively defining areas of significant difference. This method, based on data distribution characteristics, automatically determines threshold intervals, allowing the screening criteria to be automatically adjusted based on the actual similarity distribution of the image set in each processing task. This avoids the bias and inadaptability caused by manually set thresholds, significantly enhancing the scientific nature, objectivity, and adaptability of the screening process to diverse scenarios, and ensuring the rationality of the screening results.

[0088] Step 13: judging the original social scene image based on the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval in combination with the spatial scene similarity value to obtain a retained original social scene image;

[0089] Step 131: Set the valley value of the first left boundary smaller than the first spatial scene similarity threshold interval as the third right boundary, set the valley value of the second right boundary larger than the second spatial scene similarity threshold interval as the third left boundary, and combine the third right boundary and the third left boundary to obtain a third spatial scene similarity threshold interval;

[0090] Step 132: extracting the spatial scene similarity value between each pair of original social scene images;

[0091] Step 133: Count the number of times that the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval respectively;

[0092] Step 134: based on the statistical results, if the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images all fall within a first spatial scene similarity threshold range, then the current original social scene image is eliminated;

[0093] If the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images all fall within the second spatial scene similarity threshold interval, retaining the current original social scene image;

[0094] If the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images fall into the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval respectively, then determine whether the number of times they fall into the first spatial scene similarity threshold interval is greater than the number of times they fall into the second spatial scene similarity threshold interval: if greater, then discard the current original social scene image; if less, then retain the current original social scene image; if equal, then determine that the current original social scene image is an uncertain original social scene image;

[0095] If the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval, the second spatial scene similarity threshold interval, and the third spatial scene similarity threshold interval, the current original social scene image is determined to be an uncertain original social scene image;

[0096] Step 135: Analyze the original social scene image determined to be an uncertain original social scene image, and obtain a retained original social scene image based on the analysis result;

[0097] It is worth noting that a well-organized and logically rigorous judgment system has been established for steps 131 to 135, enabling refined classification and decision-making of raw social scene images. This effectively handles complex situations that a simple "keep / remove" strategy cannot address. This system not only makes a preliminary judgment based on the number of times an image pair's similarity value falls into the first (highly similar) and second (significantly different) intervals (those falling entirely into the highly similar zone are eliminated to eliminate duplication, while those falling entirely into the significantly different zone are retained to preserve diversity), but more importantly, it introduces the concept of "uncertain raw social scene images." When the similarity status of an image pair fluctuates between highly similar and significantly different (with unequal or equal similarity) or involves borderline areas (the third interval), the system labels it as "uncertain." This design avoids arbitrary decisions made when information is insufficient or ambiguous, emphasizing prudent decision-making. By separating images that are difficult to judge directly for subsequent specialized analysis, steps 131 to 135 provide a structured framework for handling complex and boundary cases. They are the core steps for achieving refined screening and reducing the false positive rate, ensuring the integrity and rigor of the screening process.

[0098] Step 1351: Calculate the uncertainty factor index of each original social scene image determined to be an uncertain original social scene image;

[0099] Step 13511: Obtain the clarity, contrast, and noise level of each original social scene image determined to be an uncertain original social scene image, and determine a quality score for each original social scene image determined to be an uncertain original social scene image based on the clarity, contrast, and noise level;

[0100] It's worth noting that for each original social scene image judged as "uncertain," its basic image quality is first assessed. The specific process is as follows: First, the Laplacian operator is used to calculate the image's gradient variance. A larger variance indicates a clearer image. The variance is normalized to the range [0, 1] as a clarity score. Next, the standard deviation of the image's grayscale histogram is calculated. A larger standard deviation indicates higher contrast. This is also normalized to the range [0, 1] as a contrast score. Next, the local variance method is used to estimate noise intensity, calculating the average of the squared differences between each pixel and its neighborhood mean. Since lower noise intensity is preferred, the inverse of this score is taken and normalized to the range [0, 1] as a noise level score. Finally, the arithmetic mean of these three scores is taken to obtain the quality score Q for each original social scene image judged as uncertain.

[0101] Step 13512: Analyze and extract content complexity features from each original social scene image determined to be uncertain, wherein the content complexity features include the number of objects, the density of object distribution, and the degree of background interference, and determine a content complexity score for each original social scene image determined to be uncertain based on the content complexity features.

[0102] It is worth noting that the specific process of analyzing the content features of each "uncertain" image to evaluate its complexity is as follows: first, for the number of targets, by counting the number of people in the image, the more people there are, the higher the complexity, and then normalizing the number of people to the interval [0, 1]. Then, for the target distribution density, by calculating the ratio of the total area of ​​all detected target bounding boxes to the total area of ​​the image, the larger the ratio, the denser the distribution and the higher the complexity, and directly using this ratio as the density score. Then, for the degree of background interference, using edge detection algorithms such as Canny, calculate the edge density of the background area. The higher the edge density, the more complex the background, and then normalizing the edge density to the interval [0, 1] as the interference degree score; finally, taking the arithmetic mean of the above three scores to obtain the content complexity score of each original social scene image judged to be uncertain .

[0103] Step 13513: Take the average of the quality score of each original social scene image determined to be an uncertain original social scene image and the content complexity score of each original social scene image determined to be an uncertain original social scene image, and obtain the comprehensive uncertainty component of each original social scene image determined to be an uncertain original social scene image. ;

[0104] Step 13514: Based on the number of times that each original social scene image determined to be an uncertain original social scene image falls within the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval in the statistics, normalize the comprehensive uncertainty component of each original social scene image determined to be an uncertain original social scene image, and convert it through a preset nonlinear mapping function to obtain an uncertainty factor index for each original social scene image determined to be an uncertain original social scene image;

[0105] It is worth noting that the performance of the combined image in similarity statistics is When making adjustments, first count the number of times the image falls into the first spatial scene similarity threshold interval in the similarity statistics , and the number of times it falls into the second spatial scene similarity threshold interval , and define a normalization factor F. Specifically, the calculation formula for the normalization factor F is: in is a very small positive number to prevent the denominator from being zero; The value range of is close to [0,1], reflecting the degree of bias in the similarity judgment of the image;

[0106] Then the normalized combined uncertainty component Input a preset nonlinear mapping function. In this embodiment, the Sigmoid function is preferably used to enhance the discrimination: in The uncertainty factor index of the original social scene image that is determined to be uncertain; k is the steepness of the control curve; t is the threshold parameter; after this transformation, the value of the uncertainty factor index is more concentrated around 0 or 1, which is convenient for subsequent binary decision-making;

[0107] It's worth noting that steps 13511 through 13514 provide a specific, implementable technical solution for calculating the "uncertainty index," objectively quantifying the inherent uncertainty of an image and providing a key foundation for refined analysis. This solution systematically evaluates multiple intrinsic image attributes: first, a quality score is determined by deriving parameters such as clarity, contrast, and noise level, directly measuring the image's usability. Second, a content complexity score is determined by analyzing the number of objects, distribution density, and background interference, reflecting the difficulty of image recognition. The quality score is then combined with the content complexity score to produce a comprehensive uncertainty component. Finally, the image's performance in similarity statistics (the number of times it falls into each interval) is combined to generate the final uncertainty index through normalization and nonlinear mapping. This logically coherent calculation process transforms the subjective concept of "uncertainty" into an objective, comparable value. The generation of this index fully considers the internal and external factors that affect the difficulty of image judgment, provides a solid and quantitative basis for the decision-making in steps 1351 to 1353, ensures the scientificity, consistency and repeatability of the processing of "uncertain" images, and is the ultimate technical closed loop for achieving refined and intelligent decision-making in the entire intelligent screening system.

[0108] Step 1352: Setting a determination threshold for the uncertainty factor index;

[0109] It's worth noting that the specific process for setting a threshold for the uncertainty index is as follows. First, a representative sample set is selected from the known dataset as a baseline sample set. This sample set should include both explicitly retained and explicitly excluded images, i.e., images directly determined by the first and second spatial scene similarity thresholds. These samples not only provide a reference standard for practical operations but also provide basic data support for subsequent threshold setting. For each image in the selected baseline sample set, its uncertainty index is calculated according to the previously defined method. This step ensures that all baseline samples have a corresponding uncertainty index value, providing a quantitative basis for subsequent analysis. Next, the distribution of the uncertainty index in the two groups of samples is analyzed. Specifically, a histogram or density plot of the uncertainty index of the two groups of samples is plotted to observe their distribution patterns and degree of overlap. The statistical characteristics of the uncertainty index of each group, such as the mean, median, and standard deviation, are calculated and recorded. Based on the results of this analysis, a threshold can be set. A common strategy is to use the mean or median. If there is a clear separation in the distribution of the uncertainty index of the retained and excluded groups, a value between the means or medians of the two groups can be selected as the threshold.

[0110] Step 1353: Based on the determination threshold, if the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is greater than the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is less than the determination threshold, then the original social scene image currently determined to be an uncertain original social scene image is eliminated;

[0111] If the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is greater than the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is greater than or equal to the determination threshold, then retaining the original social scene image currently determined to be an uncertain original social scene image;

[0112] If the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is less than the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is less than the determination threshold, then the original social scene image currently determined to be an uncertain original social scene image is eliminated;

[0113] If the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is less than the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is greater than or equal to the determination threshold, then retaining the original social scene image currently determined to be an uncertain original social scene image;

[0114] If the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is equal to the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is less than the determination threshold, then the original social scene image currently determined to be an uncertain original social scene image is eliminated;

[0115] If the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is equal to the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is greater than or equal to the determination threshold, then retaining the original social scene image currently determined to be an uncertain original social scene image;

[0116] It is worth noting that steps 1351 through 1353 propose a refined analysis and decision-making mechanism based on an "uncertainty index," successfully resolving the processing challenges of these fuzzy images and achieving a clear "retention / rejection" decision. This mechanism calculates an uncertainty index for each uncertain image and sets a threshold. This combines the image's quality and complexity attributes (quantified by the index) with its relative position within the population (the number of times it falls into different intervals) for a comprehensive assessment. Whether there are many highly similar pairs, many significantly different pairs, or both, the final decision on retention or rejection is based on whether the uncertainty index exceeds the threshold. This dual judgment criterion (external relationship + internal attributes) makes the decision more comprehensive and reliable: even if the similarity statistics are biased in one direction, an image with a high uncertainty index (e.g., poor image quality or complex content) may still be retained, avoiding false rejections due to a single statistical result. Conversely, a low index is more likely to be rejected. Therefore, this effectively utilizes the inherent characteristics of "uncertain" images and transforms them from the "blind spot" of the screening process into a quantifiable and decision-making object, significantly improving the intelligence and accuracy of the overall screening strategy.

[0117] Step 14: Aggregate the retained original social scene images to obtain the optimal social scene images;

[0118] It is worth noting that steps 11 through 14 introduce a systematic automated screening process, completely replacing the traditional, inefficient and misleading image screening method that relies on human visual inspection. This process achieves automated and intelligent conversion from a massive raw image collection to a high-quality, prioritized set of images. This process first calculates the spatial scene similarity between any two images, providing a data foundation for objectively quantifying the relationship between them. It then automatically determines the first (highly similar) and second (significantly different) spatial scene similarity thresholds based on these similarity values, eliminating the subjectivity of fixed thresholds and enabling the screening criteria to adapt to the actual data distribution. Finally, the original images are judged based on the threshold intervals and retained, forming a prioritized set of social scene images. This series of operations not only significantly improves the efficiency and consistency of image screening, effectively removes a large number of redundant images, and preserves representative key perspectives, but also provides cleaner and more diverse input data for subsequent object detection and identity recognition. This fundamentally addresses the problem of reduced recognition accuracy due to improper screening and is a critical prerequisite for improving overall system performance.

[0119] Step 2: Input the pre-processed preferred social scene image into a pre-trained convolutional neural network model, detect all potential targets in the preferred social scene image, identify each detected target, and assign a bounding box to each detected target, the bounding box including its position and size;

[0120] It is worth noting that the training process of the convolutional neural network model is as follows: First, a training set is obtained. The training set includes a large number of social scene images. These images come from multiple channels, such as public datasets such as COCO and Pascal VOC. They contain rich annotation information, covering various complex scenes and diverse target categories, as well as custom collected data. These data are images taken by fixed monitoring equipment or mobile devices (such as smartphones) in different social situations. These images need to be manually annotated to mark all potential targets and their bounding boxes; each image contains at least one target, and each target has detailed annotation information, including location coordinates, i.e., the upper left corner x of the bounding box, y coordinates, width, height and target category labels, such as people, animals, objects, etc.; then set the training labels, where the training labels mainly include two types of information: 1. Bounding box coordinates: For each target, provide the coordinates of the upper left corner of its bounding box as well as the width and height to locate the position and size of the target; 2. Category label: assign a category label to each target to indicate which category the target belongs to, such as "people", "cars", etc.; before using the above training set and training labels to obtain the trained convolutional neural network model, it is also necessary to select evaluation indicators, where the main indicators for evaluating model performance include: average precision ... Value, which is one of the most commonly used evaluation indicators in target detection tasks, calculates the average precision of each category, and then takes the average of all categories, as well as recall and precision. The recall rate measures the proportion of targets correctly identified by the model to all actual targets, while the precision rate measures the actual correct proportion of targets predicted by the model as positive. It also includes the F1 score, which is the harmonic mean of precision and recall, and is suitable for imbalanced classification problems. In addition, the cross entropy loss function is also needed to measure the accuracy of the model's prediction of the target category. Through the above steps, a trained convolutional neural network model can be obtained;

[0121] It is worth further explaining that when using the trained convolutional neural network model for target detection, the preferred social scene image is first input, that is, the pre-processed preferred social scene image is input into the trained convolutional neural network model. At this time, the model first extracts the appearance feature vector of the image through a series of convolutional layers. , then use the region proposal network to generate candidate regions, that is, regions that may contain targets, and then further refine these candidate regions, and finally output the specific location and category of each detected target; Specifically, the input image is first processed through convolutional layers, pooling layers, etc., to obtain a multi-level appearance feature vector , based on the appearance feature vector The region proposal network generates a series of candidate boxes to preliminarily determine the areas where the target may exist, and then classifies the features within each candidate box. At the same time, the position and size of the candidate box are adjusted to more accurately match the true bounding box of the target. For each identified target, the model assigns it a bounding box. The information of the bounding box includes the coordinates of the upper left corner of the bounding box, that is, the position, and the width and height of the bounding box, that is, the size.

[0122] Step 3: Based on the position and size of each detection target, the relative position, distance, and angle between each detection target are calculated, and a spatial relationship graph between each detection target is constructed. The behavioral characteristics of each detection target are extracted based on the spatial relationship graph. The behavioral pattern of each detection target is analyzed based on the behavioral characteristics of each detection target, and the feature vector of each detection target is extracted. The feature vector, behavioral pattern, and spatial relationship graph are input into the pre-trained deep learning model to confirm the identity of each detection target.

[0123] It is worth noting that the process of constructing the spatial relationship diagram between the detection targets is as follows: first, in step 2, a bounding box is assigned to each detection target in the preferred social scene image, and its position, i.e., the center coordinates of the bounding box, is obtained. and size, i.e. width ,high Based on this information, a spatial relationship graph between each detection target is constructed. Specifically, the node definition is first performed, that is, each node in the spatial relationship graph represents a detection target, and the attributes of the node include the center coordinates of its bounding box ,size and the appearance feature vector extracted from the convolutional neural network model ; Then, edge definition is performed, that is, there is an edge between any two nodes in the graph, that is, detection targets i and j. The weight of the edge is determined by the relative position, distance and angle between them. The relative position can be obtained by calculating the horizontal and vertical offsets of target i relative to target j. At the same time, the Euclidean distance algorithm can be used to obtain the spatial distance between the center points of the two targets. The angle between the vector pointing from target j to target i and the horizontal axis is calculated to obtain the angle. The weight of the edge can be a multidimensional vector, or it can be fused into a scalar through a simple function, such as the inverse of the normalized distance. However, this embodiment adopts the form of a multidimensional vector to retain richer spatial information. Finally, the spatial relationship graph A graph is constructed, where V is a set of nodes and E is a set of edges. This graph fully describes the spatial topological structure between all detected targets in the image. The spatial relationship graph is then combined to extract the behavioral features of each detected target i. The behavioral features include not only its own motion state, but more importantly, its interaction with other targets. Specific features include: local neighborhood features, which calculate the average distance, average angular deviation, and nearest neighbor distance between target i and all its neighboring nodes, reflecting its position and interaction intensity in the local group; graph structural features, which calculate the degree centrality and closeness centrality of target i in the graph G, quantifying its "centrality" or "isolation" in the social group.

[0124] Then, based on the extracted behavioral features, the behavioral pattern of each detected target is analyzed. For example, if a target i has a small average distance, a small nearest neighbor distance, and a high degree centrality, its behavioral pattern can be analyzed as "at the center of the group" or "interacting with multiple people"; if a target j has a large average distance and a low degree centrality, its behavioral pattern can be analyzed as "at the edge" or "relatively isolated"; if the distance between a pair of targets k and l is very small and the angle is close to o or π, their behavioral pattern can be analyzed as "standing side by side" or "talking face to face". These behavioral patterns are qualitative or semi-quantitative descriptions that provide contextual information for subsequent identity confirmation;

[0125] Then extract the feature vector and fuse the following information of each detection target i into a high-dimensional feature vector , including the appearance feature vector , behavioral feature vector Including local neighborhood features, graph structure features, and encoding of behavioral patterns , which uses one-hot encoding to represent behavioral patterns such as "center", "edge", and "pairwise interaction"; the final feature vector , where the semicolon indicates vector concatenation, which simultaneously encodes the target’s appearance, behavior, and social context, i.e., the encoding of the behavior pattern;

[0126] It is worth further explaining that the training process of a deep learning model begins with obtaining a training set, which contains a large number of labeled social scene image sequences or single-frame images. Each image undergoes the same preprocessing and target detection process as described above. Crucially, each detected target in each image has a real identity label, which is usually obtained through manual annotation or other methods such as RFID in a controlled environment. Next, a training label is set, which is the real identity of each detected target. Before using the above training set and training labels to obtain a trained deep learning model, it is necessary to select model evaluation metrics, including identity recognition accuracy, which is the ratio of correctly recognized identities to the total number of detected targets; Top-K accuracy, which is the proportion of real identities that appear in the top K most likely identities predicted by the model, where K is usually 1 or 5; and a confusion matrix, which analyzes the confusion between different identities. Furthermore, the cross-entropy loss function is used as the main loss function. For each detected target i, the model outputs a probability distribution, representing the probability of it belonging to each identity category. Through the above steps, a trained deep learning model is obtained.

[0127] It is worth further explaining that when confirming the identity of each detected target, the extracted feature vector, behavior pattern, and spatial relationship graph of the entire image of each detected target i are input into the trained deep learning model. The model receives the feature vector, behavior pattern, and spatial relationship graph of the entire image as input. At this time, the deep learning model can directly process the graph structure and use graph convolution operations to aggregate neighbor information, so as to incorporate its social context when updating each node, that is, the representation of the detected target. Then, after multiple layers of calculation, the model outputs a probability distribution about the identity category, and finally selects the identity category with the highest probability as the final identity of the detected target i.

[0128] It should be noted that the terms used in the present invention are only for describing specific embodiments and are not intended to limit the scope of this application. As shown in the present specification, unless the context clearly indicates an exception, the words "one", "a", "a kind of" and / or "the" do not specifically refer to the singular and may also include the plural. The terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method or device comprising a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of further restrictions, the elements defined by the sentence "comprise a..." do not exclude the presence of other identical elements in the process, method or device comprising the elements.

[0129] It should also be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", etc. should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be a communication between the internal parts of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A multi-objective spatial relationship driven human recognition method in social scene images, characterized by: The following steps are involved: Step 1: obtaining and screening original social scene images to obtain preferred social scene images, and preprocessing the preferred social scene images; Step 2: Input the pre-processed preferred social scene image into a pre-trained convolutional neural network model, detect all potential targets in the preferred social scene image, identify each detected target, and assign a bounding box to each detected target, the bounding box including its position and size; Step 3. Based on the position and size of each detection target, calculate the relative position, distance and angle between each detection target, construct a spatial relationship diagram between each detection target, extract the behavioral characteristics of each detection target in combination with the spatial relationship diagram, analyze the behavioral pattern of each detection target based on the behavioral characteristics of each detection target, and extract the feature vector of each detection target. Input the feature vector, behavioral pattern and spatial relationship diagram into the pre-trained deep learning model to confirm the identity of each detection target.

2. The multi-objective spatial relationship driven human recognition method in social scene images according to claim 1, characterized in that: The obtaining and screening the original social scene images to obtain the preferred social scene images includes: Step 11: Calculate the spatial scene similarity value between any two original social scene images; Step 12: determining a spatial scene similarity threshold interval based on the spatial scene similarity value, wherein the spatial scene similarity threshold interval includes a first spatial scene similarity threshold interval and a second spatial scene similarity threshold interval; Step 13: judging the original social scene image based on the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval in combination with the spatial scene similarity value to obtain a retained original social scene image; Step 14: Aggregate the retained original social scene images to obtain the preferred social scene images.

3. The multi-objective spatial relationship driven human recognition method in social scene images according to claim 2, characterized in that: The calculating of the spatial scene similarity value between any two original social scene images includes: Step 111: extract key points and descriptors of each original social scene image based on the SIFT algorithm; Step 112: construct a high-level semantic feature vector for each original social scene image using key points and descriptors; Step 113: Calculate the semantic similarity between any two original social scene images based on the extracted high-level semantic feature vectors; Step 114: Apply the perceptual hash algorithm to generate a perceptual hash value for each image, and calculate the hash difference between any two original social scene images; Step 115: normalize the hash difference to obtain the hash similarity between any two original social scene images; Step 116: Take the average of the semantic similarity and the hash similarity between any two original social scene images to obtain the spatial scene similarity value between any two original social scene images.

4. The multi-objective spatial relationship driven human recognition method in social scene images according to claim 3, characterized in that: The determining of the spatial scene similarity threshold interval based on the spatial scene similarity value includes: Step 121: All spatial scene similarity values ​​are aggregated, and a probability density distribution curve of all spatial scene similarity values ​​is constructed using a probability density estimation method based on the aggregated results; Step 122: Identify all peak values ​​and valley values ​​on both sides of the peak values ​​according to the probability density distribution curve; Step 123: sort all identified peaks from largest to smallest; Step 124: Select the maximum peak value based on the sorting result, and use the nearest valley value to the left of the maximum peak value as the first left boundary, and the nearest valley value to the right of the maximum peak value as the first right boundary; the first left boundary is greater than or equal to the nearest valley value to the left of the maximum peak value; and the first right boundary is less than or equal to the nearest valley value to the right of the maximum peak value. Step 125: Combine the first left boundary and the first right boundary to obtain a first spatial scene similarity threshold interval; Step 126: Select the minimum peak value based on the sorting result, and use the nearest valley value to the left of the minimum peak value as the second left boundary, and use the nearest valley value to the right of the minimum peak value as the second right boundary; the second left boundary is greater than or equal to the nearest valley value to the left of the minimum peak value; and the second right boundary is less than or equal to the nearest valley value to the right of the minimum peak value. Step 127: Combine the second left boundary and the second right boundary to obtain a second spatial scene similarity threshold interval.

5. The multi-objective spatial relationship driven human recognition method in social scene images according to claim 4, characterized in that: The determining the original social scene image based on the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval in combination with the spatial scene similarity value to obtain the retained original social scene image includes: Step 131: Set the valley value of the first left boundary smaller than the first spatial scene similarity threshold interval as the third right boundary, set the valley value of the second right boundary larger than the second spatial scene similarity threshold interval as the third left boundary, and combine the third right boundary and the third left boundary to obtain a third spatial scene similarity threshold interval; Step 132: extracting the spatial scene similarity value between each pair of original social scene images; Step 133: Count the number of times that the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval respectively; Step 134: based on the statistical results, if the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images all fall within a first spatial scene similarity threshold range, then the current original social scene image is eliminated; If the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images all fall within the second spatial scene similarity threshold interval, retaining the current original social scene image; If the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images fall into the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval respectively, then determine whether the number of times they fall into the first spatial scene similarity threshold interval is greater than the number of times they fall into the second spatial scene similarity threshold interval: if greater, then discard the current original social scene image; if less, then retain the current original social scene image; if equal, then determine that the current original social scene image is an uncertain original social scene image; If the spatial scene similarity values ​​between the current original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval, the second spatial scene similarity threshold interval, and the third spatial scene similarity threshold interval, the current original social scene image is determined to be an uncertain original social scene image; Step 135 : Analyze the original social scene image determined to be an uncertain original social scene image, and obtain a retained original social scene image based on the analysis result.

6. The multi-objective spatial relationship driven human recognition method in social scene images according to claim 5, characterized in that: The analyzing the current original social scene image determined to be an uncertain original social scene image and obtaining a retained original social scene image based on the analysis result includes: Step 1351: Calculate the uncertainty factor index of each original social scene image determined to be an uncertain original social scene image; Step 1352: Setting a determination threshold for the uncertainty factor index; Step 1353: Based on the determination threshold, if the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is greater than the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is less than the determination threshold, then the original social scene image currently determined to be an uncertain original social scene image is eliminated; If the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is greater than the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is greater than or equal to the determination threshold, then retaining the original social scene image currently determined to be an uncertain original social scene image; If the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is less than the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is less than the determination threshold, then the original social scene image currently determined to be an uncertain original social scene image is eliminated; If the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is less than the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is greater than or equal to the determination threshold, then retaining the original social scene image currently determined to be an uncertain original social scene image; If the number of times the spatial scene similarity values ​​between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images fall within the first spatial scene similarity threshold interval is equal to the number of times they fall within the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is less than the determination threshold, then the original social scene image currently determined to be an uncertain original social scene image is eliminated; If the number of times the spatial scene similarity value between the original social scene image currently determined to be an uncertain original social scene image and the remaining original social scene images falls into the first spatial scene similarity threshold interval is equal to the number of times it falls into the second spatial scene similarity threshold interval, and the uncertainty factor index of the original social scene image currently determined to be an uncertain original social scene image is greater than or equal to the determination threshold, then the original social scene image currently determined to be an uncertain original social scene image is retained.

7. The multi-objective spatial relationship driven human recognition method in social scene images according to claim 6, characterized in that: The calculating of the uncertainty factor index of each original social scene image determined to be an uncertain original social scene image includes: Step 13511: Obtain the clarity, contrast, and noise level of each original social scene image determined to be an uncertain original social scene image, and determine a quality score for each original social scene image determined to be an uncertain original social scene image based on the clarity, contrast, and noise level; Step 13512: Analyze and extract content complexity features from each original social scene image determined to be uncertain, wherein the content complexity features include the number of objects, the density of object distribution, and the degree of background interference, and determine a content complexity score for each original social scene image determined to be uncertain based on the content complexity features. Step 13513: averaging the quality score of each original social scene image determined to be an uncertain original social scene image and the content complexity score of each original social scene image determined to be uncertain, to obtain a comprehensive uncertainty component of each original social scene image determined to be an uncertain original social scene image. Step 13514: Based on the number of times that each original social scene image determined to be an uncertain original social scene image falls into the first spatial scene similarity threshold interval and the second spatial scene similarity threshold interval in the statistics, the comprehensive uncertainty component of each original social scene image determined to be an uncertain original social scene image is normalized, and converted through a preset nonlinear mapping function to obtain the uncertainty factor index of each original social scene image determined to be an uncertain original social scene image.

Images