AI-based multi-source information comparison black and grey production detection method

Abstract

The invention discloses an AI-based multi-source information comparison black and grey production detection method, which comprises the following steps: S1, collecting and preprocessing multi-source heterogeneous information data to generate a multi-source information data set; s2, performing semantic coding on the multi-source intelligence data set to obtain an intelligence semantic vector set with unified dimensions; s3, constructing a locality sensitive hash index structure, and generating a preliminary candidate retrieval space; s4, performing screening by using a random search algorithm to generate a candidate comparison sample set; s5, executing a fine comparison operation, and outputting a comprehensive similarity score; s6, black and grey product behavior judgment is carried out based on the comprehensive similarity score; and S7, the high-risk sample is used for updating the Hash bucket index structure. According to the method, semantic vector coding, locality sensitive hashing and random search technologies are fused, so that efficient comparison of multi-source heterogeneous intelligence and intelligent detection of black and grey product behaviors are realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and information security technology, and in particular to an AI-based multi-source intelligence comparison method for detecting black and gray industries. Background Art

[0002] In today's information society, the increasing sophistication of black and gray market activities has become a significant threat to network security, financial risk management, and public opinion governance. Black and gray market behaviors are characterized by high concealment, frequent cross-platform migration, and highly variable behavioral patterns. These behaviors pose significant challenges to traditional detection methods in terms of accuracy, real-time performance, and scalability. Existing black and gray market detection technologies mostly rely on rule engines, keyword matching, or static graph analysis, primarily identifying suspected samples by setting fixed risk labels and feature templates. While these methods initially demonstrate some effectiveness against known samples, they often struggle with new variants, ambiguous expressions, and combined fraudulent behaviors, leading to both false positives and false negatives. Furthermore, the integration of multi-source, heterogeneous intelligence is challenging. Data across different platforms suffers from structural inconsistencies, inconsistent semantics, and time-series asynchrony, making cross-domain behavior difficult to track and comprehensively analyze.

[0003] With the development of artificial intelligence, some research has attempted to introduce deep learning models to encode intelligence text and behavioral data and perform similarity assessments. However, these approaches suffer from high computational costs in large-scale data scenarios and lack effective structured indexing mechanisms, making it difficult to rapidly screen potential black and gray market samples. Furthermore, most existing methods employ fixed models and static scoring strategies for sample comparison, lacking joint modeling of semantic structures, behavioral pathways, and device maps between samples. This results in limited ability to identify potential connections between complex samples. Furthermore, anomaly assessments often rely on a single threshold, failing to incorporate multidimensional similarity fusion, dynamic risk differentiation, and confidence mechanisms, making it difficult to support the real-time, accurate, and explainable high-risk identification requirements.

[0004] Therefore, how to provide an AI-based multi-source intelligence comparison and black and gray industry detection method is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0005] One purpose of the present invention is to propose an AI-based multi-source intelligence comparison method for black and gray market detection. The present invention fully integrates semantic embedding coding, local sensitive hash indexing and random search strategy, and describes in detail the technical process of realizing efficient screening and multi-dimensional similarity comparison in a multi-source heterogeneous intelligence data environment. It has the advantages of high processing efficiency, high similarity recognition accuracy and timely risk warning response.

[0006] According to an embodiment of the present invention, a method for detecting black and gray industries by comparing multi-source intelligence based on AI includes the following steps:

[0007] S1. Collect multi-source heterogeneous intelligence data and pre-process the multi-source heterogeneous intelligence data to generate a multi-source intelligence dataset;

[0008] S2. Semantically encode multi-source intelligence datasets and use a unified embedding model to vectorize text information, behavior sequences, and device features to obtain a unified-dimensional intelligence semantic vector set.

[0009] S3. Based on the intelligence semantic vector set, a locality-sensitive hash index structure is constructed. A hash function family that meets the similarity measurement requirements is selected to perform a hash mapping operation on the vector set. Similar semantic vectors are assigned to the same or adjacent hash buckets to generate a preliminary candidate retrieval space.

[0010] S4. In the preliminary candidate search space, a random search algorithm is used to screen candidate intelligence samples, random perturbation rules and jump probabilities are set, and candidate intelligence vectors are randomly selected from multiple related hash buckets to generate a candidate comparison sample set;

[0011] S5. Perform a detailed comparison operation on the candidate comparison sample set, calculate the multi-dimensional similarity between the candidate sample and the target sample in semantic content, behavior path sequence, and device entity relationship diagram, and output a comprehensive similarity score based on the set weighted scoring function;

[0012] S6: Determine black and gray market behavior based on the comprehensive similarity score. When the score exceeds the set risk threshold, the corresponding sample is marked as high-risk intelligence and an early warning mechanism is triggered.

[0013] S7. Use intelligence samples judged as high-risk to update the hash bucket index structure, add intelligence semantic vectors to the hash index, and rebuild the local hash map.

[0014] Optionally, the multi-source heterogeneous intelligence data specifically includes social platform text content, network communication behavior logs, e-commerce transaction records, device access trajectory information and account identity association data.

[0015] Optionally, the preprocessing of multi-source heterogeneous intelligence data specifically includes format unification, field standardization, noise filtering, time alignment and missing value filling operations.

[0016] Optionally, the S2 specifically includes:

[0017] S21. Divide the generated standardized multi-source intelligence dataset into three subsets: text information, behavior sequence, and device features, and perform vector embedding processing on each subset.

[0018] S22, use the semantic encoding model to perform context semantic modeling on the text information. The original text sequence is T = {t1, t2, ..., t n}, where ti Represents the i-th term, and the semantic encoding model outputs the text semantic vector set V T ={v1,v2,...,v n}, where v i Represents the term t i The corresponding semantic embedding vector, i∈[1,n], n represents the number of terms in the original text sequence;

[0019] S23, the behavior sequence is vectorized using the time series modeling method, assuming that the behavior sequence is A = {a1, a2, ..., a m}, where a j Represents the jth behavior node, generating a behavior sequence embedding vector set V A ={u1,u2,…,u m}, where u j is the embedding vector of the corresponding behavior node, j∈[1,m], m represents the total number of behavior nodes in each behavior sequence;

[0020] S24, the device features are embedded using a static attribute encoding method, and the device feature set is D = {d1, d2, ..., d k}, where d k Represents the kth device attribute and generates the device feature vector set V D ={w1,w2,...,w k}, where w k Indicates device attribute d k The embedding vector of , k represents the total dimension or number of attributes of the device features;

[0021] S25, the generated text semantic vector set V T , behavior sequence embedding vector set V A and the device feature vector set V D Splicing and fusion are performed to construct an intelligence semantic vector E of unified dimension. The intelligence semantic vector constitutes an intelligence semantic vector set, which serves as input data for constructing a local sensitive hash index structure.

[0022] Optionally, the S3 specifically includes:

[0023] S31. Obtain the generated intelligence semantic vector set. The semantic vector of each intelligence sample is E i , where i∈[1,N], N is the total number of samples;

[0024] S32, based on the vector dimension and sample size N, adaptively set the hash mapping dimension L and the number of hash clusters C, and construct a multi-cluster hash function family H, each cluster contains L c A hash function;

[0025] S33. For each semantic vector E i Apply the hash function clusters under all clusters in turn to generate the hash signature matrix H(E i );

[0026] S34, according to the hash signature matrix, the intelligence semantic vector E i Mapped to hash buckets under multiple clusters, the lth hash function under the cth cluster will be mapped to bucket Forming multi-cluster mapping relationships;

[0027] S35. Build a hash bucket weight table for all hash buckets and record the sample density ρ of each bucket j , average similarity center vector And set the matching priority score ω for each bucket j ;

[0028] S36, introduce a conflict mediation mechanism. When there are high-overlap samples between multiple high-density buckets, execute the hash conflict detection algorithm and calculate the collision based on the vector distance. Redistribute boundary samples to the optimal bucket;

[0029] S37. For each sample E i Build a bucket ownership list And add bucket weight ω j , generate a priority index path;

[0030] S38. List all buckets Summarize and construct a multi-path, multi-cluster, weighted hash index structure, and use this structure to generate a preliminary candidate search space, which includes all hash buckets, sample distributions, bucket weights, and index paths;

[0031] S39. Introducing an incremental clustering learning module to perform feature center update and local bucket reconstruction on newly added semantic vectors, thus achieving dynamic self-learning of the preliminary candidate retrieval space;

[0032] S310, set the index refresh cycle, when the cumulative number of new samples reaches the threshold T u When , the full hash function reconstruction and bucket structure reconstruction are triggered to maintain the stability and search performance of the preliminary candidate retrieval space.

[0033] Optionally, the S4 specifically includes:

[0034] S41. From the generated preliminary candidate retrieval space, target intelligence semantic vector E q , retrieve the attributed hash bucket set And mark the priority weight ωq of each bucket l ;

[0035] S42, build a dynamic priority sampling pool, according to the bucket weight ωq l and sample density ρq l Sort each bucket into different levels and set high priority pool, normal pool and low priority pool;

[0036] S43. Set the multipath perturbation search parameters, including the local perturbation coefficient α, the cross-bucket jump threshold γ, and the maximum path width W. max and adaptive sampling threshold τ;

[0037] S44. For each hash bucket in the high-priority pool, for each intelligence semantic vector sample in the bucket, add a random perturbation amount generated according to a zero-mean unit variance normal distribution to each component of the sample according to the local perturbation parameter to generate a corresponding perturbation sample vector.

[0038] S45. According to the current sample distribution, the probability distribution of samples in each hash bucket is counted, and the information entropy of the probability distribution is calculated. The information entropy is the negative value of the sum of the products of all sample probabilities and logarithms.

[0039] S46. When the entropy of the sampling path distribution is lower than the set threshold, or the similarity of samples between buckets is higher than the threshold, the cross-bucket jump mechanism is triggered, and a new path is selected in the common pool or the low-priority pool with probability γ to enrich the diversity of the candidate set;

[0040] S47. Introduce an interactive memory mechanism during the sampling process to record the sample index, perturbation parameters, and sampling bucket number of each round of sampling to avoid repeated sampling and path loops;

[0041] S48. After each round of multipath perturbation and cross-bucket jump is completed, sampling distribution statistics are performed on the sample set, and the weight of the dynamic priority sampling pool is updated;

[0042] S49, the candidate comparison sample set generated by the final sampling and metadata structure as input collection;

[0043] S410: For samples marked as high confidence in the sampling results, the hash bucket priority is automatically increased based on the metadata and sampling history, and the relevant sampling path, perturbation parameters and distribution entropy information are synchronously fed back to the preliminary candidate retrieval space.

[0044] Optionally, the S5 specifically includes:

[0045] S51, obtaining the generated candidate comparison sample set and intelligence semantic vector;

[0046] S52. For each candidate sample, extract the corresponding text information, behavior path sequence, and device features, and compare them with the text information, behavior path sequence, and device features of the target intelligence sample, respectively, to calculate semantic similarity, behavior path sequence similarity, and device entity relationship graph structural similarity. The semantic similarity is calculated using cosine similarity, the behavior path sequence similarity is calculated using a dynamic time warping algorithm, and the device entity relationship graph structural similarity is calculated using an optimal subgraph matching algorithm.

[0047] S53. Based on the obtained comparison results, perform multi-scale feature fusion processing on each candidate sample, and fuse different time series granularities, behavior windows and multi-level device entity features into a comprehensive feature representation;

[0048] S54, for each candidate sample comprehensive feature representation, combined with historical label information, feature variability and category consistency, using adaptive weight adjustment mechanism to generate sample adaptive weight parameters

[0049] S55, based on the obtained adaptive weight parameters, a weighted scoring function Score (E q ,E i ) Calculate the comprehensive similarity score, and count the historical comparison scores to generate the comprehensive similarity confidence interval

[0050]

[0051] Among them, E q Represents the semantic vector of the target intelligence sample, E i is the semantic vector of the candidate intelligence sample, Sim sem (E q ,E i ) represents E q With E i The semantic similarity between act (A q ,A i ) represents the behavior path similarity between the target sample and the candidate sample, Sim dev (G q ,G i ) represents the graph structure similarity between the target and the candidate sample in the device entity structure, A q Represents the behavioral path sequence of the target intelligence sample, A i represents the behavioral path sequence of candidate samples, G q The device entity relationship diagram representing the target sample, G i Device entity relationship diagram representing candidate samples;

[0052] S56. Perform an anomaly analysis on the comprehensive similarity scores and confidence intervals of all candidate samples, use the isolation forest algorithm to detect abnormal candidate samples whose comprehensive similarity significantly deviates from the overall distribution, and mark abnormal samples with abnormal scores;

[0053] S57. For the obtained candidate samples, the adaptive weighted comprehensive score, anomaly score, confidence interval information and historical labels are combined to output the final high-risk sample list and the risk level probability of each sample;

[0054] S58. Generate a feature explanatory analysis report for the top-ranked high-risk samples, and output the main judgment basis such as semantic similarity contribution, abnormal behavior path, and key equipment entity association.

[0055] Optionally, the S6 specifically includes:

[0056] S61, receiving the output comprehensive similarity scores and ranking results of the candidate samples, and establishing a corresponding judgment record entry for each candidate sample;

[0057] S62. Set multiple preset threshold levels required for risk determination, including a high risk threshold, a medium risk threshold, and a low risk reference threshold, and load the corresponding risk determination rule table;

[0058] S63. Perform threshold comparison on the comprehensive similarity score of each candidate sample, and classify the sample into high-risk, medium-risk, low-risk, or non-risk categories based on the score interval matching judgment rules;

[0059] S64. For samples classified as high-risk, record the reason for the determination, the associated target sample number, the trigger threshold type, and the risk characteristics, and mark them as high-risk intelligence samples;

[0060] S65. Based on the determination result of the high-risk intelligence sample, generate warning data content including sample identification information, similarity comparison source information, key feature summary and corresponding risk level;

[0061] S66. Send the generated warning data content to the preset response channel, and trigger manual review, behavior isolation, and historical behavior tracing in the specified information processing process.

[0062] The beneficial effects of the present invention are:

[0063] The present invention achieves efficient indexing and rapid retrieval of multi-source heterogeneous intelligence data by constructing a unified intelligence semantic vector space and introducing a multi-path candidate screening mechanism that combines local sensitive hashing with a random search algorithm, significantly improving the efficiency of initial screening of suspected black and gray market samples. In the semantic comparison stage, the present invention adopts a multi-dimensional similarity fusion method to jointly model multiple features such as semantic content, behavioral path sequence, and device entity relationship diagram, and dynamically adjusts the weight parameters of various similarities through an adaptive weighting strategy, effectively enhancing the system's recognition robustness under complex semantic variations and cross-platform behavioral interference. By constructing a confidence interval and anomaly analysis mechanism, the system can not only accurately identify high-risk samples, but also has a strong false alarm suppression capability. The final output warning information forms a closed loop based on feature interpretation and behavior tracking, which improves the interpretability of the model judgment and the operability of the results.

[0064] Compared to existing black and gray market detection technologies that rely on fixed rules or static models, this proposed method offers greater semantic matching flexibility and structural adaptability, dynamically adapting to sample heterogeneity and changing attack patterns. While maintaining recognition accuracy, this overall solution significantly reduces comparison latency and possesses strong engineering capabilities. It is applicable to a variety of complex scenarios, including public security anti-fraud, e-commerce risk management, and financial anti-money laundering, possessing significant practical value and promotional significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0066] Figure 1 This is a flowchart of the AI-based multi-source intelligence comparison method for detecting illegal and gray industries proposed by the present invention;

[0067] Figure 2 This is a flowchart of the candidate sample screening based on local sensitive hashing and random search for the AI-based multi-source intelligence comparison black and gray market detection method proposed by the present invention. DETAILED DESCRIPTION

[0068] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0069] refer to Figure 1 and Figure 2 A multi-source intelligence comparison method for detecting black and gray industries based on AI includes the following steps:

[0070] S1. Collect multi-source heterogeneous intelligence data and pre-process the multi-source heterogeneous intelligence data to generate a multi-source intelligence dataset;

[0071] S2. Semantically encode multi-source intelligence datasets and use a unified embedding model to vectorize text information, behavior sequences, and device features to obtain a unified-dimensional intelligence semantic vector set.

[0072] S3. Based on the intelligence semantic vector set, a locality-sensitive hash index structure is constructed. A hash function family that meets the similarity measurement requirements is selected to perform a hash mapping operation on the vector set. Similar semantic vectors are assigned to the same or adjacent hash buckets to generate a preliminary candidate retrieval space.

[0073] S4. In the preliminary candidate search space, a random search algorithm is used to screen candidate intelligence samples, random perturbation rules and jump probabilities are set, and candidate intelligence vectors are randomly selected from multiple related hash buckets to generate a candidate comparison sample set;

[0074] S5. Perform a detailed comparison operation on the candidate comparison sample set, calculate the multi-dimensional similarity between the candidate sample and the target sample in semantic content, behavior path sequence, and device entity relationship diagram, and output a comprehensive similarity score based on the set weighted scoring function;

[0075] S6: Determine black and gray market behavior based on the comprehensive similarity score. When the score exceeds the set risk threshold, the corresponding sample is marked as high-risk intelligence and an early warning mechanism is triggered.

[0076] S7. Use intelligence samples judged as high-risk to update the hash bucket index structure, add intelligence semantic vectors to the hash index, and rebuild the local hash map.

[0077] In this embodiment, the multi-source heterogeneous intelligence data specifically includes social platform text content, network communication behavior logs, e-commerce transaction records, device access trajectory information and account identity association data.

[0078] In this embodiment, the preprocessing of multi-source heterogeneous intelligence data specifically includes format unification, field standardization, noise filtering, time alignment and missing value filling operations.

[0079] In this embodiment, S2 specifically includes:

[0080] S21. Divide the generated standardized multi-source intelligence dataset into three subsets: text information, behavior sequence, and device features, and perform vector embedding processing on each subset.

[0081] S22, use the semantic encoding model to perform context semantic modeling on the text information. The original text sequence is T = {t1, t2, ..., t n}, where t i Represents the i-th term, and the semantic encoding model outputs the text semantic vector set VT ={v1,v2,...,v n}, where v i Represents the term t i The corresponding semantic embedding vector, i∈[1,n], n represents the number of terms in the original text sequence;

[0082] S23, the behavior sequence is vectorized using the time series modeling method, assuming that the behavior sequence is A = {a1, a2, ..., a m}, where a j Represents the jth behavior node, generating a behavior sequence embedding vector set V A ={u1,u2,…,u m}, where u j is the embedding vector of the corresponding behavior node, j∈[1,m], m represents the total number of behavior nodes in each behavior sequence;

[0083] S24, the device features are embedded using a static attribute encoding method, and the device feature set is D = {d1, d2, ..., d k}, where d k Represents the kth device attribute and generates the device feature vector set V D ={w1,w2,...,w k}, where w k Indicates device attributes d k The embedding vector of , k represents the total dimension or number of attributes of the device features;

[0084] S25, the generated text semantic vector set V T , behavior sequence embedding vector set V A and the device feature vector set V D Splicing and fusion are performed to construct an intelligence semantic vector E of unified dimension. The intelligence semantic vector constitutes an intelligence semantic vector set, which serves as input data for constructing a local sensitive hash index structure.

[0085] In this embodiment, S3 specifically includes:

[0086] S31. Obtain the generated intelligence semantic vector set. The semantic vector of each intelligence sample is E i , where i∈[1,N], N is the total number of samples;

[0087] S32, based on the vector dimension and sample size N, adaptively set the hash mapping dimension L and the number of hash clusters C to construct a multi-cluster hash function family Each cluster contains L c A hash function;

[0088] S33. For each semantic vector E iApply the hash function clusters under all clusters in turn to generate the hash signature matrix H(E i );

[0089] S34, according to the hash signature matrix, the intelligence semantic vector E i Mapped to hash buckets under multiple clusters, the lth hash function under the cth cluster will be mapped to bucket Forming multi-cluster mapping relationships;

[0090] S35. Build a hash bucket weight table for all hash buckets and record the sample density ρ of each bucket j , average similarity center vector And set the matching priority score ω for each bucket j ;

[0091] S36, introduce a conflict mediation mechanism. When there are high-overlap samples between multiple high-density buckets, execute the hash conflict detection algorithm and calculate the collision based on the vector distance. Redistribute boundary samples to the optimal bucket;

[0092] S37. For each sample E i Build a bucket ownership list And add bucket weight ω j , generate a priority index path;

[0093] S38. List all buckets Summarize and construct a multi-path, multi-cluster, weighted hash index structure, and use this structure to generate a preliminary candidate search space, which includes all hash buckets, sample distributions, bucket weights, and index paths;

[0094] S39. Introducing an incremental clustering learning module to perform feature center update and local bucket reconstruction on newly added semantic vectors, thus achieving dynamic self-learning of the preliminary candidate retrieval space;

[0095] S310, set the index refresh cycle, when the cumulative number of new samples reaches the threshold T u When , the full hash function reconstruction and bucket structure reconstruction are triggered to maintain the stability and search performance of the preliminary candidate retrieval space.

[0096] In this embodiment, the S4 specifically includes:

[0097] S41. From the generated preliminary candidate retrieval space, target intelligence semantic vector E q , retrieve the attributed hash bucket set And mark the priority weight ωq of each bucket l ;

[0098] S42, build a dynamic priority sampling pool, according to the bucket weight ωq land sample density ρq l Sort each bucket into different levels and set high priority pool, normal pool and low priority pool;

[0099] S43. Set the multipath perturbation search parameters, including the local perturbation coefficient α, the cross-bucket jump threshold γ, and the maximum path width W. max and adaptive sampling threshold τ;

[0100] S44. For each hash bucket in the high-priority pool, for each intelligence semantic vector sample in the bucket, add a random perturbation amount generated according to a zero-mean unit variance normal distribution to each component of the sample according to the local perturbation parameter to generate a corresponding perturbation sample vector.

[0101] S45. According to the current sample distribution, the probability distribution of samples in each hash bucket is counted, and the information entropy of the probability distribution is calculated. The information entropy is the negative value of the sum of the products of all sample probabilities and logarithms.

[0102] S46. When the entropy of the sampling path distribution is lower than the set threshold, or the similarity of samples between buckets is higher than the threshold, the cross-bucket jump mechanism is triggered, and a new path is selected in the common pool or the low-priority pool with probability γ to enrich the diversity of the candidate set;

[0103] S47. Introduce an interactive memory mechanism during the sampling process to record the sample index, perturbation parameters, and sampling bucket number of each round of sampling to avoid repeated sampling and path loops;

[0104] S48. After each round of multipath perturbation and cross-bucket jump is completed, sampling distribution statistics are performed on the sample set, and the weight of the dynamic priority sampling pool is updated;

[0105] S49, the candidate comparison sample set generated by the final sampling and metadata structure as input collection;

[0106] S410: For samples marked as high confidence in the sampling results, the hash bucket priority is automatically increased based on the metadata and sampling history, and the relevant sampling path, perturbation parameters and distribution entropy information are synchronously fed back to the preliminary candidate retrieval space.

[0107] In this embodiment, the S5 specifically includes:

[0108] S51, obtaining the generated candidate comparison sample set and intelligence semantic vector;

[0109] S52. For each candidate sample, extract the corresponding text information, behavior path sequence, and device features, and compare them with the text information, behavior path sequence, and device features of the target intelligence sample, respectively, to calculate semantic similarity, behavior path sequence similarity, and device entity relationship graph structural similarity. The semantic similarity is calculated using cosine similarity, the behavior path sequence similarity is calculated using a dynamic time warping algorithm, and the device entity relationship graph structural similarity is calculated using an optimal subgraph matching algorithm.

[0110] S53. Based on the obtained comparison results, perform multi-scale feature fusion processing on each candidate sample, and fuse different time series granularities, behavior windows and multi-level device entity features into a comprehensive feature representation;

[0111] S54, for each candidate sample comprehensive feature representation, combined with historical label information, feature variability and category consistency, using adaptive weight adjustment mechanism to generate sample adaptive weight parameters

[0112] S55, based on the obtained adaptive weight parameters, a weighted scoring function Score (E q ,E i ) Calculate the comprehensive similarity score, and count the historical comparison scores to generate the comprehensive similarity confidence interval

[0113]

[0114] Among them, E q Represents the semantic vector of the target intelligence sample, E i is the semantic vector of the candidate intelligence sample, Sim sem (E q ,E i ) represents E q With E i The semantic similarity between act (A q ,A i ) represents the behavior path similarity between the target sample and the candidate sample, Sim dev (G q ,G i ) represents the graph structure similarity between the target and the candidate sample in the device entity structure, A q Represents the behavioral path sequence of the target intelligence sample, A i Represents the behavioral path sequence of candidate samples, G q The device entity relationship diagram representing the target sample, G i Device entity relationship diagram representing candidate samples;

[0115] S56. Perform an anomaly analysis on the comprehensive similarity scores and confidence intervals of all candidate samples, use the isolation forest algorithm to detect abnormal candidate samples whose comprehensive similarity significantly deviates from the overall distribution, and mark abnormal samples with abnormal scores;

[0116] S57. For the obtained candidate samples, the adaptive weighted comprehensive score, anomaly score, confidence interval information and historical labels are combined to output the final high-risk sample list and the risk level probability of each sample;

[0117] S58. Generate a feature explanatory analysis report for the top-ranked high-risk samples, and output the main judgment basis such as semantic similarity contribution, abnormal behavior path, and key equipment entity association.

[0118] In this embodiment, S6 specifically includes:

[0119] S61, receiving the output comprehensive similarity scores and ranking results of the candidate samples, and establishing a corresponding judgment record entry for each candidate sample;

[0120] S62. Set multiple preset threshold levels required for risk determination, including a high risk threshold, a medium risk threshold, and a low risk reference threshold, and load the corresponding risk determination rule table;

[0121] S63. Perform threshold comparison on the comprehensive similarity score of each candidate sample, and classify the sample into high-risk, medium-risk, low-risk, or non-risk categories based on the score interval matching judgment rules;

[0122] S64. For samples classified as high-risk, record the reason for the determination, the associated target sample number, the trigger threshold type, and the risk characteristics, and mark them as high-risk intelligence samples;

[0123] S65. Based on the determination result of the high-risk intelligence sample, generate warning data content including sample identification information, similarity comparison source information, key feature summary and corresponding risk level;

[0124] S66. Send the generated warning data content to the preset response channel, and trigger manual review, behavior isolation, and historical behavior tracing in the specified information processing process.

[0125] Example 1:

[0126] To verify the feasibility of the present invention, the present invention was applied to a commercial bank. From August to October 2024, the bank's risk control center frequently encountered a typical problem in actual business operations: suspected black market accounts displayed significant differences in identity, device information, and behavioral paths across different platforms or business systems. This resulted in the original detection system, which relied on single-platform logs or manual label analysis, being unable to effectively identify potential risks. The fraudulent activities in multiple important cases showed semantic similarities or device overlap, but due to the lack of a cross-source data joint comparison mechanism, the system failed to issue timely warnings, resulting in the transfer of some of the funds involved.

[0127] To this end, the bank incorporated the AI-based multi-source intelligence comparison method for black and gray market detection into its risk control system, enabling unified analysis and risk identification of incoming data. The system first standardized preprocessing of incoming data, including social media comments, e-commerce orders, app clickstreams, registration information, device fingerprints, and IP tracking data, and then employed an embedding model to construct a unified semantic vector representation of the intelligence. For these vectors, the system employed a locality-sensitive hash index structure to cluster similar information into the same hash bucket, thereby constructing a preliminary candidate search space.

[0128] During the risk sample detection phase, the system uses the account to be assessed as the target intelligence sample. By introducing random perturbations and cross-bucket jump strategies in the hash index space, it quickly generates a set of candidate samples with similar behavioral structures. The system then compares the target sample with the candidate samples based on three dimensions: semantic expression, behavioral sequence, and device entity graph. A dynamic weighting mechanism is used to calculate a comprehensive similarity score. If the comprehensive score exceeds a preset threshold, the system immediately marks the sample as high risk and submits it to risk control.

[0129] Table 1 Performance comparison of traditional method and the method of the present invention in black and gray product detection

[0130]

[0131] The various indicators in Table 1 allow for a systematic analysis of the performance advantages of the present invention in the detection of illegal and gray industries. From the perspective of "average processing time / sample," the traditional detection system's processing flow based on rule engines and manual labeling has a certain delay, requiring an average of 9.8 seconds to judge each sample. However, the present invention achieves efficient comparison through semantic vector encoding and locally sensitive hash indexing, shortening the processing time for a single sample to 2.4 seconds, increasing processing efficiency by approximately 75.5%, and significantly enhancing the system's real-time responsiveness.

[0132] In terms of "single-day sample processing capacity", the original system was limited by computing resources and manual review processes, with a daily processing capacity of approximately 12,000 samples. After adopting the method of the present invention, the system's parallel processing capability has been significantly enhanced, and it can stably process more than 48,000 samples, achieving a four-fold increase in processing capacity, supporting detection tasks in large-scale multi-source data scenarios.

[0133] In terms of detection accuracy, the "comprehensive recognition accuracy" was increased from 79.6% of the original system to 93.2%, indicating that the present invention has significantly improved the system's ability to identify complex black and gray market behaviors by introducing a multi-dimensional similarity weighted fusion mechanism; and the "high-risk identification recall rate" increased from 64.8% to 89.5%, which means that the system has a stronger ability to capture potential high-risk intelligence samples and significantly reduced missed detections.

[0134] In terms of the "missing negative rate," this invention has reduced the previously high rate of 21.3% to 6.2%, demonstrating that the system's screening mechanism is more accurate and stable, effectively preventing the subsequent risk spread caused by missing critical samples. The "false alarm rate" has also been reduced from 13.4% to 8.1%, reflecting that the system has minimal interference with normal samples and significantly reduces the burden of unnecessary manual verification.

[0135] In the key operational indicator of "average manual review volume / day", the system reduced the number of samples that need to be reviewed daily from 450 to 160, greatly alleviating the workload of risk control personnel while improving verification efficiency and decision-making quality.

[0136] Finally, in terms of the "proportion of explainable information output", traditional systems are often unable to provide specific traceable evidence, which increases the difficulty of user trust; while the present invention, when outputting risk samples, is fully accompanied by semantic similarity scores, key path characteristics, device behavior diagrams and other information, providing visual support and explanation capabilities for the system output results, thereby improving regulatory compliance and user acceptance.

[0137] In summary, the above data fully verifies that the present invention has significant advantages in processing performance, recognition accuracy, risk control and labor costs in actual black and gray industry detection scenarios, and has good practical value and promotion prospects.

[0138] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. An AI-based multi-source intelligence comparison method for detecting black and gray industries, characterized by: The steps include: S1. Collect multi-source heterogeneous intelligence data and pre-process the multi-source heterogeneous intelligence data to generate a multi-source intelligence dataset; S2. Semantically encode multi-source intelligence datasets and use a unified embedding model to vectorize text information, behavior sequences, and device features to obtain a unified-dimensional intelligence semantic vector set. S3. Based on the intelligence semantic vector set, a locality-sensitive hash index structure is constructed. A hash function family that meets the similarity measurement requirements is selected to perform a hash mapping operation on the vector set. Similar semantic vectors are assigned to the same or adjacent hash buckets to generate a preliminary candidate retrieval space. S4. In the preliminary candidate search space, a random search algorithm is used to screen candidate intelligence samples, random perturbation rules and jump probabilities are set, and candidate intelligence vectors are randomly selected from multiple related hash buckets to generate a candidate comparison sample set; S5. Perform a detailed comparison operation on the candidate comparison sample set, calculate the multi-dimensional similarity between the candidate sample and the target sample in semantic content, behavior path sequence, and device entity relationship diagram, and output a comprehensive similarity score based on the set weighted scoring function; S6: Determine black and gray market behavior based on the comprehensive similarity score. When the score exceeds the set risk threshold, the corresponding sample is marked as high-risk intelligence and an early warning mechanism is triggered. S7. Use intelligence samples judged as high-risk to update the hash bucket index structure, add intelligence semantic vectors to the hash index, and rebuild the local hash map.

2. The AI-based multi-source intelligence comparison method for detecting illegal and gray industries according to claim 1 is characterized by: The multi-source heterogeneous intelligence data specifically includes social platform text content, network communication behavior logs, e-commerce transaction records, device access trajectory information and account identity association data.

3. The AI-based multi-source intelligence comparison method for detecting illegal and gray industries according to claim 1 is characterized in that: The preprocessing of multi-source heterogeneous intelligence data specifically includes format unification, field standardization, noise filtering, time alignment and missing value filling operations.

4. The AI-based multi-source intelligence comparison method for detecting illegal and gray industries according to claim 1 is characterized in that: The S2 specifically includes: S21. Divide the generated standardized multi-source intelligence dataset into three subsets: text information, behavior sequence, and device features, and perform vector embedding processing on each subset. S22, use the semantic encoding model to perform context semantic modeling on the text information. The original text sequence is T = {t1, t2, ..., t n }, where t i Represents the i-th term, and the semantic encoding model outputs the text semantic vector set V T ={v1,v2,...,v n }, where v i Represents the term t i The corresponding semantic embedding vector, i∈[1,n], n represents the number of terms in the original text sequence; S23, the behavior sequence is vectorized using the time series modeling method, assuming that the behavior sequence is A = {a1, a2, ..., a m }, where a j Represents the jth behavior node, generating a behavior sequence embedding vector set V A ={u1,u2,…,u m }, where u j is the embedding vector of the corresponding behavior node, j∈[1,m], m represents the total number of behavior nodes in each behavior sequence; S24, the device features are embedded using a static attribute encoding method, and the device feature set is D = {d1, d2, ..., d k }, where d k Represents the kth device attribute and generates the device feature vector set V D ={w1,w2,...,w k }, where w k Indicates device attributes d k The embedding vector of , k represents the total dimension or number of attributes of the device features; S25, the generated text semantic vector set V T , behavior sequence embedding vector set V A and the device feature vector set V D Splicing and fusion are performed to construct an intelligence semantic vector E of unified dimension. The intelligence semantic vector constitutes an intelligence semantic vector set, which serves as input data for constructing a local sensitive hash index structure.

5. The AI-based multi-source intelligence comparison method for detecting illegal and gray industries according to claim 1 is characterized in that: The S3 specifically includes: S31. Obtain the generated intelligence semantic vector set. The semantic vector of each intelligence sample is E i , where i∈[1,N], N is the total number of samples; S32, based on the vector dimension and sample size N, adaptively set the hash mapping dimension L and the number of hash clusters C to construct a multi-cluster hash function family Each cluster contains L c A hash function; S33. For each semantic vector E i Apply the hash function clusters under all clusters in turn to generate the hash signature matrix H(E i ); S34, according to the hash signature matrix, the intelligence semantic vector E i Mapped to hash buckets under multiple clusters, the lth hash function under the cth cluster will be mapped to bucket Forming multi-cluster mapping relationships; S35. Build a hash bucket weight table for all hash buckets and record the sample density ρ of each bucket j , average similarity center vector And set the matching priority score ω for each bucket j ; S36, introduce a conflict mediation mechanism. When there are high-overlap samples between multiple high-density buckets, execute the hash conflict detection algorithm and calculate the collision based on the vector distance. Redistribute boundary samples to the optimal bucket; S37. For each sample E i Build a bucket ownership list And add bucket weight ω j , generate a priority index path; S38. List all buckets Summarize and construct a multi-path, multi-cluster, weighted hash index structure, and use this structure to generate a preliminary candidate search space, which includes all hash buckets, sample distributions, bucket weights, and index paths; S39. Introducing an incremental clustering learning module to perform feature center update and local bucket reconstruction on newly added semantic vectors, thus achieving dynamic self-learning of the preliminary candidate retrieval space; S310, set the index refresh cycle, when the cumulative number of new samples reaches the threshold T u When , the full hash function reconstruction and bucket structure reconstruction are triggered to maintain the stability and search performance of the preliminary candidate retrieval space.

6. The AI-based multi-source intelligence comparison method for detecting illegal and gray industries according to claim 1 is characterized in that: The S4 specifically includes: S41. From the generated preliminary candidate retrieval space, target intelligence semantic vector E q , retrieve the attributed hash bucket set And mark the priority weight of each bucket S42, build a dynamic priority sampling pool based on bucket weights and sample density Sort each bucket into different levels and set high priority pool, normal pool and low priority pool; S43. Set the multipath perturbation search parameters, including the local perturbation coefficient α, the cross-bucket jump threshold γ, and the maximum path width W. max and adaptive sampling threshold τ; S44. For each hash bucket in the high-priority pool, for each intelligence semantic vector sample in the bucket, add a random perturbation amount generated according to a zero-mean unit variance normal distribution to each component of the sample according to the local perturbation parameter to generate a corresponding perturbation sample vector. S45. According to the current sample distribution, the probability distribution of samples in each hash bucket is counted, and the information entropy of the probability distribution is calculated. The information entropy is the negative value of the sum of the products of all sample probabilities and logarithms. S46. When the entropy of the sampling path distribution is lower than the set threshold, or the similarity of samples between buckets is higher than the threshold, the cross-bucket jump mechanism is triggered, and a new path is selected in the common pool or the low-priority pool with probability γ to enrich the diversity of the candidate set; S47. Introduce an interactive memory mechanism during the sampling process to record the sample index, perturbation parameters, and sampling bucket number of each round of sampling to avoid repeated sampling and path loops; S48. After each round of multipath perturbation and cross-bucket jump is completed, sampling distribution statistics are performed on the sample set, and the weight of the dynamic priority sampling pool is updated; S49, the candidate comparison sample set C generated by the final sampling q and metadata structure as input collection; S410: For samples marked as high confidence in the sampling results, the hash bucket priority is automatically increased based on the metadata and sampling history, and the relevant sampling path, perturbation parameters and distribution entropy information are synchronously fed back to the preliminary candidate retrieval space.

7. The AI-based multi-source intelligence comparison method for detecting illegal and gray industries according to claim 1 is characterized in that: The S5 specifically includes: S51, obtaining the generated candidate comparison sample set and intelligence semantic vector; S52. For each candidate sample, extract the corresponding text information, behavior path sequence, and device features, and compare them with the text information, behavior path sequence, and device features of the target intelligence sample, respectively, to calculate semantic similarity, behavior path sequence similarity, and device entity relationship graph structural similarity. The semantic similarity is calculated using cosine similarity, the behavior path sequence similarity is calculated using a dynamic time warping algorithm, and the device entity relationship graph structural similarity is calculated using an optimal subgraph matching algorithm. S53. Based on the obtained comparison results, perform multi-scale feature fusion processing on each candidate sample, and fuse different time series granularities, behavior windows and multi-level device entity features into a comprehensive feature representation; S54, for each candidate sample comprehensive feature representation, combined with historical label information, feature variability and category consistency, using adaptive weight adjustment mechanism to generate sample adaptive weight parameters S55, based on the obtained adaptive weight parameters, a weighted scoring function Score (E q ,E i ) Calculate the comprehensive similarity score, and count the historical comparison scores to generate the comprehensive similarity confidence interval Among them, E q Represents the semantic vector of the target intelligence sample, E i is the semantic vector of the candidate intelligence sample, Sim sem (E q ,E i ) represents E q With E i The semantic similarity between act (A q ,A i ) represents the behavior path similarity between the target sample and the candidate sample, Sim dev (G q ,G i ) represents the graph structure similarity between the target and the candidate sample in the device entity structure, A q Represents the behavioral path sequence of the target intelligence sample, A i represents the behavioral path sequence of candidate samples, G q The device entity relationship diagram representing the target sample, G i Device entity relationship diagram representing candidate samples; S56. Perform an anomaly analysis on the comprehensive similarity scores and confidence intervals of all candidate samples, use the isolation forest algorithm to detect abnormal candidate samples whose comprehensive similarity significantly deviates from the overall distribution, and mark abnormal samples with abnormal scores; S57. For the obtained candidate samples, the adaptive weighted comprehensive score, anomaly score, confidence interval information and historical labels are combined to output the final high-risk sample list and the risk level probability of each sample; S58. Generate a feature explanatory analysis report for the top-ranked high-risk samples, and output the main judgment basis such as semantic similarity contribution, abnormal behavior path, and key equipment entity association.

8. The AI-based multi-source intelligence comparison method for detecting illegal and gray industries according to claim 1 is characterized in that: The S6 specifically includes: S61, receiving the output comprehensive similarity scores and ranking results of the candidate samples, and establishing a corresponding judgment record entry for each candidate sample; S62. Set multiple preset threshold levels required for risk determination, including a high risk threshold, a medium risk threshold, and a low risk reference threshold, and load the corresponding risk determination rule table; S63. Perform threshold comparison on the comprehensive similarity score of each candidate sample, and classify the sample into high-risk, medium-risk, low-risk, or non-risk categories based on the score interval matching judgment rules; S64. For samples classified as high-risk, record the reason for the determination, the associated target sample number, the trigger threshold type, and the risk characteristics, and mark them as high-risk intelligence samples; S65. Based on the determination result of the high-risk intelligence sample, generate warning data content including sample identification information, similarity comparison source information, key feature summary and corresponding risk level; S66. Send the generated warning data content to the preset response channel, and trigger manual review, behavior isolation, and historical behavior tracing in the specified information processing process.

Images