A cross-domain copywriting online tracing method and device

Abstract

The present invention provides a cross-domain online document traceability method and device, comprising: time slicing acquired streaming incremental internet document information according to the time of data entry to obtain data within each time slice, extracting the earliest uncalculated batch data from the data within the time slice to form an incremental dataset; parallel processing of the incremental datasets from different time slices, establishing local semantic clusters within each incremental dataset, and greedily merging them with historical clustering results to obtain clustering results; introducing a dynamic monitoring mechanism for active classes to continuously track active classes with changes in the number of members in the clustering results to obtain traceability results; and storing the clustering and traceability results in a distributed search engine. The present invention can accurately identify cross-domain document semantic associations and support real-time and efficient processing of massive amounts of data.

Description

Technical Field

[0001] The present invention relates to the technical field of online copywriting tracing, and in particular to a cross-domain online copywriting tracing method and device. Background Art

[0002] In the internet age, information dissemination is experiencing explosive growth, and a continuous stream of diverse content is being generated. Copywriting provenance technology, a fundamental requirement, relies on semantic analysis to identify groups of documents with similar content and intent, and to determine their origins and scale. This technology has significant application value in a variety of fields, including intellectual property protection, brand reputation monitoring, public opinion analysis, and marketing effectiveness evaluation. These applications all rely on accurate and rapid copywriting provenance analysis results to support decision-making.

[0003] However, traditional keyword matching methods rely on domain-specific keyword vocabularies, making them less versatile when applied across domains. Methods based on pre-trained language models like BERT are computationally complex and struggle to meet the demands of real-time processing of massive amounts of data. Therefore, there is an urgent need for a document traceability method that can accurately identify semantic associations across domains and support efficient, real-time processing of massive amounts of data. Summary of the Invention

[0004] The present invention provides a cross-domain document online traceability method and device to address the shortcomings of existing technologies, such as poor versatility in cross-domain applications and difficulty in meeting the real-time processing requirements of massive data. The method accurately identifies semantic associations between cross-domain documents and supports real-time and efficient processing of massive data. The technical solutions proposed by the present invention are as follows:

[0005] In a first aspect, the present invention provides a cross-domain document online tracing method, comprising:

[0006] The acquired streaming incremental internet copywriting information is time-sliced according to the data entry time to obtain the data within each time slice, and the earliest uncalculated batch data is extracted from the data within the time slice to form an incremental data set;

[0007] Incremental datasets of different time slices are processed in parallel, local semantic clusters are established in each incremental dataset, and clustering results are obtained by greedy merging with historical clustering results.

[0008] Introducing a dynamic monitoring mechanism for active classes to continuously track active classes with changed member numbers in the clustering results to obtain traceability results;

[0009] The clustering results and the tracing results are stored in a distributed search engine.

[0010] Optionally, establishing a local semantic cluster in each incremental data set and greedily merging it with the historical clustering results to obtain a clustering result includes:

[0011] For each incremental data set, feature extraction and similarity calculation are performed on the incremental data set, and a corresponding semantic graph is constructed based on the extracted semantic features and the calculated similarity relationships; wherein the vertex set of the semantic graph corresponds to all texts in the incremental data set, the parameters of the vertex represent the semantic features of the corresponding text, and the edge set of the semantic graph is generated according to the similarity relationships;

[0012] A depth-first search algorithm is used to identify all connected components in the semantic graph. Each connected component forms an online cluster, and the centroid of each online cluster is extracted.

[0013] For the centroid of each online cluster, a beam search is performed using a fuzzy query mechanism, and a greedy strategy based on large cluster priority is adopted to merge clusters to obtain the clustering results.

[0014] Optionally, the centroid of each online cluster is subjected to beam search using a fuzzy query mechanism, and clusters are merged using a greedy strategy based on large cluster priority to obtain a clustering result, including:

[0015] For the centroid of each online cluster, a beam search is performed using a fuzzy query mechanism to retrieve historical similar texts and their category identifiers that are similar to the centroid in the historical clustering results, and a corresponding candidate category set is formed based on the retrieval results;

[0016] A greedy strategy based on large cluster priority is used to merge online clusters and historical clusters, and the category with the most nodes in the candidate category set is selected as the current online cluster to obtain the clustering result.

[0017] Optionally, the centroid of an online cluster is determined as follows:

[0018] In the online cluster, the text with the earliest publishing time is selected as the centroid of the online cluster.

[0019] Optionally, the introduction of a dynamic monitoring mechanism for active classes to continuously track active classes with a changed number of members in the clustering results to obtain tracing results includes:

[0020] For each time, the number of members in each cluster is determined based on the clustering results at that time, and compared with the number of members in the previous time window to determine the active class set at that time;

[0021] For each time, an active class tracing function is calculated, and the active class tracing function outputs the tracing results of all active classes in the active class set that meet the activity threshold; wherein the tracing results include source information, time information, propagation path and propagation scale.

[0022] Optionally, the active class function tracing function is:

[0023]

[0024] in, For active class function tracing function, is the active class set, is the first in the active class set Active classes, For active class The traceability results, For time, For active class In time The number of members, For active class In time The number of members, is the activity threshold, is the time window.

[0025] In a second aspect, the present invention further provides a cross-domain document online tracing device, comprising the following modules:

[0026] The time slicing module is used to time slice the acquired streaming incremental internet copywriting information according to the data storage time to obtain the data within each time slice, and extract the earliest uncalculated batch data from the data within the time slice to form an incremental data set;

[0027] The parallel copy aggregation module is used to process incremental data sets of different time slices in parallel, establish local semantic clusters in each incremental data set, and greedily merge them with historical clustering results to obtain clustering results;

[0028] A global copywriting traceability module is used to introduce a dynamic monitoring mechanism for active classes, continuously tracking active classes with changed member numbers in the clustering results to obtain traceability results;

[0029] The tracing result storage module is used to store the clustering result and the tracing result in a distributed search engine.

[0030] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the cross-domain document online tracing method as described in the first aspect above is implemented.

[0031] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cross-domain document online tracing method as described in the first aspect above.

[0032] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the cross-domain document online tracing method as described in the first aspect above.

[0033] Based on the above technical solution, the present invention has the following beneficial effects compared with the prior art:

[0034] The cross-domain online copywriting traceability method and device provided by the present invention partitions streaming incremental internet copywriting information into time slices and extracts the earliest uncomputed batch of data from each time slice to form an incremental dataset. This processing approach avoids processing the entire dataset all at once, thereby reducing computational complexity. Furthermore, since only the incremental data within each time slice is processed, real-time processing of new data is possible. This method processes incremental datasets from different time slices in parallel, fully utilizing multi-core processors or distributed computing resources. Through parallel processing and a greedy merging algorithm, data processing speed can be significantly improved, meeting the real-time processing requirements of massive data volumes. Furthermore, by introducing a dynamic active class monitoring mechanism, active classes whose membership changes within clustering results can be continuously tracked, allowing for rapid traceability results. Furthermore, this method establishes local semantic clusters within each incremental dataset. These clusters are divided based on semantic similarity, rather than relying on domain-specific keyword lexicons. Therefore, even in data from different domains, copywriting information with similar themes or content will be grouped into the same semantic cluster, enhancing the versatility of cross-domain applications. Through a greedy merging algorithm, new incremental datasets are integrated with existing historical clustering results to gradually build more comprehensive and accurate clustering results. This integration method does not rely on specific domain knowledge, but rather merges data based on semantic similarity, making it suitable for cross-domain data processing.

[0035] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood through implementation of the present invention. The purposes and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0036] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 It is a flow chart of the cross-domain copywriting online tracing method provided by the present invention.

[0039] Figure 2 It is a schematic diagram of the overall framework of the cross-domain copywriting online tracing method provided by the present invention.

[0040] Figure 3 This is a schematic diagram of the streaming copywriting clustering process provided by the present invention.

[0041] Figure 4 It is a structural diagram of the cross-domain copywriting online tracing device provided by the present invention.

[0042] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0044] The following combination Figures 1-4 The present invention describes a cross-domain online copywriting traceability method and device. The input is incremental streaming internet copywriting information, including data entry time, posting time, posting content, and posting account. The output is the copywriting group, copywriting group size, and copywriting source. The copywriting source includes key information such as the time of first release, the first user, and the platform.

[0045] The present invention proposes corresponding solutions to the three core technical problems in the online copywriting traceability system. First, in view of the fact that traditional keyword matching algorithms are difficult to effectively handle cross-domain copywriting with significant differences in semantic features, a cross-domain adaptive copywriting semantic clustering algorithm based on similar connectivity is proposed; secondly, in order to solve the dual challenges of efficient clustering of stock data and real-time response to incremental data in a massive copywriting data environment, the system throughput and response timeliness are significantly improved by reconstructing the clustering algorithm into a streaming computing mode and combining the parallel stream clustering algorithm of the dynamic semantic graph, the real-time update mechanism of category information and the efficient indexing mechanism; in response to the diversified copywriting traceability requirements in different application scenarios, a clustering-traceability asynchronous decoupling mechanism is constructed. By realizing the computational separation of copywriting traceability and streaming clustering, the system can flexibly adapt to the traceability logic of different scenarios such as advertising marketing and public opinion supervision, significantly improving the scalability of the application level. Reference Figure 1 As shown, the cross-domain copywriting online tracing method includes the following:

[0046] Step S110: time-slice the acquired streaming incremental Internet copywriting information according to the data storage time to obtain data within each time slice, and extract the earliest uncalculated batch data from the data within the time slice to form an incremental data set.

[0047] Streaming incremental copywriting information obtained from the internet is divided into several time slices based on the time of data entry. Each time slice contains data from a specific period of time. Next, the earliest batch of data that has not yet been calculated or processed is extracted from the data in each time slice. This data constitutes the incremental dataset. This step ensures data timeliness and processing priority.

[0048] Because the speed of information dissemination on social networks decays exponentially over time, most content duplication and adaptation occurs shortly after the original information is published, resulting in higher similarity between texts from similar times. Therefore, this paper introduces a data sharding mechanism based on the incoming time window. This sharding process ensures the local clustering of highly similar texts while significantly reducing cross-shard computation overhead.

[0049] The present invention proposes an adaptive parallel scheduling mechanism for time series data streams. This mechanism uses dynamic time windows to slice data based on the temporal locality characteristics of data, and constructs each time segment into an independent computing task unit. Specifically, the present invention proposes an information slicing and data set construction method based on time series consistent hashing. First, the streaming incremental copywriting information obtained from the Internet is divided according to the time when the data is entered into the database. The purpose of this step is to divide the data into several time slices, each of which contains data within a certain time period. The size of the time slice (denoted as ) can be adjusted based on the dynamic availability of actual computing resources to ensure balanced computing load. Time sharding can divide data into more manageable and processable units, while ensuring local clustering of highly similar texts and reducing cross-shard computing overhead.

[0050] Next, the earliest batch of data that has not been calculated or processed is extracted from the data in each time slice, and these data constitute the incremental dataset. This step is called micro-batch dataset construction. Specifically, for the time slice [ , ], extract the earliest batch of data that has not been calculated in the order of storage time to form an incremental data set This mechanism ensures that each piece of data is processed only once, avoiding repeated calculations while maintaining the time sequence of data processing and ensuring the completeness of information traceability as much as possible. It can be expressed as:

[0051]

[0052] in, Represents a single data record, which refers to one of the streamed incremental copy information obtained from the Internet. Indicates the starting time of the time slice. Indicates the processing status of the data, which is a Boolean function. Returns true if the object has been processed, otherwise returns false. Represents data records The state that has not been processed yet. Is the logical NOT symbol. Represents data records The storage time. Indicates the size of the time slice, that is, the time span of the data contained in the time slice.

[0053] The above formula means: from all the storage times in [ , ]Interval, filter out those data records that have not been processed to form a time slice Incremental dataset within This index preparation phase lays the data foundation for subsequent parallel clustering and global traceability.

[0054] During the incremental dataset construction process, we leveraged the Elasticsearch distributed search engine to establish a multi-dimensional index structure, including fundamental attribute fields such as text content (content), publication time (publish_time), and insertion time (insert_time). For clustered data, we also indexed its category identifier (cluster_id) and similarity features (similarity_features). These indexes laid the data foundation for subsequent parallel clustering and global traceability.

[0055] At the task scheduling level, the present invention designs an adaptive scheduling algorithm based on resource awareness. This algorithm realizes on-demand allocation of computing resources and dynamic scheduling of tasks by maintaining a fixed-scale computing resource pool and a dynamic task queue. , building them into independent computing task units and dynamically scheduling them based on the current state of computing resources and the length of the task queue. This mechanism achieves efficient utilization of computing resources through task lifecycle management while ensuring the integrity and continuity of time series data processing.

[0056] Step S120 : Process the incremental data sets of different time slices in parallel, establish a local semantic cluster in each incremental data set, and greedily merge it with the historical clustering results to obtain a clustering result.

[0057] Incremental datasets from different time slices are processed in parallel to improve processing efficiency. Within each incremental dataset, local semantic clusters are established using specific algorithms or techniques. These clusters represent collections of documents with similar themes or content within the dataset. The local semantic clusters established within each time slice are then greedily merged with the historical clustering results. Greedy merging is a method that incrementally builds a solution, selecting the optimal option at each step in the current state in the hope of achieving a globally optimal solution. Through greedy merging, new incremental data can be gradually integrated with existing clustering results, resulting in more comprehensive and accurate clustering results.

[0058] Multi-core processors or distributed computing resources are used to process incremental data sets of different time slices in parallel. Each incremental data set of each time slice is processed by an independent computing unit (such as a thread, process, or computing node). The work of each computing unit is managed and coordinated by a parallel computing framework (such as Apache Spark, Dask, etc.). Figure 2 As shown in , each time slice’s incremental dataset (Micro-Chunk) corresponds to a streaming copywriting clustering task, such as Figure 2In the flow copy clustering task 1, flow copy clustering task 2, ..., flow copy clustering task n. The number of flow copy clustering tasks corresponds to the number of time slices.

[0059] Specifically, each piece of text in the incremental dataset is preprocessed, including steps such as word segmentation, stop word removal, and stemming, to extract key information. Using techniques such as TF-IDF, word embeddings (such as Word2Vec and GloVe), or contextual embeddings (such as BERT and GPT), the preprocessed text is converted into a numerical feature vector. A clustering algorithm (such as K-means, DBSCAN, or hierarchical clustering) is selected to establish local semantic clusters within the incremental dataset based on the feature vectors. The choice of clustering algorithm should be balanced based on the characteristics of the data and the clustering requirements. This allows for the identification and extraction of sets of text with similar themes or content within each incremental dataset, forming local semantic clusters.

[0060] Load existing historical clustering results from the storage system. These results may include clustering information from previous time slices. Design and implement a greedy merging algorithm that selects the optimal merging method for the current state at each step. Specific greedy strategies may include: distance-based merging. Calculate the distance (such as Euclidean distance, cosine similarity, etc.) between the local semantic cluster and each cluster in the historical clustering results, and select the cluster with the closest distance for merging. Similarity-based merging utilizes the similarity characteristics of the copy information (such as text similarity score) to select the cluster with the highest similarity for merging. Cluster size-based merging considers the cluster size (i.e., the number of copy information within the cluster) and selects clusters of similar size for merging to avoid forming overly large clusters. Post-process the merged clustering results, such as removing duplicate clusters and adjusting cluster boundaries, to ensure the accuracy and consistency of the clustering results. Store the updated clustering results in the storage system for subsequent use. Simultaneously, update the index structure to ensure that the new clustering results can be efficiently retrieved and queried.

[0061] Step S130: introduce an active class dynamic monitoring mechanism to continuously track active classes with changed member numbers in the clustering results to obtain tracing results.

[0062] The present invention proposes an asynchronous global tracing method based on active class perception. It introduces an active class dynamic monitoring mechanism into the global tracing framework to continuously track active classes whose number of members changes in clustering results. These active classes represent the copywriting information topics that are currently popular or widely watched on the Internet. By monitoring the changes in active classes, the source and propagation path of cross-domain copywriting can be discovered and tracked in a timely manner, thereby obtaining tracing results. By continuously tracking active classes whose number of members changes significantly in clustering results, this mechanism perceives the propagation dynamics of copywriting information on the Internet in real time, and realizes a rapid response to popular or widely watched copywriting information topics.

[0063] The system adopts an asynchronous decoupling architecture, decoupling active class monitoring from traceability analysis. This means that even if traceability analysis is still in progress, the system can continue to monitor new active classes, thereby improving overall processing efficiency. All traceability results are uniformly stored in a distributed search engine (such as ElasticSearch). This design not only ensures data consistency and traceability, but also facilitates subsequent data retrieval and analysis. The system supports the expansion of multi-dimensional traceability rules based on the needs of different application scenarios. For example, new algorithms or indicators can be introduced to optimize processes such as source identification and propagation path construction. Through the loosely coupled design concept, the various modules of the system (such as the parallel copy aggregation module, the global copy traceability module, etc.) can evolve and upgrade independently, thereby reducing the overall complexity and maintenance costs of the system.

[0064] Step S140: storing the clustering result and the tracing result in a distributed search engine.

[0065] The clustering results and tracing results are stored in a distributed search engine. A distributed search engine is a system that can efficiently process and retrieve large-scale data. It improves retrieval efficiency and scalability by distributing and storing data on multiple nodes. This architecture can not only effectively manage massive amounts of data, but also provide flexible data retrieval and analysis functions while maintaining high performance. In a distributed search engine, users can easily find relevant document information and its tracing results through keywords or other search conditions. The present invention stores the clustering results and tracing results in a distributed search engine, which can ensure that the clustering results and tracing results can be preserved for a long time, facilitating subsequent data analysis and retrieval. By utilizing the efficient retrieval capabilities of a distributed search engine, users can quickly find relevant document information and its tracing results, thereby improving work efficiency. Through a unified data storage layer, the consistency between clustering results and tracing results is ensured, avoiding data conflicts or inconsistencies.

[0066] Before storage, clustering and traceability results undergo necessary preprocessing, such as data cleaning and format conversion, to ensure data quality and compatibility. The distributed search engine's indexing capabilities are used to construct indexes for the clustering and traceability results. These indexes can be based on fields such as keywords, timestamps, and propagation paths to improve retrieval efficiency. The preprocessed clustering and traceability results are uploaded to each node of the distributed search engine. During the upload process, the system ensures data integrity and consistency to prevent data loss or duplication. After the upload is complete, the system performs data verification to ensure that all clustering and traceability results have been correctly stored and can be searched using the index. Users can use the search interface or API provided by the distributed search engine to enter search criteria such as keywords, time ranges, and propagation paths to quickly find relevant copywriting information and its traceability results. Based on the stored clustering and traceability results, users can conduct further data analysis, such as trend prediction and impact assessment, to gain a deeper understanding of the spread dynamics and effectiveness of copywriting information. Visualization tools are used to display clustering and traceability results in charts, maps, and other formats, helping users more intuitively understand the data and analysis results.

[0067] As data volumes increase and retrieval requirements diversify, distributed search engines require continuous expansion and upgrades. This includes increasing the number of nodes, optimizing indexing algorithms, and improving retrieval speeds. At the same time, the system must maintain compatibility with clustering algorithms and provenance analysis modules to ensure seamless data integration and efficient processing. The aforementioned system refers to the overall technical solution or software architecture for performing clustering analysis, provenance analysis, and related data storage and retrieval functions.

[0068] The cross-domain online document traceability method provided by the present invention segments streaming incremental internet document information into time slices and extracts the earliest uncomputed batch of data from each time slice to form an incremental dataset. This processing approach avoids processing the entire dataset all at once, thereby reducing computational complexity. Furthermore, since only the incremental data within each time slice is processed, real-time processing of new data is possible. This method processes incremental datasets from different time slices in parallel, fully utilizing multi-core processors or distributed computing resources. Through parallel processing and a greedy merging algorithm, data processing speed can be significantly improved, thus meeting the real-time processing requirements of massive data volumes. Furthermore, by introducing a dynamic monitoring mechanism for active classes, active classes whose membership changes in clustering results can be continuously tracked, allowing for rapid traceability results. Furthermore, this method establishes local semantic clusters within each incremental dataset. These clusters are divided based on semantic similarity of the data, rather than relying on domain-specific keyword lexicons. Therefore, even in data from different domains, documents with similar themes or content will be grouped into the same semantic cluster, enhancing the versatility of cross-domain applications. Through a greedy merging algorithm, new incremental datasets are integrated with existing historical clustering results to gradually build more comprehensive and accurate clustering results. This integration method does not rely on specific domain knowledge, but rather merges data based on semantic similarity, making it suitable for cross-domain data processing.

[0069] In an optional embodiment, a streaming semantic clustering algorithm for cross-domain documents is proposed. This method uses the Simhash algorithm combined with the Hamming distance to capture the latent semantic connections between text documents, overcoming the limitations of traditional keyword matching methods in cross-domain scenarios. By calculating the semantic similarity between documents and constructing a dynamically evolving similarity graph structure, an adaptive clustering algorithm based on graph topology is employed to effectively avoid the reliance of parameterized clustering methods (such as k-means) on the number of prior categories. At the streaming processing level, this framework designs an incremental clustering strategy based on semantic consistency. This strategy establishes local semantic clusters within small batches of data and greedily merges them with historical clustering results, enabling dynamic updating and maintenance of clustering results. This incremental clustering scheme not only ensures computational efficiency but also maintains semantic consistency across time windows. Experimental results demonstrate that this framework demonstrates significant advantages in semantic discovery and organization of cross-domain documents, making it particularly suitable for large-scale text clustering scenarios that require real-time processing and dynamic evolution.

[0070] The step S120 described above is to establish a local semantic cluster in each incremental data set and greedily merge it with the historical clustering results to obtain a clustering result, including:

[0071] S1201. For each incremental data set, perform feature extraction and similarity calculation on the incremental data set, and construct a corresponding semantic graph based on the extracted semantic features and the calculated similarity relationship; wherein the vertex set of the semantic graph corresponds to all texts in the incremental data set, the parameters of the vertex represent the semantic features of the corresponding text, and the edge set of the semantic graph is generated according to the similarity relationship.

[0072] For each incremental dataset, feature extraction is first performed. The text data in the incremental dataset (micro-chunk) is converted into numerical feature vectors that represent its semantic content. Feature extraction methods can use bag-of-words models, TF-IDF, word embeddings (such as Word2Vec and BERT), etc. Next, similarity calculation is performed. Based on the extracted semantic features, the similarity between each text in the incremental dataset is calculated. Similarity calculation can use methods such as cosine similarity, Euclidean distance, and Manhattan distance.

[0073] After obtaining similarity information, a corresponding semantic graph (SimGraph) is constructed based on the extracted semantic features and the calculated similarity relationships. The vertices of the semantic graph correspond to all texts in the incremental dataset, and the parameters of each vertex represent the semantic features of the corresponding text. The edges of the semantic graph are generated based on the similarity relationships between texts: if the similarity between two texts exceeds a preset threshold, an edge is established between them. In this way, the semantic graph intuitively reflects the similarity relationships between texts in the incremental dataset.

[0074] The above similarity algorithm can use Simhash algorithm. Figure 3 For example, taking the dynamic generation of online semantic graph based on Simhash as an example, the SimHash algorithm is used to extract features and calculate similarity of each text Doc1, Doc2, Doc3...Docm obtained from the incremental data set (Micro-Chunk). The SimHash algorithm maps the text into a 64-bit binary fingerprint to obtain the feature vectors W1, W2, W3...W of each text. m The Hamming distance is used to measure text similarity. When the Hamming distance (HD) between the SimHash values of two texts is ≤ 1, the two texts are considered similar. Based on the calculated similarity relationship, a semantic graph simG(V,E) is constructed, where the vertex set V corresponds to all texts in the dataset, and the edge set E is generated based on the similarity relationship: an edge e(i,j) is established between two texts i and j if and only if the SimHash Hamming distance between them is not greater than 1.

[0075] S1202: Use a depth-first search algorithm to identify all connected components in the semantic graph, each connected component forming an online cluster, and extract the centroid of each online cluster.

[0076] After obtaining the semantic graph, we perform online micro-cluster identification based on the semantic connectivity of the graph. We use the depth-first search algorithm (DFS) to identify all connected components in the semantic graph. Each connected component naturally forms an online cluster (CluOnline) C k , these online clusters reflect the local semantic structure of the text in the incremental dataset. In each online cluster C k In the process, we extract the centroid (C k ) as the representative of the cluster. The centroid can be the earliest published text in the cluster. The centroid will be used in the subsequent inter-cluster merging step. Figure 3 As shown, each online cluster Micro CluOnline includes the following information: centroid, insertion time, and feature. The centroids corresponding to Micro CluOnline1 and MicroCluOnline2 are Doc a and Doc z, respectively, and the corresponding features are w a 、w z . The texts of each online cluster are collected to obtain an online cluster text list (CluOnline doc list), which includes doc a, doc b, doc c, doc d, doc e, and doc f.

[0077] S1203 , for the centroid of each online cluster, a beam search is performed using a fuzzy query mechanism, and clusters are merged using a greedy strategy based on large cluster priority to obtain a clustering result.

[0078] After extracting the centroid of each online cluster, online cluster-historical cluster fusion is performed based on fuzzy retrieval. For the centroid of each online cluster, a beam search is performed using a fuzzy query mechanism. The fuzzy query mechanism allows the consideration of approximate matching relationships between texts during the merging process, thereby increasing the flexibility and accuracy of the merging. Based on the beam search, a greedy strategy based on large cluster priority is adopted for inter-cluster merging. This strategy tends to prioritize merging larger clusters because larger clusters usually contain more information and are more stable. During the merging process, the centroid of the newly merged cluster is continuously calculated and the clustering results are updated. The centroid of the old cluster is called the Old centroid, and the centroid of the new cluster is called the New centroid.

[0079] The above step S120 is not only applicable to the clustering processing of incremental data sets, but also can be seamlessly connected with historical clustering results. Through the greedy merging strategy, the dynamic update and maintenance of clustering results are realized, which not only ensures the computational efficiency but also maintains the semantic consistency across time windows. In addition, the present invention also combines a variety of advanced technologies such as a parallel-stream hybrid processing framework, an adaptive parallel scheduling mechanism for time series data streams, and an asynchronous global traceability method based on active class perception, which further improves the overall performance and practicality of the system. Experimental results show that the framework shows significant advantages in the semantic discovery and organization of cross-domain documents, and is particularly suitable for large-scale text clustering scenarios that require real-time processing and dynamic evolution. At the macro level, for multiple incremental data sets of different time slices To achieve parallel processing, each incremental data set can be assigned to an independent computing node for processing. Internally, the clustering process is performed in a strictly serial manner. This hybrid clustering approach improves system throughput through parallel processing between incremental datasets, while simultaneously employing a strict serial processing mechanism within a single incremental dataset to maximize the stability of clustering results. Furthermore, a hybrid indexing mechanism based on SimHash and ElasticSearch enables efficient similar text retrieval and category merging while maintaining clustering flexibility.

[0080] The present invention uses incremental clustering to process only newly added data sets each time, avoiding the need to recalculate the clustering results of the entire data set, thereby improving clustering efficiency. The method can flexibly handle ever-changing data sets and adapt to dynamic changes in data by continuously updating the clustering results. Through steps such as feature extraction, similarity calculation, semantic graph construction, and greedy merging, the method can maintain a high clustering quality. In particular, the fuzzy query mechanism and the greedy strategy based on large cluster priority help to maintain the stability and representativeness of the clusters during the merging process. The constructed semantic graph and extracted centroid can be used to support complex query operations, such as semantic-based similarity search, text classification, etc.

[0081] In an optional embodiment, the centroid of each online cluster in step S1203 is subjected to beam search using a fuzzy query mechanism, and clusters are merged using a greedy strategy based on large cluster priority to obtain a clustering result, including:

[0082] S12031. For the centroid of each online cluster, a beam search is performed using a fuzzy query mechanism to retrieve historical similar texts and their category identifiers that are similar to the centroid in the historical clustering results, and a corresponding candidate category set is formed based on the search results.

[0083] For the centroid of each online cluster, a fuzzy query mechanism (such as fuzzy matching, similarity calculation, etc.) is used to search for historical texts similar to the centroid in the historical clustering results to obtain a similar text list (similar doc list). Fuzzy query allows a certain degree of error, so that even if the match is not complete, relevant historical data can still be found. In the results of the fuzzy query, a beam search algorithm is used to limit the search space, retaining only the historical texts and their category identifiers (Cluster IDs) that are most similar to the centroid, such as Figure 3 in Beam search avoids search space explosion by maintaining a fixed-size candidate set. Based on the results of beam search, a candidate category set corresponding to each online cluster is formed. These sets contain category identifiers related to historically similar texts.

[0084] S12032. Use a greedy strategy based on large cluster priority to merge online clusters and historical clusters, and select the category with the most nodes in the category from the candidate category set as the current online cluster to obtain a clustering result.

[0085] After obtaining a set of candidate categories for each online cluster, a greedy strategy is used to merge clusters. The core idea of the greedy strategy is to select the currently optimal option at each step—that is, the option that appears to be optimal under the current state—in the hope of achieving a globally optimal (or near-global optimal) result. In the greedy strategy, larger clusters (i.e., clusters containing more nodes) are prioritized as historical clusters for merging. This is because large clusters typically contain more information and are more likely to represent stable categories. For each online cluster, the category with the most nodes within its candidate category set is selected as the current online cluster's category. This step ensures that online clusters are classified into the most similar and stable categories. After the above steps, the final clustering result is obtained, in which each online cluster is classified into a specific category.

[0086] Specifically, a greedy strategy based on large cluster priority is used to merge online clusters with historical clusters, and the category with the most nodes in the candidate category set is selected as the current online cluster. The process is as follows:

[0087] S210: Initialize first: create an empty cluster set clusters to store all online clusters, and create a candidate category set categories to which each cluster belongs to store the category to which each cluster belongs.

[0088] S220: For each online cluster that has not yet been assigned to a category, the categories set is traversed to find the cluster with the largest number of nodes, i.e., the largest cluster in size, as the historical cluster to be merged. The category of this historical cluster is obtained and used as the merged category. The current online cluster is assigned to this category, the merged category is assigned to all text data in the online cluster, and the number of nodes in the corresponding category in the candidate category set is updated.

[0089] S230, repeat step S220, continue to receive new text data, repeat the above steps to perform cluster merging and category attribution selection with historical clusters, until the stopping condition is met (such as reaching a preset time window, processing all input data, etc.).

[0090] S240: The cluster set "clusters" contains all the merged online clusters, each of which has a clear category. The clustering results are output, which may include information such as the text content, number of nodes, and category of each online cluster.

[0091] By merging online and historical clusters based on a greedy strategy that prioritizes large clusters, we can retain larger clusters while reducing the number of small clusters, thereby improving the overall quality of clustering. Selecting the category of the historical cluster with the most nodes ensures that each cluster has a clear category label, which facilitates subsequent analysis and processing. Furthermore, the amount of updated historical information is minimized, accelerating computation. Compared to other complex clustering algorithms, such as hierarchical clustering and spectral clustering, this algorithm has lower computational complexity and is suitable for processing large-scale online data. Setting a similarity threshold reduces unnecessary similarity calculations, further improving algorithm efficiency. This algorithm is adaptable to different data types and feature representations, simply by selecting an appropriate similarity calculation method. The algorithm is highly adaptable to the addition of new data and can update clustering results in real time.

[0092] Through a fuzzy query mechanism and beam search, the present invention can find online clusters that are most similar to historical data, thereby improving clustering accuracy. Furthermore, a greedy strategy that prioritizes large clusters ensures the stability of clustering results. The beam search algorithm limits the size of the search space, avoiding excessive consumption of computing resources. The greedy strategy reduces overall computational complexity by selecting the optimal solution at each step. This process can process online data (i.e., data arriving in real time) and dynamically classify it into appropriate categories, making it suitable for real-time clustering tasks. Due to the use of fuzzy query and a greedy strategy, the process can be easily scaled to large datasets while maintaining good performance and accuracy.

[0093] In an optional embodiment, the centroid of the online cluster is determined as follows:

[0094] In an online cluster, the text with the earliest publication time is selected as the centroid of the cluster. Specifically, in each online cluster, the earliest published text is found and recorded. This earliest published text is used as the centroid of the cluster. This selection ensures the temporal nature of traceability while reducing computational complexity.

[0095] By selecting the earliest published text as the centroid, the present invention can reflect the origin and early development of the online cluster, which helps to understand the formation background and initial content of the cluster. Moreover, selecting the earliest published text as the centroid makes calculation simpler and more direct, reducing the amount of calculation. In an online environment, text data is updated in real time. Selecting the earliest published text as the centroid can quickly determine the representative of each cluster, which is very beneficial for real-time analysis and processing. As the representative of the cluster, the content and characteristics of the centroid are easy to understand and explain. Selecting the earliest published text as the centroid makes its content and background easier to understand and accept.

[0096] In an optional embodiment, based on cluster analysis, the present invention introduces a dynamic activity class monitoring mechanism to continuously track active classes whose membership changes within the clustering results and obtain their attribution results. The core of this mechanism is to analyze the clustering results for each time window, determine the set of active classes, and calculate the active class attribution function to obtain the attribution results.

[0097] The introduction of the active class dynamic monitoring mechanism described in step S130 above, which continuously tracks the active classes whose number of members changes in the clustering results to obtain the tracing results, includes:

[0098] S1301 : For each time, determine the number of members of each cluster according to the clustering result of the time, and compare it with the number of members in the previous time window to determine the active class set at the time.

[0099] The present invention will be at a specific time The set of clustering results in which the number of members changes significantly is defined as the active cluster set , For All active classes at the moment Each active class Represents a topic of copywriting information that is currently receiving widespread attention. Definition of traceability results: The number of members in each time window is pre-stored in ES and can be obtained from ES in real time when generating the active class set.

[0100] in, is the source information function; is the time information function; is the propagation path function; is the propagation scale function.

[0101] The system regularly counts the number of members in each cluster and compares adjacent time windows (e.g. and ) The change in the number of members in a cluster. If the number of members in a cluster changes Exceeding the preset activity threshold , it is considered as an active class. After the active class is detected, the centroid and size of the online cluster are updated in real time.

[0102] For each time window, the number of members in each cluster is first determined based on the clustering results at that time. The number of members in the current time window is then compared with the number of members in the previous time window. If the number of members in a cluster changes significantly (increases or decreases), it is considered an active cluster and added to the set of active clusters for the current time window.

[0103] S1302. For each time, calculate an active class tracing function, where the active class tracing function outputs tracing results for all active classes in the active class set that meet the activity threshold; wherein the tracing results include source information, time information, propagation path, and propagation scale.

[0104] After determining the set of active classes, the active class traceability function is calculated to obtain the traceability results. The active class traceability function is a complex calculation process that takes into account multiple factors, including the source information, time information, propagation path, and scale of the active class. The function output is a set of traceability results for all active classes that meet the activity threshold.

[0105] The active class function tracing function can be defined as:

[0106]

[0107] in, For active class function tracing function, is the active class set, is the first in the active class set Active classes, For active class The traceability results, For time, For active class In time The number of members, For active class In time The number of members, is the activity threshold, is the time window.

[0108] Traceability results It can include multiple aspects of information, such as source information (i.e., the origin or initial formation of the active class), time information (including the time when the active class appeared, the duration of the active class, etc.), propagation path (i.e., the path of how the active class spreads or evolves over time), and propagation scale (i.e., the number of members or the size of the influence of the active class in different time windows).

[0109] For each class identified as active , the system performs the following traceability result calculation steps: source information acquisition, time information recording, propagation path construction and propagation scale assessment. Source information acquisition refers to Function, the system traces and determines the original source of the copy information. Time information recording refers to the use of Function, records the key time points when the copywriting information starts to spread from the source. Function, builds the propagation path of copywriting information from the source to the current active class. Propagation scale evaluation refers to the use of Function, to evaluate the scope and scale of influence of copywriting information during the dissemination process.

[0110] By introducing a dynamic monitoring mechanism for active classes, the present invention can monitor changes in the number of members in clustering results in real time and respond to these changes quickly. This helps to promptly discover and deal with potential hotspots or trends. The active class traceability function comprehensively considers multiple factors such as source information, time information, propagation path and propagation scale, thereby improving the accuracy of traceability. This helps to gain a deeper understanding of the formation causes and propagation process of active classes. By obtaining the traceability results of active classes, strong support can be provided for decision-making. For example, in the field of marketing, this information can be used to formulate targeted marketing strategies or adjust product directions. This mechanism can adapt to different data types and feature representations, and only requires appropriate adjustments to the active class traceability function. In addition, it can also be integrated with other algorithms or modules to form a complete processing flow.

[0111] The cross-domain online document tracing method proposed in the present invention involves adaptive parallel scheduling for time-series data streams, adaptive stream clustering based on dynamic semantic graphs, and an asynchronous global tracing framework based on active class perception. Through the adaptive parallel scheduling method for time-series data streams of the present invention, the computing resource allocation strategy can be dynamically adjusted according to the characteristics of the data stream, thereby improving the system processing capacity and resource utilization efficiency. This method adaptively performs task allocation and load balancing by sensing the timing characteristics and load changes of the data stream in real time, effectively solving the performance bottleneck problem of traditional fixed scheduling schemes when facing burst traffic.

[0112] Compared to traditional static clustering algorithms, this adaptive stream clustering method based on a dynamic semantic graph can update the semantic feature space in real time and dynamically adjust clustering results, significantly improving the versatility and real-time performance of semantic clustering. By constructing and maintaining a dynamic semantic graph, this method enables clustering results to adapt as text semantics evolve, overcoming the lag and inaccuracy of traditional clustering methods when processing dynamic text streams.

[0113] The proposed asynchronous global provenance framework, based on active class awareness, employs an asynchronous processing strategy that prioritizes active classes, significantly improving the system's response speed and processing efficiency. By monitoring and tracking the evolution of active classes in real time and prioritizing resource allocation for provenance analysis of key clusters, the framework effectively addresses the performance and real-time challenges of traditional approaches to large-scale text provenance. Its asynchronous design also ensures the system's scalability and fault tolerance.

[0114] In summary, this invention significantly improves the timeliness of online document traceability, enhances domain adaptability and portability, and enhances the system's scalability and stability. These technological innovations effectively address key issues in document traceability, such as efficiency bottlenecks, lack of universality, and resource scheduling, providing a comprehensive and reliable solution for large-scale document traceability applications.

[0115] The cross-domain document online tracing device provided by the present invention is described below. The cross-domain document online tracing device described below and the cross-domain document online tracing method described above can refer to each other.

[0116] The cross-domain copywriting online tracing device provided by the present invention refers to Figure 4 Shown, including:

[0117] The time slicing module 310 is used to time slice the acquired streaming incremental Internet copywriting information according to the data storage time to obtain data within each time slice, and extract the earliest uncalculated batch data from the data within the time slice to form an incremental data set;

[0118] The parallel copy aggregation module 320 is used to process the incremental data sets of different time slices in parallel, establish local semantic clusters in each incremental data set, and greedily merge them with the historical clustering results to obtain clustering results;

[0119] The global copywriting tracing module 330 is used to introduce a dynamic monitoring mechanism for active classes, continuously tracking active classes with a changed number of members in the clustering results to obtain tracing results;

[0120] The tracing result storage module 340 is used to store the clustering result and the tracing result in a distributed search engine.

[0121] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440. The processor 410, the communication interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the cross-domain document online tracing method.

[0122] Furthermore, the logic instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0123] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the cross-domain document online tracing method provided by the above methods.

[0124] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the cross-domain document online tracing method provided by the above methods.

[0125] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0126] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A cross-domain copywriting online tracing method, characterized by: include: The acquired streaming incremental internet copywriting information is time-sliced according to the data entry time to obtain the data within each time slice, and the earliest uncalculated batch data is extracted from the data within the time slice to form an incremental data set; Incremental datasets of different time slices are processed in parallel, local semantic clusters are established in each incremental dataset, and clustering results are obtained by greedy merging with historical clustering results. Introducing a dynamic monitoring mechanism for active classes to continuously track active classes with changed member numbers in the clustering results to obtain traceability results; The clustering results and the tracing results are stored in a distributed search engine.

2. The cross-domain copywriting online tracing method according to claim 1 is characterized in that: The above method establishes local semantic clusters in each incremental data set and greedily merges them with historical clustering results to obtain clustering results, including: For each incremental data set, feature extraction and similarity calculation are performed on the incremental data set, and a corresponding semantic graph is constructed based on the extracted semantic features and the calculated similarity relationships; wherein the vertex set of the semantic graph corresponds to all texts in the incremental data set, the parameters of the vertex represent the semantic features of the corresponding text, and the edge set of the semantic graph is generated according to the similarity relationships; A depth-first search algorithm is used to identify all connected components in the semantic graph. Each connected component forms an online cluster, and the centroid of each online cluster is extracted. For the centroid of each online cluster, a beam search is performed using a fuzzy query mechanism, and a greedy strategy based on large cluster priority is adopted to merge clusters to obtain the clustering results.

3. The cross-domain copywriting online tracing method according to claim 2 is characterized in that: For the centroid of each online cluster, a beam search is performed using a fuzzy query mechanism, and clusters are merged using a greedy strategy based on large cluster priority to obtain clustering results, including: For the centroid of each online cluster, a beam search is performed using a fuzzy query mechanism to retrieve historical similar texts and their category identifiers that are similar to the centroid in the historical clustering results, and a corresponding candidate category set is formed based on the retrieval results; A greedy strategy based on large cluster priority is used to merge online clusters and historical clusters, and the category with the most nodes in the candidate category set is selected as the current online cluster to obtain the clustering result.

4. The cross-domain copywriting online tracing method according to claim 2 is characterized in that: The centroid of an online cluster is determined as follows: In the online cluster, the text with the earliest publishing time is selected as the centroid of the online cluster.

5. The cross-domain copywriting online tracing method according to claim 1 is characterized in that: The active class dynamic monitoring mechanism is introduced to continuously track the active classes whose number of members in the clustering results has changed to obtain the traceability results, including: For each time, the number of members in each cluster is determined based on the clustering results at that time, and compared with the number of members in the previous time window to determine the active class set at that time; For each time, an active class tracing function is calculated, and the active class tracing function outputs the tracing results of all active classes in the active class set that meet the activity threshold; wherein the tracing results include source information, time information, propagation path and propagation scale.

6. The cross-domain copywriting online tracing method according to claim 5 is characterized in that: The active class function tracing function is: in, For active class function tracing function, is the active class set, is the first in the active class set Active classes, For active class The traceability results, For time, For active class In time The number of members, For active class In time The number of members, is the activity threshold, is the time window.

7. A cross-domain copywriting online tracing device, characterized in that: include: The time slicing module is used to time slice the acquired streaming incremental internet copywriting information according to the data storage time to obtain the data within each time slice, and extract the earliest uncalculated batch data from the data within the time slice to form an incremental data set; The parallel copy aggregation module is used to process incremental data sets of different time slices in parallel, establish local semantic clusters in each incremental data set, and greedily merge them with historical clustering results to obtain clustering results; A global copywriting traceability module is used to introduce a dynamic monitoring mechanism for active classes, continuously tracking active classes with changed member numbers in the clustering results to obtain traceability results; The tracing result storage module is used to store the clustering result and the tracing result in a distributed search engine.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the cross-domain copywriting online tracing method as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the cross-domain copywriting online tracing method as described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the cross-domain copywriting online tracing method as described in any one of claims 1 to 6 is implemented.

Images