Information deduplication method

Abstract

The invention discloses an information deduplication method, which belongs to the technical field of information processing, and comprises the following steps: acquiring original public opinion data sets related to a specified object from different media platforms; after data in the original public opinion data set is subjected to primary deduplication processing, residual text content obtained after primary deduplication processing is input into a preset rule engine, a secondary deduplication mode corresponding to the residual data is determined, the rule engine calculates current load information, the secondary deduplication mode is determined based on the result of the load information, and the current load information of the current data is calculated according to the secondary deduplication mode. Comprising the following steps: if current load information does not exceed a specified load, determining a secondary deduplication mode based on a source corresponding to residual data and a processing requirement type specified by a current user; and performing secondary de-duplication on the residual data based on the secondary de-duplication mode. Different modes are adopted for duplicate removal through the rule engine, so that the real-time performance of duplicate removal can be ensured, and the duplicate removal precision can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of information processing, and in particular to an information deduplication method. Background Art

[0002] Monitoring public opinion on public events and mastering public opinion information on a certain event is one of the very important goals for information control. When a public opinion event occurs, data retrieval and acquisition are usually achieved through big data technology, and the amount of data acquired is relatively large. Part of the reason for this is the problem of duplication of public opinion information.

[0003] In related technologies, deduplication of public opinion information is usually carried out by identifying semantic similarity, such as using NLP models for processing. This method usually results in high computational consumption, poor real-time performance, and inability to cope with the deduplication of variable data. Summary of the Invention

[0004] The main purpose of the present invention is to provide an information deduplication method to solve the deficiencies in the related art.

[0005] In order to achieve the above object, according to a first aspect of the present invention, a method for deduplication of information is provided, comprising obtaining an original public opinion dataset D={d1, d2, ..., d n}, where each data d i Contains text content T i , and metadata, wherein the metadata includes release time and source; after deduplication processing is performed on each piece of data in the original public opinion data set, the remaining text content after the deduplication processing is input into the preset rule engine to determine the secondary deduplication method corresponding to the remaining data, wherein the rule engine calculates the current load information and determines the secondary deduplication method based on the result of the load information, including if the current load information does not exceed the specified load, determining the secondary deduplication method based on the source corresponding to the remaining data and the processing requirement type specified by the current user; and performing secondary deduplication on the remaining data based on the secondary deduplication method.

[0006] Optionally, determining the secondary deduplication method based on the corresponding sources of the remaining data includes: for any two pieces of data to be deduplicated in the remaining data, if the corresponding sources are all of the first source type, determining the secondary deduplication method as the first processing method based on TF-IDF; if the corresponding sources are all of the second source type, determining the secondary deduplication method as the multi-sub-level second processing method; and / or, if the corresponding sources are respectively of the first source type and the second source type, determining the secondary deduplication method as the third processing method.

[0007] Optionally, if the current load information does not exceed the specified load, the secondary deduplication method is determined to be a TF-IDF-based processing method.

[0008] Optionally, performing secondary deduplication on the remaining data based on the secondary deduplication method includes: if the secondary deduplication method is a multi-sub-level second processing method, first using the first sub-level processing method to remove the text content T in each of the remaining data i Split the text into blocks of a specified length and calculate the hash value of each block; build an inverted index based on the hash value:

[0009] Among them, h k is the hash value of the kth text block, d i is the i-th public opinion data, k is the block in the text content T i The serial number in, k starts from 1, pl i is the character offset of the kth text block in the original text, pl i =(start k , end k ), start k The starting character position of the kth text block, which starts at 0, ends k The end character position of the k-th text block; based on the pre-built inverted index, compare the hash values ​​and block positions of different text blocks across the data to determine whether there is overlap between the text blocks across the data, determine whether there is overlap across the data based on the repeated text blocks, and perform deduplication operations on the overlapping data, wherein, based on the information of the repeated text blocks and the current threshold in the obtained cache, determine whether different data overlap.

[0010] Optionally, performing secondary deduplication on the remaining data based on the secondary deduplication method includes: if the secondary deduplication method is a multi-sub-level second processing method, performing second sub-level processing on the target data obtained by the first sub-level processing method: re-blocking the text content of each target data, wherein different blocking parameters are used to dynamically block the target data; extracting the semantic feature vector and keyword feature vector of the text block of each data obtained by the dynamic blocking; splicing the semantic feature vector and the keyword feature vector to obtain a splicing vector; calculating the position similarity of the text blocks between each data, and performing deduplication based on the position similarity and the similarity of the splicing vector.

[0011] Optionally, calculating the text block position similarity between each piece of data includes: based on Calculation, where sim pos The overlap of different text blocks in different pieces of data; For the d i The kth text block of the data; For the d j The mth text block of the data; K is the dth i The total number of text blocks of data; M is d j The total number of text blocks of data; is the indicator function, if the block and If there is overlap, the value is 1, otherwise it is 0; overlap is the overlap judgment function.

[0012] Optionally, performing deduplication based on the position similarity and the similarity of the splicing vectors includes: final (d i , d j )=γ·sim semantic (d i , d j )+(1-γ)·sin pos (d i , d j ) calculates the target similarity, where sim semantic is the semantic similarity; γ is the semantic weight coefficient; deduplication is performed based on the relationship between the target similarity and the specified threshold.

[0013] Optionally, the specified threshold is θ base is the basic threshold, μ sim is the mean cosine similarity of all text block pairs between different data, σ sim Based on μ sim The determined standard deviation; λ is the adjustment coefficient.

[0014] According to a second aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute any one of the methods described in the first aspect.

[0015] According to a third aspect of the present invention, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor executes the method described in any one implementation of the first aspect.

[0016] The information deduplication method of this embodiment includes obtaining the original public opinion dataset D={d1, d2, ..., d n}, where each data d i Contains text content T i, and metadata, wherein the metadata includes the release time and source; after deduplication processing is performed on each piece of data in the original public opinion data set, the remaining text content after the deduplication processing is input into the preset rule engine to determine the secondary deduplication method corresponding to the remaining data, wherein the rule engine calculates the current load information and determines the secondary deduplication method based on the result of the load information, including if the current load information does not exceed the specified load, determining the secondary deduplication method based on the source corresponding to the remaining data and the processing requirement type specified by the current user; and performing secondary deduplication on the remaining data based on the secondary deduplication method. Using different deduplication methods by the rule engine can ensure the real-time deduplication and improve the deduplication accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 is a flow chart of a data processing method according to an embodiment of the present invention;

[0019] Figure 2 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first," "second," and the like in the specification and claims of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate for the embodiments of the present invention described herein. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0022] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0023] According to an embodiment of the present invention, a method for deduplication of information is provided. Figure 1 ,include:

[0024] Step 101: Obtain the original public opinion dataset D = {d1, d2, ..., d n}, where each data d i Contains text content T i , and metadata, wherein the metadata includes release time and source.

[0025] In this step, in order to satisfy the user's targeted understanding and tracking of public opinion information for different events, an interactive interface can be provided to the user. After the user enters the specified object information in the interactive interface, the specified object information is sent to the server. The specified object can be specific event information configured by the user through the terminal interactive network page, including core keywords; or it can be a specified subject configured through the interactive network page, such as an enterprise or user. Multiple events can be associated with the same subject, and deduplication can be performed on each event within one or more events under a certain subject.

[0026] The server automatically obtains public opinion data from different media platforms (interfaces with these platforms have been established in advance). The data from different media platforms varies significantly, so it is possible to pre-integrate this multi-source heterogeneous data to obtain the raw public opinion data to be processed. The integration method is not limited here. This raw public opinion data includes multiple pieces of raw public opinion data for a particular event, and these multiple pieces of raw public opinion data constitute the raw public opinion dataset.

[0027] Each piece of data is defined as including text content and metadata. The metadata is defined as including the release event and source of the data. The release time and source are used to participate in the process of information deduplication.

[0028] Step 102: After deduplication of each piece of data in the original public opinion data set, the remaining text content after the deduplication process is input into a preset rule engine to determine the secondary deduplication method corresponding to the remaining data, wherein the rule engine calculates the current load information and determines the secondary deduplication method based on the result of the load information, including if the current load information does not exceed the specified load, determining the secondary deduplication method based on the corresponding source of the data and the processing requirement type specified by the current user.

[0029] Step 103: Perform secondary deduplication on the remaining data based on the secondary deduplication method.

[0030] In this step, data investigation revealed that, in public opinion information, whether it is the same media platform or different media platforms, there are usually situations such as reprinting, crawlers repeatedly crawling the same page due to network fluctuations, and storing the same content under different domain names. For the data in this situation, a global hash fingerprint of each data can be calculated, and directly compared to remove duplicates. By calculating the global unique identifier (such as hash value) of the data entry, duplicate data entries are identified and removed. For example, when performing a deduplication, a unique hash value such as MD5 is generated for the text content. If the text content is repeated, the MD5 hash value is the same. Furthermore, if the hash value is the same, the release time of the metadata corresponding to the data is eliminated. For example, if the difference exceeds a threshold (such as 1 day), the latest data can be selectively retained because the data may be updated, so the latest data is retained. If the difference does not exceed the threshold, it can be randomly eliminated.

[0031] Specifically, the following method is used to remove duplicates from different data text contents, and the data with a result of 1 is removed, and the data with a result of 0 is retained. During the deletion process, the publishing time information is retained. After a single deduplication process, completely duplicate content in the above specific situations can be quickly filtered out, significantly improving system efficiency and reducing costs.

[0032] Furthermore, after the first deduplication process, duplicate data may still exist, and the text content of the remaining data is input into the rule engine to determine a second deduplication method.

[0033] As an optional implementation of this embodiment, if the current load information does not exceed the specified load, the secondary deduplication method is determined to be a TF-IDF-based processing method.

[0034] The rule engine can calculate the current load in real time This is the ratio of the current number of tasks to the set total resource threshold. If the load exceeds the specified value, TF-IDF processing is forced to be used, which can greatly improve processing efficiency. If the load does not exceed the specified value, the deduplication processing method is determined based on the source in the metadata of the text content.

[0035] The TF-IDF-based processing method includes vectorization, similarity calculation, and threshold determination stages, which are not limited here.

[0036] In this optional implementation, the load adaptation strategy achieves the optimal balance between efficiency, accuracy, and resource consumption by dynamically switching deduplication algorithms. For high-load scenarios such as bursty traffic and data spikes, a TF-IDF-based processing approach is used. When the monitored load rate exceeds a preset value, such as 80%, the system automatically switches to TF-IDF mode.

[0037] As an optional implementation method of this embodiment, determining the secondary deduplication method based on the corresponding sources of the remaining data and the processing requirement type specified by the current user includes: for any two pieces of data to be deduplicated in the remaining data, if the corresponding sources are all of the first source type, determining the secondary deduplication method as a TF-IDF-based processing method; if the corresponding sources are all of the second source type, determining the secondary deduplication method as a multi-sub-level second processing method; and / or, if the corresponding sources are respectively of the first source type and the second source type, determining the secondary deduplication method as a layered deduplication method.

[0038] In this optional implementation, if any two sources to be deduplicated are from social media, which typically consists of mainly short text, a TF-IDF-based processing approach is used. If the source is from a news website, which typically consists of mainly long text, a multi-sublevel processing approach is used.

[0039] If the types are social media, news websites, etc., then the third processing method is adopted. For example, key information such as core keywords and paragraphs are extracted from the long text of the news website type, and the core elements of the short text are extracted, such as keywords. Then, a layered method is used to remove duplicates, including determining keyword coverage based on the ratio of the same keywords to the maximum number of keywords (short text or long text); determining the semantic similarity between the short text and each core paragraph, taking the maximum value as the semantic similarity value; determining the semantic role matching degree, annotating the short text with semantic roles, and annotating it as the semantic information of the structured representation sentence, such as triple structured data, and then matching the triple structured data in the long text. The ratio of the number of matched triple structured data to the triple structured data of the short text is the semantic role matching degree. Different weight coefficients are set for the three to obtain a final score. Based on the score and the set threshold size, it is determined whether the data is repeated. If it is repeated, it is removed.

[0040] As an optional implementation of this embodiment, performing secondary deduplication on the remaining data based on the secondary deduplication method includes: if the secondary deduplication method is a multi-sub-level second processing method, first using the first sub-level processing method to remove the text content T in each of the remaining data i Split into text blocks of specified length and calculate the hash value of each text block; build an inverted index based on the hash value Among them, h k is the hash value of the kth text block, d i is the i-th public opinion data, k is the block in the text content T i The serial number in, k starts from 1, pl i is the character offset of the kth text block in the original text, pl i=(start k , end k ), start k The starting character position of the kth text block, which starts at 0, ends k The end character position of the kth text block; based on the pre-built inverted index, compare the hash values ​​and block positions of different text blocks across the data to determine whether there is overlap between the text blocks across the data, determine whether there is overlap between different data based on the repeated text blocks, and perform deduplication operations on the overlapping data, wherein, based on the information of the repeated text blocks and the current threshold in the obtained cache, determine whether there is overlap between different data.

[0041] In this optional implementation, the length L is used as the block length to calculate the hash value of each block. SimHash is preferably used to generate a 64-bit collision-resistant hash fingerprint. The hash value of any block is:

[0042] Hash(B i，k )=SimHash(T norm [kL:(k+1)L]).

[0043] You can build an inverted index, using hash values ​​as keys to record the text and location of the blocks:

[0044] Query the hash value h in the inverted index for each block k The corresponding entry set, if there is an entry (d j ,m,pl j ) hash value and (d i ,k,pl i ) are the same, and That is, the block positions overlap, then it is determined that any two data d i ,d j There are duplicate chunks.

[0045] Furthermore, when determining whether different data overlap based on the information of the repeated text blocks and the current threshold in the obtained cache, the method includes: calculating the repetition degree of the two data based on the repeated blocks, where repetition degree = the number of repeated blocks / min (data d j number of blocks); get the threshold θ in the current cache repeat , perform deduplication based on the repetition degree and the threshold value. For example, if the repetition degree is greater than the threshold θ repeat , deduplication is performed to retain the most recently published data. Using an inverted index improves deduplication efficiency, overcoming the inefficiency of one-to-one hash value comparisons. Deduplication based on repetition and dynamic thresholds also enhances deduplication effectiveness, making it more accurate and reducing false negatives and missed detections.

[0046] The threshold can be dynamically adjusted based on experience, θ repeat =γ·θ initial +(1-γ)·(α,θ base ), where γ is the initial value weight coefficient, θ initial is the initial threshold, which can be preset based on historical data or field experience, α is the empirical sensitivity coefficient, θ base is a constant value. Through the deduplication effect of historical deduplication data, the empirical sensitivity coefficient α is dynamically adjusted, and then the deduplication accuracy is adjusted to improve the deduplication accuracy. When dynamically adjusting the empirical sensitivity coefficient, it can be increased or decreased according to a specified step size such as 0.01 based on the historical deduplication effect index. The false kill rate FP and / or the missed kill rate FN can be used as the deduplication effect index. This index can be calculated based on the historical deduplication data. The empirical sensitivity coefficient can be adjusted by this index. For example, increasing it can reduce the false kill rate, and lowering it can reduce the missed kill rate. By adjusting this value, the missed kill rate and the false kill rate can be balanced, thereby improving the deduplication accuracy.

[0047] As an optional implementation method of this embodiment, performing secondary deduplication on the remaining data based on the secondary deduplication method includes: if the secondary deduplication method is a multi-sub-level second processing method, performing second sub-level processing on the target data obtained by the first sub-level processing method: re-blocking the text content of each target data, wherein different blocking parameters are used to dynamically block the target data; extracting the semantic feature vector and keyword feature vector of the text block of each data obtained by the dynamic blocking; splicing the semantic feature vector and the keyword feature vector to obtain a splicing vector; calculating the position similarity of the text blocks between each data, and performing deduplication based on the position similarity and the similarity of the splicing vector.

[0048] In this optional implementation, after the first sub-level deduplication process, the remaining data undergoes a second sub-level process, in which complex repetitions such as rewriting and synonymous substitutions are identified. When performing dynamic chunking, chunking is performed based on a sliding window, for example, with a window size of ω and a step size of δ, to extract semantic chunks. For sentences with clear sentence boundaries, the window size is determined by punctuation, resulting in a window size based on sentence length. For sentences with no sentence boundaries, the window size can be fixed.

[0049] Furthermore, semantic feature vectors and keyword feature vectors are extracted based on the blocks and then spliced ​​together. By integrating domain-adaptive BERT and TF-IDF statistical features, the block vector not only retains deep semantic information but also explicitly strengthens the keyword weight, significantly improving its adaptability to texts in the public opinion field.

[0050] Calculating the similarity of text block positions between each piece of data includes: Calculation, where sim pos The overlap of different text blocks in different pieces of data; For the d i The kth text block of the data; For the d j The mth text block of the data; K is the dth i The total number of text blocks of data; M is d j The total number of text blocks of data; is the indicator function, if the block and If there is overlap, the value is 1, otherwise it is 0; overlap is the overlap judgment function.

[0051] In this optional implementation, position similarity is used to measure the degree of overlap between the positions of two document blocks. pos The value ranges from 0 to 1. The specific rule definition of the overlap judgment function overlap(pl1,pl2): Assume that the block position is represented by a character interval (for example, the start and end position of the block):

[0052] pl1=[s1,e1], pl2=[s2,e2]

[0053] Where s is the starting position, e is the ending position, and the overlapping condition is:

[0054] A value of 1 indicates that the intervals intersect. The numerator in the above formula is used as an identifier. All K × M combinations of blocks from the two data points are traversed, and the number of block pairs that meet the overlap condition is counted. Normalization is performed using max(K, M) to avoid biasing similarity towards data with more blocks due to differences in the number of blocks. A value of 0 indicates no overlap; otherwise, there is overlap.

[0055] As an optional implementation of this embodiment, deduplication based on the position similarity and the similarity of the splicing vector includes:

[0056] sim final (d i ,d j )=γ·sim semantic (d i ,d j )+(1-γ)·sim pos (d i ,d j ) calculates the target similarity, where sim semantic is the semantic similarity; γ is the semantic weight coefficient; deduplication is performed based on the relationship between the target similarity and the specified threshold.

[0057] In this optional implementation, the semantic similarity is determined by cosine similarity. After the semantic similarity is determined, a deduplication operation is performed.

[0058] Specifically, assuming that the data d i Contains block {A1,A2}, data d j Contains blocks {B1, B2, B3}, calculates the similarity values ​​of all cross-data text block pairs, the total number of block pairs N pairs For data d i The number of blocks multiplied by the data d j The number of blocks, such as 2*3 is 6, and the similarity set S is obtained by the similarity between different blocks, such as sim(A1,B1), sim(A1,B2), etc. The average similarity is If μ sim A high value indicates that there are a large number of similar text blocks. sim Increase, through the above specified threshold calculation formula The threshold can be dynamically increased to allow only text blocks with extremely high similarity to be judged as duplicates, otherwise they will not be judged as duplicates; μ sim By using the above-mentioned first specified threshold calculation formula, the threshold can be dynamically lowered to avoid missed detection. sim Increased, indicating that the similarity is more dispersed, indicating that there are highly similar and completely dissimilar data blocks in the blocks. If σ sim A low value indicates that the similarity of the data blocks is concentrated. The above-mentioned specified threshold can balance the two, and then a more accurate and objective specified threshold can be determined. This precise first specified threshold lays the foundation for improving the deduplication accuracy.

[0059] If the target similarity is greater than a specified threshold, the block is determined to be duplicated; otherwise, the block is not duplicated. The duplication between the data can be further determined based on the aforementioned duplication calculation formula. If the duplication is greater than a second threshold (which can be preset), the two data are determined to be duplicates and deduplication is performed; otherwise, they are not duplicates and deduplication is not performed.

[0060] Through multi-level deduplication, it is possible to achieve the greatest possible accuracy in deduplication, avoid missed or incorrect deduplication, and improve deduplication efficiency, ensuring real-time deduplication.

[0061] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0062] According to an embodiment of the present invention, a public opinion information deduplication device is also provided, comprising an acquisition unit, which acquires an original public opinion data set D={d1, d2, ..., d n}, where each data d i Contains text content T i , and metadata, wherein the metadata includes release time and source; a deduplication unit, configured to perform a deduplication process on each piece of data in the original public opinion data set, input the remaining text content after the deduplication process into a preset rule engine, and determine a secondary deduplication method corresponding to the remaining data, wherein the rule engine calculates current load information and determines a secondary deduplication method based on the result of the load information, including, if the current load information does not exceed a specified load, determining a secondary deduplication method based on the source corresponding to the remaining data and the processing requirement type specified by the current user;

[0063] The remaining data is deduplicated for the second time based on the second deduplication method.

[0064] According to an embodiment of the present invention, the present invention also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the method described in any of the above embodiments when executing.

[0065] According to an embodiment of the present invention, the present invention further provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the method described in any of the above embodiments when executed.

[0066] According to an embodiment of the present invention, the present invention further provides a computer program product, which can implement the method described in any of the above embodiments when executed by a processor.

[0067] Figure 2 A schematic block diagram of an example electronic device 300 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0068] like Figure 2As shown, the electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. Various programs and data required for the operation of the electronic device 300 can also be stored in the RAM 303. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0069] Multiple components in the electronic device 300 are connected to the I / O interface 305, including an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a magnetic disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0070] The computing unit 301 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above, such as the object matching method. For example, in some embodiments, the object matching method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 308. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the method described above can be performed.

[0071] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0072] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0073] In the context of the present invention, machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

Claims

1. A method for deduplication of information, characterized in that: include: Obtain the original public opinion dataset D = {d1, d2, ..., d n }, where each data d i Contains text content T i , and metadata, wherein the metadata includes release time and source; After performing a primary deduplication process on each piece of data in the original public opinion data set, the remaining text content after the primary deduplication process is input into a preset rule engine to determine a secondary deduplication method corresponding to the remaining data, wherein the rule engine calculates current load information and determines a secondary deduplication method based on the result of the load information, including, if the current load information does not exceed a specified load, determining a secondary deduplication method based on the source corresponding to the remaining data and the processing requirement type specified by the current user; The remaining data is deduplicated for the second time based on the second deduplication method.

2. The information deduplication method according to claim 1, characterized in that: The secondary deduplication methods based on the corresponding sources of the remaining data include: For any two pieces of data to be deduplicated in the remaining data, if the corresponding sources are both of the first source type, the second deduplication method is determined to be the first processing method based on TF-IDF; If the corresponding sources are all of the second source type, determining the secondary deduplication method to be the second multi-sub-level processing method; And / or, if the corresponding sources are respectively the first information source type and the second information source type, the secondary deduplication method is determined to be the third processing method.

3. The information deduplication method according to claim 2, characterized in that: If the current load information does not exceed the specified load, the secondary deduplication method is determined to be a TF-IDF-based processing method.

4. The information deduplication method according to claim 2, characterized in that: Performing secondary deduplication on the remaining data based on the secondary deduplication method includes: If the secondary deduplication method is the second processing method of multiple sub-levels, the first sub-level processing method is first used to remove the text content T in the remaining data. i Split into text blocks of specified length and calculate the hash value of each text block; Build an inverted index based on the hash value: Among them, h k is the hash value of the kth text block, d i is the i-th public opinion data, k is the block in the text content T i The serial number in, k starts from 1, pl i is the character offset of the kth text block in the original text, pl i =(start k , end k ), start k The starting character position of the kth text block, which starts at 0, ends k The ending character position of the kth text block; Based on the pre-built inverted index, the hash values ​​and block positions of different text blocks across the data are compared to determine whether there is overlap between the text blocks across the data. Based on the repeated text blocks, whether there is overlap across the data is determined, and deduplication operations are performed on the overlapping data. Among them, whether there is overlap between different data is determined based on the information of the repeated text blocks and the current threshold in the obtained cache.

5. The information deduplication method according to claim 4, characterized in that: Performing secondary deduplication on the remaining data based on the secondary deduplication method includes: If the secondary deduplication method is the multi-sub-level second processing method, the target data obtained by the first sub-level processing method is processed by the second sub-level processing method: Re-blocking the text content of each target data, wherein the target data is dynamically blocked using different blocking parameters; Extracting semantic feature vectors and keyword feature vectors of the text block of each piece of data obtained by the dynamic segmentation; Splicing the semantic feature vector and the keyword feature vector to obtain a spliced ​​vector; The position similarity of the text blocks between the pieces of data is calculated, and duplicates are removed based on the position similarity and the similarity of the splicing vectors.

6. The information deduplication method according to claim 5, characterized in that: Calculating the similarity of text block positions between each piece of data includes: based on Calculation, where sim pos The overlap of different text blocks in different pieces of data; For the d i The kth text block of the data; For the d j The mth text block of the data; K is the dth i The total number of text blocks of data; M is d j The total number of text blocks of data; is the indicator function, if the block and If there is overlap, the value is 1, otherwise it is 0; overlap is the overlap judgment function.

7. The information deduplication method according to claim 6, characterized in that: Deduplication based on the position similarity and the similarity of the splicing vectors includes: sim-based final (d i ,d j )=γ·sim semantic (d i ,d j )+(1-γ)·sim pos (d i ,d j ) calculates the target similarity, where sim semantic is the semantic similarity; γ is the semantic weight coefficient; Deduplication is performed based on the magnitude relationship between the target similarity and a specified threshold.

8. The information deduplication method according to claim 7, characterized in that: The specified threshold is θ base is the basic threshold, μ sim is the mean cosine similarity of all text block pairs between different data, σ sim Based on μ sim The determined standard deviation; λ is the adjustment coefficient.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor executes the method according to any one of claims 1 to 8.

Images