Information deduplication method

Abstract

The application discloses an information deduplication method, and belongs to the technical field of information processing. The method comprises the following steps: obtaining original public opinion data sets related to a specified object from different media platforms; performing a first deduplication processing on each piece of data in the original public opinion data sets, and inputting the remaining text content after the first deduplication processing into a preset rule engine to determine a secondary deduplication mode corresponding to the remaining data, wherein the rule engine calculates current load information, determines the secondary deduplication mode based on the result of the load information, and the secondary deduplication mode is determined based on the source corresponding to the remaining data if the current load information does not exceed a specified load; and performing secondary deduplication on the remaining data based on the secondary deduplication mode. Different deduplication modes are adopted by the rule engine, so that the real-time performance of deduplication can be ensured and the deduplication accuracy can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information processing, and particularly relates to an information deduplication method. BACKGROUND

[0002] Monitoring public opinion of public events and grasping public opinion information of an event are one of the important purposes for information management and control. After the occurrence of an opinion event, data retrieval and acquisition are usually realized through big data technology, and the amount of acquired data is relatively large. One of the reasons is the existence of duplicate opinion information.

[0003] In related technologies, opinion information deduplication is usually identified by semantic similarity judgment, such as using an NLP model for processing. This method usually brings about large computational consumption, poor real-time performance, and cannot cope with variable data deduplication situations. SUMMARY

[0004] The main purpose of the present application is to provide an information deduplication method to solve the problems in related technologies.

[0005] In order to achieve the above purpose, according to the first aspect of the present application, an information deduplication method is provided, which comprises obtaining an original public opinion data set related to a specified object from different media platforms , wherein each data contains text content and metadata, wherein the metadata includes the publication time and the source; after one deduplication processing of each data in the original public opinion data set, the remaining text content after the one deduplication processing is input into a preset rule engine to determine the secondary deduplication mode corresponding to the remaining data, wherein the rule engine calculates the current load information, determines the secondary deduplication mode based on the result of the load information, including if the current load information does not exceed the specified load, determining the secondary deduplication mode based on the source corresponding to the remaining data; and performing secondary deduplication on the remaining data based on the secondary deduplication mode.

[0006] Optionally, determining the secondary deduplication mode based on the source corresponding to the remaining data comprises: for any two pieces of data to be deduplicated in the remaining data, if the corresponding sources are both of a first source type, determining the secondary deduplication mode as a first processing mode based on TF-IDF; if the corresponding sources are both of a second source type, determining the secondary deduplication mode as a second processing mode of multiple sub-levels; and / or if the corresponding sources are of the first source type and the second source type respectively, determining the secondary deduplication mode as a third processing mode.

[0007] Optionally, if the current load information does not exceed the specified load, the secondary deduplication mode is determined as a processing mode based on TF-IDF.

[0008] Optionally, performing secondary deduplication on the remaining data based on the secondary deduplication method includes: if the secondary deduplication method is a multi-level second processing method, then the first-level processing method is first used to deduplicate the text content in each of the remaining data entries. The text is divided into blocks of a specified length, and the hash value of each text block is calculated; an inverted index is constructed based on the hash values. ,in, The hash value of the k-th text block. Let k be the i-th piece of public opinion data, and k be the number of blocks in the text content. The sequence number in the sequence, k starts from 1, Let pl be the character offset of the k-th text block in the original text. i = (start k end k ), start k The starting character position of the k-th text block, which starts from 0, is end. k The end character position of the k-th text block; based on the pre-built inverted index, compare the hash value and block position of different text blocks across data to determine whether there is overlap between text blocks across data. Based on the duplicate text blocks, determine whether there is overlap across data and perform deduplication operation on the overlapping data. Among them, the information of the duplicate text blocks and the current threshold obtained in the cache determine whether there is overlap between different data.

[0009] Optionally, performing secondary deduplication on the remaining data based on the secondary deduplication method includes: if the secondary deduplication method is a multi-sub-level second processing method, then performing second-sub-level processing on the target data obtained by the first sub-level processing method: re-segmenting the text content of each target data item, wherein different segmentation parameters are used to dynamically segment the target data; extracting the semantic feature vector and keyword feature vector of the text block of each data item obtained by the dynamic segmentation; concatenating the semantic feature vector and the keyword feature vector to obtain a concatenated vector; calculating the text block position similarity between each data item, and performing deduplication based on the position similarity and the similarity of the concatenated vector.

[0010] Optionally, calculating the text block position similarity between data points includes: based on

[0011] Calculate, where, The degree of overlap between different text blocks in different data; For the first The k-th text block of the data; For the first The m-th text block of the data; For the first total number of text blocks of the data; for total number of text blocks of the data; is an indicator function, and if the blocks and overlap, the value is 1, otherwise 0; overlap is an overlap judgment function.

[0012] Optionally, the deduplication based on the position similarity and the similarity of the concatenation vectors comprises: calculating a target similarity based on , wherein is a semantic similarity; is a semantic weight coefficient; and the deduplication is performed based on a size relationship between the target similarity and a specified threshold.

[0013] Optionally, the specified threshold is , is a basic threshold, is an average value of cosine similarities of all text block pairs between different data, is a standard deviation determined based on ; and is an adjustment coefficient.

[0014] According to a second aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for causing a computer to execute the method of any one of the first aspect.

[0015] According to a third aspect of the present application, an electronic device is provided, which comprises at least one processor and a memory connected in communication with the at least one processor; wherein the memory stores computer programs executable by the at least one processor, and the computer programs are executed by the at least one processor to cause the at least one processor to execute the method of any one of the implementation manners of the first aspect.

[0016] The information deduplication method of the embodiment comprises: acquiring an original public opinion data set related to a specified object from different media platforms , wherein each data contains text content and metadata, wherein the metadata comprises a publishing time and a source; after one-time deduplication processing of each piece of data in the original public opinion data set, inputting remaining text content after the one-time deduplication processing into a preset rule engine to determine a secondary deduplication manner corresponding to the remaining data, wherein the rule engine calculates current load information, and determines the secondary deduplication manner based on a result of the load information, including determining the secondary deduplication manner based on the source corresponding to the remaining data if the current load information does not exceed a specified load; and performing secondary deduplication on the remaining data based on the secondary deduplication manner. Different manners of deduplication are adopted through the rule engine, which can ensure the real-time performance of deduplication and improve the deduplication accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the specific embodiments or the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0018] Figure 1 is a data processing method flowchart of an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the technical personnel in the art better understand the present application, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort should be within the scope of protection of the present application.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0022] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0023] According to an embodiment of the present application, an information deduplication method is provided, referring to Figure 1 , comprising:

[0024] Step 101: Obtain a set of original public opinion data related to a specified object from different media platforms , wherein each piece of data contains text content and metadata, wherein the metadata includes the time of publication and the source.

[0025] In this step, in order to meet the user's understanding and tracking of public opinion information of different events, an interactive interface can be provided to the user, and when the user inputs the specified object information in the interactive interface, the specified object information is sent to the server. The specified object can be specific event information configured by the user through the terminal interactive web page, including the core keyword; or it can be a specified subject configured through the interactive web page, such as an enterprise, a user, etc. There can be multiple events associated with the same subject, and each event in one or more events of a certain subject can be deduplicated.

[0026] The server automatically obtains public opinion data from different media platforms (interfaces are pre-established with different media platforms). The data from different media platforms is quite different, so the multi-source heterogeneous data can be integrated in advance to obtain the original public opinion data to be processed, and the integration method is not limited herein. The original public opinion data includes multiple pieces of original public opinion data for a certain event, and the multiple pieces of original public opinion data form an original public opinion data set.

[0027] Each piece of data is defined to include text content and metadata, and the metadata is defined to include the publication event and source of the data, the time of publication and the source are used in the process of participating in information deduplication.

[0028] Step 102: After one-time deduplication processing of each piece of data in the original public opinion data set, input the remaining text content after one-time deduplication processing into a pre-set rule engine to determine the secondary deduplication mode corresponding to the remaining data, wherein the rule engine calculates the current load information, and determines the secondary deduplication mode based on the result of the load information, including if the current load information does not exceed a specified load, determining the secondary deduplication mode based on the source corresponding to the data.

[0029] Step 103: Secondary deduplication of the remaining data based on the secondary deduplication mode.

[0030] In this step, it is found through data investigation that, whether in the same media platform or different media platforms, there are usually situations such as reprinting, repeated crawling of the same page due to network fluctuations, and the same content stored under different domain names. For data in this situation, the global hash fingerprint of each piece of data can be calculated and directly compared for deduplication. By calculating the globally unique identifier (such as a hash value) of the data entry, duplicate content data entries are identified and removed. For example, when performing deduplication once, a unique hash value such as MD5 is generated for the text content. If the text content is duplicated, the MD5 hash values are the same. Further, if the hash values are the same, the publication time of the metadata corresponding to the data is removed, such as a difference of more than a threshold value (such as 1 day). The latest data can be selectively retained because the data may be updated. If the difference does not exceed the threshold value, the data can be randomly removed.

[0031] Specifically, the following method is used to deduplicate the text content of different data entries. Data with a result of 1 is removed, and data with a result of 0 is retained. When removing, the retention is based on the publication time information. After one-time deduplication processing, the completely duplicated content in the above-mentioned specific situation can be quickly screened out, significantly improving system efficiency and reducing costs.

[0032] Further, after one-time deduplication processing, there may still be duplicate data. The text content of the remaining data is input into a rule engine to determine a secondary deduplication method.

[0033] As an optional implementation manner of the embodiment, if the current load information does not exceed the specified load, the secondary deduplication method is determined to be a processing method based on TF-IDF.

[0034] The rule engine can calculate the current load in real time , that is, the ratio of the current task number to the set total resource threshold value. If the load exceeds the specified value, the processing method based on TF-IDF is forced to be used. Through this method, the processing efficiency can be greatly improved. If the load does not exceed the specified value, the deduplication processing method is determined based on the source in the metadata of the text content.

[0035] The processing method based on TF-IDF includes a vectorization process, a similarity calculation, and a threshold determination stage, which are not limited here.

[0036] In this optional implementation manner, the load adaptive strategy dynamically switches the deduplication algorithm to achieve an optimal balance of efficiency, accuracy, and resource consumption. For high-load scenarios such as burst traffic and data flood, the processing method based on TF-IDF is used. When the load rate is greater than a preset value, such as 80%, the system automatically switches to the TF-IDF mode.

[0037] As an optional implementation manner of the embodiment, the determining the secondary deduplication manner based on the corresponding sources of the residual data comprises: for any two pieces of data to be deduplicated in the residual data, if the corresponding sources are both the first source type, the secondary deduplication manner is determined as the TF-IDF-based processing manner; if the corresponding sources are both the second source type, the secondary deduplication manner is determined as the second processing manner of multiple sub-levels; and / or if the corresponding sources are the first source type and the second source type respectively, the secondary deduplication manner is determined as the hierarchical deduplication.

[0038] In the optional implementation manner, if the source types of any two pieces of data to be deduplicated are both the social media type, the text of the social media type is usually short text, and the TF-IDF-based processing manner is adopted. If the source is a news website or the like which usually includes long text, the multiple sub-level processing is adopted.

[0039] If the source types are the social media type and the news website type respectively, the third processing manner is adopted. For example, for the long text of the news website type, the key information such as core keywords and paragraphs is extracted, and for the short text, the core elements are extracted such as keywords, and then the hierarchical manner is adopted for deduplication, including determining the keyword coverage rate based on the ratio of the same keywords to the maximum number of keywords (short text or long text), determining the semantic similarity of the short text and each core paragraph, and taking the maximum value as the semantic similarity value, determining the semantic role matching degree, and marking the semantic role of the short text as the semantic information of the structured sentence such as the triple structured data, and then matching the triple structured data in the long text, and the ratio of the matched triple structured data to the triple structured data possessed by the short text is the semantic role matching degree. Different weight coefficients are set for the three, and the final score can be obtained, and based on the score and the set threshold size, it is determined whether the data is duplicated, and if duplicated, the data is removed.

[0040] As an optional implementation manner of the embodiment, the secondary deduplication of the residual data based on the secondary deduplication manner comprises: if the secondary deduplication manner is the second processing manner of multiple sub-levels, the text content of each piece of residual data is first divided into text blocks of a specified length, and the hash value of each text block is calculated; and an inverted index is constructed based on the hash values , wherein, is the hash value of the kth text block, is the i-th public opinion data, k is the sequence number of the block in the text content , k starts from 1, is the character offset of the kth text block in the original text, pl i = (start k , end​k ), start k start character position of the kth text block, which starts from 0, end k end character position of the kth text block; based on the pre-constructed inverted index, the hash values and the block positions of different text blocks across data are compared to determine whether there is overlap between the text blocks across the data, whether there is overlap between different data is determined based on the repeated text blocks, and a deduplication operation is performed on the overlapping data, wherein whether there is overlap between different data is determined based on the information of the repeated text blocks and the current threshold value obtained from the cache.

[0041] In this optional implementation, the hash value of each block is calculated with a block length L, and a 64-bit collision-resistant hash fingerprint is generated by using SimHash first. The hash value of any block is:

[0042] .

[0043] An inverted index can be constructed to record the text and position of the block with the hash value as the key: For each block, the hash value is queried in the inverted index The corresponding entry set, if there is an entry The hash value is the same as , and , that is, the block positions overlap, it is determined that any two pieces of data , exist repeated blocks.

[0044] Further, when determining whether different data overlap based on the information of the repeated text blocks and the current threshold value obtained from the cache, the following steps are included: obtaining the repetition degree of the two pieces of data based on the repeated blocks, the repetition degree = the number of repeated blocks / min(data block number); obtaining the threshold value in the current cache , and performing a deduplication operation based on the size of the repetition degree and the threshold value. For example, if the repetition degree is greater than the threshold value , a deduplication operation is performed, and the data with the latest publishing time can be retained. The inverted index method can improve the deduplication efficiency and overcome the low efficiency of one-to-one comparison of hash values. At the same time, performing a deduplication operation based on the repetition degree and the dynamic threshold value can improve the deduplication effect, make the deduplication more accurate, and reduce false positives and false negatives.

[0045] The threshold value can be dynamically adjusted based on experience values, wherein, the initial value weight coefficient, is the initial threshold value, which can be preset based on historical data or field experience, the experience sensitivity coefficient, This is a constant value. The empirical sensitivity coefficient is dynamically adjusted based on the deduplication results of historical deduplication data. This allows for adjustments to the deduplication accuracy, improving it. When dynamically adjusting the empirical sensitivity coefficient, it can be adjusted up or down in specified increments, such as 0.01, based on historical deduplication performance indicators. False positive rate (FP) and / or false negative rate (FN) can be used as deduplication performance indicators. These indicators can be calculated from historical deduplication data. Adjusting the empirical sensitivity coefficient using these indicators—for example, increasing it to reduce the false positive rate and decreasing it to reduce the false negative rate—can balance the false positive and false negative rates, thereby improving deduplication accuracy.

[0046] As an optional implementation of this embodiment, the secondary deduplication of the remaining data based on the secondary deduplication method includes: if the secondary deduplication method is a multi-sub-level second processing method, then the target data obtained by the first sub-level processing method is processed by the second sub-level processing method: the text content of each target data is re-blocked, wherein different blocking parameters are used to dynamically block the target data; the semantic feature vector and keyword feature vector of the text block of each data obtained by the dynamic blocking are extracted; the semantic feature vector and the keyword feature vector are concatenated to obtain a concatenated vector; the text block position similarity between each data is calculated, and deduplication is performed based on the position similarity and the similarity of the concatenated vector.

[0047] In this optional implementation, after the first-level deduplication process, the remaining data is processed in the second-level process, which identifies complex repetitions such as rewriting and synonym replacement.

[0048] When performing dynamic chunking, chunking is implemented based on a sliding window, such as window size ω and step size δ, to extract semantic chunks. For chunks with clear sentence boundaries, the window size is determined by punctuation, resulting in a window size based on sentence length; for chunks with five sentence boundaries, the window size can be a fixed value.

[0049] Furthermore, semantic feature vectors and keyword feature vectors extracted in blocks are then concatenated. By integrating domain-adaptive BERT and TF-IDF statistical features, the block vectors retain deep semantic information and explicitly strengthen keyword weights, significantly improving the adaptability to texts in the public opinion domain.

[0050] Calculating the text block position similarity between different data points includes: based on Calculate, where, The degree of overlap between different text blocks in different data; For the first The k-th text block of the data; For the first The m-th text block of the data; For the first The total number of text blocks in the data; for The total number of text blocks in the data; For indicator functions, if block-based and If there is overlap, the value is 1; otherwise, it is 0. The overlap function is used to determine overlap.

[0051] In this optional implementation, positional similarity is used to measure the degree of overlap between the block positions of two documents. Values ​​range from 0 to 1. Overlap detection function. Specific rule definition: Assume that the block position is represented by a character range (e.g., the start and end positions of the block): ,

[0052] in Starting position The ending position has the following overlap condition: Where a value of 1 indicates that the intervals intersect. The numerator in the above formula is used to identify the intersection of the two data blocks. Yes, count the number of all pairs of blocks that satisfy the overlap condition. Using... Normalization helps avoid bias in similarity scores towards data with more blocks due to differences in the number of blocks. A value of 0 indicates that blocks do not overlap; otherwise, they overlap.

[0053] As an optional implementation of this embodiment, deduplication based on the positional similarity and the similarity of the concatenated vectors includes: based on Calculate target similarity, where, For semantic similarity; The semantic weight coefficient is used to remove duplicates based on the relationship between the target similarity and the specified threshold.

[0054] In this optional implementation, semantic similarity is determined using cosine similarity. After determining semantic similarity, a deduplication operation is performed.

[0055] Specifically, assuming the data Contains block {A1, A2}, data Given blocks {B1, B2, B3}, calculate the similarity value for all cross-text block pairs, and the total number of block pairs. For data Number of blocks multiplied by data The number of blocks, for example, 2*3 equals 6, and the similarity set S is obtained by the similarity between different blocks, for example... , Etc. The average similarity is ,

[0056] If the value is high, it means that there are currently a large number of similar text blocks, If the value is high, it means that there are currently a large number of similar text blocks, If the value is high, it means that there are currently a large number of similar text blocks, If the value is high, it means that there are currently a large number of similar text blocks, If the value is high, it means that there are currently a large number of similar text blocks, If the value is high, it means that there are currently a large number of similar text blocks, If the value is low, it means that the data block similarity is concentrated, and the specified threshold can be balanced to determine a more accurate and objective specified threshold, which lays a foundation for improving the accuracy of the first specified threshold.

[0057] If the target similarity is greater than the specified threshold, it is determined that the block is repeated, otherwise the block is not repeated. Further, the degree of repetition between the data can be determined based on the aforementioned repetition degree calculation formula, if the degree of repetition is greater than a second threshold (which can be preset), it is determined that the two data are repeated, and the deduplication operation is performed, otherwise it is not repeated and not deduplicated.

[0058] Through the multi-level deduplication method, the accuracy of deduplication can be maximized, and the efficiency of deduplication can be improved to ensure the real-time performance of deduplication.

[0059] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0060] According to the embodiment of the present application, a public opinion information deduplication device is also provided, which comprises an acquisition unit for acquiring original public opinion data sets related to a specified object from different media platforms Each data contains text content and metadata, wherein the metadata includes the publishing time and the source; a deduplication unit for performing a one-time deduplication processing on each data in the original public opinion data set, and inputting the remaining text content after the one-time deduplication processing into a preset rule engine to determine the secondary deduplication mode corresponding to the remaining data, wherein the rule engine calculates the current load information, and determines the secondary deduplication mode based on the result of the load information, including determining the secondary deduplication mode based on the source corresponding to the remaining data if the current load information does not exceed a specified load.

[0061] performing secondary deduplication on the remaining data based on the secondary deduplication manner.

[0062] According to an embodiment of the present application, the present application also provides an electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to implement the method described in any of the above embodiments.

[0063] According to an embodiment of the present application, the present application also provides a readable storage medium storing computer instructions, which, when executed by a computer, enable the computer to implement the method described in any of the above embodiments.

[0064] According to an embodiment of the present application, the present application also provides a computer program product, which, when executed by a processor, enables the processor to implement the method described in any of the above embodiments.

[0065] Figure 2 A schematic block diagram of an example electronic device 300 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices.

[0066] As shown in Figure 2 The electronic device 300 includes a computing unit 301 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. Various programs and data required for the operation of the electronic device 300 can also be stored in the RAM 303. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0067] Various components in the electronic device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, a speaker, etc.; a storage unit 308, such as a magnetic disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0068] The computing unit 301 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 301 performs various methods and processes described above, such as the object matching method. For example, in some embodiments, the object matching method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 308. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded onto the RAM 303 and executed by the computing unit 301, one or more steps of the methods described above can be performed.

[0069] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0070] Program code for carrying out methods of the present application can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0071] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine- readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include one or more lines of electrical connections, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

Claims

1. A method for deduplicating information, characterized in that, include: Obtain raw public opinion datasets related to the specified target from different media platforms. Each data item Includes text content and metadata, wherein the metadata includes publication time and source; After deduplication of each data item in the original public opinion dataset, the remaining text content after the first deduplication is input into a preset rule engine to determine the secondary deduplication method corresponding to the remaining data. The rule engine calculates the current load information and determines the secondary deduplication method based on the result of the load information, including determining the secondary deduplication method based on the source of the remaining data if the current load information does not exceed the specified load. The remaining data is further deduplicated based on the aforementioned secondary deduplication method, including: for any two data points to be deduplicated in the remaining data, if the corresponding sources are both of the first source type, then the secondary deduplication method is determined to be the first processing method based on TF-IDF, where the first source type is social media type mainly consisting of short text; if the corresponding sources are both of the second source type, then the secondary deduplication method is determined to be the second processing method with multiple sub-levels, where the second source type is news website type mainly consisting of long text; if the corresponding sources are the first source type and the second source type respectively, then the secondary deduplication method is determined to be the third processing method. The secondary deduplication of the remaining data based on the aforementioned secondary deduplication method includes: if the secondary deduplication method is a multi-level second processing method, then the first-level processing method is first used to deduplicate the text content T in each of the remaining data. i The data is divided into text blocks of a specified length, and the hash value of each text block is calculated. An inverted index is constructed based on the hash values, using the hash values ​​as keys to record the text and position of each text block. Based on the pre-constructed inverted index, the hash values ​​and block positions of different text blocks across the data are compared to determine whether there are duplicate text blocks. The duplication rate of the data is calculated based on the duplicate text blocks to determine whether there is overlap in the data, and a deduplication operation is performed on the overlapping data. Among these, the determination of whether there is overlap in the data is based on the information of the duplicate text blocks and the current threshold obtained in the cache. The target data obtained from the first sub-level processing is then processed in the second sub-level: the text content of each piece of target data is re-blocked, wherein different blocking parameters are used to dynamically block the target data; the semantic feature vector and keyword feature vector of each text block obtained from the dynamic blocking are extracted; the semantic feature vector and the keyword feature vector are concatenated to obtain a concatenated vector; the text block position similarity between each piece of data is calculated, and deduplication is performed based on the position similarity and the similarity of the concatenated vector.

2. The information deduplication method according to claim 1, characterized in that, When constructing the inverted index, the method includes constructing the inverted index based on the hash value: ,in, The hash value of the k-th text block. Let k be the i-th piece of public opinion data, and k be the number of blocks in the text content. The sequence number in the sequence, k starts from 1, Let pl be the character offset of the k-th text block in the original text. i = (start k end k ), start k Let be the starting character position of the k-th text block, starting from 0, and end. k Let K be the position of the end character of the k-th text block, and K be the position of the d-th text block. i The total number of text blocks in the data.

3. The information deduplication method according to claim 1, characterized in that, Calculating the text block position similarity between different data points includes: based on Calculate, where, For the first The k-th text block of the data; For the first The m-th text block of the data; For the first The total number of text blocks in the data; For the first The total number of text blocks in the data; For indicator functions, if text block and If there is overlap, the value is 1; otherwise, it is 0. The overlap function is used to determine overlap.

4. The information deduplication method according to claim 1, characterized in that, Deduplication based on the positional similarity and the similarity of the concatenated vectors includes: based on Calculate target similarity, where, For semantic similarity; These are semantic weight coefficients; Deduplication is performed based on the relationship between the target similarity and the specified threshold.

5. The information deduplication method according to claim 4, characterized in that, The specified threshold is , Based on the threshold, This represents the mean cosine similarity of all text block pairs across different data sets. For based on A defined standard deviation; This is for adjusting the coefficient.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method according to any one of claims 1-5.

7. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the at least one processor to perform the method according to any one of claims 1-5.

Images